Add observability capabilities to functions

[tpiperatgod]

We need to add observability capabilities to functions, which facilitate observing and tracking the operation of functions in large-scale scenarios.

Referring to #7, we can take the form of a plugin in functions-framework to wake up the observability component to run at a specific node.

For example, in functions-framework-go, we can add a plugin hook before and after the function is run, and run the logic related to the observability component in the hook.

Reference this:

func registerHTTPFunction(path string, fn func(http.ResponseWriter, *http.Request), h *http.ServeMux) error {
	h.HandleFunc(path, func(w http.ResponseWriter, r *http.Request) {
		defer recoverPanicHTTP(w, "Function panic")
		// execute pre-run plugins
		fn(w, r)
		// execute post-run plugins
	})
	return nil
}

We should also consider as much as possible the consistency of the scheme's implementation in different languages.

We can complete the details of the design of this solution in this document.

[benjaminhuo]

We might need to adjust all sync function and async function signature to the same as below, this way we can put tracing options such as skywalking, opentelemetry or none into OpenFunctionContext.

And then we can use these tracing parameters in OpenFunctionContext to create a wrapper function to wrap user function with tracing ability.

The tracing options skywalking, opentelemetry or none can be put into function crd maybe.

What do you think?
@tpiperatgod @wanjunlei @FeynmanZhou @arugal :

func func1(ctx *ofctx.OpenFunctionContext, in []byte) ofctx.RetValue 

[tpiperatgod]

I think it would make sense to use the OpenFunctionContext to pass tracing options to the functions-framework, which is the job the OpenFunctionContext should take on.

And I agree with putting the options about function tracing in function crd.

[wu-sheng]

I want to give a heads up to the OpenFunction team. I am going to put a core-level proposal to SkyWalking project, which means we are going to officially move to SkyWalking v9.
About the immigration part, OpenFunction project doesn't need to worry about the break, because we are going to do that. All agents, go2sky, nodejs and python, are still as same as before, v3 tracing protocol will not be changed.

The thing I want to mention is, a new concept is going to be added in v9 core, which is layer. I suggest to add layer=faas as a specific tag into the root span of segment, which would help SkyWalking to ship the logic service and endpoint into FAAS page.

More information will be share next week or this weekend. Once the 8.9.0 release(In releasing process) is done, the new proposal will be out.

[wu-sheng]

Besides the APIs you are discussing, we also should consider

1. Shipping logs, some agents(SkyWalking, not OpenTelemetry) have bundled channel to forward this directly, rather than collecting logs from K8s or files.
2. Manual instrumentation Metrics APIs. There should be some kinds Prometheus concepts, but more closing to a metric API rather than implementation.
3. Tracing part, beside before/after/context as mentioned, we should provide at least manually tagging APIs to add more custom information when needed.

[benjaminhuo]

Besides the APIs you are discussing, we also should consider

1. Shipping logs, some agents(SkyWalking, not OpenTelemetry) have bundled channel to forward this directly, rather than collecting logs from K8s or files.
2. Manual instrumentation Metrics APIs. There should be some kinds Prometheus concepts, but more closing to a metric API rather than implementation.
3. Tracing part, beside before/after/context as mentioned, we should provide at least manually tagging APIs to add more custom information when needed.

Thanks a lot for these suggestions @wu-sheng! we'll think about these points

[benjaminhuo]

I want to give a heads up to the OpenFunction team. I am going to put a core-level proposal to SkyWalking project, which means we are going to officially move to SkyWalking v9. About the immigration part, OpenFunction project doesn't need to worry about the break, because we are going to do that. All agents, go2sky, nodejs and python, are still as same as before, v3 tracing protocol will not be changed.

The thing I want to mention is, a new concept is going to be added in v9 core, which is layer. I suggest to add layer=faas as a specific tag into the root span of segment, which would help SkyWalking to ship the logic service and endpoint into FAAS page.

More information will be share next week or this weekend. Once the 8.9.0 release(In releasing process) is done, the new proposal will be out.

Great to have this info! Sure, we'll add layer=faas tag to the root span

[benjaminhuo]

I've created an initial proposal for tracing : https://hackmd.io/@UrcJbEg9R_mxQy4aRXO5tA/H1A4vDe9K
@wu-sheng @arugal @webup @tpiperatgod @wanjunlei @FeynmanZhou

[wu-sheng]

I think we needs to provide tags(in the proposal) for users, and also consider Correlation context. OpenTracing(OpenTelemetry should have too) has a same concept called baggage.

Also, to @arugal , we should consider how to add timestamp of previous function end, to propagate through sw8-x. Then SkyWalking server could have the scheduling latency from functionA to functionB.

[benjaminhuo]

Correlation context

I think we needs to provide tags(in the proposal) for users, and also consider Correlation context. OpenTracing(OpenTelemetry should have too) has a same concept called baggage.

Also, to @arugal , we should consider how to add timestamp of previous function end, to propagate through sw8-x. Then SkyWalking server could have the scheduling latency from functionA to functionB.

There is customTags section to add tags a user want to add, change it to tags ?

      customTags:
      - func: function-with-tracing
      - layer: faas
      - tag1: value1
      - tag2: value2

[wu-sheng]

I think tags work.