Add observability capabilities to functions #9

tpiperatgod · 2021-12-02T09:25:00Z

We need to add observability capabilities to functions, which facilitate observing and tracking the operation of functions in large-scale scenarios.

Referring to #7, we can take the form of a plugin in functions-framework to wake up the observability component to run at a specific node.

For example, in functions-framework-go, we can add a plugin hook before and after the function is run, and run the logic related to the observability component in the hook.

Reference this:

func registerHTTPFunction(path string, fn func(http.ResponseWriter, *http.Request), h *http.ServeMux) error {
	h.HandleFunc(path, func(w http.ResponseWriter, r *http.Request) {
		defer recoverPanicHTTP(w, "Function panic")
		// execute pre-run plugins
		fn(w, r)
		// execute post-run plugins
	})
	return nil
}

We should also consider as much as possible the consistency of the scheme's implementation in different languages.

We can complete the details of the design of this solution in this document.

benjaminhuo · 2021-12-03T06:49:54Z

We might need to adjust all sync function and async function signature to the same as below, this way we can put tracing options such as skywalking, opentelemetry or none into OpenFunctionContext.

And then we can use these tracing parameters in OpenFunctionContext to create a wrapper function to wrap user function with tracing ability.

The tracing options skywalking, opentelemetry or none can be put into function crd maybe.

What do you think?
@tpiperatgod @wanjunlei @FeynmanZhou @arugal :

func func1(ctx *ofctx.OpenFunctionContext, in []byte) ofctx.RetValue

tpiperatgod · 2021-12-03T08:46:43Z

I think it would make sense to use the OpenFunctionContext to pass tracing options to the functions-framework, which is the job the OpenFunctionContext should take on.

And I agree with putting the options about function tracing in function crd.

wu-sheng · 2021-12-03T09:52:19Z

I want to give a heads up to the OpenFunction team. I am going to put a core-level proposal to SkyWalking project, which means we are going to officially move to SkyWalking v9.
About the immigration part, OpenFunction project doesn't need to worry about the break, because we are going to do that. All agents, go2sky, nodejs and python, are still as same as before, v3 tracing protocol will not be changed.

The thing I want to mention is, a new concept is going to be added in v9 core, which is layer. I suggest to add layer=faas as a specific tag into the root span of segment, which would help SkyWalking to ship the logic service and endpoint into FAAS page.

More information will be share next week or this weekend. Once the 8.9.0 release(In releasing process) is done, the new proposal will be out.

wu-sheng · 2021-12-03T09:57:57Z

Besides the APIs you are discussing, we also should consider

Shipping logs, some agents(SkyWalking, not OpenTelemetry) have bundled channel to forward this directly, rather than collecting logs from K8s or files.
Manual instrumentation Metrics APIs. There should be some kinds Prometheus concepts, but more closing to a metric API rather than implementation.
Tracing part, beside before/after/context as mentioned, we should provide at least manually tagging APIs to add more custom information when needed.

benjaminhuo · 2021-12-03T11:08:41Z

Besides the APIs you are discussing, we also should consider

Shipping logs, some agents(SkyWalking, not OpenTelemetry) have bundled channel to forward this directly, rather than collecting logs from K8s or files.

Manual instrumentation Metrics APIs. There should be some kinds Prometheus concepts, but more closing to a metric API rather than implementation.

Tracing part, beside before/after/context as mentioned, we should provide at least manually tagging APIs to add more custom information when needed.

Thanks a lot for these suggestions @wu-sheng! we'll think about these points

benjaminhuo · 2021-12-03T11:09:19Z

I want to give a heads up to the OpenFunction team. I am going to put a core-level proposal to SkyWalking project, which means we are going to officially move to SkyWalking v9. About the immigration part, OpenFunction project doesn't need to worry about the break, because we are going to do that. All agents, go2sky, nodejs and python, are still as same as before, v3 tracing protocol will not be changed.

The thing I want to mention is, a new concept is going to be added in v9 core, which is layer. I suggest to add layer=faas as a specific tag into the root span of segment, which would help SkyWalking to ship the logic service and endpoint into FAAS page.

More information will be share next week or this weekend. Once the 8.9.0 release(In releasing process) is done, the new proposal will be out.

Great to have this info! Sure, we'll add layer=faas tag to the root span

benjaminhuo · 2021-12-10T09:33:02Z

I've created an initial proposal for tracing : https://hackmd.io/@UrcJbEg9R_mxQy4aRXO5tA/H1A4vDe9K
@wu-sheng @arugal @webup @tpiperatgod @wanjunlei @FeynmanZhou

wu-sheng · 2021-12-10T09:50:31Z

I think we needs to provide tags(in the proposal) for users, and also consider Correlation context. OpenTracing(OpenTelemetry should have too) has a same concept called baggage.

Also, to @arugal , we should consider how to add timestamp of previous function end, to propagate through sw8-x. Then SkyWalking server could have the scheduling latency from functionA to functionB.

benjaminhuo · 2021-12-10T09:58:11Z

Correlation context

I think we needs to provide tags(in the proposal) for users, and also consider Correlation context. OpenTracing(OpenTelemetry should have too) has a same concept called baggage.

Also, to @arugal , we should consider how to add timestamp of previous function end, to propagate through sw8-x. Then SkyWalking server could have the scheduling latency from functionA to functionB.

There is customTags section to add tags a user want to add, change it to tags ?

      customTags:
      - func: function-with-tracing
      - layer: faas
      - tag1: value1
      - tag2: value2

wu-sheng · 2021-12-10T10:14:57Z

I think tags work.

benjaminhuo · 2021-12-10T10:22:06Z

Changed customTags to tags already

benjaminhuo · 2021-12-10T11:53:49Z

Added baggage like below:

apiVersion: core.openfunction.io/v1alpha2
kind: Function
metadata:
  name: function-with-tracing
spec:
  serving:
    runtime: "OpenFuncAsync"
    tracing:
      # Switch to tracing, default to false
      enabled: true
      # Provider name can be set to "skywalking", "opentelemetry"
      # A valid provider must be set if tracing is enabled.
      provider: 
        name: "skywalking"
      # Custom tags to add to tracing
      tags:
      - func: function-with-tracing
      - layer: faas
      - tag1: value1
      - tag2: value2
      baggage:
      # baggage key is `sw8-correlation` for skywalking and `baggage` for opentelemetry
      # Correlation context for skywalking: https://skywalking.apache.org/docs/main/latest/en/protocols/skywalking-cross-process-correlation-headers-protocol-v1/
      # baggage for opentelemetry: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/baggage/api.md
      # W3C Baggage Specification/: https://w3c.github.io/baggage/
        key: sw8-correlation # key should be baggage for opentelemetry
        value: "base64(string key):base64(string value),base64(string key2):base64(string value2)"

wu-sheng · 2021-12-10T11:54:48Z

Make sense. Good to me.

benjaminhuo · 2021-12-17T08:50:42Z

Proposal for plugin mechanism to function framework by @tpiperatgod https://hackmd.io/O8o01-mjT6uv6L9F25pYsA?view=

wu-sheng · 2021-12-31T01:40:20Z

@arugal apache/skywalking#8367 SkyWalking v9 core upgrade is almost done.

Do we have any update about OpenFunction side? As we are going to add tracing(go2sky) to it first, @arugal you need to follow this v9 update, and we need to make sure OpenFunction's trace could be identified as a faas layer service, instance.

Also, we need a definition about what are the service and instance in the OpenFunction or general FAAS scope.
@benjaminhuo @tpiperatgod Any suggestion about this?

benjaminhuo · 2021-12-31T01:54:03Z

@wu-sheng，Thanks very much for the reminder.

To add skywalking tracing we need to refactor functions-framework and add a plugin mechanism and the design is almost finished.

With the previous tracing proposal and this design, the skywalking tracing function is now our current most important work to do.

Once the coding of the plugin mechanism is finished, we'll need @arugal's help to add skywalking tracing code as a plugin.

From my understanding, a function is a service and its replica is an instance. Do we need to add the service and instance tag to skywalking tracing?

wu-sheng · 2021-12-31T02:04:24Z

SkyWalking has service and instance fields directly(not need tags) to declare that. The reason I am asking for this, usually, an FAAS level function seems(from my little FAAS understanding, please CMIIW) more closing to an endpoint concept in SkyWalking.

So, I just recheck, whether OpenFunction has a higher level concept for a group of function replica(instance) grouped as a unit or something. There is no issue for function as service, it is just if we are using like this, the SkyWalking's endpoint concept seems not very useful for OpenFunction case. Or do I miss anything in the OpenFunction could be defined as a subset of function to be a function.

benjaminhuo · 2021-12-31T02:17:44Z

OpenFunction has sync functions and it can be accessed through HTTP, endpoint could be valuable for sync functions.
Regarding async functions, it's triggered by events from middleware like MQ and maybe it's not applicable here.
We'll take a look at skywalking's Service/Instance/Endpoint concept to find out how to integrate with it.

wu-sheng · 2021-12-31T02:27:15Z

Sync and async all work in SkyWalking. We have Kafka consumer or async scheduled task in SkyWalking is defined as an endpoint.
My question is more focusing on, should we have endpoint still works in OpenFunction, as here, Function is the executable unit. Do we have larger concept for service?

benjaminhuo · 2021-12-31T02:31:04Z

Got you, we'll add serviceless workflow capability and it's a set of related functions, so maybe a serverless workflow is a skywalking servcie

wu-sheng · 2021-12-31T02:33:56Z

Is a workflow always running in one process(OS level)? Because service-to-service is better to measure network performance comparing to endpoint-to-endpoint in today's SkyWalking.

benjaminhuo · 2021-12-31T02:51:46Z

A workflow itself will run in different processes (functions) actually.

arugal · 2021-12-31T03:29:48Z

To add skywalking tracing we need to refactor functions-framework and add a plugin mechanism and the design is almost finished.

With the previous tracing proposal and this design, the skywalking tracing function is now our current most important work to do.
Once the coding of the plugin mechanism is finished, we'll need @arugal's help to add skywalking tracing code as a plugin.

Good to me, I'll start after the framework is complete :)

wu-sheng · 2021-12-31T07:27:54Z

A workflow itself will run in different processes (functions) actually.

OK, then, we need to consider more how to define service in OpenFunction. Let's set each function as service for now as a PoC version.

benjaminhuo · 2021-12-31T07:32:55Z

Sure, the refactoring of functions framework is almost done.
Maybe we can start the integration with Skywalking the first community meeting of 2022(Jan 6th)
@tpiperatgod @arugal

A workflow itself will run in different processes (functions) actually.

OK, then, we need to consider more how to define service in OpenFunction. Let's set each function as service for now as a PoC version.

wu-sheng · 2021-12-31T07:35:37Z

Sure, the refactoring of functions framework is almost done.
Maybe we can start the integration with Skywalking the first community meeting of 2022(Jan 6th)

This seems good. SkyWalking's first release plans on March.
For developers, new backend core should be available in the first week of Jan., and the first draft will be around Chinese New Year. @Fine0830 Do you have a solid timeline for booster UI?

Fine0830 · 2021-12-31T08:19:45Z

Sure, the refactoring of functions framework is almost done.
Maybe we can start the integration with Skywalking the first community meeting of 2022(Jan 6th)

This seems good. SkyWalking's first release plans on March. For developers, new backend core should be available in the first week of Jan., and the first draft will be around Chinese New Year. @Fine0830 Do you have a solid timeline for booster UI?

Uh...I don't sure about the timeline. Probably March is okay for me.

wu-sheng · 2021-12-31T08:20:51Z

Uh...I don't sure about the timeline. Probably March is okay for me.

OK, let's see. Anyway, I think OpenFunction will move faster than SkyWalking itself :)

benjaminhuo · 2021-12-31T09:11:15Z

Now the functions-framework refactoring proposal is ready:
https://github.com/OpenFunction/OpenFunction/blob/main/docs/proposals/202112_functions_framework_refactoring.md

Skywalking tracing will be implemented as a plugin of the above functions-framework
https://github.com/OpenFunction/OpenFunction/blob/main/docs/proposals/202112_support_function_tracing.md

tpiperatgod · 2022-01-21T03:00:45Z

Skywalking tracing will be implemented as a plugin of the above functions-framework https://github.com/OpenFunction/OpenFunction/blob/main/docs/proposals/202112_support_function_tracing.md

On the basis of this proposal, how about setting the configuration of the plugin section to this?

apiVersion: core.openfunction.io/v1alpha2
kind: Function
metadata:
  name: function-with-tracing
  annotations:
    plugins.pre:
      - pluginA
      - pluginB
      - pluginC
    plugins.post:
      - pluginC
      - pluginA
    plugins.tracing: |
      # Switch for tracing, default to false
      enabled: true
      # Provider name can be set to "skywalking", "opentelemetry"
      # A valid provider must be set if tracing is enabled.
      provider:
        name: "skywalking"
        oapServer: "localhost:xxx"
      # Custom tags to add to tracing
      tags:
      - func: function-with-tracing
      - layer: faas
      - tag1: value1
      - tag2: value2
      baggage:
      # baggage key is `sw8-correlation` for skywalking and `baggage` for opentelemetry
      # Correlation context for skywalking: https://skywalking.apache.org/docs/main/latest/en/protocols/skywalking-cross-process-correlation-headers-protocol-v1/
      # baggage for opentelemetry: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/baggage/api.md
      # W3C Baggage Specification/: https://w3c.github.io/baggage/
        key: sw8-correlation # key should be baggage for opentelemetry
        value: "base64(string key):base64(string value),base64(string key2):base64(string value2)"

benjaminhuo · 2022-01-21T03:03:44Z

Skywalking tracing will be implemented as a plugin of the above functions-framework https://github.com/OpenFunction/OpenFunction/blob/main/docs/proposals/202112_support_function_tracing.md

On the basis of this proposal, how about setting the configuration of the plugin section to this?

apiVersion: core.openfunction.io/v1alpha2
kind: Function
metadata:
  name: function-with-tracing
  annotations:
    plugins.pre:
      - pluginA
      - pluginB
      - pluginC
    plugins.post:
      - pluginC
      - pluginA
    plugins.tracing: |
      # Switch for tracing, default to false
      enabled: true
      # Provider name can be set to "skywalking", "opentelemetry"
      # A valid provider must be set if tracing is enabled.
      provider:
        name: "skywalking"
        oapServer: "localhost:xxx"
      # Custom tags to add to tracing
      tags:
      - func: function-with-tracing
      - layer: faas
      - tag1: value1
      - tag2: value2
      baggage:
      # baggage key is `sw8-correlation` for skywalking and `baggage` for opentelemetry
      # Correlation context for skywalking: https://skywalking.apache.org/docs/main/latest/en/protocols/skywalking-cross-process-correlation-headers-protocol-v1/
      # baggage for opentelemetry: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/baggage/api.md
      # W3C Baggage Specification/: https://w3c.github.io/baggage/
        key: sw8-correlation # key should be baggage for opentelemetry
        value: "base64(string key):base64(string value),base64(string key2):base64(string value2)"

Annotation is just a map[string]string, make sure you know how to fit the data you want to this data structure

wu-sheng · 2022-01-21T03:10:45Z

As annotation is just a map[string]string, this seems the value(such as key=plugins.tracing) has to be parsed by tracer implementation. I am not sure what does OpenFunction recommends? Do you prefer key/value pairs or proto-obj oriented like Envoy?

benjaminhuo · 2022-01-21T03:17:10Z

@wu-sheng Yes, the value of the key has to be parsed by OpenFunction itself before passing it to skywalking.
No need for skywalking to parse it in my opinion.

tpiperatgod · 2022-01-21T03:20:03Z

My mistake. It has been adjusted to the following format:

apiVersion: core.openfunction.io/v1alpha2
kind: Function
metadata:
  name: function-with-tracing
  annotations:
    plugins: |
      # Default order option. During the preHooks phase the plugins will be executed in the following order:
      #   pluginA -> pluginB -> pluginC
      # In the postHooks phase the plugins will be executed in the following order:
      #   pluginC -> pluginB -> pluginA
      order:
      - pluginA
      - pluginB
      - pluginC
      # The "pre" and "post" options will override the order in the "order" option,
      # and you can specify the order of execution of the plugins in the prehooks and posthooks phases separately
      pre:
      - pluginA
      - pluginC
      - pluginB
      post:
      - pluginB
      - pluginA
    plugins.tracing: |
      # Switch for tracing, default to false
      enabled: true
      # Provider name can be set to "skywalking", "opentelemetry"
      # A valid provider must be set if tracing is enabled.
      provider:
        name: "skywalking"
        oapServer: "localhost:xxx"
      # Custom tags to add to tracing
      tags:
      - func: function-with-tracing
      - layer: faas
      - tag1: value1
      - tag2: value2
      baggage:
      # baggage key is `sw8-correlation` for skywalking and `baggage` for opentelemetry
      # Correlation context for skywalking: https://skywalking.apache.org/docs/main/latest/en/protocols/skywalking-cross-process-correlation-headers-protocol-v1/
      # baggage for opentelemetry: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/baggage/api.md
      # W3C Baggage Specification/: https://w3c.github.io/baggage/
        key: sw8-correlation # key should be baggage for opentelemetry
        value: "base64(string key):base64(string value),base64(string key2):base64(string value2)"

wu-sheng · 2022-01-21T03:20:58Z

So, there will be an object to define the configuration in the OpenFunction codebase, carrying the parsed configurations. Then SkyWalking tracer accepts it and sets it to the go2sky kernal.

benjaminhuo · 2022-01-21T03:23:46Z

So, there will be an object to define the configuration in the OpenFunction codebase, carrying the parsed configurations. Then SkyWalking tracer accepts it and sets it to the go2sky kernel.

Yes, that's correct!

benjaminhuo · 2022-03-08T08:47:45Z

@wu-sheng @arugal OpenFunction v0.6.0-rc.0 has been released and now SkyWalking has a perfect integration with OpenFunction Async and Sync functions!
Thanks, @arugal for the huge effort on this integration!

Skywalking tracing can be enabled either as a global option or as a per-function option as described in https://github.com/OpenFunction/OpenFunction/blob/main/docs/proposals/202112_support_function_tracing.md

wu-sheng · 2022-03-08T09:43:54Z

Fantastic! We are going to prepare the v9 release in the next 2 weeks, I will ask @arugal to set up the FAAS dashboard for you. This dashboard will be included as a default active function(on the top-level menu), I think you would love that.

I will update after we have that.

benjaminhuo · 2022-03-08T09:45:05Z

Looking forward to SkyWalking v9!

tpiperatgod added the enhancement New feature or request label Dec 2, 2021

benjaminhuo mentioned this issue Dec 3, 2021

[Discussion] Integration possibility of bundled go2sky OpenFunction/OpenFunction#146

Closed

tpiperatgod mentioned this issue Dec 4, 2021

Add plug-in mechanism(interface) for functions-framework #8

Closed

benjaminhuo assigned tpiperatgod and benjaminhuo Dec 10, 2021

benjaminhuo mentioned this issue Jan 24, 2022

add tracing configuration to the Context OpenFunction/functions-framework-go#26

Merged

benjaminhuo closed this as completed Jun 13, 2022

Add observability capabilities to functions #9

Add observability capabilities to functions #9

Comments

tpiperatgod commented Dec 2, 2021

benjaminhuo commented Dec 3, 2021 • edited

tpiperatgod commented Dec 3, 2021

wu-sheng commented Dec 3, 2021

wu-sheng commented Dec 3, 2021

benjaminhuo commented Dec 3, 2021

benjaminhuo commented Dec 3, 2021

benjaminhuo commented Dec 10, 2021

wu-sheng commented Dec 10, 2021

benjaminhuo commented Dec 10, 2021 • edited

wu-sheng commented Dec 10, 2021

benjaminhuo commented Dec 10, 2021

benjaminhuo commented Dec 10, 2021

wu-sheng commented Dec 10, 2021

benjaminhuo commented Dec 17, 2021 • edited

wu-sheng commented Dec 31, 2021

benjaminhuo commented Dec 31, 2021 • edited

wu-sheng commented Dec 31, 2021

benjaminhuo commented Dec 31, 2021

wu-sheng commented Dec 31, 2021

benjaminhuo commented Dec 31, 2021

wu-sheng commented Dec 31, 2021

benjaminhuo commented Dec 31, 2021

arugal commented Dec 31, 2021

wu-sheng commented Dec 31, 2021

benjaminhuo commented Dec 31, 2021

wu-sheng commented Dec 31, 2021

Fine0830 commented Dec 31, 2021

wu-sheng commented Dec 31, 2021

benjaminhuo commented Dec 31, 2021

tpiperatgod commented Jan 21, 2022

benjaminhuo commented Jan 21, 2022

wu-sheng commented Jan 21, 2022

benjaminhuo commented Jan 21, 2022

tpiperatgod commented Jan 21, 2022 • edited

wu-sheng commented Jan 21, 2022

benjaminhuo commented Jan 21, 2022

benjaminhuo commented Mar 8, 2022 • edited

wu-sheng commented Mar 8, 2022

benjaminhuo commented Mar 8, 2022

benjaminhuo commented Dec 3, 2021 •

edited

benjaminhuo commented Dec 10, 2021 •

edited

benjaminhuo commented Dec 17, 2021 •

edited

benjaminhuo commented Dec 31, 2021 •

edited

tpiperatgod commented Jan 21, 2022 •

edited

benjaminhuo commented Mar 8, 2022 •

edited