Switch from sh.keptn.context to TraceContext/ParentContext (transparent)

[christian-kreuzberger-dtx]

Keptn should support w3c tracecontext.

Research

• Explore the usage of the cloudevents spec extension
• Investigate the usage of the cloudevents-sdk used in ktpen

Prototype

As a POC implementation, an existing keptn endpoint shall be used to demonstrate the usage of trace context.
The interaction between the keptn cli and the keptn endpoint shall be shown.

Tasks

• implement poc
• Define additional issues that came up during investigation

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[johannes-b]

An extension of the CloudEvents focuses on exactly this topic. Let's have a look at: https://github.com/cloudevents/spec/blob/master/extensions/distributed-tracing.md when working on that issue.

[johannes-b]

How is this issue related to: #2422 ?

[christian-kreuzberger-dtx]

#2422 is meant to allow capturing certain custom metrics (e.g., git pulls in prometheus-service) that would allow us to track Keptns performance, while #1614 (this issue) is meant to introduce tracecontext etc...

[warber]

After a very helpful discussion with @slinkydeveloper, it seems like the distributed tracing extension offered by the cloudevents spec is not that useful and misunderstood by many people. First of all, thought the name would suggest it, the distributed tracing extension is not really usable for doing distributed tracing. It does not even relate to the use cases defined in the W3C standard. A lot of the confusion originates from the fact that the extension spec does follow the naming and format of the W3C Trace Context standard. There is quite a chance that this feature will be removed and reworked in the cloudevents specification (see: cloudevents/spec#751 for more information)
As for now using the distributed tracing feature is equivalent to renaming the shkeptncontext field to traceparent and change its format to the one mentioned in the W3C standard. (@johannes-b @agrimmer should we still do that? what's the benefit? I believe it brings also a lot of confusion into the keptn project. And why should the keptn context follow the format of W3C trace context? )

To truly introduce distributed tracing we should use the tools and APIs offered by opentelemetry. (@johannes-b @agrimmer follow up issue?)

[johannes-b]

Thanks for reaching out to the CloudEvents team. It is unfortunate that the extension does not help us and is still subject to change. Consequently, don't let's continue research this direction.

However, I would suggest figuring out how to make use of the official APIs offered by OpentTelementry as part of this issue.

I agree with you that replacing shkeptncontext with traceparent is quite a big change. But what do you think about adding traceparent as a header attribute to introduce it to Keptn CloudEvents?

[warber]

But what do you think about adding traceparent as a header attribute to introduce it to Keptn CloudEvents?

Yes, we could do that. This basically just mean: introduce OpenTelemetry into Keptn.

However, it is still a bit unclear about what we want to achieve. (1) Is it the goal to introduce distributed tracing into the architecture of keptn. Meaning, being able to trace all calls between our microservices using the capabilities of OpenTelemetry?
(2) Or, do we just want to correlate our CloudEvents, i.e., being able to see which CloudEvents belong together in a trace of an e2e usecase (user triggers some action, and we want to easily see the chain of cloud events that belong to this action)

IMO, these are two different things, (1) means introducing OpenTelemetry, while (2) means adding information to our CloudEvents to correlate them. It does not mean that (1) and (2) cant be solved using the same technology, though ;)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[thisthat]

Semantically, the sh.keptn.context is different than the trace context. Hence we cannot replace one with another.
Trace Context can be propagated via CloudEvent natively now with this extension: https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/extensions/distributed-tracing.md