Avoiding conflicting context headers

[dyladan]

tl;dr should we overwrite user-set headers when injecting context? Existing propagation API may be insufficient to solve this problem in a satisfying way.

Context

W3C Trace Context specifies 2 headers, traceparent and tracestate. tracestate depends on traceparent. That is, tracestate is meaningless on its own, and the specification says it should be dropped in the absence of a well-formed traceparent.

This problem may apply to other context propagation methods as well, but I am most familiar with W3C Trace Context.

The Problem

If the user creates a request which already contains a traceparent when we inject context we have to do one of the following:

1. Overwrite the user-set header, potentially causing a change in application behavior.
2. Set our header in addition to user-set header. In some cases such as W3C, this is against specification. In the example below, we will see that this sometimes results in spec-breaking context.
3. Do nothing, potentially breaking the OpenTelemetry trace. The existing propagator API may make this option impossible in some cases.

In all cases, we are a bit limited by the existing API. The TextMapPropagator#Inject interface includes only a Carrier (which contains headers) and Setter (which defines how to set a header). Without at Getter, there is no way for the Inject function to read existing headers. In simple cases where Carrier is a plain object or map this may be a trivial operation, but in some cases it is not (which is the reason the Setter parameter exists).

Example

In undici (built-in NodeJS implementation of fetch), the request object has a method to set a header, but not to read already-set headers. If a user has already set a traceparent, the instrumentation will still attempt to inject context. The Setter function in this case calls setHeader, which sets the new traceparent in addition to the existing one by appending it to the existing header with comma-separated-values as specified in rfc9110#5.2. It may also inject tracestate. This results in an invalid traceparent, and potentially an orphaned tracestate, which causes both traces (ours and the user-set context) to be dropped by a spec-compliant receiver.