Otel activation for message traces

[ripienaar]

Proposed change

In #5014 a basic capability to do on-demand message tracing was introduced for core pub-sub messages.

This capability is activated using a NATS specific header. The bigger picture is that users want to trigger traces based on trace headers from the likes of Open Telemetry etc.

There is prior-art in the server around ADR-3 that already supports triggering based on the headers we need and there are already functions for parsing the headers and determining if it's a continuation of a trace that started outside of NATS. See the shouldSample() function.

This leaves the question of where should traces be delivered when these headers like traceparent are present, of course we don't know as previously we configured the delivery per message. The only option I can think of is to add an account scope config item that configures a per-account subject for these externally-activated traces to be delivered.

Some other cases to consider, there might be more:

• If the configuration is not set the account does not support this style of activation of tracing.
• If the Nats-Trace-Only is present AND otel headers are set, we honor Nats-Trace-Only and do not deliver the message
• If Nats-Trace-Dest is set AND otel headers are set, we honor Nats-Trace-Dest and deliver to that subject

/cc @kozlovic

Use case

Integration with external tracing systems on account level.

Contribution

No response

[kozlovic]

@ripienaar Can you clarify:

If the Nats-Trace-Only is present AND otel headers are set, we honor Nats-Trace-Only and do not deliver the message

Are you saying that we do not deliver the message to the apps (like we do with Nats-Trace-Only) but deliver the resulting trace to the account's destination, or are you saying that we do not deliver the message nor the trace, in which case I would simply say that trace is not triggered at all?

[ripienaar]

@ripienaar Can you clarify:

If the Nats-Trace-Only is present AND otel headers are set, we honor Nats-Trace-Only and do not deliver the message

Are you saying that we do not deliver the message to the apps (like we do with Nats-Trace-Only) but deliver the resulting trace to the account's destination, or are you saying that we do not deliver the message nor the trace, in which case I would simply say that trace is not triggered at all?

Basically the Nats-Trace-* headers have precedence. If they are present we do what they say and do not deliver to the otel compatible destination. If Nats-Trace-Only is not set we'd deliver to the app as normal.

What we don't do is deliver to the otel subject when the Nats-Trace-* headers are present. Use case is CLI testing, and we wont want those to go to otel

[kozlovic]

@ripienaar Understood. What confused me was the reference to Nats-Trace-Only (alone) with Otel. Since Nats-Trace-Only without Nats-Trace-Dest has no effect, I wasn't sure what you meant.

But I get it now:

• If Nats-Trace-Dest is present, ignore account trace dest and work as it is working now, meaning trigger trace and deliver message to apps, unless Nats-Trace-Only is set.
• If Nats-Trace-Dest is not present and Otel is, trigger trace if sampling is on (last -01). But we ignore Nats-Trace-Only if present, and we deliver message to apps and send the trace to the account trace destination. If no trace destination is defined in the account, the trace will not be triggered (even if sampling is on).