Tracing for passthrough requests (CDN use case)

[bobrik]

Imagine a reverse proxy / CDN company wants to implement tracing. This CDN company also has customers that control clients and origins that may want to implement tracing themself:

Client -> CDN -> Origin

Current design assumes that CDN is supposed to trust trace information passed from the client. This may not be the best course of action for a few reasons:

• Clients are not always trustworthy and tracing is not zero cost. A flood of requests with forced tracing may increase CPU usage and be used as a DDoS vector.
• Generally, CDN and its clients do not want to mix their tracing data and traces will end up in two different sinks anyway.
• CDN should not decide if tracing is supposed to be enabled for origin.

One possible solution for this is for the CDN company to:

• Use separate headers for own tracing data
• Sign and verify tracing headers to prevent forging (think hmac)
• Pass client tracing headers untouched

This is not exactly possible when the spec defines just one set of header names to use by any involved party. Keep in mind that CDN is likely to have lots of components that cannot be all changed at the same time (and some may never be instrumented).

I would like to see this scenario addressed in the spec.

[SergeyKanzhelev]

In your example CDN has many components between Client and Origin and you need to trace thru those components without loosing the connection between Client and Origin?

If so - we've discussed this scenario. You can restart the traceparent while preserving original traceparent and tracestate in a new tracestate. Use those to do distributed tracing for CDN components. And then restore original traceparent back when you are done.

Also see this section of rationale: https://github.com/w3c/trace-context/blob/master/spec/21-http_header_format_rationale.md#restarting-trace

Is it something that would work for your scenario?

[SergeyKanzhelev]

I also filed https://github.com/w3c/trace-context/issues/279 as we lost explanation of multi-graph identity that tracestate carries that may help understand how it is intended to be used.

[bobrik]

The problem with stacking trace state is that there's no easy way to see where the boundaries are:

• Clients may be owned by CDN
• Origins may be owned by CDN
• CDN itself may have several hops with some of them being a third party

In all of those cases trace information for both internal and external trace has to be preserved to be able to stitch whole trace together.

There's no easy way for a CDN to tell if an origin (aka upstream) is its own service or a third party service, and the decision whether to pop a trace state stack depends on this very fact. Clients also need to signify that they are somehow a part of a "secure network".

I'll read through the spec and removed parts from #259 again to see if I missed a clear way of how this should be done.

[SergeyKanzhelev]

With the requirements you listed - there is hardly a way to implement what you want. Spec allows to do as much as you'd do with alternative headers as tracestate can be used as a collection of those alternative headers.

Let us know if you have more questions. Also if you want to discuss them in realtime, join the next W3C call to discuss your scenario.