Add the HTTP header format proposal for TraceContext propagation. #1
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| # Trace Context HTTP Header Format | ||
|
|
||
| Date: 31/03/2017 | ||
|
There was a problem hiding this comment. I see why this is here, though it will be maintenance causing.. wonder if there's a badge to auto-include the git timestamp? |
||
|
|
||
| A trace context header is used to pass trace context information across systems | ||
| for a HTTP request. Our goal is to share this with the community so that various | ||
|
There was a problem hiding this comment. I hear a lot of pain around tracing things like messages buses / kafka / etc. Might be worth intentionally decoupling from HTTP and making this about any messaging tech that has room for metadata/headers. |
||
| tracing and diagnostics products can operate together, and so that services can | ||
| pass context through them, even if they're not being traced (useful for load | ||
| balancers, etc.) | ||
|
|
||
| # Format | ||
|
|
||
| ## Header name | ||
|
|
||
| `Trace-Context` | ||
|
There was a problem hiding this comment. 'Via' is a standard header with pretty similar semantics. It is also in hpack. There was a problem hiding this comment.
|
||
|
|
||
| ## Field value | ||
|
|
||
| `base16(<version><version_format>)` | ||
|
|
||
| The value will be US-ASCII encoded (which is UTF-8 compliant). | ||
|
|
||
| ### Version | ||
|
There was a problem hiding this comment. Version or variant? maybe mention in docs this could be used by variants of the spec? or just wait and see.. There was a problem hiding this comment. (As a random, interested passerby:) The problem with having an explicit version field is that you risk breaking all receivers as soon as you increment the field, which means a user can't do a staged rollout of a system that uses a new version of the protocol. A better strategy would be to reserve some of the options bits and then requiring current implementations to set them to zero and ignore any trailing data. This way, you can evolve your protocol gradually. |
||
|
|
||
| Is a 1-byte representing a uint8 value. | ||
|
|
||
| ### Version = 0 | ||
|
|
||
| #### Format | ||
|
|
||
| `<trace-id><span-id><trace-options>` | ||
|
|
||
| All fields are required. | ||
|
|
||
| #### Trace-id | ||
|
|
||
| Is the ID of the whole trace forest. It is represented as a 16-bytes array, | ||
|
There was a problem hiding this comment. The room in 🇩🇪 seemed to favor a variable-length field here... I thought that was for good reason, too: 128 bits of precision is completely unnecessary for the purposes of most tracing systems, yet insufficient for others we heard about that needed 192. Why does this spec need to try and get this right? (We can set a hard max at 32 bytes or something if you are concerned about unbounded allocations) There was a problem hiding this comment. fwiw As a unit, zipkin usually implement things a lot later than the first request after significant cross-project interest exists, and there are hands to do it. Ex 128bit trace ids was implemented probably 2 years after the first sites started forking zipkin to accomplish this. This happened after a couple more recent glitches occurred including users of AWS lambda not being able to reuse the uuid there, as well zipkin users who have hybrid sites that employ stackdriver (which is 128bit). This topic has been discussed at length on other forums, so not trying to replicate it here. Suffice to say that 128bit is definitely in use in zipkin sites. To make things less conjectury, it would be nice to have a matrix of tracing systems and their trace context requirements someplace, maybe on a google doc or otherwise. In practice, Zipkin has never been asked directly to support a bit length higher than 128. However, we have been asked for more "squishy" things like opaque strings. Also, some tracers like sleuth were asked for lambda, which has an interesting timestamp+identifier trace id format. Having a matrix around can make things more objective as people can see if all of the systems they care about can work with a spec or not. It won't be about most, rather what site is running. For example, 9/10 is meaningless if that last one is the only one they use. There was a problem hiding this comment. If we accept variable size context we should definitely have a max size for both trace_id/span_id if 32 bytes fits all the systems then we should probably change to that. Do you know any system that is willing to change to a common format and uses more than 128bits? As @adriancole mentioned for the moment the systems that are interested in using this format are Zipkin and Google and both are fine with 128bits. One option to move forward is to make v0 having fixed sizes and we can define the v1 later when we have a clear system that wants to use this and requires more than 128bits. There was a problem hiding this comment. Also filed an issue to keep track of who wants/is willing to use this format. Please add your project here #4. Add any extra requirements. There was a problem hiding this comment. Tracelytics uses 160-bit trace IDs currently, and we are interested in a common format. FWIW AWS X-Ray uses a 128-bit trace ID format consisting of a 64-bit timestamp concatenated (using a hyphen) to a 96-bit unique ID, both encoded in hex. |
||
| e.g., `0x4bf92f3577b34da6a3ce929d0e0e4736`. All bytes 0 is considered invalid. | ||
|
|
||
|
|
||
| Implementation may decide to completely ignore the trace-context if the trace-id | ||
| is invalid. | ||
|
|
||
| #### Span-id | ||
|
|
||
| Is the ID of the caller span (parent). It is represented as a 8-bytes array, | ||
| e.g., `0x00f067aa0ba902b7`. All bytes 0 is considered invalid. | ||
|
|
||
| Implementation may decide to completely ignore the trace-context if the span-id | ||
| is invalid. | ||
|
|
||
| #### Trace-options | ||
|
There was a problem hiding this comment. it's odd to me that we support a versioning bit, yet have this 4-byte thing we only have 1 bit specified for... we could alternatively just have a single byte for version 0 and skip all of the endianness discussion. There was a problem hiding this comment. I was expecting this to fill up (the 4-byte options) with things like what you proposed (sampling probability). Uber also suggested an extra bit for deferred sampling decision. I am trying to get for the moment the minimum requirement. There was a problem hiding this comment. I am trying to collect data about how many bytes we need. So far the list contains 3 (based on Jager requirements). If the list is less than 8 we can definitely go with 1 byte for the options in v0. Should we have the sampling probability that @bhs proposed as a separate field? There was a problem hiding this comment. Done. 1-byte used for options. |
||
|
|
||
| Controls tracing options such as sampling, trace level etc. It is a 4-bytes | ||
| representing a 32-bit unsigned integer in little-endian order. The least | ||
|
There was a problem hiding this comment. java's default is big-endian.. there are a lot of java programmers There was a problem hiding this comment. ps to elaborate on this, there's a couple things in play. On one hand, if you have only one flag, the endianness isn't an issue. Most users have sampled set as a single bit, and that's the only flag used in zipkin routinely. On the other hand, the platform default encoding of java as big-endian results in many not being aware of endian-ness in general. Mistakes coding anything binary have been difficult in zipkin, resulting in json actually. While the case endianness is "little" elsewhere, anectodally I've noticed those who have little endian platforms already know how to do big endian. Main thing is that they get annoyed when endian isn't documented. Ex when folks were doing thrift encoding I remember someone rather annoyed that thrift tbinary format was big endian without saying so (because it was implicitly that in java). food for thought There was a problem hiding this comment.
not only that, but big-endian is also "network byte order". Wikipedia says:
If I can write the 32bit flags number as 0x00000001, it seems more natural to have There was a problem hiding this comment. I know that network byte order is big-endian. But probably 99% of the CPU in used are little-endian. Let's vote here #3 for this. There was a problem hiding this comment. I think @bhs proposal to reduce this to 1 byte is an alternative option. Updated the issue with this possibility. There was a problem hiding this comment. Done. 1-byte used for options. |
||
| significant bit provides recommendation whether the request should be traced or | ||
| not (1 recommends the request should be traced, 0 means the caller does not make | ||
| a decision to trace and the decision might be deferred). The flags are | ||
| recommendations given by the caller rather than strict rules to follow for 3 | ||
| reasons: | ||
|
|
||
| 1. Trust and abuse. | ||
| 2. Bug in caller | ||
| 3. Different load between caller service and callee service might force callee | ||
| to down sample. | ||
|
There was a problem hiding this comment. Assuming sampling logic may differ much between vendors - why place sampling flag to the trace context? Why not pass it as a separate header that is subject for vendor-specific logic? In some cases to resolve the issues you describe - extension libraries will need more than just a sampling flag. Most probably some additional information from You may also have multi-tier sampling. Take an example of local agent mode that @bogdandrutu demo-ed for census. You may want to sample data that you send to backend with sampling rate Another consideration - properties of There was a problem hiding this comment. @SergeyKanzhelev consistent sampling across the whole trace is very important characteristic. If the sampling decision is not propagated and the trace spans multiple implementations There was a problem hiding this comment. @yurishkuro if you are making a sampling decision based on When you do sampling you may need to estimate the count of spans exhibit certain properties based on sampled data. In case of statistical sampling you may just multiple the raw count of spans to sampling percentage to get statistically accurate number. If sampling decision forced from above without the information on sampling percentage of originator - you cannot do this type of estimations. There was a problem hiding this comment. statistical sampling on every layer is just one example of different type of sampling you may want to implement and you will need more than a bit of informaiton There was a problem hiding this comment.
Having the sampling bit does not prevent you from passing extra data or making more elaborate decisions, but it's not part of the proposed standard. NOT having the sampling bit pretty much guarantees that the trace will be broken, unless every implementation makes the decision based on the exact same formula, like Sampling bit is a recommendation. If a service can respect it and handle the volume - great. If it cannot respect it 100%, maybe it can respect it for "more important" spans like RPCs, and shed the load by dropping in-process spans & metrics. Essentially it does not put any restrictions on how you want to implement sampling, but still provides a way to achieve consistent sampling across the stack, if the volume allows it. There was a problem hiding this comment. That is precisely my question. Do you think it will be typical for libraries to trust this flag? Or every vendor and library will implement own logic so this flag will never be used? Should this standard define the flag or let customers decide on sampling algorithm across services in their org as a decision separate from the data correlation protocol? Beauty of sampling flag is ability to implement solutions like forced data collection for period of time or specific span name. Also it removes the need to synchronize the sampling decision algorithm. On negative side - services looses control of the data volume and statistical accuracy of collected data. Protocol is optimized for a single team owning many components with relatively similar load. It is not always the case. Every component may be owned by a team which want to play nice and contribute to the overall correlation story. But have a bigger priority to fully control telemetry volume and distribution collected from this component. There was a problem hiding this comment.
I don't agree with this assessment - the flag is recommendation, an implementation does not have to respect it if it thinks it can do a better job. There was a problem hiding this comment.
Yes, I think it's a very common implementation to simply trust the flag, in the absence of other knowledge about the system. There was a problem hiding this comment.
What I want to avoid is the situation when you heavily rely on implementation detail of upstream component sampling algo. Also I want to make sure there is an easy to replicate on any language mechanism to control the flood of telemetry in case of non-matching load patterns. Third, for many Application Insights scenarios we need to keep the sampling percentage that was used to sample telemetry out so we can run statistical algorithms on telemetry and recognize patterns. So single bit will not generally work for us as a universal sampling mechanism. I'd propose to have a |
||
|
|
||
| The behavior of other bits is currently undefined. | ||
|
|
||
| #### Examples of HTTP headers | ||
|
|
||
| *Valid sampled Trace-Context:* | ||
|
|
||
| ``` | ||
| Value = 004bf92f3577b34da6a3ce929d0e0e473600f067aa0ba902b701000000 | ||
|
There was a problem hiding this comment. IMO this is over-optimized / pre-optimized for performance. In the context of an RPC, parsing these things is going to be a joke from an overhead standpoint... I'd rather have an id format that's human-readable, or at least human-parseable (without counting characters by hand, that is). E.g., this: There was a problem hiding this comment. made comment elsewhere but I also agree a delimited format is near required from an operator POV. Especially as people cross-reference trace ids in other systems. Like splunk search shows something where one system logs the trace id as a logging key. Copy/paste this into zipkin or another system. There's also the "curl"ability which is "can a user work only having curl?" Straightforwardness is important for adoption, or rejection prevention. There was a problem hiding this comment. I think it is not unreasonable to add an extra character especially for the HTTP/String version to make this more readable. If this is an agreement I think we should do it this way. I will start few issues where we can vote for certain things to get all these concerns/issues/questions resolved and CC interested people. There was a problem hiding this comment. Please vote/comment here: #2 There was a problem hiding this comment. Done. Used a delimiter between all fields. |
||
| Version = 0 (0x00) | ||
| TraceId = 0x4bf92f3577b34da6a3ce929d0e0e4736 | ||
|
There was a problem hiding this comment. 0x prefix will likely lead to encoding bugs as people will almost certainly copy/paste if we give opportunity to |
||
| SpanId = 0x00f067aa0ba902b7 | ||
| TraceOptions = 0x1 // sampled | ||
| ``` | ||
|
|
||
| *Valid not-sampled Trace-Context:* | ||
|
|
||
| ``` | ||
| Value = 004bf92f3577b34da6a3ce929d0e0e473600f067aa0ba902b700000000 | ||
| Version = 0 (0x00) | ||
| TraceId = 0x4bf92f3577b34da6a3ce929d0e0e4736 | ||
| SpanId = 0x00f067aa0ba902b7 | ||
| TraceOptions = 0x0 // not-sampled | ||
|
There was a problem hiding this comment. make an example that breaks with endian, especially if we use little endian There was a problem hiding this comment. @adriancole This is terrific initiative. The proposal is looking quite reasonable. I think the only debatable piece is the presence of There was a problem hiding this comment. No more endianness. I like the idea of maybe having multiple supported systems, but one of the main purpose is to make all these systems work together so having separate encodings for each system will not be that much of a win. |
||
| ``` | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which systems are committed to support this? Which systems would you like to support this? Might be helpful to
@-mention people from the latter so we can have any debates before this is merged.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
putting in 2p eventhough the question was for @bogdandrutu :)
TL;DR; I'd expect tracing systems which maintain all of their tracers to make a more top-down decision, but yeah not currently the case in zipkin as this trace context is compatible with zipkin's wire format.
Currently, some zipkin-compatible tracers support vendor-specific or not widely used formats. Those types of tracers will have an easier time with this since zipkin's trace context isn't inherently incompatible with this specification (at the moment). I'd expect the cross-section of google+zipkin users to be first to ask, as it is likely google will land some variant of this first (grpc, cloud services and stackdriver instrumentation).
Similar to other things that happen, when that demand occurs it is usually in a repo or two. For example, our first requests for StackDriver and X-Ray trace support came from sleuth issues list. Tracers run independently and can move to support something sooner or later. I often ping people across tracers on things like this so that they can weigh-in before organic demand hits.
Regardless, support or not support lies in the scope of each tracer to decide until there are server implications like the trace context is incompatible or too wide to store in zipkin.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the current format (that is compatible with ZIpkin and Google) we try to make at least these systems to work with this format. Having a common place for the specs (maybe some simple implementation in multiple languages) is one of the goal.
Anyone who is interested in using this format is welcome to join the effort and send patches/PRs etc.