programatically drop and keep traces / delaying sampling decisions #586
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is addressing two needs that we found while upgrading instrumentation on HTTP servers:
First, we always try to start the HTTP Server Spans as early as possible in the request processing, so early that it might happen (in fact, it is almost always the case) that at the point where we are starting the Span, we don't have an operation name to assign to it. So far when we got into this situation we would just assign a default name and then the instrumentation would take care of assigning a decent operation name down the road, where "decent" means that it doesn't have any high cardinality parts on it as it is used for the
span.xxx-time
metric. In the pre-2.0 world, we only had the constant and random samplers so it just wouldn't matter what the operation name was, but since we are introducing an adaptive sampler that actually needs an operation name to work, we should allow for the sampling decision to be taken at a point where we actually have an operation name that could be given to the adaptive sampler if it is being used.Second need is, to have some manual control over sampling decisions. This comes in two forms: one is to be able to "suggest" a Sampling Decision, so that if a Span is going to start a new Trace and a sampling decision must be taken, the tracer will use the suggestion instead of using the sampler; the other form is by explicitly asking to keep or drop a trace, which can be really useful if the user somehow detects that they really want to keep/drop a trace, regardless of the sampling decision already available. When a trace is dropped/kept, all Spans in the current process will immediately see the change. This might lead to missing data in some cases where, for example, the decision to keep a trace is made after several calls to external services were fired with the previous sampling decision, but that is something we can't influence much.
In the case of our own instrumentation, we will suggest a Unknown sampling decision when we start the Spans, then let the instrumentation hit the points where we can actually assign a proper operation name and then make the Span take a sampling decision.