Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

programatically drop and keep traces / delaying sampling decisions #586

Merged
merged 1 commit into from
May 21, 2019

Conversation

ivantopo
Copy link
Contributor

@ivantopo ivantopo commented May 15, 2019

This PR is addressing two needs that we found while upgrading instrumentation on HTTP servers:

First, we always try to start the HTTP Server Spans as early as possible in the request processing, so early that it might happen (in fact, it is almost always the case) that at the point where we are starting the Span, we don't have an operation name to assign to it. So far when we got into this situation we would just assign a default name and then the instrumentation would take care of assigning a decent operation name down the road, where "decent" means that it doesn't have any high cardinality parts on it as it is used for the span.xxx-time metric. In the pre-2.0 world, we only had the constant and random samplers so it just wouldn't matter what the operation name was, but since we are introducing an adaptive sampler that actually needs an operation name to work, we should allow for the sampling decision to be taken at a point where we actually have an operation name that could be given to the adaptive sampler if it is being used.

Second need is, to have some manual control over sampling decisions. This comes in two forms: one is to be able to "suggest" a Sampling Decision, so that if a Span is going to start a new Trace and a sampling decision must be taken, the tracer will use the suggestion instead of using the sampler; the other form is by explicitly asking to keep or drop a trace, which can be really useful if the user somehow detects that they really want to keep/drop a trace, regardless of the sampling decision already available. When a trace is dropped/kept, all Spans in the current process will immediately see the change. This might lead to missing data in some cases where, for example, the decision to keep a trace is made after several calls to external services were fired with the previous sampling decision, but that is something we can't influence much.

In the case of our own instrumentation, we will suggest a Unknown sampling decision when we start the Spans, then let the instrumentation hit the points where we can actually assign a proper operation name and then make the Span take a sampling decision.

@ivantopo ivantopo changed the title programatically drop and keep traces and delaying sampling decisions programatically drop and keep traces / delaying sampling decisions May 15, 2019
@ivantopo ivantopo force-pushed the delay-sampling-decision branch from cf3896d to 248c81f Compare May 19, 2019 20:47
@ivantopo ivantopo marked this pull request as ready for review May 19, 2019 20:47
Copy link
Contributor

@dpsoft dpsoft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ivantopo LGTM!

* Name assigned to the operation that triggers the sampler. The name is typically assigned by either the user or
* automatic instrumentation.
*/
def operationName(): String
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for the record, I really don't like repeating the class name in any of the members, but with this name it was quite easy to make both Span and SpanBuilder implement this trait :)

@ivantopo ivantopo merged commit 2740a42 into kamon-io:master May 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants