Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix synthetics-derived contexts being ignored #856

Merged
merged 3 commits into from
Nov 20, 2019

Conversation

delner
Copy link
Contributor

@delner delner commented Nov 17, 2019

The following does not produce a distributed trace with the designated trace ID:

curl -H "x-datadog-trace-id: 3238677264721744442" -H "x-datadog-parent-id: 0" -H "x-datadog-sampling-priority: 1" -H "x-datadog-origin: synthetics" localhost:3000

When synthetics traces are submitted via HTTP headers, they often are missing parent_id or have a value of 0, which is not a valid span ID, and thus the context's parent_id becomes nil.

When the tracer encounters parent_id = nil, it assumes the context is malformed and ignores its trace ID and parent ID, opting to use the ones generated from the local span instead. Ultimately this causes synthetics traces to go missing, because the trace IDs don't match what's sent by synthetics.

This pull request ensures that the trace ID from synthetics is respected, regardless of what parent ID is, so that the resulting traces receive the correct trace ID.

@delner delner added bug Involves a bug core Involves Datadog core libraries labels Nov 17, 2019
@delner delner added this to the 0.29.0 milestone Nov 17, 2019
@delner delner requested review from marcotc, brettlangdon and a team November 17, 2019 00:00
@delner delner self-assigned this Nov 17, 2019
Copy link
Contributor Author

@delner delner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This first iteration is the bare-minimum sort of implementation; a simple check and a simple integration test. As far as getting an immediate fix out, this is "okay" but really isn't net positive in terms of making the tracer code more manageable and less error prone.

It does not do a good job of identifying/addressing the source of the bug (parent IDs parsed to nil in HTTP propagation) and generally reveals some coupling and complexity that should be refactored. Some of these possible improvements are noted below.

lib/ddtrace/tracer.rb Outdated Show resolved Hide resolved
spec/ddtrace/tracer_integration_spec.rb Show resolved Hide resolved
@@ -76,4 +76,53 @@ def lang_tag(span)
it { expect(@child_root_span).to be child_span }
end
end

context 'with synthetics' do
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a good idea to have a few top level integration tests like this: this bug manifested because we never actually had any basic tests to verify the most simple happy path for synthetics.

In the future, I'd like to extract integration tests like this to their own suite of sorts, separate from unit tests, as each kind of test serves an important purpose, but integration tests like this have a habit of creeping into the space unit tests should possess.

lib/ddtrace/tracer.rb Outdated Show resolved Hide resolved
@delner delner force-pushed the fix/synthethics_distributed_tracing branch from 83eb348 to ce7ba16 Compare November 18, 2019 21:50
@delner delner force-pushed the fix/synthethics_distributed_tracing branch from ce7ba16 to 14deece Compare November 18, 2019 21:59
@delner
Copy link
Contributor Author

delner commented Nov 19, 2019

Should be ready for final review @brettlangdon.

context 'with synthetics' do
context 'which applies the context from distributed tracing headers' do
let(:trace_id) { 3238677264721744442 }
let(:parent_id) { 0 }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we test when parent id header is not set at all?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm, I do see one above in #extract test cases.

@delner delner merged commit 537d921 into master Nov 20, 2019
@delner delner deleted the fix/synthethics_distributed_tracing branch November 20, 2019 17:52
@delner delner added this to Released in Active work Nov 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Involves a bug core Involves Datadog core libraries
Projects
Active work
  
Released
Development

Successfully merging this pull request may close these issues.

None yet

3 participants