fix(w3c): fix traceparent validation (backport #4791 to 1.7)

[mabdinur]

Description

Running the W3C trace context parametric tests (introduced here) surfaced four issues with how the python tracer propagates traceparent and tracestate headers. These issues are described below and are fixed in this PR.

Edge Case 1 (most important)

Currently trace_parent.islower() is used to check if a traceparent contains lower case letters and digits (uppercase letters are invalid). Unfortunately trace_parent.islower() also evaluates to False if there are no lowercase letters are present in the traceparent (ex: traceparent=00-12312312003-1232131321 would be marked as invalid and dropped). This is incorrect. Fix:
ddb9dc5.

Since the traceparent should contain 48 randomly generated hex values it is unlikely to only contain digits. This edge case is very unlikely to occur.

Current behavior:

• 00-123123123-23123213AAA -> uppercase letter detected traceparent is ignored
• 00-123123123-23123213132 -> no lowercase letter detected traceparent is ignored [WRONG]
• 00-123123123-2312322aaaa -> lowercase letter detected traceparent is propagated

Expected behavior:

• 00-123123123-23123213132AAAA -> uppercase detected traceparent is ignored
• 00-123123123-23123213132 -> no uppercase detected traceparent is propagated
• 00-123123123-23123213132aaaa -> no uppercase detected traceparent is propagated

Edge Case 2

If traceparent contains an invalid characters (ex: period) do not propagate the traceparent. Start a new trace. Currently we can propagate invalid trace_ids (this is because we truncate the trace_id and only convert the last 16 digits to hex, we ignore all the characters in the first 16 digits). Fix: ae1e73e

Edge Case 3

Ensure that traceflags contain ONLY two digits (00 or 01). 000 and 001 are examples of invalid values that should not be propagated. Fix: adc4f80

Edge Case 4

Ensure traceparent version contain ONLY two digits (00 or 01). 000 and 001 are examples of invalid values that should not be propagated. Fix: 2e416c5

Edge Case 5

The W3C specification is additive. Although only traceparents with the version 00 are supported we should attempt to parse traceparent with different formats.

Example:
"01-12345678901234567890123456789012-1234567890123456-01-what-the-future-will-be-like"

In the traceparent example above we should parse the version, trace id, span id, and sample flag and ignore the trailing values. Fix: 4cebe7514682787f6068754037fba569f4af3d60

Edge Case 6

This edge case is not addressed in this PR. I am including it here for completeness. In the W3C tracecontext specification a tracer SHOULD set two http headers, one header should set the tracestate and the other should set the traceparent. However, if duplicate traceparent and tracestate headers are received, the tracer must processes and reconcile these headers (logic: 1) receive duplicate traceparent headers with different values -> drop these values and start a new trace. 2) receive duplicate tracestates with different tags -> combine the tracestates and propagate it).

In the ddtrace library http headers are added to a dictionary, so duplicate http header values are overwritten. To address this edge case the tracer must be able to store detect and store all key value pairs in http headers. Since this edge case requires significant changes to distributed tracing and does not resolve a critical issue this work can be deferred.

Testing

Testing will covered by system tests.

Reviewer Checklist

• Title is accurate.
• Description motivates each change.
• No unnecessary changes were introduced in this PR.
• Avoid breaking API changes unless absolutely necessary.
• Tests provided or description of manual testing performed is included in the code or PR.
• Release note has been added and follows the library release note guidelines, or else changelog/no-changelog label added.
• All relevant GitHub issues are correctly linked.
• Backports are identified and tagged with Mergifyio.

Co-authored-by: Brett Langdon brett.langdon@datadoghq.com
Co-authored-by: Tahir H. Butt tahir.butt@datadoghq.com

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.02%. Comparing base (1a308d1) to head (a7b2ba3).
Report is 35 commits behind head on 1.7.

Additional details and impacted files

@@           Coverage Diff           @@
##              1.7    #4803   +/-   ##
=======================================
  Coverage   78.02%   78.02%           
=======================================
  Files         808      808           
  Lines       62053    62053           
=======================================
  Hits        48416    48416           
  Misses      13637    13637           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.