[tracing] Assert that OTLP spans have full 128-bit trace IDs

[zacharycmontoya]

Motivation

End-to-end testing identified that traces exported as OTLP by dd-trace-py had incorrect trace IDs. The HTTP server span had a 128-bit trace ID while its child spans had that same trace ID except the upper 64 bits were zeroed out. This test will ensure that we track and remediate this behavior across all languages.

Changes

Add test/otel_test_tracing_otlp.py::Test_Otel_Tracing_OTLP::test_128bit_trace_id_consistent_across_spans to validate that implementations export all OTLP spans with the full 128-bit trace ID. This was identified as an issue in the libdatadog implementation

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/otel/test_tracing_otlp.py                                         @DataDog/system-tests-core
utils/interfaces/_open_telemetry.py                                     @DataDog/system-tests-core

[datadog-prod-us1-3]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 33 Pipeline jobs failed

Testing the test | System Tests (python, dev) / End-to-end #1 / flask-poc 1     

🔧 Fix in code (Fix with Cursor).

3 failed tests due to assertion errors: 'Snapshot was not received'

🧪 3 Tests failed

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions.test_log_line_capture_expressions[flask-poc] from system_tests_suite   (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f2e9c42b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f2e9c42b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions object at 0x7f2e9490f7a0>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions object at 0x7f2e9490f7a0>

    def test_log_line_capture_expressions(self):
>       self._assert()

...

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions.test_complex_capture_expressions[flask-poc] from system_tests_suite   (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f2e9c42b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f2e9c42b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7f2e9490dd00>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7f2e9490dd00>

    def test_complex_capture_expressions(self):
>       self._assert()

...

View all 3 test failures

Testing the test | System Tests (python, dev) / End-to-end #1 / python3.12 1     

🔧 Fix in code (Fix with Cursor).

3 failed tests due to assertion errors related to probe diagnostics and trace ID propagation.

🧪 2 Tests failed

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions.test_log_line_capture_expressions[python3.12] from system_tests_suite   (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f8e4762b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f8e4762b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions object at 0x7f8e3de423f0>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions object at 0x7f8e3de423f0>

    def test_log_line_capture_expressions(self):
>       self._assert()

...

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions.test_complex_capture_expressions[python3.12] from system_tests_suite   (Fix with Cursor)

AssertionError: assert 'Probes did not reach INSTALLED status' is None
 +  where 'Probes did not reach INSTALLED status' = <built-in method join of str object at 0x7f8e4762b720>(['Probes did not reach INSTALLED status'])
 +    where <built-in method join of str object at 0x7f8e4762b720> = '\n'.join
 +    and   ['Probes did not reach INSTALLED status'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7f8e3de41df0>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7f8e3de41df0>

    def test_complex_capture_expressions(self):
>       self._assert()

...

Testing the test | System Tests (python, dev) / End-to-end #1 / uds-flask 1     

🔧 Fix in code (Fix with Cursor).

3 failed tests due to mismatched trace IDs. AssertionError: Found 6 span(s) in the same logical trace with a different 128-bit traceId than the server span.

🧪 2 Tests failed

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions.test_log_line_capture_expressions[uds-flask] from system_tests_suite   (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f8a0f82b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f8a0f82b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions object at 0x7f8a01e323f0>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions object at 0x7f8a01e323f0>

    def test_log_line_capture_expressions(self):
>       self._assert()

...

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions.test_complex_capture_expressions[uds-flask] from system_tests_suite   (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f8a0f82b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f8a0f82b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7f8a01e331d0>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7f8a01e331d0>

    def test_complex_capture_expressions(self):
>       self._assert()

...

View all 33 failed jobs.

🧪 3 Tests failed in 1 job

Testing the test | main  

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions.test_log_line_capture_expressions[django-poc] from system_tests_suite   (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7fd88e42b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7fd88e42b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions object at 0x7fd884d24d70>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Line_Capture_Expressions object at 0x7fd884d24d70>

    def test_log_line_capture_expressions(self):
>       self._assert()

...

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions.test_complex_capture_expressions[django-poc] from system_tests_suite   (Fix with Cursor)

AssertionError: assert 'Probes did not reach INSTALLED status' is None
 +  where 'Probes did not reach INSTALLED status' = <built-in method join of str object at 0x7fd88e42b720>(['Probes did not reach INSTALLED status'])
 +    where <built-in method join of str object at 0x7fd88e42b720> = '\n'.join
 +    and   ['Probes did not reach INSTALLED status'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7fd884d24b00>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7fd884d24b00>

    def test_complex_capture_expressions(self):
>       self._assert()

...

View all 3 test failures

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 110f38a | Docs | Datadog PR Page | Give us feedback!}