instrumentation telemetry: validate session id headers by mabdinur · Pull Request #6510 · DataDog/system-tests

mabdinur · 2026-03-16T20:07:03Z

Motivation

Enable telemetry session ID header tests (DD-Session-ID, DD-Root-Session-ID, DD-Parent-Session-ID) across process forks per the Stable Service Instance Identifier RFC.

Changes

GET /spawn_child – New endpoint in weblogs (Python flask, Node.js express4, Ruby rails72, PHP, Go net-http, Java spring-boot, .NET poc). Params: sleep, crash, fork. Uses fork when supported, exec otherwise. Runtimes without fork (Java, Go, PHP, .NET) return 400 for fork=true.
Tests – test_session_id_headers_across_forks and test_session_id_headers_across_spawned validate session ID headers in lifecycle telemetry. Uses get_lifecycle_events() to avoid lib-datadog metrics/log events. Asserts: DD-Session-ID = runtime_id, one root per app instance, at least two runtimes (parent + child).
Library interface – get_lifecycle_events() added to filter lifecycle events.
Docs – Endpoint spec in docs/weblog/end-to-end_weblog.md.
Manifests – Enabled for Ruby rails72; missing_feature for other weblogs and non-fork runtimes.

Workflow

⚠️ Create your PR as draft ⚠️
Work on your PR until the CI passes
Mark it as ready for review
- Test logic is modified? → Get a review from RFC owner.
- Framework is modified, or non-obvious usage of it → Get a review from R&P team

🚀 Once your PR is reviewed and the CI is green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified? I have the approval from R&P team
A docker base image is modified?
- The relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

docs/understand/weblogs/end-to-end_weblog.md

tests/test_telemetry.py

khanayan123 · 2026-03-17T16:26:03Z

As per https://dd.slack.com/archives/D032MDTSCR1/p1773765779731369

We need to assert:

Headers Present & Valid For Every Telemetry Event: DD-Session-ID always present, DD-Root-Session-ID present when a child process is forked/spawned
Root Stability Across Fork: Session-ID regenerates per process, Root-Session-ID inherited and never changes

docs/understand/weblogs/end-to-end_weblog.md

Co-authored-by: Munir Abdinur <munir.abdinur@datadoghq.com>

tests/test_telemetry.py

khanayan123 · 2026-03-18T14:14:23Z

Remaining gaps I believe are:

Gap 1: The test should assert that for every event where DD-Session-ID != root_session_id (i.e. every child event), DD-Root-Session-ID must be present not just that at least one event has it globally.

Gap 2 (exec vs fork): Same validation function for both test cases, exec propagation via env vars isn't distinctly tested.

mabdinur · 2026-03-18T14:47:39Z

Remaining gaps I believe are:

Gap 1: The test should assert that for every event where DD-Session-ID != root_session_id (i.e. every child event), DD-Root-Session-ID must be present not just that at least one event has it globally.

Gap 2 (exec vs fork): Same validation function for both test cases, exec propagation via env vars isn't distinctly tested.

Both cases should be covered by the current test. We can discuss it in our next sync

…ders

github-actions · 2026-03-18T15:14:01Z

CODEOWNERS have been resolved as:

utils/build/docker/dotnet/weblog/Endpoints/SpawnChildEndpoint.cs        @DataDog/apm-dotnet @DataDog/asm-dotnet @DataDog/system-tests-core
utils/build/docker/golang/app/_shared/common/spawn_child.go             @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/nodejs/express/fork_child.js                         @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/php/common/spawn_child.php                           @DataDog/apm-php @DataDog/system-tests-core
docs/understand/weblogs/end-to-end_weblog.md                            @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_kong.yml                                                  @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/test_telemetry.py                                                 @DataDog/libdatadog-telemetry @DataDog/apm-sdk-capabilities @DataDog/system-tests-core
utils/build/docker/dotnet/weblog/Program.cs                             @DataDog/apm-dotnet @DataDog/asm-dotnet @DataDog/system-tests-core
utils/build/docker/golang/app/net-http/main.go                          @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/java/spring-boot/src/main/java/com/datadoghq/system_tests/springboot/App.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/nodejs/express/app.js                                @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/install_ddtrace.sh                            @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/php/apache-mod/php.conf                              @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm/php-fpm.conf                             @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/python/flask/app.py                                  @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/ruby/rails72/app/controllers/system_test_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails72/config/routes.rb                        @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/interfaces/_library/core.py                                       @DataDog/system-tests-core

khanayan123

Thanks for addressing the comments, tests LGTM

datadog-datadog-prod-us1-2 · 2026-03-18T15:49:14Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 82 Tests failed

tests.test_config_consistency.Test_Config_HttpServerErrorStatuses_Default.test_status_code_400[uds-flask] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Agent received the incorrect amount of spans
assert 3 == 1
 +  where 3 = len([<utils.dd_types._datadog_agent_trace.DataDogAgentSpanLegacy object at 0x7f2052e87350>, <utils.dd_types._datadog_agent...egacy object at 0x7f2052836a80>, <utils.dd_types._datadog_agent_trace.DataDogAgentSpanLegacy object at 0x7f2052836630>])

self = <tests.test_config_consistency.Test_Config_HttpServerErrorStatuses_Default object at 0x7f2081f97470>

    def test_status_code_400(self):
        assert self.r.status_code == 400
    
        interfaces.library.assert_trace_exists(self.r)
...

tests.test_config_consistency.Test_Config_HttpServerErrorStatuses_Default.test_status_code_400[uds-flask] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Agent received the incorrect amount of spans
assert 3 == 1
 +  where 3 = len([<utils.dd_types._datadog_agent_trace.DataDogAgentSpanLegacy object at 0x7fe9330e9e20>, <utils.dd_types._datadog_agent...egacy object at 0x7fe933106480>, <utils.dd_types._datadog_agent_trace.DataDogAgentSpanLegacy object at 0x7fe93310eed0>])

self = <tests.test_config_consistency.Test_Config_HttpServerErrorStatuses_Default object at 0x7fe964576330>

    def test_status_code_400(self):
        assert self.r.status_code == 400
    
        interfaces.library.assert_trace_exists(self.r)
...

tests.test_config_consistency.Test_Config_HttpServerErrorStatuses_Default.test_status_code_400[uwsgi-poc] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Agent received the incorrect amount of spans
assert 2 == 1
 +  where 2 = len([<utils.dd_types._datadog_agent_trace.DataDogAgentSpanLegacy object at 0x7f0e625e9100>, <utils.dd_types._datadog_agent_trace.DataDogAgentSpanLegacy object at 0x7f0e625fa9f0>])

self = <tests.test_config_consistency.Test_Config_HttpServerErrorStatuses_Default object at 0x7f0e91186360>

    def test_status_code_400(self):
        assert self.r.status_code == 400
    
        interfaces.library.assert_trace_exists(self.r)
...

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 43bd633 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

mabdinur added 3 commits March 16, 2026 16:05

instrumentation telemetry: validate session id headers

8d9511f

add spawn test and clean up manifest

f387188

enable ruby tests

4974b0f

mabdinur mentioned this pull request Mar 16, 2026

[WIP] chore(telemetry): track process creation via it headers DataDog/dd-trace-rb#5464

Draft