Skip to content

New - expand trace flags and temp workaround for upstream bug.#460

Merged
cleverchuk merged 4 commits intomainfrom
cc/NH-133007
Apr 10, 2026
Merged

New - expand trace flags and temp workaround for upstream bug.#460
cleverchuk merged 4 commits intomainfrom
cc/NH-133007

Conversation

@cleverchuk
Copy link
Copy Markdown
Contributor

Summary

Fixes an OTel SDK bug that silently drops X-Trace-Options for root spans, updates trace flag handling to support the full W3C hex byte spec instead of only 00/01, pins Netty to 4.1.132.Final to address a CVE-2026-33871, and parallelizes the release smoke test in CI.

Changes

ThreadLocal fallback for XtraceOptions propagation

The OTel SDK has a bug (opentelemetry-java#8012) where it replaces the parent Context with Context.root() when constructing root spans. This discards any XtraceOptions placed in the context during extract(), causing trigger-trace and custom KVs from the X-Trace-Options header to be silently lost.

A new PropagatedContext class stores XtraceOptions in a ThreadLocal as a side-channel. This is safe because the OTel agent's PropagatingFromUpstreamInstrumenter.start() calls extract() and shouldSample() synchronously in the same method — no async gap, no thread switch — across all server instrumentations (Servlet, Netty, WebFlux, gRPC). The sampler reads from the ThreadLocal only when the context-based lookup returns null, and unconditionally clears it in a finally block.

Trace flags handling

Previously, trace flags were hardcoded to "00" or "01" (sampled/not-sampled). The W3C trace context spec defines trace flags as a full hex byte, and upstream OTel now emits values beyond 01 (e.g., 02, 03). Changes:

  • Replaces isSampled() ? "01" : "00" with getTraceFlags().asHex() in the context propagator and response header customizer
  • Relaxes sw tracestate flag validation from exact "00"/"01" to a two-character hex pattern
  • k6 smoke tests now use bitwise check (parseInt(flag, 16) & 1) instead of exact string comparison to determine sampled status, matching the W3C spec where bit 0 indicates the sampled flag
  • Java smoke test assertions updated to accept any valid two-character hex flag

Netty BOM for CVE remediation

Adds the Netty BOM (io.netty:netty-bom:4.1.132.Final) to centralized dependency management, pinning all transitive Netty dependencies to a version that addresses CVE-2026-33871.

CI: parallelize release smoke test

The release smoke test job now depends on s3-stage-upload instead of release-test, removing the serial dependency and allowing it to run in parallel.

Misc

  • Adds @ToString (Lombok) to TraceDecision and XtraceOptions for better debug logging

Test services data

  1. e-1712644058766987264
  2. e-1712643928659124224
  3. e-1742334541200846848
  4. e-1777406072376840192

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes loss of X-Trace-Options on root spans due to an upstream OTel SDK context handling bug, expands trace-flags handling to support full W3C hex-byte semantics, pins Netty for CVE remediation, and parallelizes a CI smoke-test job.

Changes:

  • Add a ThreadLocal “side channel” (PropagatedContext) populated during extract() and consumed by SolarwindsSampler as a fallback for root-span sampling.
  • Replace hardcoded "00"/"01" flag logic with TraceFlags.asHex() and relax sw tracestate validation to accept any 2-hex-char byte.
  • Pin Netty via BOM, update smoke tests (Java + k6) to accept/interpret full hex-byte trace flags, and adjust CI job dependencies to increase parallelism.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
smoke-tests/src/test/java/com/solarwinds/SmokeTestV2.java Updates log regex to accept any 2-hex-char trace_flags.
smoke-tests/src/test/java/com/solarwinds/SmokeTest.java Updates log regex to accept any 2-hex-char trace_flags.
smoke-tests/k6/basic.js Updates sampled/not-sampled checks to use W3C bitwise semantics.
libs/shared/src/test/java/com/solarwinds/opentelemetry/extensions/SolarwindsSamplerTest.java Adds coverage for root-span sampler fallback to PropagatedContext.
libs/shared/src/test/java/com/solarwinds/opentelemetry/extensions/SolarwindsContextPropagatorTest.java Adds coverage that extract() populates PropagatedContext.
libs/shared/src/test/java/com/solarwinds/opentelemetry/extensions/SamplingUtilTest.java Extends tests for tracestate flag validation beyond 00/01.
libs/shared/src/main/java/com/solarwinds/opentelemetry/extensions/SolarwindsSampler.java Reads XtraceOptions from PropagatedContext fallback and clears it in finally.
libs/shared/src/main/java/com/solarwinds/opentelemetry/extensions/SolarwindsContextPropagator.java Injects full hex-byte flags into sw tracestate and sets PropagatedContext on extract.
libs/shared/src/main/java/com/solarwinds/opentelemetry/extensions/SamplingUtil.java Validates sw tracestate flags as any 2-hex-char byte.
libs/shared/src/main/java/com/solarwinds/opentelemetry/extensions/ResponseHeaderCustomizer.java Emits X-Trace using TraceFlags.asHex() instead of 00/01.
libs/shared/src/main/java/com/solarwinds/opentelemetry/extensions/PropagatedContext.java Introduces ThreadLocal holder for propagated XtraceOptions.
libs/sampling/src/main/java/com/solarwinds/joboe/sampling/XtraceOptions.java Adds Lombok @ToString for debug visibility.
libs/sampling/src/main/java/com/solarwinds/joboe/sampling/TraceDecision.java Adds Lombok @ToString for debug visibility.
dependencyManagement/build.gradle.kts Adds Netty BOM pin to 4.1.132.Final.
.github/workflows/push.yml Changes smoke-test job dependency to run in parallel with other release checks.

Comment thread smoke-tests/k6/basic.js
Comment thread smoke-tests/k6/basic.js
Comment thread smoke-tests/k6/basic.js
Comment thread smoke-tests/k6/basic.js
Comment thread smoke-tests/k6/basic.js
Comment thread smoke-tests/k6/basic.js
@cleverchuk cleverchuk changed the title Cc/nh 133007 New - expand trace flags and temp workaround for upstream bug. Apr 9, 2026
jerrytfleung
jerrytfleung previously approved these changes Apr 9, 2026
Copy link
Copy Markdown

@jerrytfleung jerrytfleung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Copy link
Copy Markdown

@jerrytfleung jerrytfleung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's go

@cleverchuk cleverchuk merged commit fa13daa into main Apr 10, 2026
27 of 29 checks passed
@cleverchuk cleverchuk deleted the cc/NH-133007 branch April 10, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants