Skip to content

fix: Fix OTEL trace context propagation race - #2450

Merged
Luca Forstner (lforst) merged 3 commits into
mainfrom
fix/trace_context_injection_race
Sep 10, 2026
Merged

fix: Fix OTEL trace context propagation race#2450
Luca Forstner (lforst) merged 3 commits into
mainfrom
fix/trace_context_injection_race

Conversation

@delner

Copy link
Copy Markdown
Contributor

Problem

Span.inject() / injectTraceContext() are synchronous, but the experiment id they need is not. _getOtelParent() builds the braintrust.parent baggage entry from parentObjectId.getSync(), which stays undefined until POST /api/experiment/register returns. An inject that happens before then emits a traceparent with no baggage: trace identity with no destination.

The failure lands in a different process. The receiving side warns ("Received traceparent without a braintrust.parent ... Starting a fresh local span instead") and starts a fresh local trace, so spans from a subprocess agent land in project logs instead of the experiment. Discovery happens while debugging the consumer, not the producer. It is also first-span-only: the id caches once resolved, so it breaks the first turn of a run and then quietly stops, which is how it ships broken and goes unnoticed.

Why this fixes it

The precondition belongs at experiment init, not in the injector, so the race becomes structurally impossible rather than narrower:

  • Eval() already awaited experiment._waitForId(), but only when globalThis.BRAINTRUST_CONTEXT_MANAGER !== undefined (the opt-in @braintrust/otel integration). That gate was correct when written and stopped being correct once inject started depending on the same resolved id. Ungating it means no span can exist before the id does.
  • inject() stays synchronous, matching the OTel propagator API, which is sync by spec.
  • SpanImpl.inject() now warns when it has no braintrust.parent, symmetric with the receive-side warning. It does not prevent the failure, but it moves discovery to the process that caused it, which matters for the paths eager resolution does not cover.
File Change
js/src/framework.ts Ungate await experiment._waitForId() in Eval()
js/src/logger.ts Warn in SpanImpl.inject() when braintrustParent is empty

Testing

  • js/src/framework.test.ts: injectTraceContext() inside the first Eval() task must carry braintrust.parent. Verified against the old gate, where it fails with expected undefined to be 'experiment_id:experiment-id' for every row, not just the first: concurrent tasks all inject before the first flush resolves the id.
  • js/src/propagation.test.ts: unresolved experiment id warns and omits the parent; resolved id injects experiment_id:<id>.
  • e2e trace-context-and-continuation, trace-primitives-basic, test-framework-evals-node, otel-compat-mixed-tracing, durable-eval-webhook pass, run twice.

Notes

  • Python is not affected. _get_parent_info() already falls back to a blocking parent_object_id.get() for experiments, and Python's LazyValue.get() computes on the calling thread, so a sync inject() there resolves the id rather than dropping it.
  • Manual init() plus startSpan() plus inject is still racy, since init() is fully lazy and lazyId is not kicked off until the first flush. It now warns instead of failing silently. Closing it would mean either firing lazyId.get() from Experiment.startSpan (narrows the window, does not close it) or documenting await experiment.id before injecting.

@lforst Luca Forstner (lforst) changed the title Fixed: trace context injection race fix: Fix OTEL trace context propagation race Sep 10, 2026
@lforst
Luca Forstner (lforst) marked this pull request as ready for review September 10, 2026 13:59
@lforst
Luca Forstner (lforst) merged commit 5361846 into main Sep 10, 2026
47 checks passed
@lforst
Luca Forstner (lforst) deleted the fix/trace_context_injection_race branch September 10, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants