Skip to content

Tracer provider defaults to OTEL default provider for evals#127

Merged
David Elner (delner) merged 3 commits intomainfrom
fix/default_tracer_provider_for_eval
Mar 22, 2026
Merged

Tracer provider defaults to OTEL default provider for evals#127
David Elner (delner) merged 3 commits intomainfrom
fix/default_tracer_provider_for_eval

Conversation

@delner
Copy link
Collaborator

Trace scorers query spans via BTQL after the task runs. For spans to be queryable, they must first be flushed from the OpenTelemetry tracer provider.

Previously, force_flush was called on eval_context.tracer_provider, which was nil when users didn't explicitly pass tracer_provider: to Eval.run (the common case). The safe navigation operator (&.) silently skipped the flush. In practice, the OTLP exporter's background thread usually shipped spans in time, but this was a race condition — not a guarantee.

Additionally, the runner maintained its own fallback (eval_context.tracer_provider || OpenTelemetry.tracer_provider) to create spans, but this resolved provider was never persisted, so downstream code couldn't rely on it.

Changes

  • Context::Factory#build now defaults tracer_provider to OpenTelemetry.tracer_provider when none is provided, so the context always carries a resolved provider.
  • The runner's force_flush call uses a respond_to? guard instead of &., since the global provider may be a ProxyTracerProvider that doesn't implement force_flush (e.g. if Braintrust.init was never called).
  • Removed the redundant fallback in the runner's initialize since the context now guarantees a provider.

Impact

Users running trace scorers without passing tracer_provider: to Eval.run should no longer need that workaround. The flush will execute against the correct provider automatically.

# @param errors [Queue] Thread-safe error collection queue
def run_eval_case(case_context, errors)
# Each eval case starts its own trace — detach from any ambient span context
eval_span = tracer.start_root_span("eval")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i feel like this would be more readable if case_context was shorter

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Practically speaking we could abbreviate the variable name to case because its semantically the same.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh whoops, forgot case is reserved. So it'd have to be kase (Ruby convention)

@delner David Elner (delner) merged commit 44c169c into main Mar 22, 2026
7 checks passed
@delner David Elner (delner) deleted the fix/default_tracer_provider_for_eval branch March 22, 2026 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants