Skip to content

fix: avoid OTel "Failed to detach context" error on early run_async close - #6560

Open
GanziMan wants to merge 1 commit into
google:mainfrom
GanziMan:fix/otel-invocation-span-detach
Open

fix: avoid OTel "Failed to detach context" error on early run_async close#6560
GanziMan wants to merge 1 commit into
google:mainfrom
GanziMan:fix/otel-invocation-span-detach

Conversation

@GanziMan

@GanziMan GanziMan commented Aug 3, 2026

Copy link
Copy Markdown

Fixes #6559.

Problem

record_invocation (schema v1, the default off Agent Engine) wraps
runners._run_node_async — an async generator — with
start_as_current_span("invocation"). When a caller stops iterating
Runner.run_async() early (a common break/return as soon as the final
response arrives), the generator is finalized (GeneratorExit /
CancelledError) in a different execution context than the one where the
span was attached. The automatic context detach() then raises
ValueError: <Token ...> was created in a different Context.

OpenTelemetry swallows that exception inside context.detach() but logs it at
ERROR (Failed to detach context) on every early-terminated run. The
invocation still completes correctly, so this is cosmetic — but it creates
significant log noise and false error-rate spikes (e.g. Datadog). Note that
OTEL_SDK_DISABLED=true does not suppress it, since attach()/detach() live
in the context API, not the SDK.

Fix

Manage the span/context explicitly and detach() only when the generator body
completed normally; skip detach() on early close. The span is always
end()ed, so trace data stays complete. Full consumption is unchanged.

Tests

Adds tests/unittests/telemetry/test_instrumentation.py:

  • test_record_invocation_no_detach_error_on_early_close — reproduces the
    early-close path (via asyncio.runloop.shutdown_asyncgens(), which
    finalizes the still-open generator in a different context) and asserts no
    Failed to detach context ERROR is logged, while the invocation span is
    still recorded and ended.
  • test_record_invocation_full_consumption_still_records_span — full
    consumption keeps recording the span with no error.

Verified the new early-close test fails without this change (with the exact
ValueError: <Token ...> was created in a different Context) and passes
with it
.

Scope

Intentionally minimal: only the default schema-v1 invocation span that users
hit today. The same start_as_current_span-around-async-generator pattern also
exists on the schema-v2 path (node_tracing._use_invoke_workflow_span) and can
get the same treatment in a follow-up.

@google-cla

google-cla Bot commented Aug 3, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@adk-bot adk-bot added the tracing [Component] This issue is related to OpenTelemetry tracing label Aug 3, 2026
…lose

The schema-v1 invocation span in record_invocation wrapped
runners._run_node_async (an async generator) with start_as_current_span.
When a caller stops iterating early, the generator is finalized in a
different execution context than where the span was attached, so the
automatic context detach raised "Token was created in a different Context"
-- OpenTelemetry swallows it but logs it at ERROR on every early-terminated
run, creating log noise and false error-rate spikes.

Manage the span/context explicitly and detach only on normal completion;
skip detach on early close. The span is always ended, so trace data stays
complete.

Fixes google#6559
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tracing [Component] This issue is related to OpenTelemetry tracing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Failed to detach context" (Token created in a different Context) logged as ERROR when caller stops runner.run_async() early

3 participants