fix(anthropic): ping-aware stream watchdog — eliminate spurious NonZeroAgentExitCodeError#734
Merged
Merged
Conversation
…ntExitCodeError on large-context agentic runs Fixes the dominant terminal-bench 2.1 failure class: 18/89 trials died with NonZeroAgentExitCodeError (mean 0.58 vs claude-code 0.72 which had ZERO such crashes). Root cause (SDK-source-verified): the Anthropic Python SDK drops `ping` keepalive events (_streaming.py:151 `if sse.event == "ping": continue`), so the idle watchdog — iterating the TYPED event stream — sees dead air whenever the model works >90s between typed events. In the agentic loop every turn re-sends the growing conversation, so every stream call pays prompt-processing time-to-first-event, scaling toward minutes near opus-4-8's 1M window (the #730 1M fix made this common). Crash duration_ms clustered at 94-100s = just past the flat 90s window. The watchdog's recovery then re-issued NON-streaming, which the SDK refuses for opus-class max_tokens ("Streaming is required for operations that may take longer than 10 minutes") -> fatal exit 1. Fix (TS-parity, three complementary layers): 1. Ping-aware liveness (primary): the dropped pings are still BYTES that httpx.Response.num_bytes_downloaded counts (httpx _models.py:952, per raw chunk in iter_bytes; the SDK consumes response.iter_bytes()). On each deadline the watchdog re-arms instead of firing when bytes advanced -> fires only on TRUE dead air. Zero changes to the accumulation path. 2. Retry the stream (stream_idle_max_attempts, default 3); raise StreamIdleTimeout on exhaustion. Fatal _fallback_to_chat deleted. 3. Two-phase deadline (300s first-event grace / 90s inter-event) as the time-based fallback when the byte counter is unavailable. Plus explicit chat() timeout (API_TIMEOUT_MS, default 600s) which also disarms the SDK >10-min guard for legit non-streaming callers. Live re-smoke of 6 previously-crashing tasks (opus-4-8 + subscription + effort high): regex-log, dna-assembly, feal-linear-cryptanalysis -> 1.0 PASS; filter-js-from-html, adaptive-rejection-sampler -> complete 0.0 (genuine model miss, no exception); write-compressor -> AgentTimeoutError (legit slow, same mode claude-code hits). ZERO spurious watchdog crashes remain. 119 tests pass across watchdog/provider/ch04/thinking/query/ context suites. Critic: "Ship the branch". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the dominant failure class in the terminal-bench 2.1 eval: 18/89 trials died with
NonZeroAgentExitCodeError(clawcodex mean 0.58 vs claude-code 0.72, which had zero such crashes on the same model).Root cause (verified at SDK/httpx source)
The Anthropic Python SDK drops
pingkeepalive events (_streaming.py:151if sse.event == "ping": continue), so the idle watchdog — iterating the typed event stream — sees dead air whenever the model works >90s between typed events. In the agentic loop every turn re-sends the growing conversation, so every call pays prompt-processing time-to-first-event, scaling toward minutes near opus-4-8's 1M window (the #730 1M fix made this common). Crashduration_msclustered at 94–100s = just past the flat 90s window. The watchdog's recovery re-issued non-streaming, which the SDK refuses for opus-classmax_tokens("Streaming is required…") → fatal exit 1.Fix (TS-parity, three layers)
httpx.Response.num_bytes_downloadedcounts (_models.py:952, per raw chunk initer_bytes; the SDK consumesresponse.iter_bytes()). Each deadline re-arms instead of firing when bytes advanced → fires only on true dead air. Zero changes to the accumulation path.stream_idle_max_attempts, default 3) →StreamIdleTimeouton exhaustion. Fatal_fallback_to_chatdeleted.Plus explicit
chat()timeout (API_TIMEOUT_MS, default 600s) which disarms the SDK's >10-min guard for legit non-streaming callers.Test plan
NonZeroAgentExitCodeErrorin the 0.58 run (opus-4-8 + subscription + effort high):regex-log,dna-assembly,feal-linear-cryptanalysis→ 1.0 PASS;filter-js-from-html,adaptive-rejection-sampler→ complete, reward 0.0, no exception (genuine misses);write-compressor→AgentTimeoutError(legit slow, same mode claude-code hits). Zero spurious watchdog crashes remain.(Clean re-base of #733, which had a stale merge-base after main history was rewritten.)
🤖 Generated with Claude Code