Skip to content

fix(anthropic): ping-aware stream watchdog — eliminate spurious NonZeroAgentExitCodeError#734

Merged
ericleepi314 merged 1 commit into
mainfrom
fix/stream-watchdog-restream-v2
Jul 21, 2026
Merged

fix(anthropic): ping-aware stream watchdog — eliminate spurious NonZeroAgentExitCodeError#734
ericleepi314 merged 1 commit into
mainfrom
fix/stream-watchdog-restream-v2

Conversation

@ericleepi314

Copy link
Copy Markdown
Collaborator

Summary

Fixes the dominant failure class in the terminal-bench 2.1 eval: 18/89 trials died with NonZeroAgentExitCodeError (clawcodex mean 0.58 vs claude-code 0.72, which had zero such crashes on the same model).

Root cause (verified at SDK/httpx source)

The Anthropic Python SDK drops ping keepalive events (_streaming.py:151 if sse.event == "ping": continue), so the idle watchdog — iterating the typed event stream — sees dead air whenever the model works >90s between typed events. In the agentic loop every turn re-sends the growing conversation, so every call pays prompt-processing time-to-first-event, scaling toward minutes near opus-4-8's 1M window (the #730 1M fix made this common). Crash duration_ms clustered at 94–100s = just past the flat 90s window. The watchdog's recovery re-issued non-streaming, which the SDK refuses for opus-class max_tokens ("Streaming is required…") → fatal exit 1.

Fix (TS-parity, three layers)

  1. Ping-aware liveness (primary): the dropped pings are still bytes that httpx.Response.num_bytes_downloaded counts (_models.py:952, per raw chunk in iter_bytes; the SDK consumes response.iter_bytes()). Each deadline re-arms instead of firing when bytes advanced → fires only on true dead air. Zero changes to the accumulation path.
  2. Retry the stream (stream_idle_max_attempts, default 3) → StreamIdleTimeout on exhaustion. Fatal _fallback_to_chat deleted.
  3. Two-phase deadline (300s first-event grace / 90s inter-event) as the time-based fallback.
    Plus explicit chat() timeout (API_TIMEOUT_MS, default 600s) which disarms the SDK's >10-min guard for legit non-streaming callers.

Test plan

  • Live re-smoke, 6 tasks that all died with NonZeroAgentExitCodeError in the 0.58 run (opus-4-8 + subscription + effort high): regex-log, dna-assembly, feal-linear-cryptanalysis1.0 PASS; filter-js-from-html, adaptive-rejection-samplercomplete, reward 0.0, no exception (genuine misses); write-compressorAgentTimeoutError (legit slow, same mode claude-code hits). Zero spurious watchdog crashes remain.
  • 119 passing across watchdog/provider/ch04/thinking/query/context suites (byte-progress re-arm + dead-air fire, retry-then-succeed, exhaustion-raises, chat timeout, two-phase transition)
  • Critic review (4 rounds) → "Ship the branch"

(Clean re-base of #733, which had a stale merge-base after main history was rewritten.)

🤖 Generated with Claude Code

…ntExitCodeError on large-context agentic runs

Fixes the dominant terminal-bench 2.1 failure class: 18/89 trials died
with NonZeroAgentExitCodeError (mean 0.58 vs claude-code 0.72 which had
ZERO such crashes).

Root cause (SDK-source-verified): the Anthropic Python SDK drops `ping`
keepalive events (_streaming.py:151 `if sse.event == "ping": continue`),
so the idle watchdog — iterating the TYPED event stream — sees dead air
whenever the model works >90s between typed events. In the agentic loop
every turn re-sends the growing conversation, so every stream call pays
prompt-processing time-to-first-event, scaling toward minutes near
opus-4-8's 1M window (the #730 1M fix made this common). Crash
duration_ms clustered at 94-100s = just past the flat 90s window. The
watchdog's recovery then re-issued NON-streaming, which the SDK refuses
for opus-class max_tokens ("Streaming is required for operations that
may take longer than 10 minutes") -> fatal exit 1.

Fix (TS-parity, three complementary layers):
1. Ping-aware liveness (primary): the dropped pings are still BYTES that
   httpx.Response.num_bytes_downloaded counts (httpx _models.py:952, per
   raw chunk in iter_bytes; the SDK consumes response.iter_bytes()). On
   each deadline the watchdog re-arms instead of firing when bytes
   advanced -> fires only on TRUE dead air. Zero changes to the
   accumulation path.
2. Retry the stream (stream_idle_max_attempts, default 3); raise
   StreamIdleTimeout on exhaustion. Fatal _fallback_to_chat deleted.
3. Two-phase deadline (300s first-event grace / 90s inter-event) as the
   time-based fallback when the byte counter is unavailable.
Plus explicit chat() timeout (API_TIMEOUT_MS, default 600s) which also
disarms the SDK >10-min guard for legit non-streaming callers.

Live re-smoke of 6 previously-crashing tasks (opus-4-8 + subscription +
effort high): regex-log, dna-assembly, feal-linear-cryptanalysis -> 1.0
PASS; filter-js-from-html, adaptive-rejection-sampler -> complete 0.0
(genuine model miss, no exception); write-compressor -> AgentTimeoutError
(legit slow, same mode claude-code hits). ZERO spurious watchdog crashes
remain. 119 tests pass across watchdog/provider/ch04/thinking/query/
context suites. Critic: "Ship the branch".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ericleepi314
ericleepi314 merged commit 0b2f643 into main Jul 21, 2026
1 of 2 checks passed
@ericleepi314
ericleepi314 deleted the fix/stream-watchdog-restream-v2 branch July 21, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant