zarvis(codex-oauth): WS keepalive pings no longer defeat the idle watchdog (fixes 91-min turn hang)#346
Merged
Merged
Conversation
… bound WS connect A stalled Responses-WebSocket response stream (server holds the socket open with keepalive Pings but sends no model output) defeated the 90s idle watchdog: read_ws_turn called sink.progress() on every frame, including Ping control frames, so the activity timer reset forever and the turn hung for the entire trial budget (observed: a 91-min trial with zero model events — 3.8x the codex wall-clock on the same task, entirely from this one hang). - read_ws_turn: call sink.progress() only for real data (Text) frames, not Ping/control frames, so a silent-but-pinging stream idle-times-out and retries. - try_ws: bound connect with WS_CONNECT_TIMEOUT (30s) -> fall back to HTTP on a stalled TCP / proxy-CONNECT / TLS handshake instead of hanging the turn. Deterministic regression test added (local WS server sends only pings -> 0 progress calls). 150 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
While benchmarking zarvis vs codex (same gpt-5.5 backend), one task showed a 3.8×
wall-clock gap — traced to a single trial that hung for 91 minutes producing zero
model events, then hit the harness trial timeout. The daemon log showed the adapter
alive but completely silent for 91 min, and the 90s provider idle-watchdog never fired.
Root cause: in the Responses-WebSocket path,
read_ws_turncalledsink.progress()onevery frame — including WS keepalive
Pingcontrol frames. The idle watchdog treatsprogress()as "the turn is alive," so a stalled response stream (server holds the socketopen with periodic pings but sends no model output) reset the activity timer forever and
the turn hung for the entire trial budget. Connection-level liveness defeated the watchdog.
(Normal trials were already at parity with codex — ~11.5m vs ~10m, ~12.6s/turn vs ~13.4s.
The entire apparent gap was this one hang.)
Fix
read_ws_turn: count only real data (Text) frames asprogress(). Ping/controlframes no longer reset the idle watchdog, so a silent-but-pinging stream idle-times-out
(90s) and retries/falls back instead of hanging. Real reasoning still emits Text deltas,
so legitimate long "thinking" turns stay alive as before.
try_ws: bound the WS connect withWS_CONNECT_TIMEOUT(30s) — a stalled TCP /proxy-CONNECT / TLS handshake falls back to HTTP instead of hanging the turn.
Validation
ws_keepalive_pings_do_not_count_as_progress: a localWS server sends only Pings (no Text) →
read_ws_turnrecords 0 progress calls (was 3),so the watchdog would fire instead of hanging.
hung_provider_turn_times_out,progress_pings_keep_a_text_less_stream_alive). 150 tests pass.agentd-adapter-zarvis.🤖 Generated with Claude Code