fix(tools): nudge off a stalled background task instead of polling forever - #759
Merged
Merged
Conversation
…rever A background command that cannot finish on its own — an unbounded search, a brute-force, a wait for something that never arrives — turns TaskOutput into an absorbing state: the model polls "still running, no new output", polls again, and never reconsiders. On terminal-bench 2.1 crack-7z-hash (deepseek-v4-pro, 2026-07-27) an agent launched an incremental john-the-ripper brute-force in the background, then spent its final 20 minutes and ~15 consecutive polls waiting on it until the harness killed the trial at its 30-minute budget. opus-5 finished the same task in 18 seconds. Durations confirm this is a loop, not slowness: on the 50 tasks both models completed, DeepSeek ran at 0.92x opus-5's wall-clock, while crack-7z-hash alone showed a 120x spread — the signature of a stuck job, not a slow one. The guard, in _task_output_call -> _guard_repeated_polls: per background bash task, count consecutive polls that return it still running with no growth in its on-disk log. Past a threshold AND a wall-clock floor, prepend a hint pointing at TaskStop. Model-agnostic and tool-level — it catches the trap for every provider, and cannot regress trajectories the way a prompt edit can. Design points, several from review: * Progress is measured by the LOG FILE SIZE, not the returned output length. read_background_output caps its output at the last 200KB and re-seeks, so a chatty progressing job's reported length plateaus at the cap and would read as "stuck". File size keeps growing, so it stays a true progress signal. * Both poll modes count. block=False returns retrieval_status="success" on a running task, not "timeout" — and 13 of the 15 crack-7z-hash polls were block=False, so gating on the timeout status (the obvious reading) would make the guard inert for the exact failure it exists to catch. The gate is status=="running", any mode. Replayed against the real trajectory, the guard fires on poll 4 of 15. * Counting fast block=False polls is safe only because of the wall-clock floor (_STUCK_MIN_WALL_SECONDS): a burst of instant non-blocking polls on a just-started job cannot fire the nudge. * The hint heads the serialized result (a leading top-level field plus a prepend into task.output). A result over ~50KB is persisted with a HEAD preview, which would truncate a tail-appended note away — verified with an integration test through maybe_persist_large_tool_result. * Counter state lives on the background_bash_tasks entry dict, which is written once at spawn and only mutated in place by the reaper — never rebuilt, so it cannot be silently wiped mid-loop. Agent subtasks are excluded; a no-new-output poll there is normal. Tests: 12, all revert-sensitive (neutering the body, the length-vs-filesize signal, the append-vs-prepend placement, and the timeout-vs-status gate each fail their specific tests). Full suite 8865 passed, 0 failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A background command that cannot finish on its own — an unbounded search, a brute-force, a wait for something that never arrives — turns
TaskOutputinto an absorbing state: the model polls "still running, no new output", polls again, and never reconsiders.The failure
terminal-bench 2.1
crack-7z-hash(deepseek-v4-pro,effort=max): the agent launched a john-the-ripper brute-force in the background, then spent its final 20 minutes and ~15 consecutive polls waiting on it until the harness killed the trial at its 30-minute budget. opus-5 finished the same task in 18 seconds.Durations prove it's a loop, not slowness: on the 50 tasks both models completed, DeepSeek ran at 0.92× opus-5's wall-clock, while
crack-7z-hashalone showed a 120× spread — the signature of a stuck job.The guard
Per background bash task, count consecutive polls that return it still running with no growth in its on-disk log. Past a threshold and a wall-clock floor, prepend a hint pointing at
TaskStop. Tool-level and model-agnostic — catches the trap for every provider, can't regress trajectories the way a prompt edit can.Replayed against the real
crack-7z-hashpoll sequence, the guard fires on poll 4 of 15 — the model gets the nudge with 11 polls to spare.Design points (several from review)
read_background_outputcaps output at the last 200KB and re-seeks, so a chatty progressing job's reported length plateaus at the cap and would read as "stuck". File size keeps growing → true progress signal. (Critic M1)block=Falsereturnsretrieval_status="success"on a running task, not"timeout"— and 13 of the 15crack-7z-hashpolls wereblock=False. Gating on the timeout status (the obvious reading) would make the guard inert for the exact failure it exists to catch. Gate isstatus=="running", any mode.block=Falsepolls are safe only because of the wall-clock floor: a burst of instant non-blocking polls on a just-started job can't fire the nudge. (Critic m4)task.output). A result over ~50KB is persisted with a head preview, which would truncate a tail-appended note away — verified end-to-end throughmaybe_persist_large_tool_result. (Critic M2)background_bash_tasksentry dict — written once at spawn, only mutated in place by the reaper, never rebuilt, so it can't be silently wiped mid-loop. Agent subtasks excluded.Scope
One retry-class of failure (poll-loop timeouts). Does not touch the two other DeepSeek failure buckets triaged in the same run: the
image_url400 (9 tasks, needs multimodal stripping for non-vision models) and genuine wrong answers (15 tasks).Tests
12, all revert-sensitive — neutering the body, the length-vs-filesize signal, append-vs-prepend placement, and timeout-vs-status gate each fail their specific tests. Full suite 8865 passed, 0 failures.
🤖 Generated with Claude Code