Skip to content

fix(tools): nudge off a stalled background task instead of polling forever - #759

Merged
ericleepi314 merged 1 commit into
mainfrom
fix/taskoutput-stuck-poll-guard
Jul 28, 2026
Merged

fix(tools): nudge off a stalled background task instead of polling forever#759
ericleepi314 merged 1 commit into
mainfrom
fix/taskoutput-stuck-poll-guard

Conversation

@ericleepi314

Copy link
Copy Markdown
Collaborator

A background command that cannot finish on its own — an unbounded search, a brute-force, a wait for something that never arrives — turns TaskOutput into an absorbing state: the model polls "still running, no new output", polls again, and never reconsiders.

The failure

terminal-bench 2.1 crack-7z-hash (deepseek-v4-pro, effort=max): the agent launched a john-the-ripper brute-force in the background, then spent its final 20 minutes and ~15 consecutive polls waiting on it until the harness killed the trial at its 30-minute budget. opus-5 finished the same task in 18 seconds.

Durations prove it's a loop, not slowness: on the 50 tasks both models completed, DeepSeek ran at 0.92× opus-5's wall-clock, while crack-7z-hash alone showed a 120× spread — the signature of a stuck job.

The guard

Per background bash task, count consecutive polls that return it still running with no growth in its on-disk log. Past a threshold and a wall-clock floor, prepend a hint pointing at TaskStop. Tool-level and model-agnostic — catches the trap for every provider, can't regress trajectories the way a prompt edit can.

Replayed against the real crack-7z-hash poll sequence, the guard fires on poll 4 of 15 — the model gets the nudge with 11 polls to spare.

Design points (several from review)

  • Progress = log file size, not returned output length. read_background_output caps output at the last 200KB and re-seeks, so a chatty progressing job's reported length plateaus at the cap and would read as "stuck". File size keeps growing → true progress signal. (Critic M1)
  • Both poll modes count. block=False returns retrieval_status="success" on a running task, not "timeout" — and 13 of the 15 crack-7z-hash polls were block=False. Gating on the timeout status (the obvious reading) would make the guard inert for the exact failure it exists to catch. Gate is status=="running", any mode.
  • Fast block=False polls are safe only because of the wall-clock floor: a burst of instant non-blocking polls on a just-started job can't fire the nudge. (Critic m4)
  • The hint heads the serialized result (leading top-level field + prepend into task.output). A result over ~50KB is persisted with a head preview, which would truncate a tail-appended note away — verified end-to-end through maybe_persist_large_tool_result. (Critic M2)
  • Counter state lives on the background_bash_tasks entry dict — written once at spawn, only mutated in place by the reaper, never rebuilt, so it can't be silently wiped mid-loop. Agent subtasks excluded.

Scope

One retry-class of failure (poll-loop timeouts). Does not touch the two other DeepSeek failure buckets triaged in the same run: the image_url 400 (9 tasks, needs multimodal stripping for non-vision models) and genuine wrong answers (15 tasks).

Tests

12, all revert-sensitive — neutering the body, the length-vs-filesize signal, append-vs-prepend placement, and timeout-vs-status gate each fail their specific tests. Full suite 8865 passed, 0 failures.

🤖 Generated with Claude Code

…rever

A background command that cannot finish on its own — an unbounded search, a
brute-force, a wait for something that never arrives — turns TaskOutput into
an absorbing state: the model polls "still running, no new output", polls
again, and never reconsiders. On terminal-bench 2.1 crack-7z-hash
(deepseek-v4-pro, 2026-07-27) an agent launched an incremental
john-the-ripper brute-force in the background, then spent its final 20
minutes and ~15 consecutive polls waiting on it until the harness killed the
trial at its 30-minute budget. opus-5 finished the same task in 18 seconds.

Durations confirm this is a loop, not slowness: on the 50 tasks both models
completed, DeepSeek ran at 0.92x opus-5's wall-clock, while crack-7z-hash
alone showed a 120x spread — the signature of a stuck job, not a slow one.

The guard, in _task_output_call -> _guard_repeated_polls: per background bash
task, count consecutive polls that return it still running with no growth in
its on-disk log. Past a threshold AND a wall-clock floor, prepend a hint
pointing at TaskStop. Model-agnostic and tool-level — it catches the trap
for every provider, and cannot regress trajectories the way a prompt edit
can.

Design points, several from review:

* Progress is measured by the LOG FILE SIZE, not the returned output length.
  read_background_output caps its output at the last 200KB and re-seeks, so a
  chatty progressing job's reported length plateaus at the cap and would read
  as "stuck". File size keeps growing, so it stays a true progress signal.

* Both poll modes count. block=False returns retrieval_status="success" on a
  running task, not "timeout" — and 13 of the 15 crack-7z-hash polls were
  block=False, so gating on the timeout status (the obvious reading) would
  make the guard inert for the exact failure it exists to catch. The gate is
  status=="running", any mode. Replayed against the real trajectory, the
  guard fires on poll 4 of 15.

* Counting fast block=False polls is safe only because of the wall-clock
  floor (_STUCK_MIN_WALL_SECONDS): a burst of instant non-blocking polls on a
  just-started job cannot fire the nudge.

* The hint heads the serialized result (a leading top-level field plus a
  prepend into task.output). A result over ~50KB is persisted with a HEAD
  preview, which would truncate a tail-appended note away — verified with an
  integration test through maybe_persist_large_tool_result.

* Counter state lives on the background_bash_tasks entry dict, which is
  written once at spawn and only mutated in place by the reaper — never
  rebuilt, so it cannot be silently wiped mid-loop. Agent subtasks are
  excluded; a no-new-output poll there is normal.

Tests: 12, all revert-sensitive (neutering the body, the length-vs-filesize
signal, the append-vs-prepend placement, and the timeout-vs-status gate each
fail their specific tests). Full suite 8865 passed, 0 failures.
@ericleepi314
ericleepi314 merged commit 2f114aa into main Jul 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant