feat(runtime): stop a turn on the clock as well as the step count - #64
Merged
Conversation
The loop has only ever measured its budget in iterations while the harness that runs it enforces a wall clock, so `make-mips-interpreter` was killed by its own 200th iteration at 406s of an 1800s budget, mid-work, with `exception_info: null` proving Harbor's timeout never fired. Across the five-task config every trial spent 2-23% of the time it was given. `WOOPCODE_MAX_WALL_SEC` gives the loop a second budget and it stops on whichever binds first. The operator passes the whole budget; `setDeadline` subtracts a 60s reserve so the last step, the final answer and the session write still land. The deadline lives in module state for the reason `runtime/sandbox/registry.ts` gives for the same shape. `onBudgetExhausted` is deliberately not consulted for the deadline: the step ceiling can afford to ask because iterations do not tick while a human thinks, and a clock does. `WallBudgetExhaustedError` is a sibling of `IterationBudgetExhaustedError` under a shared `BudgetExhaustedError`, so both exit 2 - one code, because a distinct one would be booked as an exception by any harness not yet updated to know it. The wind-down now converts remaining time into steps at the turn's own measured rate, so one message and one flag serve both budgets. That flag replaces an equality test that silently never fired when the ceiling was below five, which is a behaviour change: a two-step budget is now announced at its first step. Prompt assembly is untouched - `bun run replay:baseline` is byte-identical before and after, across all ten fixtures and the corpus totals. Reasoning and the per-task measurements: docs/adr/0001-wall-clock-budget-for-the-agent-loop.md. Deferred by design: clamping tool timeouts to the remaining budget (#61) and wiring the budget through `agent.py`, `job.yaml` and CLAUDE.md (#62). Until #61 lands the deadline is advisory for a command already running. Closes #60
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #60 (already closed). ADR committed alongside the code at
docs/adr/0001-wall-clock-budget-for-the-agent-loop.md.Why
The loop has only ever measured its budget in iterations while the harness enforces a wall clock.
make-mips-interpreterwas killed by its own 200th iteration at 406s of an 1800s budget, mid-work, withexception_info: nullproving Harbor's timeout never fired. Across the five-task config every trial spent 2–23% of the time it was given.What
runtime/deadline.ts— module singleton in the shape ofruntime/sandbox/registry.ts.WALL_RESERVE_SEC = 60held back so the last step, the final answer and the session write land before the harness kills the process. The clock is injectable, which is how elapsed time is tested without stubbingDateglobally.runtime/loop.ts—BudgetExhaustedErrorbase withIterationBudgetExhaustedErrorand a newWallBudgetExhaustedErroras siblings;WOOPCODE_MAX_WALL_SECread once per turn; the deadline checked at the top of an iteration, inside thetry, so it takes the sameonError-then-rethrow path.commands/agent.tsx— oneinstanceof BudgetExhaustedErrorin place of two, so both budgets exit 2. A distinct code would be booked as an exception by any harness not yet updated to know it.onBudgetExhaustedis deliberately not consulted for the deadline: the step ceiling can afford to ask because iterations do not tick while a human thinks, and a clock does.Behaviour change
The wind-down flag replaces an equality test that silently never fired when the ceiling was below five steps — the turns with least room were the ones told nothing. A two-step budget is now announced at its first step.
iterationBudget.test.tswas updated to assert the new behaviour.Verification
bun run verify --all— 4 gates passed (docs lint, docs surface, type check, tests).bun run replay:baseline— byte-identical before and after, across all ten fixtures and the corpus totals. Expected: this decides when a turn stops, not what the prompt carries.clearDeadlineinfinally(1), the wind-down reset (1), the wall term instepsRemaining(3), the reserve subtraction (11), the clamp floor (1). The first disarm test survived its mutation and was rewritten to assertremainingMs()directly.Not checked: a live Harbor run, a live provider, interactive TUI input.
Deferred, and load-bearing
This makes the failure fixable, not fixed. Both of these are needed before it stops happening:
run_terminal's 300s default is 40% ofoverfull-hbox's budget.agent.pyorjob.yaml, so no benchmark run gets a wall budget. CLAUDE.md's contradictory "Wall clock is the binding budget, not iterations" line is Wire the wall budget through Harbor, and correct CLAUDE.md #62's to correct.Two things a reviewer should weigh
Both were raised in review and left as the ADR decided them, rather than redesigned here:
WOOPCODE_MAX_WALL_SECin a.envwould break the TUI. Bun auto-loads.env, the deadline anchors to process start, and the interactive path skipsonBudgetExhausted— so after the budget elapses every turn throws before its first request, with no checkpoint. The ADR's reasoning rests on "the interactive path will not have the variable set," which the code does not enforce.meanStepMsis a whole-turn mean and the warning flag is permanent. One slow step can tell the model to wrap up while plenty of budget remains, and it is never taken back. It nudges rather than ends the turn.