feat(runtime): ask an unattended turn to prove the task's requirements - #67
Merged
Conversation
… answer The reminder guarded only `iterations < budget`, which cannot see a clock. Injected against a wall budget with nothing left, the turn goes round, the deadline check at the top of the next iteration fires, and a turn holding a finished answer exits as WallBudgetExhaustedError with status 2. Both budgets are now read through `canAffordAnotherRound`: stepsRemaining against a floor of three, plus the deadline consulted directly, because the clock-to-steps conversion is not trusted until MIN_RATE_SAMPLES steps have gone into it and a short turn would otherwise slip past.
recentMessages counts user messages, and the loop pushes user messages of its own — the wind-down warning, the verification reminder, a truncated-stream resume. Six of those and the request no longer contains the task being worked on, while the turn carries on working on it. The loop captures the turn-initiating message at entry, as an index because the array only grows, and recentMessages carries it back in when the window has moved past it. A window that still holds it is assembled byte-identically. The replay corpus cannot measure this: its recordings hold one conversation turn each, because injected messages were never written to the event log, so the pin never fires there and the baseline is unchanged. Also gates agentController.test.ts's recentMessages stub on stubActive, like the session stubs beside it. Ungated, its identity window was installed for the whole run — under it these tests would have passed vacuously.
Two of three failed trials in tb2-post-1.1 ended early, confident and wrong, with ~70% of both budgets unspent. The only completion gate looked for edits that had gone unchecked, and theirs had been checked — overfull-hbox ran its chosen check three times. It verified a property the task never asked about. So a second finish gate: an unattended turn that responds with no tool calls, with ten steps left and no wind-down warning outstanding, is asked once to enumerate every stated requirement and quote the command output proving each. Both gates answer with one message when both apply, because a second injection costs one of the six conversation turns the window keeps. The duplicate threshold is cleared as the gate fires: the check a turn most needs to re-run is usually the one it has already run twice, where the loop would answer that the result is already in a conversation the window dropped. TurnSummary gains requirementReminders and requirementGateActedOn so a run can be read for "asked and ignored" rather than only "asked". Reasoning, rejected alternatives and how this gets judged are in the ADR; CONTEXT.md is new and holds the turn-lifecycle vocabulary.
run_end already carried the two fields; this lifts them beside woopcode_unverified_edits so a job can be read without opening a trajectory. Kept as two keys because they answer different questions: how often a trial tried to stop early, and how often being asked sent it back to run something. A gate that fires and is answered in prose changes nothing, and in the score alone that is indistinguishable from a gate that never fired.
Review findings from the two-axis pass over this branch. Standards: both new test files hand-rolled the deadline incantation the bun-test skill says to take from packages/tests/shared — so the movable-clock case joins budgetWith there as budgetDrivenBy, summaryOf moves to shared as turnSummaryOf instead of being copied per file, and the four inline stream generators become createStreamingProvider. canAffordAnotherRound's ceiling parameter is named for what it is. Spec: the merged injection reported only the verification gate on the status channel, which is what a headless operator watches and what the event log records — the notice now names every gate that fired. canAffordAnotherRound guards deadlineReached on whether this turn set a wall budget at all, mirroring the loop's own check, so two readers of one piece of module state cannot disagree. The ADR gains the revert-to-red evidence per mechanism, which was run but never written down.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
mangit955
marked this pull request as ready for review
August 30, 2026 13:27
…'s bound Review follow-ups. The pin makes MAX_TURNS a bound on the tail rather than on the window: a pinned request carries one conversation turn more than the constant names. That is the only exception in a budget every other context decision treats as absolute, so it is now stated at recentMessages, in the ADR and in CONTEXT.md, and pinned by a test that asserts the arithmetic in both directions. The amnesty is wholesale — every exhausted call may run again, not only the one the gate asks about — and what bounds it is that suppression resumes at once. There was a test for the intended repeat running again and none for the downside; now a third identical call after the gate is asserted to be refused, which fails if the threshold is disabled rather than reset. No change to windDownWarned, which is not latched for the turn: shouldWarnWindDown clears it above twice the threshold and requirementGate.test.ts covers the gate firing after it clears. The gate's comment now says so, since a reader of `!state.windDownWarned` could reasonably assume otherwise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A turn ends when the model responds without calling a tool. One thing could overrule that: a turn that had changed files and run nothing to check them was asked, once, to verify. That gate fires on evidence the loop can see. The failure it cannot see is a turn that verified something thoroughly, and verified the wrong thing.
Two of the three failed trials in
jobs/tb2-post-1.1ended that way — early, confident, and wrong, with most of both budgets unspent.overfull-hboxis the instructive one: it ranpdflatexand a search for overfull boxes three times over, so the unverified-edits gate had nothing to say — the edits were checked. The task also constrained which wording was permitted, and nothing it ran tested that. CLAUDE.md already recorded the pattern from an earlier run: two trials that reported success with accurate self-verification still scored 0, because they verified the wrong property.Separately, and found while reading that gate: the verification reminder was clock-blind. It guarded only
iterations < budget, so injected against a spent wall budget it sent the turn round to a deadline check that throws — turning a turn holding a finished answer intoWallBudgetExhaustedErrorand exit status 2. Reachable on every wall-budgeted run. That fix is the first commit, on its own.What
A second finish gate. An unattended turn that responds with no tool calls, has ten steps left and no wind-down warning outstanding, is asked once to enumerate every stated requirement — including constraints on what is not allowed — and quote the command output proving each. Recollection is explicitly refused as evidence, since the turn being interrupted is one whose recollection is already wrong.
Ten steps is twice the wind-down threshold. The gate asks for work, and inside that zone the loop is telling the model the opposite: finish what you started, begin nothing new. The
windDownWarnedflag is read as well as the count, because the count comes from a measured rate that moves — it can recover past the floor while the model is still under a warning issued earlier.Headless only, by an explicit option.
AgentLoopOptions.unattended, set byrunHeadlessand threaded throughAgentController. Deliberately not inferred from a missingonBudgetExhaustedcallback, which would hand the behaviour to every embedder that happens not to pass one, and not read fromWOOPCODE_NON_INTERACTIVE, which only the benchmark harness sets. Interactive turns are excluded because a person is reading the answer and can correct it for the cost of one sentence.Both gates answer with one message. They are cheap in round trips and expensive in window: each injection costs one of the six conversation turns the window keeps, and losing the window is what this gate exists to correct.
The task statement is pinned into the window. The gate says "go back to the task statement above", and that has to be true. The window counts user messages and the loop pushes its own — the wind-down warning, the finish gates, a truncated-stream resume — so six of them and the question being worked on has left the request.
agentLoopcaptures the turn-initiating message at entry andrecentMessagescarries it back in when the window has moved past it. Rejected: quoting the task into the gate's message instead, which leaves a long turn still arguing from a question it cannot see; and pinningmessages[0], which is a greeting in an interactive session and a trimmed leftover under--resume.The duplicate threshold is cleared when the gate fires. The two collide head on: the check a turn most needs to re-run is usually the one it has already run twice, where the loop answers that the result is already in the conversation — pointing at output the window dropped long ago.
Observability.
TurnSummarygainsrequirementRemindersandrequirementGateActedOn, both onrun_endand lifted into Harbor trial metadata. They separate three outcomes a score collapses into one: never fired, fired and the model ran commands, fired and the model answered in prose from memory. The last is this mechanism's likeliest failure and is otherwise invisible.docs/adr/0002-finish-gates-for-an-unattended-turn.mdcarries the reasoning and the rejected alternatives.CONTEXT.mdis new and holds the turn-lifecycle vocabulary as a glossary.Verified
bun run verify --allon the tree as pushed: docs lint, docs surface, type check andbun testall pass. The reverse-order sweep runs 1965 tests across 120 files with 0 failures, andpython3 -m pytest harbor_woopcode/test_agent.pypasses 46.Every mechanism was proved by reverting it and watching its test go red, each revert confirmed applied before the suite ran:
WallBudgetExhaustedErrorin place of the finished answerunattendedconditionKeyErrorin the harness testsWhile writing the pin tests,
agentController.test.tswas found stubbingrecentMessageswith an ungated identity function registered for the whole run — themock.moduletrap CLAUDE.md documents, and the reason the new tests would otherwise have passed vacuously. It is now gated onstubActivelike the session stubs beside it.What is not verified.
bun run replay:baselineis unchanged before and after, and that is not evidence: the corpus holds one conversation turn per fixture, because the loop's injected messages were never written to the event log, so the pin cannot fire there. The harness cannot see this change.Nothing here has faced a live provider or a real task. Whether the model obeys the gate or answers it in prose is exactly what
requirementGateActedOnexists to measure, and only the benchmark answers it —overfull-hboxandvideo-processingfirst, then the five-task job to check the three passing tasks did not regress. Draft until those have run.