[AUTOMATED] fix(worker): a builder budget cap is not a build failure either - #436
Merged
Conversation
…either
Found in production: round 4's `get-pc-helper-loses` builder stopped at 200 turns and
$25.07 -- `REPIPE_BUILDER_USD` exhausted -- and worker.sh recorded the generic
`claude rc=1`. The round-4 captain had to open the result JSON itself to work out it
was "the budget-cap salvage case", which is exactly the diagnosis the note should
have handed it.
The classifier added for account limits reads only `result`, and the two stops do not
look alike:
account/session limit subtype "success", is_error true, reason ONLY in `result`
("You've hit your session limit ...")
--max-budget-usd cap subtype "error_max_budget_usd", and `result` is NULL
So the budget case had nothing for a `result`-only matcher to see. `subtype` is now
checked FIRST -- where it is informative it is exact, and it is the only signal the
budget case carries -- and the note becomes
`capped: builder budget cap ($25.07 over 200 turns) (claude rc=1)`.
The log line and note say "STOPPED BY A CAP" rather than "ACCOUNT LIMIT", since it now
covers both.
Verified against the three shapes, using round 4's REAL result JSON for the first:
budget cap -> priced message; session limit -> unchanged, still matches; a genuine
refusal ("I could not fix this; the tests fail") -> neither, falls through to the
generic note as it must.
Worth noting the pattern: this is the third metric this loop reported as something it
was not -- provider refusals counted as kuna failures (#419), un-refuted hypotheses
indistinguishable from undecided ones (#420), and now a budget cap indistinguishable
from a broken build. Each was invisible until a number was checked for another reason.
tools/repipe/smoke.sh 127/127.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mahaloz
added a commit
that referenced
this pull request
Sep 6, 2026
…437) Found by round 4's captain, which wrote in its own notes that it had checked the builders with `ps`, "not with status.py, whose stale_s only ticks on a phase change and still lists b-r4-function-disasse as running". It was right, and it had to work around the tool this loop gives it for exactly this question. `stale_s` is `now - updated_at`, and `updated_at` moves on a `state update`, i.e. on a PHASE CHANGE -- not on any sign of life. Neither status module calls `os.kill`, so nothing ever asks whether the pid exists. `reap()` does ask, but only past `stale_seconds` (1800) and only when something calls it. So a worker that dies mid-phase reads `running` with a growing counter for up to half an hour, and the operator cannot tell it from one working hard on a long phase. One label, two very different states. `collect()` now records `alive` from a `kill(pid, 0)`: True, False, or None when no pid was recorded or the check itself failed -- unknowable is not the same as dead, and PermissionError means the process exists under another uid. The repipe table prints `DEAD` in the status column when a row claims `running` and the pid is gone. Verified by planting a `running` worker with pid 999999 in the inventory: the row prints DEAD, and the inventory was restored afterwards. `scripts.pipeline.status --json` still works, so the angr lane is unaffected. tools/repipe/smoke.sh 127/127. Fourth instance of one pattern this session, and the last three were found the same way -- a number checked for an unrelated reason. Provider refusals counted as kuna failures (#419); un-refuted hypotheses indistinguishable from undecided ones (#420); a builder budget cap indistinguishable from a broken build (#436); now a dead worker indistinguishable from a busy one. Each time, a status collapsed two causes into one label, and nothing failed loudly. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found in production: round 4's
get-pc-helper-losesbuilder stopped at 200 turns and$25.07 --
REPIPE_BUILDER_USDexhausted -- and worker.sh recorded the genericclaude rc=1. The round-4 captain had to open the result JSON itself to work out itwas "the budget-cap salvage case", which is exactly the diagnosis the note should
have handed it.
The classifier added for account limits reads only
result, and the two stops do notlook alike:
account/session limit subtype "success", is_error true, reason ONLY in
result("You've hit your session limit ...")
--max-budget-usd cap subtype "error_max_budget_usd", and
resultis NULLSo the budget case had nothing for a
result-only matcher to see.subtypeis nowchecked FIRST -- where it is informative it is exact, and it is the only signal the
budget case carries -- and the note becomes
capped: builder budget cap ($25.07 over 200 turns) (claude rc=1).The log line and note say "STOPPED BY A CAP" rather than "ACCOUNT LIMIT", since it now
covers both.
Verified against the three shapes, using round 4's REAL result JSON for the first:
budget cap -> priced message; session limit -> unchanged, still matches; a genuine
refusal ("I could not fix this; the tests fail") -> neither, falls through to the
generic note as it must.
Worth noting the pattern: this is the third metric this loop reported as something it
was not -- provider refusals counted as kuna failures (#419), un-refuted hypotheses
indistinguishable from undecided ones (#420), and now a budget cap indistinguishable
from a broken build. Each was invisible until a number was checked for another reason.
tools/repipe/smoke.sh 127/127.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
🤖 Generated with Claude Code
https://claude.ai/code/session_01YcFmfZndNjgfqLQVBZCdkY