[AUTOMATED] fix(tester): a provider content refusal is not a kuna failure - #419
Merged
Conversation
…lure
A round-3 tester died 78 seconds in, `codex exited rc=1`, no report, nine tool calls
of evidence lost. The cause is only in the event stream:
{"type":"turn.failed","error":{"message":"This content was flagged for possible
cybersecurity risk. ... join the Trusted Access for Cyber program"}}
The tester MODEL declined the work. Measured across the runs on disk: **6 of 36**
tester runs hit it, and **all three** attempts at challenge 63d5a26a did -- so that
challenge is systematically unmeasurable with this tester model rather than hard.
Three other runs hit it mid-session and still filed reports, so it is not always
fatal.
It matters because it silently corrupts the two numbers this loop exists to produce.
A refusal exits 1, indistinguishable from a crash, so it was landing as a generic
`failed` -- and a challenge nobody was permitted to attempt looks exactly like one
kuna could not support. Both the solve rate and `gave_up: kuna-blocked` drift by
however often it happens, and round 3's rate is 17%.
tester.sh now reads the event stream on a non-zero exit and records `--phase refused`
with a `provider-refusal:` note, so status and grading can separate it from a kuna
failure and from a harness fault. The generic rc path is unchanged.
RECORDED, NOT WORKED AROUND. The comment and the runbook both say so explicitly: do
not reword prompts to get past the classifier. The refusal is the provider's call and
the sanctioned route is the authorization programme its own message names. If the rate
ever rises far enough to starve a round, the answers are to take it up with the
provider or to run the tester on a model licensed for this work -- not for the loop to
route around it.
tools/repipe/smoke.sh 127/127.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mahaloz
force-pushed
the
fix/tester-classify-provider-refusal
branch
from
September 5, 2026 20:46
e491153 to
c8d59f5
Compare
mahaloz
added a commit
that referenced
this pull request
Sep 6, 2026
…either (#436) Found in production: round 4's `get-pc-helper-loses` builder stopped at 200 turns and $25.07 -- `REPIPE_BUILDER_USD` exhausted -- and worker.sh recorded the generic `claude rc=1`. The round-4 captain had to open the result JSON itself to work out it was "the budget-cap salvage case", which is exactly the diagnosis the note should have handed it. The classifier added for account limits reads only `result`, and the two stops do not look alike: account/session limit subtype "success", is_error true, reason ONLY in `result` ("You've hit your session limit ...") --max-budget-usd cap subtype "error_max_budget_usd", and `result` is NULL So the budget case had nothing for a `result`-only matcher to see. `subtype` is now checked FIRST -- where it is informative it is exact, and it is the only signal the budget case carries -- and the note becomes `capped: builder budget cap ($25.07 over 200 turns) (claude rc=1)`. The log line and note say "STOPPED BY A CAP" rather than "ACCOUNT LIMIT", since it now covers both. Verified against the three shapes, using round 4's REAL result JSON for the first: budget cap -> priced message; session limit -> unchanged, still matches; a genuine refusal ("I could not fix this; the tests fail") -> neither, falls through to the generic note as it must. Worth noting the pattern: this is the third metric this loop reported as something it was not -- provider refusals counted as kuna failures (#419), un-refuted hypotheses indistinguishable from undecided ones (#420), and now a budget cap indistinguishable from a broken build. Each was invisible until a number was checked for another reason. tools/repipe/smoke.sh 127/127. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
mahaloz
added a commit
that referenced
this pull request
Sep 6, 2026
…437) Found by round 4's captain, which wrote in its own notes that it had checked the builders with `ps`, "not with status.py, whose stale_s only ticks on a phase change and still lists b-r4-function-disasse as running". It was right, and it had to work around the tool this loop gives it for exactly this question. `stale_s` is `now - updated_at`, and `updated_at` moves on a `state update`, i.e. on a PHASE CHANGE -- not on any sign of life. Neither status module calls `os.kill`, so nothing ever asks whether the pid exists. `reap()` does ask, but only past `stale_seconds` (1800) and only when something calls it. So a worker that dies mid-phase reads `running` with a growing counter for up to half an hour, and the operator cannot tell it from one working hard on a long phase. One label, two very different states. `collect()` now records `alive` from a `kill(pid, 0)`: True, False, or None when no pid was recorded or the check itself failed -- unknowable is not the same as dead, and PermissionError means the process exists under another uid. The repipe table prints `DEAD` in the status column when a row claims `running` and the pid is gone. Verified by planting a `running` worker with pid 999999 in the inventory: the row prints DEAD, and the inventory was restored afterwards. `scripts.pipeline.status --json` still works, so the angr lane is unaffected. tools/repipe/smoke.sh 127/127. Fourth instance of one pattern this session, and the last three were found the same way -- a number checked for an unrelated reason. Provider refusals counted as kuna failures (#419); un-refuted hypotheses indistinguishable from undecided ones (#420); a builder budget cap indistinguishable from a broken build (#436); now a dead worker indistinguishable from a busy one. Each time, a status collapsed two causes into one label, and nothing failed loudly. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A round-3 tester died 78 seconds in,
codex exited rc=1, no report, nine tool callsof evidence lost. The cause is only in the event stream:
{"type":"turn.failed","error":{"message":"This content was flagged for possible
cybersecurity risk. ... join the Trusted Access for Cyber program"}}
The tester MODEL declined the work. Measured across the runs on disk: 6 of 36
tester runs hit it, and all three attempts at challenge 63d5a26a did -- so that
challenge is systematically unmeasurable with this tester model rather than hard.
Three other runs hit it mid-session and still filed reports, so it is not always
fatal.
It matters because it silently corrupts the two numbers this loop exists to produce.
A refusal exits 1, indistinguishable from a crash, so it was landing as a generic
failed-- and a challenge nobody was permitted to attempt looks exactly like onekuna could not support. Both the solve rate and
gave_up: kuna-blockeddrift byhowever often it happens, and round 3's rate is 17%.
tester.sh now reads the event stream on a non-zero exit and records
--phase refusedwith a
provider-refusal:note, so status and grading can separate it from a kunafailure and from a harness fault. The generic rc path is unchanged.
RECORDED, NOT WORKED AROUND. The comment and the runbook both say so explicitly: do
not reword prompts to get past the classifier. The refusal is the provider's call and
the sanctioned route is the authorization programme its own message names. If the rate
ever rises far enough to starve a round, the answers are to take it up with the
provider or to run the tester on a model licensed for this work -- not for the loop to
route around it.
tools/repipe/smoke.sh 127/127.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
🤖 Generated with Claude Code
https://claude.ai/code/session_01YcFmfZndNjgfqLQVBZCdkY