[AUTOMATED] feat(needs): a CLI to record a refutation verdict, which T_REFUTE never had - #420
Merged
Conversation
…T_REFUTE never had
Round 3 filed 16 needs and recorded a hypothesis verdict on 5. The other 11 read
`inconclusive`, which looks like "a refuter ran and could not decide" and is
indistinguishable in the record from it. It is not what happened.
`hypothesis_status` had exactly two writers in the whole tree, and both are DEFAULTS:
cluster.py sets `inconclusive` at filing, needs.py repeats it in the schema. Nothing
ever changed it. The captain prompt says T_REFUTE must "record upheld/overturned
either way", and there was no command that could -- the only route was hand-editing
YAML front matter in a markdown file. So the instruction had no mechanism, and 69% of
round 3's needs went to builders with an un-refuted hypothesis.
That is the expensive kind of gap, because refutation is what the two-arm design rests
on: the sibling decbench campaign overturned the filed diagnosis on 3 of 8 cases while
the symptom stood in all 8. Round 3's own numbers say the same thing -- of the five
verdicts that WERE recorded, two were overturned. Extrapolated over eleven unexamined
hypotheses, that is roughly four builders working from a wrong cause.
python3 -m scripts.repipe.needs refute <need_id> \
--verdict upheld|overturned|inconclusive --note "what the refuter did and found"
Writes `hypothesis_status` and appends a dated line to the Decision log, so the verdict
and its reasoning live with the need rather than in a captain tick's transcript. Refuses
to write a record that fails `validate()`.
The captain prompt now names the command in both places it describes T_REFUTE, and says
plainly that `inconclusive` is legitimate ONLY when a refuter ran and could not decide --
leaving a need at the filed default is not a verdict.
Verified against a real record: front matter flips, the decision-log line lands with the
round number and the previous value, and the record still validates. tools/repipe/smoke.sh
127/127.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mahaloz
added a commit
that referenced
this pull request
Sep 6, 2026
…either (#436) Found in production: round 4's `get-pc-helper-loses` builder stopped at 200 turns and $25.07 -- `REPIPE_BUILDER_USD` exhausted -- and worker.sh recorded the generic `claude rc=1`. The round-4 captain had to open the result JSON itself to work out it was "the budget-cap salvage case", which is exactly the diagnosis the note should have handed it. The classifier added for account limits reads only `result`, and the two stops do not look alike: account/session limit subtype "success", is_error true, reason ONLY in `result` ("You've hit your session limit ...") --max-budget-usd cap subtype "error_max_budget_usd", and `result` is NULL So the budget case had nothing for a `result`-only matcher to see. `subtype` is now checked FIRST -- where it is informative it is exact, and it is the only signal the budget case carries -- and the note becomes `capped: builder budget cap ($25.07 over 200 turns) (claude rc=1)`. The log line and note say "STOPPED BY A CAP" rather than "ACCOUNT LIMIT", since it now covers both. Verified against the three shapes, using round 4's REAL result JSON for the first: budget cap -> priced message; session limit -> unchanged, still matches; a genuine refusal ("I could not fix this; the tests fail") -> neither, falls through to the generic note as it must. Worth noting the pattern: this is the third metric this loop reported as something it was not -- provider refusals counted as kuna failures (#419), un-refuted hypotheses indistinguishable from undecided ones (#420), and now a budget cap indistinguishable from a broken build. Each was invisible until a number was checked for another reason. tools/repipe/smoke.sh 127/127. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
mahaloz
added a commit
that referenced
this pull request
Sep 6, 2026
…437) Found by round 4's captain, which wrote in its own notes that it had checked the builders with `ps`, "not with status.py, whose stale_s only ticks on a phase change and still lists b-r4-function-disasse as running". It was right, and it had to work around the tool this loop gives it for exactly this question. `stale_s` is `now - updated_at`, and `updated_at` moves on a `state update`, i.e. on a PHASE CHANGE -- not on any sign of life. Neither status module calls `os.kill`, so nothing ever asks whether the pid exists. `reap()` does ask, but only past `stale_seconds` (1800) and only when something calls it. So a worker that dies mid-phase reads `running` with a growing counter for up to half an hour, and the operator cannot tell it from one working hard on a long phase. One label, two very different states. `collect()` now records `alive` from a `kill(pid, 0)`: True, False, or None when no pid was recorded or the check itself failed -- unknowable is not the same as dead, and PermissionError means the process exists under another uid. The repipe table prints `DEAD` in the status column when a row claims `running` and the pid is gone. Verified by planting a `running` worker with pid 999999 in the inventory: the row prints DEAD, and the inventory was restored afterwards. `scripts.pipeline.status --json` still works, so the angr lane is unaffected. tools/repipe/smoke.sh 127/127. Fourth instance of one pattern this session, and the last three were found the same way -- a number checked for an unrelated reason. Provider refusals counted as kuna failures (#419); un-refuted hypotheses indistinguishable from undecided ones (#420); a builder budget cap indistinguishable from a broken build (#436); now a dead worker indistinguishable from a busy one. Each time, a status collapsed two causes into one label, and nothing failed loudly. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Round 3 filed 16 needs and recorded a hypothesis verdict on 5. The other 11 read
inconclusive, which looks like "a refuter ran and could not decide" and isindistinguishable in the record from it. It is not what happened.
hypothesis_statushad exactly two writers in the whole tree, and both are DEFAULTS:cluster.py sets
inconclusiveat filing, needs.py repeats it in the schema. Nothingever changed it. The captain prompt says T_REFUTE must "record upheld/overturned
either way", and there was no command that could -- the only route was hand-editing
YAML front matter in a markdown file. So the instruction had no mechanism, and 69% of
round 3's needs went to builders with an un-refuted hypothesis.
That is the expensive kind of gap, because refutation is what the two-arm design rests
on: the sibling decbench campaign overturned the filed diagnosis on 3 of 8 cases while
the symptom stood in all 8. Round 3's own numbers say the same thing -- of the five
verdicts that WERE recorded, two were overturned. Extrapolated over eleven unexamined
hypotheses, that is roughly four builders working from a wrong cause.
Writes
hypothesis_statusand appends a dated line to the Decision log, so the verdictand its reasoning live with the need rather than in a captain tick's transcript. Refuses
to write a record that fails
validate().The captain prompt now names the command in both places it describes T_REFUTE, and says
plainly that
inconclusiveis legitimate ONLY when a refuter ran and could not decide --leaving a need at the filed default is not a verdict.
Verified against a real record: front matter flips, the decision-log line lands with the
round number and the previous value, and the record still validates. tools/repipe/smoke.sh
127/127.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
🤖 Generated with Claude Code
https://claude.ai/code/session_01YcFmfZndNjgfqLQVBZCdkY