Releases: ncoevoet/claude-review-all
Release list
v0.8.1 — doc-staleness rule reverted; REVIEW.md support now measured
A measurement release. One v0.8.0 feature is removed because an A/B showed it changed nothing, and another is now backed by evidence rather than argument.
Removed: the CLAUDE.md staleness rule
Two independent fixture designs, N=3, both arms. The first named a symbol the diff changed — reachable by the consistency agent's existing stale-reference grep, so it proved nothing. The second was built specifically to be out of that mechanic's reach: an architectural invariant stated in prose sharing no identifier with the diff, falsified by a fetch() added inside a request handler. The v0.7.1 baseline caught it anyway, PASS 3/3, exactly as the treatment did.
An instruction that changes no outcome is prompt weight in every consistency-agent spawn, so it is gone: the persona section, the stale-docs root-cause category, tests/check-doc-staleness.sh, and the README paragraph. Eval case 91 stays, reclassified as a regression guard — the behavior is real and emergent, it just was not caused by the rule.
Measured: REVIEW.md support works
Case 90 is the suite's first genuine evidence for a v0.8.0 feature: baseline FAIL (1/3), treatment PASS (3/3). Both directions of the rule require the feature — a DEBT-tier rounding duplication promoted to CRITICAL, and a real unvalidated-input defect under a suppressed path staying unreported.
Its first version was non-discriminating (it used a defect default calibration already rated CRITICAL, and bait the baseline already ignored), which is what prompted the rewrite.
Honestly unmeasured
- Per-agent diff ordering. Case 02 is the only fixture that exercises it, and it returned
PASS (2/3)thenFAIL (1/3)on identical baseline code, with no timeouts or errors in the raw logs. It cannot resolve a difference, so ordering is neither proven nor regressing. An earlier apparent regression from this case was that flicker. <gate_results>. Case 92 passes on both arms: a failing gate is already caught by Phase 1 without the block. It is kept because it is the only fixture whose gates genuinely run and fail — infrastructure the suite did not previously have.
The rule this produced
A case proves nothing until the baseline arm has been shown to fail it. Run the baseline on every new case, not just the treatment. A case both arms pass is a regression guard; only a case the baseline fails and the treatment passes is evidence. evals/README.md now records this along with the full A/B table.
Suite: 92 cases, 73 unit tests, 9 gates, shellcheck clean.
v0.8.0 — REVIEW.md support, per-agent diff ordering, gate output to agents
Four web-grounded improvements, plus four documentation-drift fixes. Research basis: Cursor's Bugbot (randomized-order parallel passes), SWR-Bench (arXiv 2509.01494, multi-review aggregation), 2026 hybrid LLM + static-analysis studies, and Anthropic's Code Review documentation, which has since formalized REVIEW.md.
REVIEW.md support
Drop a REVIEW.md at the repository root to change what the review flags, at what severity, and where. It is read fresh on every run — never cached — and injected verbatim into all ten agents and the verifier as the highest-priority block. Where it conflicts with a persona or the shared rules, it wins. No configuration key is involved; the file is the interface, and its per-path raised-bar idiom ("in scripts/, only report if near-certain and severe") replaces the previously planned pathBars key.
It steers what gets reviewed, never how it is proven. The 3-question gate, claim classes, file:line evidence, and Phase 2.5 verification are integrity rules: a severity REVIEW.md promotes still has to be earned by proof at that tier, and an instruction to skip verification is ignored while the rest of the file is honored.
In gate mode it moves findings across the blocking floor in both directions — a promotion becomes CI-blocking, and a demotion can un-block a real defect. Same trust model as CLAUDE.md. When REVIEW.md is itself modified inside the reviewed diff, the gate summary now says so, because a change to the rules that grade a diff, arriving inside that diff, is the shape of a self-approving PR.
Per-agent diff ordering
scripts/agent-order.py gives each agent the same diff in a different file order, from sha256("<agent>:<path>"). Attention is not uniform across a long prompt, so ten agents reading an identically ordered diff shared one weak middle. The permutation is reproducible across machines and runs, and filter-stable: an agent reviewing a narrower slice gets the order it would have had if that subset were permuted directly, so one call serves all ten slices.
Chunk composition is computed on the canonical git order first and the permutation applied within each chunk, so chunk membership stays identical across agents. Hunks inside a file are never reordered.
Failing gate output reaches the agents
When a Phase 1 gate fails, its output is passed to every agent and to the verifier as a truncated, failures-only <gate_results> block — omitted entirely on a green run, so the common path costs nothing. A compiler or test runner is ground truth that reading cannot match, which is why hybrid static-analysis plus LLM review outperforms either alone. It is framed as a lead to the underlying defect, never a finding to restate: the failure is already on the report.
dedupe.py now records corroborating_agents, and the report renders "Flagged independently by N agents". The verifier uses the count to prioritize re-read effort and break borderline ties, never as a substitute for the citation gate — agreement is not independence, so the scoring rubric is deliberately unchanged.
CLAUDE.md staleness
The consistency agent flags a doc claim the diff falsifies, but only when it can quote both the doc sentence and the contradicting diff lines. A doc that merely mentions the changed area is not stale. An explicit rule keeps this disjoint from the standards agent, which reads the same sentence from the opposite direction.
Drift fixes
The Phase 4 menu-skip condition now carries the unverified clause in all four places it is stated; SKILL.md's inline config schema regained the nine keys it had fallen behind on; a duplicated Fix Results heading is gone; an orphaned pycache artifact was removed.
Measurement — read this before trusting the new eval cases
A targeted A/B (N=3, v0.7.1 baseline versus this release) ran the two precision counter-cases and the two new cases. Both arms scored 4/4.
- No precision regression. Counter-cases
05and55held at PASS 3/3, with zero critical or important findings on05. The corroboration count is not being read as proof. - No recall lift is claimed. The two new cases (
90REVIEW.md,91CLAUDE.md staleness) also passed on the pre-feature baseline, so they demonstrate nothing about the features and stand only as regression guards.evals/README.mdrecords why each fixture fails to discriminate —90picked a defect default calibration already rates CRITICAL, and91is already caught by the consistency agent's generic stale-reference mechanic — and what a discriminating rewrite needs.
The verifier is bumped 6 → 7 once for all three verifier-affecting changes, so cached verdicts invalidate once rather than three times.
Tests
Five new static doc gates, each proven non-vacuous by deleting its asserted sentence on a scratch copy and confirming failure. check-config-sync.sh set-diffs SKILL.md's config schema against the documented key table, making that drift class structurally unrepeatable. Nine unit tests for agent-order.py, four for corroborating_agents. Suite: 91 eval cases, 73 unit tests, 10 gates, shellcheck clean.
v0.7.1 — claim-class discipline, gate provenance, unverified verdict
Evidence-provenance hardening across the review pipeline. The target is the failure mode where a review reads real code, cites real lines, and is still wrong — because a static read was silently substituted for a claim about runtime behaviour, backend data, or rendered output. Same shape for the gates: an assumed build looks exactly like one that ran.
🧭 Claim classes (agents/_shared.md)
Every finding agent now classifies its core claim as static / runtime / data / rendering, and checks it holds the proof that class demands. Reading a template proves what the template says; it is zero evidence about what the server returned or what the user saw. A runtime/data/rendering claim backed only by a source read is capped at ⚪ QUESTION / MEDIUM confidence and must state the missing observation inline.
| Class | Admissible proof | Not proof |
|---|---|---|
| Runtime | a test exercising the path, a log line from a real run, a documented framework contract | reading the function that would do it |
| Data | an actual payload — fixture, recorded response, network capture | inferring from a schema, model, or DTO |
| Rendering | rendered DOM, snapshot test, screenshot | reading the template |
🔬 The unverified verdict (agents/verifier.md, v5 → v6)
A claim that demands observation, where nobody has one, is neither a false positive nor a confirmed defect — forcing it into keep/drop destroys information either way. The verifier now emits a fourth verdict, unverified, plus claim_class and needs_observation (the specific check that would settle it — "open a row of the variant subtype and assert the resolved URL", not "test it"). It is orthogonal to the score bands, so a well-argued, high-scoring claim still lands there. These render in a new 🔬 Unverified — needs observation report section, never as 🔴/🟠, and are never dropped.
The cross-agent +10 confirmation bonus now requires methodologically independent evidence: N agents re-reading the same file and drawing the same inference is one inference reported N times, and the bonus was rewarding that correlation.
🧾 Gate provenance (SKILL.md Phase 1, references/phase-3-report.md)
Every gate row now records the exact command, its exit code, and the wall-clock time it ran, surfaced as a mandatory Provenance column in the report's gate table. Hard rules: PASS requires a command executed in this run — a result from a log, a prior session, or CI is SKIP(not-run-this-session); a gate that applies but did not run is SKIP(reason), not N/A; one gate never infers another (passing tests do not imply a passing typecheck); and template compilation is not covered by a plain tsc --noEmit. Substituting a running dev server for the build gate now requires a four-point liveness proof — port open this run, log identified, log mtime newer than the newest file in the diff, success marker in the tail.
⚠️ Behaviour change for gate mode
A runtime claim held on static evidence that would previously have blocked as 🔴 now lands in 🔬 and does not block /review-all gate. This is a deliberate precision-over-recall trade for the CI/autonomous-loop path: a blocking oracle should not fire on a claim nobody observed. The finding stays fully visible to human readers.
🧪 Testing
New tests/check-claim-class.sh doc-invariant gate (24 assertions across _shared.md, verifier.md, phase-2.5-verification.md, phase-3-report.md, phase-gate.md, SKILL.md, README.md), wired into tests/run.sh alongside the four existing gates. Like the menu and vote gates, it guards behaviour that is instruction-driven and cannot be exercised headlessly.
Validation
Guarded by the deterministic suite only: shellcheck -S warning clean, tests/run.sh green — 5 doc-invariant gates CLEAN, eval schema validation, shell tests, 60 Python unit tests OK.
No headless eval A/B was run for this release, unlike previous cycles. The precision/recall effect on the 89-case suite is therefore unmeasured. The change is additive and instruction-level — it introduces a new report section and a new verdict rather than altering existing detection heuristics — but that is a reasoned expectation, not a measurement.
Full changelog: v0.7.0...v0.7.1
v0.7.0 — eval scorecard, opt-in majority-vote verifier, dismissed-finding digest
Three additive / opt-in improvements, each validated for non-regression against the eval suite.
📊 Eval scorecard (scripts/eval-scorecard.py)
Aggregates the headless runner's per-case PASS/FAIL into a suite-level recall % / precision % / F1 / SNR scorecard, so a prompt change can be A/B'd against an aggregate precision number, not just per-case pass rates. The headless runner now emits SCORE lines (from the report's severity tally) and accepts a REVIEW_ALL_CONFIG_JSON hook so config-driven features can be A/B'd at all. The SNR is an honest suite-derived proxy, documented as such — not a per-comment metric.
🗳️ Opt-in majority-vote verifier (verifierVotes, default 1)
Set an odd N > 1 (e.g. 3) in .claude/review-all.json and each 🔴 CRITICAL / 🟠 IMPORTANT finding reaches the main report only if a majority of N independent hostile verifiers keep it (per-batch, median score, ties demote to appendix). Scoped to top severity — debt/suggested/question stay single-pass — and off by default, since it multiplies verifier cost on the highest-stakes findings. Targets single-verifier mis-scoring on borderline findings.
{ "verifierVotes": 3 }🔁 Dismissed-finding digest
The finding agents now receive a <previously_dismissed> digest of the team's wontfix / non-expired-snoozed decisions (from state.json) and skip re-deriving them at unchanged locations — saving generation and verifier spend. The Phase 2.5 central drop remains the guarantee.
🧪 Testing
Two new doc-invariant gates (check-verifier-votes.sh, check-dismissed-digest.sh) guard the headlessly-untestable prompt behaviors, wired into tests/run.sh alongside the existing menu gate.
Validation
A headless A/B (verifierVotes:3 vs default, via claude -p) confirmed non-regression: recall and precision held at 100% on the tested subset. A beyond-the-diff "blast radius" feature (grep callers of changed exports) was prototyped and evaluated but removed before release — the review already performs cross-file caller analysis (the control arm caught a designed cross-file break and executed the code to confirm the runtime error), so the feature was redundant.
Full changelog: v0.6.1...v0.7.0
v0.6.1 — README polish, report ToC, verifier fix
- README: CI / version / license / Claude-Code-plugin badges, a demo placeholder, and refreshed
code.claude.comdoc links. - references/phase-3-report.md: added a Contents table of contents.
- Verifier (from the prior commit): destructive claims must cite the destructive op.
No behavioural changes to the review flow. Bumps plugin + marketplace to 0.6.1.
v0.6.0
Fixed
- Marketplace name collision.
claude-review-allandclaude-loopboth declared marketplacename: ncoevoet, so adding one clobbered the other and/plugin install review-all@ncoevoetfailed with "not found in marketplace ncoevoet". This marketplace is nowncoevoet-review-all(the repo pathncoevoet/claude-review-allis unchanged). - Version drift.
marketplace.json(0.4.0) andplugin.json(0.5.0) disagreed; both are now 0.6.0.
Install / migrate
/plugin marketplace add ncoevoet/claude-review-all
/plugin install review-all@ncoevoet-review-all
If you installed the old ncoevoet marketplace, remove it first: claude plugin marketplace remove ncoevoet, then re-add and install with the new @ncoevoet-review-all identifier.