You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The judge lists everything before it judges. The review report opens with ## 📋 Everything raised (N): every finding from every reviewer, numbered, with its author, its file:line and its claim in the reviewer's words, and the buckets below cite those numbers. Two findings merge only when they name the same mechanism; the same line with a different mechanism stays two, and a dropped finding keeps its own text and a reason that cites the line contradicting it. Measured on 8 real bugs (commit that introduced the bug, headless, same reviewers): the old judge delivered 3 of 8 while its own reviewers had found 5 — Codex's real finding at one line was folded into a sub-agent's speculation at a neighbouring line and dropped with it, and a bug three reviewers agreed on was merged into a broader neighbour and lost its mechanism. With the inventory the report delivers what the reviewers found, 6 of 8, at the same cost per review; the one finding still lost is a compound sentence from Codex whose second half is the bug, and the rule now says a two-mechanism sentence is two lines — that clarification is checked on the judge step alone, not end-to-end. Swapping the judge model did not fix it: Opus on the same inputs lost the same finding two times out of three, by the same merge.
Control arm against the built-in /code-review high, same bugs, same judge model: its report found 3 of 8. Scoreboard and protocol in evals/RESULTS.md; the harness is evals/run-arms.sh and evals/grade-arms.sh (blind grading: reports flattened to file:line | claim, three votes, majority). The comparison is one run per arm on eight bugs — a one-bug difference is noise; the reviewers' 5-vs-3 and the report's 6-vs-3 are the gaps that clear it.
evals/run.sh and run-arms.sh accept a seventh severity column in the case file.