Skip to content

1.12.0 — the judge lists everything before it judges

Latest

Choose a tag to compare

@szarkans szarkans released this 08 Sep 16:51
  • The judge lists everything before it judges. The review report opens with ## 📋 Everything raised (N): every finding from every reviewer, numbered, with its author, its file:line and its claim in the reviewer's words, and the buckets below cite those numbers. Two findings merge only when they name the same mechanism; the same line with a different mechanism stays two, and a dropped finding keeps its own text and a reason that cites the line contradicting it. Measured on 8 real bugs (commit that introduced the bug, headless, same reviewers): the old judge delivered 3 of 8 while its own reviewers had found 5 — Codex's real finding at one line was folded into a sub-agent's speculation at a neighbouring line and dropped with it, and a bug three reviewers agreed on was merged into a broader neighbour and lost its mechanism. With the inventory the report delivers what the reviewers found, 6 of 8, at the same cost per review; the one finding still lost is a compound sentence from Codex whose second half is the bug, and the rule now says a two-mechanism sentence is two lines — that clarification is checked on the judge step alone, not end-to-end. Swapping the judge model did not fix it: Opus on the same inputs lost the same finding two times out of three, by the same merge.
  • Control arm against the built-in /code-review high, same bugs, same judge model: its report found 3 of 8. Scoreboard and protocol in evals/RESULTS.md; the harness is evals/run-arms.sh and evals/grade-arms.sh (blind grading: reports flattened to file:line | claim, three votes, majority). The comparison is one run per arm on eight bugs — a one-bug difference is noise; the reviewers' 5-vs-3 and the report's 6-vs-3 are the gaps that clear it.
  • evals/run.sh and run-arms.sh accept a seventh severity column in the case file.