Apply the eight-reviewer manuscript audit

[MaxGhenis]

Section-by-section multi-agent review of the manuscript before arXiv/SSRN submission: eight parallel reviewers (six section readers, a numbers auditor over ~290 quantitative claims, and a whole-paper editor), every finding verified computationally against the frozen snapshot before fixing.

Blockers fixed

• Weighting mechanism misstated: "zero-reference outputs contribute zero weight" was categorically false — zero rows hold 66.9% of weight mass, which is exactly why the always-zero baseline scores 66.9%. The corrected mechanism (weight concentrates in high-dollar, rarely-zero output groups) now lives once in §3.1; the abstract, related work, and baselines section shrink to references, which also collapses the four near-verbatim restatements of the zero-inflation argument.
• Table clipping: provenance, model-run, scope, deviation-audit, and parse-audit tables truncated cells mid-word at the page margin (the appendix findings were unreadable). Wide tables now render as wrapping pipe tables; the leaderboard drops its overflowing constant Parsed/Total columns; the model-run table drops two boilerplate columns repeated 24× (moved to the caption).

Majors fixed

• Abstract's "newest flagship is not the strongest" was false on this board (Claude Fable 5 is Anthropic's newest and strongest Claude at 79.9%); reframed as the within-family Opus 4.8 < 4.7 observation.
• §4.3 stale prose claimed GPT-5.5 ran pinned-low with a default rerun "scheduled," contradicting the very table it cites; the frozen wave is the default-effort run.
• Claude Fable 5's cost/latency showed $0.000/0 s (empty per-row usage); the table now reconciles against modelStats ($0.541/98 s — the priciest model, not the cheapest), sub-second batch dispatch renders as — (unmeasured), and a serving-regime comparability caveat replaces an editorializing aside.
• "Joint hit rate is consistently lower than either marginal" failed for GPT-5.6 Sol (joint = state marginal); Limitations said failed attempts are "preserved separately" while the appendix and data say they live inside the canonical files; reference-defect terminology unified as "reference-computation defects" with the retained scenario mis-encoding disclosed distinctly.
• Stale June-era annotations orphan inside the snapshot dir (3,300 rows failing the manifest's own hashes) removed; a reproducibility note documents the frozen scenarios' stale source_dataset label.

Minors (~15)

Dangling "Impact weighting" cross-reference → anchored section refs; Table 11 caption falsified by its own columns → accurate rationale; formal indicator 1{pred = ref} → 1{|pred − ref| ≤ 1}; bounded-score zero-branch strictness caveated; within-5% promise dropped; figure now cited; front/back matter unnumbered; ACA expanded; 18-group reconciliation explained; the two unrelated 84% figures disambiguated; "31 of 35 (89%)"; run-on splits; % formatting.

Verification

• 18 content assertions against the rendered PDF (each fix present, each stale claim absent); clipping canaries absent; wide tables visually inspected.
• All 16 snapshot-artifact tests pass after the re-freeze (which is byte-stable except the new reproducibility note and re-pinned render hashes); full suite 481 passed; Ruff clean.
• arXiv source bundle rebuilt and compiles standalone to the same 29-page content (~/Downloads/policybench-arxiv-source.tar.gz).

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policybench-site	Ready	Preview, Comment	Jul 17, 2026 5:41pm