2.1.0

No scores change in this release. It closes the gap that 2.0.0 left behind.

Rubric v2 made an independent audit mandatory — but the audit itself only existed as a throwaway script inside the session that ran it. Its prompt, scope split and output shape were nowhere in the repo, so the next benchmark run could not have reproduced a v2 grade. That is the one thing a version bump is supposed to guarantee.

Pinned like every other pass

• grading/AUDIT.md — the canonical auditor prompt.
• grading/audit-schema.json — what a single auditor returns, and the merged shape grade.mjs --adjudicate consumes.
• scripts/audit.mjs — runs the ten scopes (five feature groups, four craft axes, one core-logic correctness check) as independent claude -p agents that are blind to each other, then merges their verdicts. --only <scope> re-runs individual scopes and merges them back into an existing audit file — scopes do die mid-run on session limits, and re-running all ten to recover one is wasteful.
• scripts/regrade.sh <id> — drives all three passes end to end; regrade.sh <id> 12|audit|3 runs a single stage.

The full v2 flow is now one command per submission, with every prompt in version control:

scripts/regrade.sh <id>
node scripts/grade.mjs --submission <id> --merge /tmp/rg-<id>-final.json --by "<judge>"

Ingest fix

add-submission.mjs now writes provenance. validate.mjs has always required it and ingest never produced it, so every newly ingested submission left the manifest in a failing state until someone hand-edited JSON. It defaults to the harness case (one-shot, autonomous, self-provisioned, harness-verified); --verified and --not-self-provisioned cover the rest.

2.0.0

Two Opus 5 submissions join the benchmark, and grading them broke the method badly enough that the whole board was re-scored under a new rubric version.

Leaderboard (rubric v2)

#	Model	Effort	Bench	Depth	Code · Arch · UX · Robust
🥇	Opus 5	ultracode	83.8	74/90	8.5 · 9 · 9 · 8
🥈	Fable 5	ultracode	78.0	66/90	8 · 8.5 · 9 · 8.5
🥉	Opus 5	low	62.0	54/90	7 · 8 · 6 · 5
4	Opus 4.8	low	56.8	41/90	7.5 · 7.5 · 7 · 7.5
5	Fable 5	low	55.8	44/90	6.5 · 6.5 · 7 · 6.5
6	GLM 5.2	max	48.7	43/90	4.5 · 5 · 6 · 4.5
7	Grok	default	41.7	37/90	4 · 4 · 5.5 · 3.5
8	DeepSeek V4 Flash	default	41.3	29/90	5.5 · 4.5 · 6.5 · 5.5
9	Laguna S-2.1	default	33.0	33/90	2.5 · 2.5 · 4 · 2

v1 scores are not comparable to these. Every entry moved down and the order changed. That is what the major version bump means.

Why v2

v1 graded in two passes: grade, then adversarial calibration. The calibration pass turned out to rubber-stamp — on both new submissions it returned the first pass unchanged, once leaving 22 of 30 features at grade 3. It was approving "correct and complete" as exceptional.

v2 makes an independent audit mandatory: ten agents, blind to each other, try to refute every grade against the source and the live site; the judge then adjudicates each dispute. Grading only the new entries this way would have made the leaderboard a comparison of grading rigor, so rubricVersion was bumped and all nine were re-graded together by one judge under one method. Calibration anchors are now stated as criteria rather than pointers at other entries' scores, so they survive a re-grade.

New submissions

Both are one-shot autonomous runs of claude-opus-5 against the verbatim brief, self-provisioning their GitHub repo and Cloudflare deployment.

• ultracode — 288 turns, 33,239 LOC across 149 files, the full PokéAPI CSV dataset compiled into 6,032 static shards, all 1025 species reachable at runtime. It ran its own adversarial review before shipping.
• low — 100 turns, 4,491 LOC. Zero console errors, and its browse page still hands the user only 60 of 1025 species: the infinite-scroll observer attaches before its sentinel mounts, with no pagination fallback.

Integrity fixes

• A recorded deployment had drifted off its submission. None of the commits deployed to Grok's Pages project exist in its repo — its runtime signal was measuring a different app. scripts/verify-live.mjs now fingerprints every live URL against the vendored source and fails on divergence. Grok's runtime is measured against a local build of its own source, and it has no demo link.
• dexReach — the smoke test now scrolls the browse page to exhaustion and counts distinct species, across anchor, hash-route and click-handler UIs, flagging whether a pager exists. A dex that throws no errors and still stops at 60 of 1025 is now visible; pagination (DeepSeek's 30/page) is not mistaken for breakage.
• validate.mjs fails any entry on a stale rubric version or missing a note when its deployment is gone; duplicate feature grades are rejected; the pinned score test recomputes the formula instead of hardcoding a number.

Caveat

The v2 judge is Opus 5, which also built two of the entries. The audit exists precisely to counter self-preference and cut those two as hard as anything in the field; an Opus 4.8 cross-check landed within one point pre-audit. A fully independent judge would be stronger still — see methodology.

1.0.0

Fix review findings: rubric-version single source + doc/anchor accuracy

- rubricVersion now has one source of truth: grading/config.json. Removed
  the redundant manifest top-level field; validate.mjs and grade.mjs both
  key off config, so a deliberate version bump can't desync them (the
  latent bug that would have failed every submission on the first bump).
- Reworded the Laguna robustness anchor everywhere (PROMPT/RUBRIC/
  methodology) to match its real runtime verdict — loads, then throws on
  the detail route (errors) — not a load crash.
- running-a-benchmark.md: the opencode grade path is two-pass; rewrote the
  Calibration section for the append-only fixed-anchor model (it still
  described the replaced co-grade / re-grade-the-whole-set approach).
- Tighter provenance validation (oneShot/autonomous/verified), dedupe
  .gitignore, skip the unused Playwright browser download in CI.