Skip to content

2.1.0

Latest

Choose a tag to compare

@guitaripod guitaripod released this 25 Jul 08:51

No scores change in this release. It closes the gap that 2.0.0 left behind.

Rubric v2 made an independent audit mandatory — but the audit itself only existed as a throwaway script inside the session that ran it. Its prompt, scope split and output shape were nowhere in the repo, so the next benchmark run could not have reproduced a v2 grade. That is the one thing a version bump is supposed to guarantee.

Pinned like every other pass

  • grading/AUDIT.md — the canonical auditor prompt.
  • grading/audit-schema.json — what a single auditor returns, and the merged shape grade.mjs --adjudicate consumes.
  • scripts/audit.mjs — runs the ten scopes (five feature groups, four craft axes, one core-logic correctness check) as independent claude -p agents that are blind to each other, then merges their verdicts. --only <scope> re-runs individual scopes and merges them back into an existing audit file — scopes do die mid-run on session limits, and re-running all ten to recover one is wasteful.
  • scripts/regrade.sh <id> — drives all three passes end to end; regrade.sh <id> 12|audit|3 runs a single stage.

The full v2 flow is now one command per submission, with every prompt in version control:

scripts/regrade.sh <id>
node scripts/grade.mjs --submission <id> --merge /tmp/rg-<id>-final.json --by "<judge>"

Ingest fix

add-submission.mjs now writes provenance. validate.mjs has always required it and ingest never produced it, so every newly ingested submission left the manifest in a failing state until someone hand-edited JSON. It defaults to the harness case (one-shot, autonomous, self-provisioned, harness-verified); --verified and --not-self-provisioned cover the rest.