v3.2.0 - The ruler shrinks
The region axis had one number and an 18-point error bar. It now has a 7-point one.
v3.2.0 runs the scaled region eval: 300 DVIDS snippets graded by the Opus judge that
validated at 100.0% region agreement against the human labels — the gate
ADR-014
set before allowing the judge anywhere near this axis.
| Workhorse vs judge | Accuracy | 95% CI (Wilson) | Macro-F1 |
|---|---|---|---|
| Region | 88.3% (265/300) | [84.2%, 91.5%] | 0.904 |
| Category | 91.7% (275/300) | [88.0%, 94.3%] | 0.763 |
| Domain | 89.3% (268/300) | [85.3%, 92.3%] | 0.868 |
The interval is the deliverable, not the accuracy. 88.3% against the gold set's 87.0% is
corroboration; what changed is that the number is now known to within 7 points instead of 18,
so a future change to the region rubric can finally be told apart from noise.
The named global cluster is confirmed systematic. On the 54-row gold set, all seven
region misses were rows whose true label is global that the model pulled to a specific
region, inferring a theater from the US actor where the snippet names no place — but seven
rows cannot separate a behavior from a run of luck. At n=300 they can: of 70 answer-key
global rows, 17 were pulled to a region (16 to americas), which is 49% of all 35
region disagreements. One identified failure mode, half the region error. The prompt clause
that would target it is deliberately not in this release — measuring first is the method.
Read alongside, never instead of, the human-graded n=54 figures: this is workhorse-vs-judge
agreement, and the judge's own measured disagreement with humans on region was 0/54, which is
itself a wide interval.
The shipped classifier is unchanged — same prompt, same single call, same
{category, operational_domain, region} contract. MINOR.
Also in this tag: one API error taxonomy plus a stop-reason assertion so a truncated response
is never scored (ADR-021), a paired-comparison layer with harness health reported separately,
the ADR-017 classical baseline made runnable in a browser behind a parity gate, two provenance
pins that stop a stale prompt being published or graded, and two CI lanes.
Full detail: the [3.2.0] CHANGELOG entry
and ADR-022.
Report: evals/scale_eval_v3.txt.