v3.1.0 — The autonomy ladder, built to the top
Milestone: the autonomy ladder, built to the top. L3's second rung and all of L4 land here, together with the classical-ML bake-off that finally measured whether the LLM was worth paying for at all.
Four verdicts, three of them negative. All four are kept as records rather than deleted, which is the point of the repo.
What landed
ADR-017 — the classical baseline, finally measured. TF-IDF + logistic regression trained on 300 judge-graded snippets, scored once against the human gold set. The LLM wins decisively: category 92.6% vs 72.2% (McNemar p=0.013), domain 92.6% vs 66.7% (p=0.0005). This repo had measured three things layered on top of an LLM without ever baselining the LLM itself. Now it has.
ADR-018 — L3 rung 2, the agent-driven ML loop. An agentic outer loop reads out-of-fold errors and proposes the next sklearn experiment, reusing rung 1's honesty architecture as shared code. The first live run is the headline: the agent improved the metric it could see and degraded the one that matters, and the harness caught it. Best-by-B iteration scored B 0.699 (+6.0 over baseline) while held-out C fell to 0.545 (−8.6).
Nothing cheated. The mechanism is distribution shift — splits A and B are both judge-labeled wire text, so keywords mined from A's errors genuinely generalize there and mislead on C's human-labeled mix. Every guard behaved: plateau fired, best-iteration selection read B alone, and C exposed the trade precisely because nothing was allowed to optimize it. The held-out set vetoed the loop's own best iteration. That veto is the artifact.
ADR-019 — kNN exemplars, a clean null. k=3 BM25-retrieved labeled exemplars in the prompt: category 91.0% vs 90.0% (p=0.70), domain flat, region guardrail exact. That closes the retrieval question in three shapes — neighbor documents harmful, mined keyword features harmful off-distribution, labeled exemplars inert.
ADR-020 — L4, the multi-agent pipeline. Triage produces verbatim evidence spans, the shipped classify() runs blind, and a critic with a narrow rubric-checkable charter can bounce a label backward for exactly one re-classify. Hypothesis confirmed: the backward edge fixed 6 of the 7 named region misses. Pipeline declined anyway — the all-axes critic over-challenged (57.4% against an expected ~13%, tripping the spec's own red-flag rule) and did net harm elsewhere, including the repo's first statistically significant harm (scale domain 91.3 → 86.7, p=0.016) at roughly four times the calls.
The shipped single call stays production.
Why MINOR
Everything above is eval and experiment machinery. src/api.py is untouched since v3.0.0 and the system prompt did not move, so the {category, operational_domain, region} contract holds and callers see nothing new.
The published gold numbers are unchanged and still describe what ships — regenerating evals/metrics.json moved only the version field, which is the mechanical proof.
Roadmap note
v3.0.0's release notes pencilled the scaled (n=300) region eval in at this version number. That work is unblocked but unscheduled, and none of the ladder work depended on it, so the ladder shipped first and the scaled region eval moves to v3.2.0. Surfaces that named the old slot now use version-free phrasing so the reference cannot rot again.
Full detail: CHANGELOG 3.1.0