1.1.0 — Tighten the measurement
Hardens the measurement around the v1.0.0 classifier. The classifier's
happy-path behavior is unchanged; this release is about trusting the numbers.
Added
- Macro-F1 — the honest single number for an imbalanced problem
(category macro-F1 0.765 vs 79.0% accuracy; domain 0.973). - Stability harness (
src/stability.py) — sizes the run-to-run noise floor
(category accuracy std 0.24 pts over 5 runs), so a config change counts only if
it clears ~2× std. Confirms the reverted prompt experiment's 2.3-pt regression
was real (~10× the noise), not luck. - Error audit (
evals/error_audit.md) — all 67 misses triaged; ~90% are
label-scheme overlap (industry/procurement/technology), not model error. - Enum-validation guard in
classify()— re-samples once on an out-of-enum
label, then raises. Caught a real defect: 1/300 predictions returned an
invalid category.
Fixed
- Corrected the docs claim that tool-use "rejects out-of-enum output at the API
layer." It's a guided prior, not a hard constraint; enum membership is now
validated in code.
Docs
docs/CASE_STUDY.mdnarrative writeup;docs/how-it-works.mdplain-language
one-pager with a precision/recall/F1 definition and a "Threats to validity"
section.
Full details in CHANGELOG.md.