Skip to content

1.1.0 — Tighten the measurement

Choose a tag to compare

@sanlee-ys sanlee-ys released this 21 Jun 02:07
· 261 commits to main since this release

Hardens the measurement around the v1.0.0 classifier. The classifier's
happy-path behavior is unchanged; this release is about trusting the numbers.

Added

  • Macro-F1 — the honest single number for an imbalanced problem
    (category macro-F1 0.765 vs 79.0% accuracy; domain 0.973).
  • Stability harness (src/stability.py) — sizes the run-to-run noise floor
    (category accuracy std 0.24 pts over 5 runs), so a config change counts only if
    it clears ~2× std. Confirms the reverted prompt experiment's 2.3-pt regression
    was real (~10× the noise), not luck.
  • Error audit (evals/error_audit.md) — all 67 misses triaged; ~90% are
    label-scheme overlap (industry/procurement/technology), not model error.
  • Enum-validation guard in classify() — re-samples once on an out-of-enum
    label, then raises. Caught a real defect: 1/300 predictions returned an
    invalid category.

Fixed

  • Corrected the docs claim that tool-use "rejects out-of-enum output at the API
    layer." It's a guided prior, not a hard constraint; enum membership is now
    validated in code.

Docs

  • docs/CASE_STUDY.md narrative writeup; docs/how-it-works.md plain-language
    one-pager with a precision/recall/F1 definition and a "Threats to validity"
    section.

Full details in CHANGELOG.md.