Skip to content

v2.1.0 — Scale the eval

Choose a tag to compare

@sanlee-ys sanlee-ys released this 18 Jul 14:06
· 138 commits to main since this release
f4c6f4f

Milestone: scale the eval. The Opus judge — validated against the human gold labels at 94.4% / 94.4% agreement — grades 300 fresh real DVIDS snippets, disjoint from both the corpus and the gold set, with 95% Wilson confidence intervals: category 93.3% [89.9, 95.6], domain 90.3% [86.5, 93.2] — half the n=54 CI width, corroborating the gold-set numbers rather than replacing them.

This tag also ships everything accumulated under [Unreleased] since v2.0.1:

  • Evals-as-CI capability gate (ADR-007) — a free offline gate on every push/PR grading the committed snapshots against thresholds.toml floors, plus a paid weekly live gate.
  • Rung-1 prompt-optimization loop (ADR-005, autonomy ladder L3) — agent-driven prompt revision with a structural Goodhart guard and an explicit done-signal, held for this release per the spec's sequencing.
  • Prompt refinement (tech-vs-ops + land) — measured on the gold set: category 90.7% → 94.4%, domain 90.7% → 92.6%, technology recall to 1.000.
  • BM25 grounding retired (ADR-012) — a fair same-prompt re-measure showed grounding no longer pays: neutral on category, 0 fixed / 4 broken on domain. Removed from the shipped path; kept dormant as the record.
  • OpenTelemetry tracing over classify(), the v2.0.2 orchestration-test backfill (src/ coverage 90% → 97%), and a parity Jenkinsfile.

Full detail: CHANGELOG 2.1.0