Skip to content

v1.3.0 — retrieval metrics, judge calibration, online monitoring

Choose a tag to compare

View all tags
@lopadova lopadova released this 16 Jun 20:18
· 24 commits to main since this release
v1.3.0
5332e17
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Additive, backward-compatible release. All v1 contracts are preserved.

Added

  • Retrieval-ranking metrics (domain-agnostic): retrieval-hit-at-k, retrieval-recall-at-k, retrieval-mrr, retrieval-ndcg-at-k (binary or graded gains), answer-containment-at-k, on a shared RankedRetrieval parser + AbstractRetrievalRankingMetric base. metrics.retrieval.default_k config + per-sample metadata.k.
  • ordinal-distance metric — partial credit for ordered labels (exact 1.0 / off-by-one 0.5 / further 0.0).
  • Judge calibration — eval-harness:calibrate-judge: verdict agreement rate, confusion matrix, length-bias signal, self-preference guard, CI gating.
  • Online / production monitoring (off by default) — OnlineMonitor::capture(), queued JudgeLiveSampleJob, first package migration (eval_harness_online_scores), OnlinePassRateDropped drift event, read-only GET /{prefix}/online/{dataset}/trend endpoint, eval-harness-migrations publish tag.
  • RuntimeOptions::normalizeUnitInterval() helper.

Companion admin UI screen: padosoft/eval-harness-admin#6 (requires this release).

🤖 Generated with Claude Code

Assets 2
Loading