Skip to content

v2.0.0 — Real-text RAG eval

Choose a tag to compare

@sanlee-ys sanlee-ys released this 22 Jun 00:03
· 261 commits to main since this release
a1068e3

v2 moves the classifier off synthetic, self-graded data and onto real public-domain text, with retrieval grounding and a non-circular, human-labeled answer key. v1 measured in-distribution consistency; v2 measures real-world accuracy.

Headline results (real text, human-graded gold set, n=54)

Field Accuracy Macro-F1
Category 88.9% 0.906
Operational domain 88.9% 0.894
  • v1's worst class fixed: industry recall 0.217 → F1 1.000 on real SEC filings (caveat: n=5 clear-cut cases).
  • Judge validated: Opus judge agrees with the human labels 88.9% (category) / 94.4% (domain).
  • Grounding measured, not assumed: BM25 grounding gave +1.9% category / flat domain → lexical retrieval doesn't justify embeddings here. The negative result is the finding.

What's new

  • Real public-domain corpus (62 docs: DVIDS news wire + SEC filings) with a BM25 retriever (src/retrieve.py).
  • Human-labeled gold set (data/gold/) + labeling guide; honest eval harness with Opus-judge validation (src/gold_eval.py).
  • Retrieval-grounded classification with citations (src/classify_rag.py) + grounding-lift eval with flip analysis (src/gold_eval_rag.py).
  • README rewritten to lead with the v2 numbers.

Full details in CHANGELOG.md. Diff: v1.1.0...v2.0.0