v2.0.0 — Real-text RAG eval
v2 moves the classifier off synthetic, self-graded data and onto real public-domain text, with retrieval grounding and a non-circular, human-labeled answer key. v1 measured in-distribution consistency; v2 measures real-world accuracy.
Headline results (real text, human-graded gold set, n=54)
| Field | Accuracy | Macro-F1 |
|---|---|---|
| Category | 88.9% | 0.906 |
| Operational domain | 88.9% | 0.894 |
- v1's worst class fixed:
industryrecall 0.217 → F1 1.000 on real SEC filings (caveat: n=5 clear-cut cases). - Judge validated: Opus judge agrees with the human labels 88.9% (category) / 94.4% (domain).
- Grounding measured, not assumed: BM25 grounding gave +1.9% category / flat domain → lexical retrieval doesn't justify embeddings here. The negative result is the finding.
What's new
- Real public-domain corpus (62 docs: DVIDS news wire + SEC filings) with a BM25 retriever (
src/retrieve.py). - Human-labeled gold set (
data/gold/) + labeling guide; honest eval harness with Opus-judge validation (src/gold_eval.py). - Retrieval-grounded classification with citations (
src/classify_rag.py) + grounding-lift eval with flip analysis (src/gold_eval_rag.py). - README rewritten to lead with the v2 numbers.
Full details in CHANGELOG.md. Diff: v1.1.0...v2.0.0