What changed
Proofrag 0.8.0 makes evaluation failures harder to hide and adds three fully retained case studies over official, hash-checked sources.
- Added exact-chunk retrieval matching to
evaluate,compare, and the GitHub Action. - Made evaluation, comparison, generation, optional backends, and regression diffs fail closed on incomplete or incompatible data.
- Corrected NDCG normalization and strengthened corpus, HTTP adapter, JSON, and output safety.
- Added opt-in provider token accounting through
PROOFRAG_USAGE_LOGwithout storing prompts or answers. - Added audited Python, HTTP RFC, and OWASP studies with golden sets, predictions, scorecards, blind comparisons, cost records, failure analysis, and reproduction steps.
What the new studies found
- Python concurrency: SQLite FTS5 won 13 blind comparisons to token overlap's 6, with 11 ties; the largest exact-match difference was on multi-document retrieval.
- HTTP RFC metadata: section-enriched BM25 improved exact NDCG@5 by 0.018, below the predeclared 0.05 materiality threshold; metadata won 4 of 5 decided comparisons.
- OWASP context depth: top-6 improved exact Recall@6 by 0.024 but reduced judged answer quality; top-3 won 7–5, with 12 ties.
The two new studies used 270 API calls and an estimated $0.1604 at the providers' published list prices. Their reports retain negative results and judge disagreements, including a manually identified false-negative refusal score.
See the changelog for the complete change list and the case-study index for reports and reproduction workflows.
The retired floating v0 Action tag is not recreated; workflows should pin unshDee/proofrag@v0.8.0.