proofrag v0.8.0

What changed

Proofrag 0.8.0 makes evaluation failures harder to hide and adds three fully retained case studies over official, hash-checked sources.

• Added exact-chunk retrieval matching to evaluate, compare, and the GitHub Action.
• Made evaluation, comparison, generation, optional backends, and regression diffs fail closed on incomplete or incompatible data.
• Corrected NDCG normalization and strengthened corpus, HTTP adapter, JSON, and output safety.
• Added opt-in provider token accounting through PROOFRAG_USAGE_LOG without storing prompts or answers.
• Added audited Python, HTTP RFC, and OWASP studies with golden sets, predictions, scorecards, blind comparisons, cost records, failure analysis, and reproduction steps.

What the new studies found

• Python concurrency: SQLite FTS5 won 13 blind comparisons to token overlap's 6, with 11 ties; the largest exact-match difference was on multi-document retrieval.
• HTTP RFC metadata: section-enriched BM25 improved exact NDCG@5 by 0.018, below the predeclared 0.05 materiality threshold; metadata won 4 of 5 decided comparisons.
• OWASP context depth: top-6 improved exact Recall@6 by 0.024 but reduced judged answer quality; top-3 won 7–5, with 12 ties.

The two new studies used 270 API calls and an estimated $0.1604 at the providers' published list prices. Their reports retain negative results and judge disagreements, including a manually identified false-negative refusal score.

See the changelog for the complete change list and the case-study index for reports and reproduction workflows.

The retired floating v0 Action tag is not recreated; workflows should pin unshDee/proofrag@v0.8.0.

v0.7.0

Summary

Corpus ingestion release.

Added

• Added proofrag corpus to inspect corpus loading before generation.
• Added source, chunk, character, average chunk, and extension counts.
• Added default skips for noisy directories like .git, .venv, node_modules, dist, build, and caches.
• Added .gitignore support.
• Added --include, --exclude, and --no-gitignore filters to corpus and generate.
• Added HTML text extraction by default.
• Added optional PDF loading through the proofrag[pdf] extra.
• Added context_metadata to generated golden sets so source path, chunk id, chunk index, character count, and extension are preserved.

Verification

• PR checks passed on Python 3.11, 3.12, and 3.13.

v0.6.0

Summary

Backend expansion release.

Added

• Added Ragas scoring backend through evaluate --backend ragas and the proofrag[ragas] extra.
• Added Ragas faithfulness and factual correctness scoring using the configured LLM provider.
• Added Ragas answer relevancy when OpenAI-compatible embeddings are configured.
• Updated DeepEval support to deepeval 4.0.6.
• Preserved DeepEval metric reasons in record rationales for scorecard notes.
• Added golden set validation with schema checks, duplicate detection, source coverage, strict mode, JSON reports, and stable fingerprints.
• Added proofrag run prediction adapters for HTTP endpoints and Python callables.
• Added CI markdown summaries and automatic scorecard/results artifacts.

Changed

• Updated plugin metadata to 0.6.0.
• Updated changelog comparison links for recent releases.

Verification

• PR checks passed on Python 3.11, 3.12, and 3.13.

proofrag v0.5.2

Changed

• Redesigned the HTML scorecard and blind A/B comparison reports with a clean, professional light theme. Adopts shadcn-style design tokens (neutral palette, thin borders, soft shadows, tabular numerals) and uses color only to flag weak scores instead of saturating every metric. Still zero-dependency, self-contained static HTML — no JS, no external assets — so it stays drop-in for PRs and CI artifacts.

Full Changelog: v0.5.1...v0.5.2

proofrag v0.5.1

proofrag v0.5.1

Provider completeness + a leaner package.

• The whole pipeline (generate / judge / compare / --semantic / DeepEval backend) now works across Anthropic, OpenAI, and OpenAI-compatible / local endpoints.
• OPENAI_BASE_URL works without an API key — point it at Ollama, vLLM, or LM Studio. Provider auto-detect picks OpenAI when only a base URL is set.
• DeepEval backend honors OPENAI_BASE_URL via GPTModel(base_url=...).
• sdist trimmed ~982 KB → ~139 KB (README demo media no longer shipped).

Verified end-to-end over an OpenAI-compatible endpoint (no key) and on Anthropic.

proofrag v0.5.0

proofrag v0.5.0

Pluggable scoring backends.

• evaluate --backend deepeval (pip install "proofrag[deepeval]", verified against deepeval 4.0.5) — scores generation with FaithfulnessMetric / AnswerRelevancyMetric / GEval correctness, using proofrag's own model config (Anthropic → AnthropicModel, OpenAI → GPTModel).
• Retrieval metrics, the HTML scorecard, diff, and compare are unchanged across backends; the scorecard renders each backend's metric set dynamically.
• Ragas backend planned next.

Install: pipx install "proofrag[anthropic]" · uvx "proofrag[anthropic]" demo

proofrag v0.4.0

proofrag v0.4.0

Blind A/B comparison of two RAG variants.

• proofrag compare — run two variants (vector vs GraphRAG, two prompts, two models) over the same golden set; the same pinned judge picks the better answer per question blind (answers shown in randomized order, position bias shuffled out).
• Reports win counts + per-variant retrieval metrics, plus a self-contained HTML comparison report. proofrag report auto-detects the format; proofrag demo --compare renders a sample with no API key.

Install: pipx install "proofrag[anthropic]" · uvx "proofrag[anthropic]" demo

proofrag v0.3.0

proofrag v0.3.0

Point your agent at your docs + your RAG app → golden test set, LLM-as-judge + retrieval scorecard, CI gate. Agent Skill (Claude Code / Codex) + Python CLI.

• Golden-set generation from your own corpus (single/multi-doc/unanswerable tiers)
• LLM-as-judge (groundedness, correctness, completeness, citation quality), pinned + fingerprinted
• Rank-aware retrieval metrics: Recall@k, Precision@k, NDCG@k, MRR
• Shareable self-contained HTML scorecard
• CI gates: absolute floor (--fail-under) + baseline regression (proofrag diff)
• Reusable GitHub Action: uses: unshDee/proofrag@v0

Install: pipx install "proofrag[anthropic]" · uvx "proofrag[anthropic]" demo