Repository navigation
v0.1.1
RAG-Forge v0.1.1 — Post-Audit Hardening Release
Release date: 2026-04-13
Branch: fix/v011-audit-bugs
Trigger: First real-world audit run against PearMedica's clinical RAG pipeline surfaced 12 distinct issues across correctness, packaging, and UX. v0.1.1 fixes all of them.
TL;DR — Why you should upgrade immediately
If you ran rag-forge audit against any production pipeline on v0.1.0, your scorecard is mathematically wrong in a direction that makes your pipeline look ~3-4× worse than reality. v0.1.1 fixes the silent parse-failure-to-zero coercion that polluted aggregate metrics, plus a hardcoded judge token budget that truncated mid-array on long structured responses (clinical, legal, financial). Re-running the same audit against v0.1.1 will give you trustworthy numbers for the first time.
Critical correctness fix
Aggregate scores no longer count parse failures as zero
Bug: Every metric wrapped json.loads() in a try/except that returned score=0.0 on any failure. The LLMJudgeEvaluator then averaged across ALL samples including the failed ones, dragging real averages down by up to 4×.
Real-world impact: PearMedica's reported context_relevance was 0.063 (15/19 samples parse-failed). Real average over the 4 successfully-scored samples was ~0.30. The reported answer_relevance of 0.768 was actually 0.913 once forced zeros were excluded — moving it from "fails threshold" to "passes threshold."
Fix: New shared parse_judge_json helper handles empty strings, code fences, leading prose, trailing prose, and truncated output uniformly. All four metrics emit MetricResult(skipped=True) on parse failure. LLMJudgeEvaluator aggregates over scored_count only and reports skipped_count separately on each metric.
Missing-field scores no longer default to worst-case
Bug: When the judge returned valid JSON but the expected field was absent, hallucination.py defaulted hallucination_rate=1.0 ("100% hallucinated") and faithfulness.py defaulted score=0.0. A judge that returned a refusal or a partial response was recorded as proof of total failure.
Fix: Missing fields now mark the result as skipped=True rather than scoring worst-case.
Generalization fixes (the "different customer" lens)
Configurable judge max_tokens — default bumped 1024 → 4096
Bug: max_tokens=1024 was structurally too small for faithfulness/hallucination metrics, which ask the judge to enumerate every factual claim. On long structured responses (~30-50 claims), the judge truncated mid-array.
Fix: ClaudeJudge and OpenAIJudge now accept max_tokens via constructor argument, RAG_FORGE_JUDGE_MAX_TOKENS env var, or a 4096 default. Cost difference for unused budget is zero — providers only charge for tokens actually generated.
Configurable judge max_retries — default 5 with SDK exponential backoff
Bug: Anthropic SDK default is 2 retries. A single 529 Overloaded killed the entire 19-sample audit mid-run after $0.40 of judge calls.
Fix: Anthropic(max_retries=5) + OpenAI(max_retries=5). Configurable via constructor arg or RAG_FORGE_JUDGE_MAX_RETRIES env var.
--judge-model CLI flag
Bug: The CLI --judge flag only accepted mock | claude | openai. A law firm wanting Claude Opus, or a startup wanting gpt-4o-mini, had no way to override.
Fix: New --judge-model flag passes through any provider-specific model id (claude-opus-4-6, gpt-4-turbo, claude-haiku-4-5, etc.). Falls back to RAG_FORGE_JUDGE_MODEL env var, then the provider default.
--evaluator ragas rejects non-OpenAI judges loudly
Bug: The RAGAS engine ignores --judge and uses its own internal OpenAI judge. Running --evaluator ragas --judge claude silently swapped models and failed mid-run with an opaque auth error if OPENAI_API_KEY wasn't set.
Fix: AuditOrchestrator.__init__ validates the combo at construction time and raises ConfigurationError with concrete remediation steps. Mock and OpenAI judges are still allowed.
RAGAS defensive result access + version pin
Bug: 'EvaluationResult' object has no attribute 'get' on ragas ≥ 0.4 because the result type changed shape between major versions.
Fix: Pin to ragas>=0.2.10,<0.3 for v0.1.1. New _extract_ragas_score helper falls through dict access → __getitem__ → attribute access → numeric cast so future upgrades degrade gracefully.
UX fixes
Pre-run banner with cost + duration estimate, plus per-sample progress streaming
Bug: A 5-15 minute audit gave zero feedback. Users had no idea whether the run was hung, how long it would take, or how much it would cost.
Fix:
- New
ProgressReporterprotocol withStderrProgressReporterdefault - Pre-run banner: sample count, judge model, evaluator engine, judge call total, estimated minutes range, estimated USD cost
- Confirmation prompt before any judge calls (
--yes/-yto bypass; required for non-interactive runs) - Per-sample stderr line streaming
faith=… ctx=… ans=… hall=… OK|WARN N skipped (elapsed) - End-of-run summary with elapsed time, scored/skipped totals, RMM level, report path
- Mock-judge runs (cost = $0) skip the prompt automatically so existing test suites are unaffected
Stdout no longer block-buffers on Windows non-TTY
Bug: python -m rag_forge_evaluator.cli audit ... from a non-TTY shell sat at 0 bytes for 5-10 minutes, then dumped everything at once. This is Python's default block-buffering when stdout isn't a terminal.
Fix: Both subprocess paths covered:
- Node bridge passes
-uto the python interpreter - Python CLI calls
sys.stdout.reconfigure(line_buffering=True)at startup
--pdf checks Playwright at audit start, not end
Bug: PearMedica audit ran $0.40 of judge calls then crashed at the very end with "Playwright not installed" because the [pdf] extra was undocumented.
Fix: New is_available() helper in report/pdf.py checks both the playwright import AND the Chromium binary. AuditOrchestrator.run() calls it as step 0 when --pdf is set, raises ConfigurationError with the exact install command if either check fails.
Packaging fixes
@rag-forge/cli is actually installable from npm
Bug: v0.1.0 was published with "@rag-forge/shared": "workspace:*" literal, making it uninstallable anywhere outside the source monorepo. Same bug affected @rag-forge/mcp.
Root cause: The publish workflow used npm publish, which does not understand the pnpm workspace protocol and ships package.json verbatim.
Fix: Switched to pnpm publish --no-git-checks for all 3 npm packages. pnpm publish rewrites workspace:* to the actual version range before uploading. Provenance attestations still emitted via --provenance.
uv tool install rag-forge-evaluator works
Bug: Failed with "no executables are provided" because the package had no [project.scripts] entry.
Fix: Added rag-forge-eval = "rag_forge_evaluator.cli:main" console script.
rag-forge-observability declared as a runtime dependency
Bug: cli.py imported rag_forge_observability.tracing at module load but the package was not declared. Installing from PyPI then running the CLI crashed immediately.
Fix: Added rag-forge-observability>=0.1.1 to dependencies.
Documentation
JudgeProvider protocol prominently documented
The protocol is two methods (judge, model_name) and ~20 lines of code to plug in any provider. The evaluator README now includes a working GeminiJudge example, the protocol contract, wiring instructions, and three implementation hints (max_tokens budget, retry config, raw response handling). First-party Gemini, Bedrock, and Ollama judges are still tracked for v0.1.2.
Deferred to v0.1.2
- Truncation-detection retry loop. When
finish_reason='length', retry with doubledmax_tokens. Requires upgrading theJudgeProviderinterface to return(text, finish_reason)instead of justtext. - "Retrieval bypass" root-cause classification. When
context_relevance < 0.3 AND faithfulness < 0.3 AND answer_relevance > 0.8, the diagnosis is "LLM is ignoring retrieval and answering from training data" — not "generator hallucination." This is the actual RMM-0 pathology PearMedica exhibited. v0.1.2 will surface it as a distinct category in the report. - Per-metric
max_tokens. Faithfulness/hallucination need bigger budgets than answer_relevance. v0.1.2 will let eachMetricEvaluatordeclare its own token requirement. --judge claudepropagation through RAGAS. Needs aLangchainLLMWrapperaroundClaudeJudge. Architecturally invasive — better in v0.1.2.- First-party Gemini, Bedrock, and Ollama judges alongside Claude and OpenAI.
Test count
v0.1.0: 136 tests · v0.1.1: 177 tests (+41 covering the parser, aggregation, judge config, audit validation, and ragas extractor)
Acknowledgments
This release exists because of a single forensic audit run against PearMedica's clinical RAG pipeline on 2026-04-13. Eight of the twelve issues fixed here would have shipped to every customer otherwise.