Skip to content

v0.1.1

Choose a tag to compare

@hallengray hallengray released this 13 Apr 15:46
· 33 commits to main since this release

RAG-Forge v0.1.1 — Post-Audit Hardening Release

Release date: 2026-04-13
Branch: fix/v011-audit-bugs
Trigger: First real-world audit run against PearMedica's clinical RAG pipeline surfaced 12 distinct issues across correctness, packaging, and UX. v0.1.1 fixes all of them.

TL;DR — Why you should upgrade immediately

If you ran rag-forge audit against any production pipeline on v0.1.0, your scorecard is mathematically wrong in a direction that makes your pipeline look ~3-4× worse than reality. v0.1.1 fixes the silent parse-failure-to-zero coercion that polluted aggregate metrics, plus a hardcoded judge token budget that truncated mid-array on long structured responses (clinical, legal, financial). Re-running the same audit against v0.1.1 will give you trustworthy numbers for the first time.

Critical correctness fix

Aggregate scores no longer count parse failures as zero

Bug: Every metric wrapped json.loads() in a try/except that returned score=0.0 on any failure. The LLMJudgeEvaluator then averaged across ALL samples including the failed ones, dragging real averages down by up to 4×.
Real-world impact: PearMedica's reported context_relevance was 0.063 (15/19 samples parse-failed). Real average over the 4 successfully-scored samples was ~0.30. The reported answer_relevance of 0.768 was actually 0.913 once forced zeros were excluded — moving it from "fails threshold" to "passes threshold."
Fix: New shared parse_judge_json helper handles empty strings, code fences, leading prose, trailing prose, and truncated output uniformly. All four metrics emit MetricResult(skipped=True) on parse failure. LLMJudgeEvaluator aggregates over scored_count only and reports skipped_count separately on each metric.

Missing-field scores no longer default to worst-case

Bug: When the judge returned valid JSON but the expected field was absent, hallucination.py defaulted hallucination_rate=1.0 ("100% hallucinated") and faithfulness.py defaulted score=0.0. A judge that returned a refusal or a partial response was recorded as proof of total failure.
Fix: Missing fields now mark the result as skipped=True rather than scoring worst-case.

Generalization fixes (the "different customer" lens)

Configurable judge max_tokens — default bumped 1024 → 4096

Bug: max_tokens=1024 was structurally too small for faithfulness/hallucination metrics, which ask the judge to enumerate every factual claim. On long structured responses (~30-50 claims), the judge truncated mid-array.
Fix: ClaudeJudge and OpenAIJudge now accept max_tokens via constructor argument, RAG_FORGE_JUDGE_MAX_TOKENS env var, or a 4096 default. Cost difference for unused budget is zero — providers only charge for tokens actually generated.

Configurable judge max_retries — default 5 with SDK exponential backoff

Bug: Anthropic SDK default is 2 retries. A single 529 Overloaded killed the entire 19-sample audit mid-run after $0.40 of judge calls.
Fix: Anthropic(max_retries=5) + OpenAI(max_retries=5). Configurable via constructor arg or RAG_FORGE_JUDGE_MAX_RETRIES env var.

--judge-model CLI flag

Bug: The CLI --judge flag only accepted mock | claude | openai. A law firm wanting Claude Opus, or a startup wanting gpt-4o-mini, had no way to override.
Fix: New --judge-model flag passes through any provider-specific model id (claude-opus-4-6, gpt-4-turbo, claude-haiku-4-5, etc.). Falls back to RAG_FORGE_JUDGE_MODEL env var, then the provider default.

--evaluator ragas rejects non-OpenAI judges loudly

Bug: The RAGAS engine ignores --judge and uses its own internal OpenAI judge. Running --evaluator ragas --judge claude silently swapped models and failed mid-run with an opaque auth error if OPENAI_API_KEY wasn't set.
Fix: AuditOrchestrator.__init__ validates the combo at construction time and raises ConfigurationError with concrete remediation steps. Mock and OpenAI judges are still allowed.

RAGAS defensive result access + version pin

Bug: 'EvaluationResult' object has no attribute 'get' on ragas ≥ 0.4 because the result type changed shape between major versions.
Fix: Pin to ragas>=0.2.10,<0.3 for v0.1.1. New _extract_ragas_score helper falls through dict access → __getitem__ → attribute access → numeric cast so future upgrades degrade gracefully.

UX fixes

Pre-run banner with cost + duration estimate, plus per-sample progress streaming

Bug: A 5-15 minute audit gave zero feedback. Users had no idea whether the run was hung, how long it would take, or how much it would cost.
Fix:

  • New ProgressReporter protocol with StderrProgressReporter default
  • Pre-run banner: sample count, judge model, evaluator engine, judge call total, estimated minutes range, estimated USD cost
  • Confirmation prompt before any judge calls (--yes/-y to bypass; required for non-interactive runs)
  • Per-sample stderr line streaming faith=… ctx=… ans=… hall=… OK|WARN N skipped (elapsed)
  • End-of-run summary with elapsed time, scored/skipped totals, RMM level, report path
  • Mock-judge runs (cost = $0) skip the prompt automatically so existing test suites are unaffected

Stdout no longer block-buffers on Windows non-TTY

Bug: python -m rag_forge_evaluator.cli audit ... from a non-TTY shell sat at 0 bytes for 5-10 minutes, then dumped everything at once. This is Python's default block-buffering when stdout isn't a terminal.
Fix: Both subprocess paths covered:

  • Node bridge passes -u to the python interpreter
  • Python CLI calls sys.stdout.reconfigure(line_buffering=True) at startup

--pdf checks Playwright at audit start, not end

Bug: PearMedica audit ran $0.40 of judge calls then crashed at the very end with "Playwright not installed" because the [pdf] extra was undocumented.
Fix: New is_available() helper in report/pdf.py checks both the playwright import AND the Chromium binary. AuditOrchestrator.run() calls it as step 0 when --pdf is set, raises ConfigurationError with the exact install command if either check fails.

Packaging fixes

@rag-forge/cli is actually installable from npm

Bug: v0.1.0 was published with "@rag-forge/shared": "workspace:*" literal, making it uninstallable anywhere outside the source monorepo. Same bug affected @rag-forge/mcp.
Root cause: The publish workflow used npm publish, which does not understand the pnpm workspace protocol and ships package.json verbatim.
Fix: Switched to pnpm publish --no-git-checks for all 3 npm packages. pnpm publish rewrites workspace:* to the actual version range before uploading. Provenance attestations still emitted via --provenance.

uv tool install rag-forge-evaluator works

Bug: Failed with "no executables are provided" because the package had no [project.scripts] entry.
Fix: Added rag-forge-eval = "rag_forge_evaluator.cli:main" console script.

rag-forge-observability declared as a runtime dependency

Bug: cli.py imported rag_forge_observability.tracing at module load but the package was not declared. Installing from PyPI then running the CLI crashed immediately.
Fix: Added rag-forge-observability>=0.1.1 to dependencies.

Documentation

JudgeProvider protocol prominently documented

The protocol is two methods (judge, model_name) and ~20 lines of code to plug in any provider. The evaluator README now includes a working GeminiJudge example, the protocol contract, wiring instructions, and three implementation hints (max_tokens budget, retry config, raw response handling). First-party Gemini, Bedrock, and Ollama judges are still tracked for v0.1.2.

Deferred to v0.1.2

  • Truncation-detection retry loop. When finish_reason='length', retry with doubled max_tokens. Requires upgrading the JudgeProvider interface to return (text, finish_reason) instead of just text.
  • "Retrieval bypass" root-cause classification. When context_relevance < 0.3 AND faithfulness < 0.3 AND answer_relevance > 0.8, the diagnosis is "LLM is ignoring retrieval and answering from training data" — not "generator hallucination." This is the actual RMM-0 pathology PearMedica exhibited. v0.1.2 will surface it as a distinct category in the report.
  • Per-metric max_tokens. Faithfulness/hallucination need bigger budgets than answer_relevance. v0.1.2 will let each MetricEvaluator declare its own token requirement.
  • --judge claude propagation through RAGAS. Needs a LangchainLLMWrapper around ClaudeJudge. Architecturally invasive — better in v0.1.2.
  • First-party Gemini, Bedrock, and Ollama judges alongside Claude and OpenAI.

Test count

v0.1.0: 136 tests · v0.1.1: 177 tests (+41 covering the parser, aggregation, judge config, audit validation, and ragas extractor)

Acknowledgments

This release exists because of a single forensic audit run against PearMedica's clinical RAG pipeline on 2026-04-13. Eight of the twelve issues fixed here would have shipped to every customer otherwise.