Repository navigation
Releases
v0.2.3
v0.2.3 — Evaluator Bug Fixes
Fixes the last two open bugs from the PearMedica Cycle 4 audit.
Bug Fixes
-
fix(evaluator): RAGAS 0.4.x score extraction (C4-3) — Rewrites
_extract_ragas_score()to handle the RAGAS 0.4.xEvaluationResult.scoresshape (list ofMetricResultobjects with.value). Adds.to_pandas()fallback. Preserves legacy 0.2.x/0.3.x extraction paths. Three consecutive cycles of RAGAS failure — this fixes the final layer (result parsing). -
fix(evaluator): sample_id propagation (C4-4) — The JSONL input loader now extracts case identifiers with priority chain
case_id > sample_id > id > sequential fallback. Reports no longer showsample_id: "(unknown)".
Packages
All 6 packages bumped in lockstep:
| Package | Version |
|---|---|
@rag-forge/cli |
0.2.3 |
@rag-forge/mcp |
0.2.3 |
@rag-forge/shared |
0.2.3 |
rag-forge-core |
0.2.3 |
rag-forge-evaluator |
0.2.3 |
rag-forge-observability |
0.2.3 |
RAGAS Failure Timeline (resolved)
| Cycle | Version | Failure | Status |
|---|---|---|---|
| 2 | 0.1.3 | OpenAIEmbeddings.embed_query missing + silent 0.0 coercion |
Fixed in 0.2.0 |
| 3 | 0.2.1 | RagForgeRagasLLM.generate() missing |
Fixed in 0.2.2 |
| 4 | 0.2.2 | Score extraction ValueError — return shape mismatch |
Fixed in 0.2.3 |
Validation
- 296 tests pass, ruff clean, mypy clean, TypeScript build/lint/typecheck clean
- Validate with PearMedica Cycle 5:
rag-forge-eval audit --evaluator ragasshould produce real scores for the first time
v0.2.2 — Honest-Measurement Repair Release
0.2.2 — Honest-Measurement Repair Release (2026-04-15)
Cycle 3 of the PearMedica clinical audit (2026-04-15) caught five gaps in v0.2.1. This release closes every one of them. No new features.
Fixed
- RAGAS engine crashed on every run (Cycle 3 Finding C3-2, HIGH).
RagForgeRagasLLMwas shipped without the.generate()/is_finished()/get_temperature()/set_run_config()methods that ragas 0.4.x'sBaseRagasLLMexposes. Every metric job crashed withAttributeError: 'RagForgeRagasLLM' object has no attribute 'generate'in roughly 6 seconds without making a single network call. v0.2.2 re-declares every public method on bothBaseRagasLLMandBaseRagasEmbeddingsas duck-typed shims, and a new contract test (tests/test_ragas_adapters_contract.py) iterates the real ragas base classes and asserts the wrappers match them — so the next ragas release that grows a new method fails fast in CI, not in a user audit. - Skipped counter silently wrong (Cycle 3 residual of Cycle 2 Finding #6, MEDIUM).
EvaluationResult.skipped_evaluationswas never populated by the RAGAS engine, so the report's top-level "Skipped: N" line read0even when every job had crashed. Individual-metric extraction failures also produced one aggregateSkipRecordper metric name instead of fanning out per (sample, metric) pair — a 12-sample x 4-metric run produced 4 skip records instead of 48, under-reporting the blast radius by 12x. v0.2.2 fans out skips across every affected sample and keeps the integer counter in lockstep with the detail list. Reason strings are truncated to 400 chars with a trailing ellipsis so long Python tracebacks don't blow up HTML/PDF rendering. __version__constant stale (Cycle 3 Finding C3-5, LOW).pip show rag-forge-evaluatorcorrectly reported0.2.1butimport rag_forge_evaluator; rag_forge_evaluator.__version__returned"0.1.0". Users programmatically tagging their audit reports with the tool version were recording wrong versions in their own records. v0.2.2 bumps the constant, adds the missing__version__torag_forge_observability, and introducestests/test_version_drift.py— a parametrized CI test that reads each package'spyproject.tomland__init__.pyand asserts they stay in lockstep. Any future release bumping one half but forgetting the other fails in CI.
Measurement rubric changes (read if upgrading from v0.1.x)
Context Relevance scores are systematically higher under v0.2.x than v0.1.x — by ~0.25 on average against identical telemetry — and this is a rubric change, not product improvement.
Cycle 3 observed every case in the 12-sample PearMedica golden set scoring higher on Context Relevance under v0.2.1's llm-judge than it did under v0.1.3's, with deltas from +0.05 to +0.48. The production retrieval code was unchanged between the two runs; only the measurement framework moved.
Root cause (mechanistic): v0.1.3 scored Context Relevance by per-chunk averaging. Its judge prompt (in packages/evaluator/src/rag_forge_evaluator/metrics/context_relevance.py) instructed the judge to rate each retrieved chunk individually on a 1-5 scale and return mean_score = mean(ratings) / 5:
Rate each context chunk's relevance to the query on a 1-5 scale.
Respond with ... "ratings": [{"chunk_index": 0, "score": 4, ...}], "mean_score": 0.8}
The mean_score should be the average rating divided by 5 (normalized to 0.0-1.0).
v0.2.0's new combined llm_judge prompt (in packages/evaluator/src/rag_forge_evaluator/metrics/llm_judge.py, introduced in commit 330465f) switched to a holistic 0-1 score across the whole context:
- context_relevance (0-1): how relevant the retrieved context is to the query.
The two methodologies are not equivalent:
- Per-chunk averaging penalises the presence of irrelevant chunks. If 5 chunks are retrieved and 2 are relevant, the average is roughly
(5+5+1+1+1)/5/5 = 0.52. - Holistic scoring typically rewards the presence of at least some relevant information. Given the same 5 chunks, a judge will often return 0.80+ because "the context does contain relevant information."
Holistic scoring is systematically more lenient than averaged per-chunk scoring when retrieval surfaces a mix of relevant and irrelevant chunks — which is the normal case for hybrid retrieval pipelines. That is the mechanism behind the +0.25 drift.
What this means for upgrading from v0.1.x:
- Do not read a Context Relevance jump as a product improvement when upgrading. Your retrieval pipeline has not gotten better; the measurement got more lenient.
- Re-baseline your CI gate thresholds after the first v0.2.x run against a known-good pipeline. If your v0.1.x gate was
context_relevance >= 0.60, you may need>= 0.80or higher under v0.2.x to catch the same drift magnitude. - The other three metrics (faithfulness, answer_relevance, hallucination) also moved to the combined-pass prompt in v0.2.0, but their deltas against Cycle 2's telemetry were within LLM-judge variance (+0.02 to +0.06). Treat them as lightly noisy, not as rubric-changed, unless you see large per-case jumps.
- Refusal-aware scoring is default-on in v0.2.0+ and adds a separate
safety_refusalrubric with explicitly softer context-relevance criteria. That path only fires when the judge classifies a sample as a refusal; in the Cycle 3 run, 0 samples were classified as refusals, so the drift above is entirely on thestandardrubric — not a side effect of refusal mode.
A future --rubric=strict-v1 flag that pins the judge prompt to the v0.1.x per-chunk-averaging shape is under design for a later release. Not in v0.2.2. Open a GitHub issue if you need it before then.
Verified fixed in this release
- Cycle 2 Finding #4 (
OpenAIEmbeddings.embed_queryAttributeError): contract test intests/test_ragas_adapters_contract.pyassertsembed_query+aembed_query+embed_documents+aembed_documents+embed_text+embed_texts+set_run_configare all declared onRagForgeRagasEmbeddings. End-to-end smoke test intests/test_ragas_adapters_e2e.pyrunsragas.evaluate()against the wrapper and asserts noAttributeErrornaming our classes. - Cycle 2 Finding #5 (
max_tokensoverflow on long structured clinical responses): v0.2.0 raisedClaudeJudgedefault from 1024 -> 4096 tokens.tests/test_cycle2_regression.py::test_cycle2_fixture_handles_long_structured_responsesasserts the absence ofInstructorRetryException/finish_reason='length'signatures in skip records against a 3-sample long-response fixture. - Cycle 2 Finding #6 (silent 0.0 coercion): two-part fix. v0.2.0 replaced the 0.0 fallback with
SkipRecordtracking. v0.2.2 (this release) closes the residual gap where theskipped_evaluationsinteger counter was not kept in lockstep with the skip detail list — see Fixed above.
v0.2.1 — v0.2.0 partial-publish recovery
v0.2.1 — v0.2.0 partial-publish recovery
Release date: 2026-04-14
Scope: One-line packaging fix to rag-forge-evaluator; all six packages bumped 0.2.0 → 0.2.1 for monorepo version alignment. No feature changes. Users upgrading from v0.1.3 should read the v0.2.0 release notes — that's where the real changes are.
Why this release exists
The v0.2.0 publish workflow got partway through before PyPI rejected the rag-forge-evaluator wheel with:
400 Invalid distribution file. ZIP archive not accepted:
Duplicate filename in local headers.
PyPI's 2025+ strict validator rejects wheels whose internal ZIP archive contains duplicate filenames. Our wheel had the entire rag_forge_evaluator/report/templates/ directory packaged twice — once via [tool.hatch.build.targets.wheel] packages = ["src/rag_forge_evaluator"] (which already recurses into every subdirectory) and a second time via a redundant [tool.hatch.build.targets.wheel.force-include] entry that was added during the v0.2.0 template work as belt-and-braces insurance.
Partial-publish state after v0.2.0:
| Package | State on v0.2.0 |
|---|---|
@rag-forge/shared (npm) |
✅ Published |
@rag-forge/mcp (npm) |
✅ Published |
@rag-forge/cli (npm) |
✅ Published |
rag-forge-core (PyPI) |
✅ Published |
rag-forge-evaluator (PyPI) |
❌ Rejected (duplicate-filename error) |
rag-forge-observability (PyPI) |
⏭️ Skipped (blocked by evaluator failure) |
What's fixed
packages/evaluator/pyproject.toml— remove the redundant[tool.hatch.build.targets.wheel.force-include]block. Thepackagesentry already covers the templates directory. Verified locally: the rebuilt wheel has 41 entries, 0 duplicates, and the Jinja template lives exactly once atrag_forge_evaluator/report/templates/audit.html.j2..github/workflows/publish.yml— addskip-existing: trueto all threepypa/gh-action-pypi-publishsteps. This lets partial-publish recoveries (like this one) re-run cleanly:rag-forge-corewas already uploaded as0.2.0→ the0.2.1upload is fresh, but if a future re-run ever needs to re-publish something that already exists, it'll skip instead of hard-failing the whole workflow.
Version skew warning
Because the v0.2.0 workflow partially completed, PyPI and npm saw different final states on v0.2.0:
- npm: all three packages (
shared,mcp,cli) published at0.2.0 - PyPI: only
rag-forge-corepublished at0.2.0;rag-forge-evaluatorandrag-forge-observabilitystayed at0.1.3
v0.2.1 restores alignment: every package is now available at 0.2.1 on both registries. New installs should use 0.2.1.
Upgrade instructions
# Python
pip install -U 'rag-forge-evaluator[ragas]==0.2.1'
# npm
npm install -g @rag-forge/cli@0.2.1No code changes, no CLI flag changes, no configuration changes between 0.2.0 and 0.2.1. This is purely a packaging fix.
Credits
Caught by PyPI's strict-ZIP validator during the v0.2.0 release. The local wheel built cleanly on uv build, so the duplicate-filename issue would not have been visible without the publish attempt — pip install tolerates the duplicate silently. Worth watching going forward: add a check-wheel-contents step to CI so we catch this class of issue before release.
v0.2.0 — Evaluator Refresh
⚠️ Partial publish — install v0.2.1 insteadThis release shipped to npm successfully (
@rag-forge/cli,@rag-forge/mcp,@rag-forge/sharedare all available at 0.2.0) butrag-forge-evaluatorwas rejected by PyPI with a duplicate-filename error, which also blockedrag-forge-observability. Onlyrag-forge-coreis available on PyPI at 0.2.0.Upgrade to v0.2.1 instead — it contains the exact same evaluator features plus the packaging fix, and all six packages are available on both registries.
$(cat docs/release-notes/v0.2.0.md)
v0.1.3
v0.1.3 — Audit resilience
Release date: 2026-04-13
Scope: rag-forge-evaluator (Python). Other packages bumped in lockstep for monorepo version alignment.
Why this release exists
The PearMedica cycle-2 validation audit on v0.1.2 surfaced two operational gaps that don't show up until a real long-running audit meets a real Anthropic capacity event:
- Mid-loop crashes vaporized all progress. When the audit loop crashed on sample 5 of 19 (or 9 of 19, depending on the run), there was no partial report file on disk. Every sample scored before the crash was lost from the report writer's perspective — the only reason we have any cycle-2 data at all is that the user reconstructed it from the terminal progress stream.
- The 529 retry budget was too small and silent. v0.1.2's retry wrapper used a hardcoded ~62-second total backoff window. That's enough for transient flaps but inadequate for sustained overload events, which can last 5–15 minutes. There was also no way to see when a retry was in flight; the audit just looked frozen.
v0.1.3 fixes both.
What's new
1. Partial reports on mid-loop abort (resolves #24)
Any mid-loop exit — KeyboardInterrupt, unhandled exception, or retry-budget exhaustion — now writes audit-report.partial.json before propagating. The file contains:
partial: truemarkercompleted_samples: N/total_samples: Maborted_reason:"keyboard_interrupt"|"retry_budget_exhausted"|"unhandled_exception"error_message: the triggering exception- Full per-sample scored results in the same shape as a complete report
metrics: null,rmm_level: null,rmm_name: null,overall_score: nullat the top level
Subset aggregates are intentionally namespaced under partial_metrics.by_metric with an in-band caveat:
"partial_metrics": {
"note": "Subset aggregates over 9/19 samples. NOT comparable to a full-run report. Not thresholded against RMM. Do not screenshot these numbers without this caveat.",
"by_metric": {
"faithfulness": {"score": 0.50, "scored_count": 9, "skipped_count": 0},
...
}
}RMM level is unconditionally omitted from partial reports. Computing it over a subset is wrong (threshold claims apply to the whole pipeline), not partial.
Exit codes:
0— audit completed successfully1— hard failure (unrecoverable error, no report)2— usage / config error (argparse default)3— partial audit;audit-report.partial.jsonwritten
CI scripts can now distinguish partial from hard failure:
rag-forge-eval audit ...
code=$?
if [ $code -eq 3 ]; then
echo "Partial audit — inspect $OUTDIR/audit-report.partial.json"
fi2. Configurable 529 retry budget + visibility (resolves #25)
New environment variable: RAG_FORGE_JUDGE_OVERLOAD_BUDGET_SECONDS (default: 300).
- Retries on 529 fire until either
max_retriesattempts are exhausted or cumulative elapsed time + next backoff would exceed the budget — whichever trips first. - Also exposed as a constructor argument on
ClaudeJudge:overload_budget_seconds=300.0.
Retry notices stream to stderr in real time, so a long-running audit no longer looks frozen during a capacity event:
[judge: 529 overload, retry 1, elapsed 0s / 300s budget]
[judge: 529 overload, retry 2, elapsed 2s / 300s budget]
[judge: 529 overload, retry 3, elapsed 6s / 300s budget]
On budget exhaustion, the CLI emits a structured fallback hint before writing the partial report:
Anthropic was in a sustained 529 overload event. Options:
(a) wait 10-15 minutes and re-run
(b) switch judges with --judge openai
(c) increase the budget: RAG_FORGE_JUDGE_OVERLOAD_BUDGET_SECONDS=600
The retry wrapper raises OverloadBudgetExhaustedError (new, exported from rag_forge_evaluator.judge.claude_judge) which wraps the underlying APIStatusError(529) on .original. Callers that want to branch on the distinction can catch the new type.
Implementation notes
ClaudeJudgegains anon_retrycallback. Kept off theProgressReporterprotocol on purpose: the judge has multiple call sites (tests, programmatic use, MCP server) and shouldn't unilaterally decide stderr is the right output channel. The CLI wires in a stderr writer; other callers get silence by default.- Partial state lives on
LLMJudgeEvaluator. The accumulator lists are promoted from local variables to instance attributes so the orchestrator can reach in after an exception. RAGAS-backed audits don't get partial reports in v0.1.3 — RAGAS batches internally and has no mid-loop hook. PartialAuditErroris the bridge between the orchestrator and the CLI. It wraps the original exception, carries the path to the written partial report, and is what the CLI catches to emit exit code 3.
Upgrade path
uv pip install --upgrade \
rag-forge-evaluator==0.1.3 \
rag-forge-observability==0.1.3No breaking changes for existing consumers:
ClaudeJudgeconstructor gains optionaloverload_budget_secondsandon_retry— both default to None and preserve v0.1.2 behavior minus the crash-on-529 bug.AuditConfiggains optionalon_judge_retry— defaults to None.- The new
OverloadBudgetExhaustedErrorinherits fromRuntimeError; any caller that previously caughtExceptionorRuntimeErrorstill catches it. - Exit code 3 is new. If you had a CI script that treated any non-zero exit as a hard failure, it will now treat partials as failures too — which was already the old behavior. Upgrade your script to branch on
3when you're ready to surface partial artifacts.
Tests
tests/test_claude_judge_529_retry.py— extended with budget-exhausts-before-max-retries, env var override, on_retry callback assertion. 7 tests, all passing.tests/test_partial_report.py— new. Covers: partial JSON on mid-loop exception, PartialAuditError wrapping, KeyboardInterrupt path, and the zero-samples-scored case where no partial report is written and the original exception propagates unchanged. 4 tests, all passing.- Full evaluator suite: 202 passing. Full monorepo suite: 527 passing.
What's not in this release
- HTML rendering for partial reports. v0.1.3 writes partial JSON only. HTML can come later if anyone asks.
- Resume-from-partial. Out of scope — partial reports are handoff artifacts, not resumable checkpoints.
- RAGAS partial reports. RAGAS doesn't expose a mid-loop hook. An LLM-judge run is the only engine that gets the resilience fix in v0.1.3.
- Partial HTML + RMM-level workarounds — intentionally omitted to avoid misleading screenshots.
Cycle-2 follow-up
With v0.1.3 installed, the PearMedica cycle-2 validation audit is fully unblocked:
- If Anthropic stays healthy, the audit runs to completion and writes the standard report.
- If Anthropic overloads again, retries absorb up to 5 minutes of transient 529s with visible stderr notices.
- If the overload outlasts the budget, the audit exits with code 3 and a partial JSON with everything scored so far — no more lost progress.
Re-run the command from the cycle-2 runbook. Everything from the midway point onward just works.
v0.1.2
v0.1.2 — 529 Overloaded retry hotfix
Release date: 2026-04-13
Scope: rag-forge-evaluator (Python). Other packages bumped in lockstep for monorepo version alignment.
Why this release exists
During the PearMedica cycle-2 validation audit on v0.1.1, the run crashed on the first faithfulness judge call with:
anthropic._exceptions.OverloadedError: Error code: 529 - 'Overloaded'
v0.1.1 replaced the cycle-1 manual retry loop with the Anthropic SDK's built-in retry (Anthropic(max_retries=5)), assuming it covered the full set of transient error codes. It does not. The SDK's retry set is 408, 429, 500, 502, 503, 504 — 529 Overloaded is Anthropic-specific and falls outside that set, so it propagates as APIStatusError and crashes long-running audits during any Anthropic capacity event.
Two back-to-back PearMedica cycle-2 runs crashed on 529 within the first 1–2 samples, confirming the issue is deterministic during capacity pressure and blocks all --judge claude audits until it clears.
The fix
packages/evaluator/src/rag_forge_evaluator/judge/claude_judge.py now wraps messages.create() in an explicit 529 retry loop layered on top of the SDK's built-in retry:
- Catches
APIStatusErrorwherestatus_code == 529. - Retries up to
max_retriestimes (default 5, overridable viaRAG_FORGE_JUDGE_MAX_RETRIES). - Exponential backoff: 2s, 4s, 8s, 16s, 32s.
- All other
APIStatusErrorcodes propagate immediately — non-529 errors remain the SDK's responsibility.
This mirrors the cycle-1 manual patch that had proven the approach in production before v0.1.1 incorrectly assumed the SDK covered 529.
Tests
New test file tests/test_claude_judge_529_retry.py:
- Retry then succeed — 529 on first call, success on second, verifies the judge returns the successful payload and called
time.sleeponce. - Exhaust retries — 529 on every call, verifies
APIStatusErroris re-raised aftermax_retries + 1attempts. - Non-529 errors bypass retry — 400 Bad Request propagates immediately with zero sleeps.
All 23 judge-related tests pass.
Upgrade path
uv pip install --upgrade rag-forge-evaluator==0.1.2 rag-forge-observability==0.1.2No code or config changes required. If you were on v0.1.1 and hit a 529 crash, v0.1.2 resolves it with no further action.
Package versions
All 6 packages bumped to 0.1.2 for monorepo alignment, even though the only functional change is in rag-forge-evaluator:
@rag-forge/cli@0.1.2@rag-forge/mcp@0.1.2@rag-forge/shared@0.1.2rag-forge-core==0.1.2rag-forge-evaluator==0.1.2← the fix is hererag-forge-observability==0.1.2
What's not in this release
- The RAGAS engine still does not honor
--judge claude— deferred to a future release. - No changes to the marketing site, docs, CLI, or MCP server.
Cycle-2 follow-up
With v0.1.2 installed, the PearMedica cycle-2 validation audit is unblocked. Re-run the command from the cycle-2 runbook; the 529 retry wrapper will absorb transient Anthropic capacity events and the audit will either complete or fail with a fundamentally different error (in which case we investigate fresh).
v0.1.1
RAG-Forge v0.1.1 — Post-Audit Hardening Release
Release date: 2026-04-13
Branch: fix/v011-audit-bugs
Trigger: First real-world audit run against PearMedica's clinical RAG pipeline surfaced 12 distinct issues across correctness, packaging, and UX. v0.1.1 fixes all of them.
TL;DR — Why you should upgrade immediately
If you ran rag-forge audit against any production pipeline on v0.1.0, your scorecard is mathematically wrong in a direction that makes your pipeline look ~3-4× worse than reality. v0.1.1 fixes the silent parse-failure-to-zero coercion that polluted aggregate metrics, plus a hardcoded judge token budget that truncated mid-array on long structured responses (clinical, legal, financial). Re-running the same audit against v0.1.1 will give you trustworthy numbers for the first time.
Critical correctness fix
Aggregate scores no longer count parse failures as zero
Bug: Every metric wrapped json.loads() in a try/except that returned score=0.0 on any failure. The LLMJudgeEvaluator then averaged across ALL samples including the failed ones, dragging real averages down by up to 4×.
Real-world impact: PearMedica's reported context_relevance was 0.063 (15/19 samples parse-failed). Real average over the 4 successfully-scored samples was ~0.30. The reported answer_relevance of 0.768 was actually 0.913 once forced zeros were excluded — moving it from "fails threshold" to "passes threshold."
Fix: New shared parse_judge_json helper handles empty strings, code fences, leading prose, trailing prose, and truncated output uniformly. All four metrics emit MetricResult(skipped=True) on parse failure. LLMJudgeEvaluator aggregates over scored_count only and reports skipped_count separately on each metric.
Missing-field scores no longer default to worst-case
Bug: When the judge returned valid JSON but the expected field was absent, hallucination.py defaulted hallucination_rate=1.0 ("100% hallucinated") and faithfulness.py defaulted score=0.0. A judge that returned a refusal or a partial response was recorded as proof of total failure.
Fix: Missing fields now mark the result as skipped=True rather than scoring worst-case.
Generalization fixes (the "different customer" lens)
Configurable judge max_tokens — default bumped 1024 → 4096
Bug: max_tokens=1024 was structurally too small for faithfulness/hallucination metrics, which ask the judge to enumerate every factual claim. On long structured responses (~30-50 claims), the judge truncated mid-array.
Fix: ClaudeJudge and OpenAIJudge now accept max_tokens via constructor argument, RAG_FORGE_JUDGE_MAX_TOKENS env var, or a 4096 default. Cost difference for unused budget is zero — providers only charge for tokens actually generated.
Configurable judge max_retries — default 5 with SDK exponential backoff
Bug: Anthropic SDK default is 2 retries. A single 529 Overloaded killed the entire 19-sample audit mid-run after $0.40 of judge calls.
Fix: Anthropic(max_retries=5) + OpenAI(max_retries=5). Configurable via constructor arg or RAG_FORGE_JUDGE_MAX_RETRIES env var.
--judge-model CLI flag
Bug: The CLI --judge flag only accepted mock | claude | openai. A law firm wanting Claude Opus, or a startup wanting gpt-4o-mini, had no way to override.
Fix: New --judge-model flag passes through any provider-specific model id (claude-opus-4-6, gpt-4-turbo, claude-haiku-4-5, etc.). Falls back to RAG_FORGE_JUDGE_MODEL env var, then the provider default.
--evaluator ragas rejects non-OpenAI judges loudly
Bug: The RAGAS engine ignores --judge and uses its own internal OpenAI judge. Running --evaluator ragas --judge claude silently swapped models and failed mid-run with an opaque auth error if OPENAI_API_KEY wasn't set.
Fix: AuditOrchestrator.__init__ validates the combo at construction time and raises ConfigurationError with concrete remediation steps. Mock and OpenAI judges are still allowed.
RAGAS defensive result access + version pin
Bug: 'EvaluationResult' object has no attribute 'get' on ragas ≥ 0.4 because the result type changed shape between major versions.
Fix: Pin to ragas>=0.2.10,<0.3 for v0.1.1. New _extract_ragas_score helper falls through dict access → __getitem__ → attribute access → numeric cast so future upgrades degrade gracefully.
UX fixes
Pre-run banner with cost + duration estimate, plus per-sample progress streaming
Bug: A 5-15 minute audit gave zero feedback. Users had no idea whether the run was hung, how long it would take, or how much it would cost.
Fix:
- New
ProgressReporterprotocol withStderrProgressReporterdefault - Pre-run banner: sample count, judge model, evaluator engine, judge call total, estimated minutes range, estimated USD cost
- Confirmation prompt before any judge calls (
--yes/-yto bypass; required for non-interactive runs) - Per-sample stderr line streaming
faith=… ctx=… ans=… hall=… OK|WARN N skipped (elapsed) - End-of-run summary with elapsed time, scored/skipped totals, RMM level, report path
- Mock-judge runs (cost = $0) skip the prompt automatically so existing test suites are unaffected
Stdout no longer block-buffers on Windows non-TTY
Bug: python -m rag_forge_evaluator.cli audit ... from a non-TTY shell sat at 0 bytes for 5-10 minutes, then dumped everything at once. This is Python's default block-buffering when stdout isn't a terminal.
Fix: Both subprocess paths covered:
- Node bridge passes
-uto the python interpreter - Python CLI calls
sys.stdout.reconfigure(line_buffering=True)at startup
--pdf checks Playwright at audit start, not end
Bug: PearMedica audit ran $0.40 of judge calls then crashed at the very end with "Playwright not installed" because the [pdf] extra was undocumented.
Fix: New is_available() helper in report/pdf.py checks both the playwright import AND the Chromium binary. AuditOrchestrator.run() calls it as step 0 when --pdf is set, raises ConfigurationError with the exact install command if either check fails.
Packaging fixes
@rag-forge/cli is actually installable from npm
Bug: v0.1.0 was published with "@rag-forge/shared": "workspace:*" literal, making it uninstallable anywhere outside the source monorepo. Same bug affected @rag-forge/mcp.
Root cause: The publish workflow used npm publish, which does not understand the pnpm workspace protocol and ships package.json verbatim.
Fix: Switched to pnpm publish --no-git-checks for all 3 npm packages. pnpm publish rewrites workspace:* to the actual version range before uploading. Provenance attestations still emitted via --provenance.
uv tool install rag-forge-evaluator works
Bug: Failed with "no executables are provided" because the package had no [project.scripts] entry.
Fix: Added rag-forge-eval = "rag_forge_evaluator.cli:main" console script.
rag-forge-observability declared as a runtime dependency
Bug: cli.py imported rag_forge_observability.tracing at module load but the package was not declared. Installing from PyPI then running the CLI crashed immediately.
Fix: Added rag-forge-observability>=0.1.1 to dependencies.
Documentation
JudgeProvider protocol prominently documented
The protocol is two methods (judge, model_name) and ~20 lines of code to plug in any provider. The evaluator README now includes a working GeminiJudge example, the protocol contract, wiring instructions, and three implementation hints (max_tokens budget, retry config, raw response handling). First-party Gemini, Bedrock, and Ollama judges are still tracked for v0.1.2.
Deferred to v0.1.2
- Truncation-detection retry loop. When
finish_reason='length', retry with doubledmax_tokens. Requires upgrading theJudgeProviderinterface to return(text, finish_reason)instead of justtext. - "Retrieval bypass" root-cause classification. When
context_relevance < 0.3 AND faithfulness < 0.3 AND answer_relevance > 0.8, the diagnosis is "LLM is ignoring retrieval and answering from training data" — not "generator hallucination." This is the actual RMM-0 pathology PearMedica exhibited. v0.1.2 will surface it as a distinct category in the report. - Per-metric
max_tokens. Faithfulness/hallucination need bigger budgets than answer_relevance. v0.1.2 will let eachMetricEvaluatordeclare its own token requirement. --judge claudepropagation through RAGAS. Needs aLangchainLLMWrapperaroundClaudeJudge. Architecturally invasive — better in v0.1.2.- First-party Gemini, Bedrock, and Ollama judges alongside Claude and OpenAI.
Test count
v0.1.0: 136 tests · v0.1.1: 177 tests (+41 covering the parser, aggregation, judge config, audit validation, and ragas extractor)
Acknowledgments
This release exists because of a single forensic audit run against PearMedica's clinical RAG pipeline on 2026-04-13. Eight of the twelve issues fixed here would have shipped to every customer otherwise.
v0.1.0 — Initial release
Initial release of RAG-Forge: framework-agnostic CLI toolkit for production-grade RAG pipelines with evaluation baked in.