Skip to content

Releases

v0.2.3

Choose a tag to compare

@hallengray hallengray released this 16 Apr 13:35
4ab9aca

v0.2.3 — Evaluator Bug Fixes

Fixes the last two open bugs from the PearMedica Cycle 4 audit.

Bug Fixes

  • fix(evaluator): RAGAS 0.4.x score extraction (C4-3) — Rewrites _extract_ragas_score() to handle the RAGAS 0.4.x EvaluationResult.scores shape (list of MetricResult objects with .value). Adds .to_pandas() fallback. Preserves legacy 0.2.x/0.3.x extraction paths. Three consecutive cycles of RAGAS failure — this fixes the final layer (result parsing).

  • fix(evaluator): sample_id propagation (C4-4) — The JSONL input loader now extracts case identifiers with priority chain case_id > sample_id > id > sequential fallback. Reports no longer show sample_id: "(unknown)".

Packages

All 6 packages bumped in lockstep:

Package Version
@rag-forge/cli 0.2.3
@rag-forge/mcp 0.2.3
@rag-forge/shared 0.2.3
rag-forge-core 0.2.3
rag-forge-evaluator 0.2.3
rag-forge-observability 0.2.3

RAGAS Failure Timeline (resolved)

Cycle Version Failure Status
2 0.1.3 OpenAIEmbeddings.embed_query missing + silent 0.0 coercion Fixed in 0.2.0
3 0.2.1 RagForgeRagasLLM.generate() missing Fixed in 0.2.2
4 0.2.2 Score extraction ValueError — return shape mismatch Fixed in 0.2.3

Validation

  • 296 tests pass, ruff clean, mypy clean, TypeScript build/lint/typecheck clean
  • Validate with PearMedica Cycle 5: rag-forge-eval audit --evaluator ragas should produce real scores for the first time

v0.2.2 — Honest-Measurement Repair Release

Choose a tag to compare

@hallengray hallengray released this 15 Apr 23:03
8f45df5

0.2.2 — Honest-Measurement Repair Release (2026-04-15)

Cycle 3 of the PearMedica clinical audit (2026-04-15) caught five gaps in v0.2.1. This release closes every one of them. No new features.

Fixed

  • RAGAS engine crashed on every run (Cycle 3 Finding C3-2, HIGH). RagForgeRagasLLM was shipped without the .generate() / is_finished() / get_temperature() / set_run_config() methods that ragas 0.4.x's BaseRagasLLM exposes. Every metric job crashed with AttributeError: 'RagForgeRagasLLM' object has no attribute 'generate' in roughly 6 seconds without making a single network call. v0.2.2 re-declares every public method on both BaseRagasLLM and BaseRagasEmbeddings as duck-typed shims, and a new contract test (tests/test_ragas_adapters_contract.py) iterates the real ragas base classes and asserts the wrappers match them — so the next ragas release that grows a new method fails fast in CI, not in a user audit.
  • Skipped counter silently wrong (Cycle 3 residual of Cycle 2 Finding #6, MEDIUM). EvaluationResult.skipped_evaluations was never populated by the RAGAS engine, so the report's top-level "Skipped: N" line read 0 even when every job had crashed. Individual-metric extraction failures also produced one aggregate SkipRecord per metric name instead of fanning out per (sample, metric) pair — a 12-sample x 4-metric run produced 4 skip records instead of 48, under-reporting the blast radius by 12x. v0.2.2 fans out skips across every affected sample and keeps the integer counter in lockstep with the detail list. Reason strings are truncated to 400 chars with a trailing ellipsis so long Python tracebacks don't blow up HTML/PDF rendering.
  • __version__ constant stale (Cycle 3 Finding C3-5, LOW). pip show rag-forge-evaluator correctly reported 0.2.1 but import rag_forge_evaluator; rag_forge_evaluator.__version__ returned "0.1.0". Users programmatically tagging their audit reports with the tool version were recording wrong versions in their own records. v0.2.2 bumps the constant, adds the missing __version__ to rag_forge_observability, and introduces tests/test_version_drift.py — a parametrized CI test that reads each package's pyproject.toml and __init__.py and asserts they stay in lockstep. Any future release bumping one half but forgetting the other fails in CI.

Measurement rubric changes (read if upgrading from v0.1.x)

Context Relevance scores are systematically higher under v0.2.x than v0.1.x — by ~0.25 on average against identical telemetry — and this is a rubric change, not product improvement.

Cycle 3 observed every case in the 12-sample PearMedica golden set scoring higher on Context Relevance under v0.2.1's llm-judge than it did under v0.1.3's, with deltas from +0.05 to +0.48. The production retrieval code was unchanged between the two runs; only the measurement framework moved.

Root cause (mechanistic): v0.1.3 scored Context Relevance by per-chunk averaging. Its judge prompt (in packages/evaluator/src/rag_forge_evaluator/metrics/context_relevance.py) instructed the judge to rate each retrieved chunk individually on a 1-5 scale and return mean_score = mean(ratings) / 5:

Rate each context chunk's relevance to the query on a 1-5 scale.
Respond with ... "ratings": [{"chunk_index": 0, "score": 4, ...}], "mean_score": 0.8}
The mean_score should be the average rating divided by 5 (normalized to 0.0-1.0).

v0.2.0's new combined llm_judge prompt (in packages/evaluator/src/rag_forge_evaluator/metrics/llm_judge.py, introduced in commit 330465f) switched to a holistic 0-1 score across the whole context:

- context_relevance (0-1): how relevant the retrieved context is to the query.

The two methodologies are not equivalent:

  • Per-chunk averaging penalises the presence of irrelevant chunks. If 5 chunks are retrieved and 2 are relevant, the average is roughly (5+5+1+1+1)/5/5 = 0.52.
  • Holistic scoring typically rewards the presence of at least some relevant information. Given the same 5 chunks, a judge will often return 0.80+ because "the context does contain relevant information."

Holistic scoring is systematically more lenient than averaged per-chunk scoring when retrieval surfaces a mix of relevant and irrelevant chunks — which is the normal case for hybrid retrieval pipelines. That is the mechanism behind the +0.25 drift.

What this means for upgrading from v0.1.x:

  • Do not read a Context Relevance jump as a product improvement when upgrading. Your retrieval pipeline has not gotten better; the measurement got more lenient.
  • Re-baseline your CI gate thresholds after the first v0.2.x run against a known-good pipeline. If your v0.1.x gate was context_relevance >= 0.60, you may need >= 0.80 or higher under v0.2.x to catch the same drift magnitude.
  • The other three metrics (faithfulness, answer_relevance, hallucination) also moved to the combined-pass prompt in v0.2.0, but their deltas against Cycle 2's telemetry were within LLM-judge variance (+0.02 to +0.06). Treat them as lightly noisy, not as rubric-changed, unless you see large per-case jumps.
  • Refusal-aware scoring is default-on in v0.2.0+ and adds a separate safety_refusal rubric with explicitly softer context-relevance criteria. That path only fires when the judge classifies a sample as a refusal; in the Cycle 3 run, 0 samples were classified as refusals, so the drift above is entirely on the standard rubric — not a side effect of refusal mode.

A future --rubric=strict-v1 flag that pins the judge prompt to the v0.1.x per-chunk-averaging shape is under design for a later release. Not in v0.2.2. Open a GitHub issue if you need it before then.

Verified fixed in this release

  • Cycle 2 Finding #4 (OpenAIEmbeddings.embed_query AttributeError): contract test in tests/test_ragas_adapters_contract.py asserts embed_query + aembed_query + embed_documents + aembed_documents + embed_text + embed_texts + set_run_config are all declared on RagForgeRagasEmbeddings. End-to-end smoke test in tests/test_ragas_adapters_e2e.py runs ragas.evaluate() against the wrapper and asserts no AttributeError naming our classes.
  • Cycle 2 Finding #5 (max_tokens overflow on long structured clinical responses): v0.2.0 raised ClaudeJudge default from 1024 -> 4096 tokens. tests/test_cycle2_regression.py::test_cycle2_fixture_handles_long_structured_responses asserts the absence of InstructorRetryException / finish_reason='length' signatures in skip records against a 3-sample long-response fixture.
  • Cycle 2 Finding #6 (silent 0.0 coercion): two-part fix. v0.2.0 replaced the 0.0 fallback with SkipRecord tracking. v0.2.2 (this release) closes the residual gap where the skipped_evaluations integer counter was not kept in lockstep with the skip detail list — see Fixed above.

v0.2.1 — v0.2.0 partial-publish recovery

Choose a tag to compare

@hallengray hallengray released this 14 Apr 20:11
18947f4

v0.2.1 — v0.2.0 partial-publish recovery

Release date: 2026-04-14
Scope: One-line packaging fix to rag-forge-evaluator; all six packages bumped 0.2.0 → 0.2.1 for monorepo version alignment. No feature changes. Users upgrading from v0.1.3 should read the v0.2.0 release notes — that's where the real changes are.

Why this release exists

The v0.2.0 publish workflow got partway through before PyPI rejected the rag-forge-evaluator wheel with:

400 Invalid distribution file. ZIP archive not accepted:
Duplicate filename in local headers.

PyPI's 2025+ strict validator rejects wheels whose internal ZIP archive contains duplicate filenames. Our wheel had the entire rag_forge_evaluator/report/templates/ directory packaged twice — once via [tool.hatch.build.targets.wheel] packages = ["src/rag_forge_evaluator"] (which already recurses into every subdirectory) and a second time via a redundant [tool.hatch.build.targets.wheel.force-include] entry that was added during the v0.2.0 template work as belt-and-braces insurance.

Partial-publish state after v0.2.0:

Package State on v0.2.0
@rag-forge/shared (npm) ✅ Published
@rag-forge/mcp (npm) ✅ Published
@rag-forge/cli (npm) ✅ Published
rag-forge-core (PyPI) ✅ Published
rag-forge-evaluator (PyPI) ❌ Rejected (duplicate-filename error)
rag-forge-observability (PyPI) ⏭️ Skipped (blocked by evaluator failure)

What's fixed

  • packages/evaluator/pyproject.toml — remove the redundant [tool.hatch.build.targets.wheel.force-include] block. The packages entry already covers the templates directory. Verified locally: the rebuilt wheel has 41 entries, 0 duplicates, and the Jinja template lives exactly once at rag_forge_evaluator/report/templates/audit.html.j2.
  • .github/workflows/publish.yml — add skip-existing: true to all three pypa/gh-action-pypi-publish steps. This lets partial-publish recoveries (like this one) re-run cleanly: rag-forge-core was already uploaded as 0.2.0 → the 0.2.1 upload is fresh, but if a future re-run ever needs to re-publish something that already exists, it'll skip instead of hard-failing the whole workflow.

Version skew warning

Because the v0.2.0 workflow partially completed, PyPI and npm saw different final states on v0.2.0:

  • npm: all three packages (shared, mcp, cli) published at 0.2.0
  • PyPI: only rag-forge-core published at 0.2.0; rag-forge-evaluator and rag-forge-observability stayed at 0.1.3

v0.2.1 restores alignment: every package is now available at 0.2.1 on both registries. New installs should use 0.2.1.

Upgrade instructions

# Python
pip install -U 'rag-forge-evaluator[ragas]==0.2.1'

# npm
npm install -g @rag-forge/cli@0.2.1

No code changes, no CLI flag changes, no configuration changes between 0.2.0 and 0.2.1. This is purely a packaging fix.

Credits

Caught by PyPI's strict-ZIP validator during the v0.2.0 release. The local wheel built cleanly on uv build, so the duplicate-filename issue would not have been visible without the publish attempt — pip install tolerates the duplicate silently. Worth watching going forward: add a check-wheel-contents step to CI so we catch this class of issue before release.

v0.2.0 — Evaluator Refresh

Choose a tag to compare

@hallengray hallengray released this 14 Apr 19:48
5cc9426

⚠️ Partial publish — install v0.2.1 instead

This release shipped to npm successfully (@rag-forge/cli, @rag-forge/mcp, @rag-forge/shared are all available at 0.2.0) but rag-forge-evaluator was rejected by PyPI with a duplicate-filename error, which also blocked rag-forge-observability. Only rag-forge-core is available on PyPI at 0.2.0.

Upgrade to v0.2.1 instead — it contains the exact same evaluator features plus the packaging fix, and all six packages are available on both registries.

$(cat docs/release-notes/v0.2.0.md)

v0.1.3

Choose a tag to compare

@hallengray hallengray released this 13 Apr 17:26

v0.1.3 — Audit resilience

Release date: 2026-04-13
Scope: rag-forge-evaluator (Python). Other packages bumped in lockstep for monorepo version alignment.

Why this release exists

The PearMedica cycle-2 validation audit on v0.1.2 surfaced two operational gaps that don't show up until a real long-running audit meets a real Anthropic capacity event:

  1. Mid-loop crashes vaporized all progress. When the audit loop crashed on sample 5 of 19 (or 9 of 19, depending on the run), there was no partial report file on disk. Every sample scored before the crash was lost from the report writer's perspective — the only reason we have any cycle-2 data at all is that the user reconstructed it from the terminal progress stream.
  2. The 529 retry budget was too small and silent. v0.1.2's retry wrapper used a hardcoded ~62-second total backoff window. That's enough for transient flaps but inadequate for sustained overload events, which can last 5–15 minutes. There was also no way to see when a retry was in flight; the audit just looked frozen.

v0.1.3 fixes both.

What's new

1. Partial reports on mid-loop abort (resolves #24)

Any mid-loop exit — KeyboardInterrupt, unhandled exception, or retry-budget exhaustion — now writes audit-report.partial.json before propagating. The file contains:

  • partial: true marker
  • completed_samples: N / total_samples: M
  • aborted_reason: "keyboard_interrupt" | "retry_budget_exhausted" | "unhandled_exception"
  • error_message: the triggering exception
  • Full per-sample scored results in the same shape as a complete report
  • metrics: null, rmm_level: null, rmm_name: null, overall_score: null at the top level

Subset aggregates are intentionally namespaced under partial_metrics.by_metric with an in-band caveat:

"partial_metrics": {
  "note": "Subset aggregates over 9/19 samples. NOT comparable to a full-run report. Not thresholded against RMM. Do not screenshot these numbers without this caveat.",
  "by_metric": {
    "faithfulness": {"score": 0.50, "scored_count": 9, "skipped_count": 0},
    ...
  }
}

RMM level is unconditionally omitted from partial reports. Computing it over a subset is wrong (threshold claims apply to the whole pipeline), not partial.

Exit codes:

  • 0 — audit completed successfully
  • 1 — hard failure (unrecoverable error, no report)
  • 2 — usage / config error (argparse default)
  • 3 — partial audit; audit-report.partial.json written

CI scripts can now distinguish partial from hard failure:

rag-forge-eval audit ...
code=$?
if [ $code -eq 3 ]; then
  echo "Partial audit — inspect $OUTDIR/audit-report.partial.json"
fi

2. Configurable 529 retry budget + visibility (resolves #25)

New environment variable: RAG_FORGE_JUDGE_OVERLOAD_BUDGET_SECONDS (default: 300).

  • Retries on 529 fire until either max_retries attempts are exhausted or cumulative elapsed time + next backoff would exceed the budget — whichever trips first.
  • Also exposed as a constructor argument on ClaudeJudge: overload_budget_seconds=300.0.

Retry notices stream to stderr in real time, so a long-running audit no longer looks frozen during a capacity event:

  [judge: 529 overload, retry 1, elapsed 0s / 300s budget]
  [judge: 529 overload, retry 2, elapsed 2s / 300s budget]
  [judge: 529 overload, retry 3, elapsed 6s / 300s budget]

On budget exhaustion, the CLI emits a structured fallback hint before writing the partial report:

Anthropic was in a sustained 529 overload event. Options:
  (a) wait 10-15 minutes and re-run
  (b) switch judges with --judge openai
  (c) increase the budget: RAG_FORGE_JUDGE_OVERLOAD_BUDGET_SECONDS=600

The retry wrapper raises OverloadBudgetExhaustedError (new, exported from rag_forge_evaluator.judge.claude_judge) which wraps the underlying APIStatusError(529) on .original. Callers that want to branch on the distinction can catch the new type.

Implementation notes

  • ClaudeJudge gains an on_retry callback. Kept off the ProgressReporter protocol on purpose: the judge has multiple call sites (tests, programmatic use, MCP server) and shouldn't unilaterally decide stderr is the right output channel. The CLI wires in a stderr writer; other callers get silence by default.
  • Partial state lives on LLMJudgeEvaluator. The accumulator lists are promoted from local variables to instance attributes so the orchestrator can reach in after an exception. RAGAS-backed audits don't get partial reports in v0.1.3 — RAGAS batches internally and has no mid-loop hook.
  • PartialAuditError is the bridge between the orchestrator and the CLI. It wraps the original exception, carries the path to the written partial report, and is what the CLI catches to emit exit code 3.

Upgrade path

uv pip install --upgrade \
  rag-forge-evaluator==0.1.3 \
  rag-forge-observability==0.1.3

No breaking changes for existing consumers:

  • ClaudeJudge constructor gains optional overload_budget_seconds and on_retry — both default to None and preserve v0.1.2 behavior minus the crash-on-529 bug.
  • AuditConfig gains optional on_judge_retry — defaults to None.
  • The new OverloadBudgetExhaustedError inherits from RuntimeError; any caller that previously caught Exception or RuntimeError still catches it.
  • Exit code 3 is new. If you had a CI script that treated any non-zero exit as a hard failure, it will now treat partials as failures too — which was already the old behavior. Upgrade your script to branch on 3 when you're ready to surface partial artifacts.

Tests

  • tests/test_claude_judge_529_retry.py — extended with budget-exhausts-before-max-retries, env var override, on_retry callback assertion. 7 tests, all passing.
  • tests/test_partial_report.py — new. Covers: partial JSON on mid-loop exception, PartialAuditError wrapping, KeyboardInterrupt path, and the zero-samples-scored case where no partial report is written and the original exception propagates unchanged. 4 tests, all passing.
  • Full evaluator suite: 202 passing. Full monorepo suite: 527 passing.

What's not in this release

  • HTML rendering for partial reports. v0.1.3 writes partial JSON only. HTML can come later if anyone asks.
  • Resume-from-partial. Out of scope — partial reports are handoff artifacts, not resumable checkpoints.
  • RAGAS partial reports. RAGAS doesn't expose a mid-loop hook. An LLM-judge run is the only engine that gets the resilience fix in v0.1.3.
  • Partial HTML + RMM-level workarounds — intentionally omitted to avoid misleading screenshots.

Cycle-2 follow-up

With v0.1.3 installed, the PearMedica cycle-2 validation audit is fully unblocked:

  • If Anthropic stays healthy, the audit runs to completion and writes the standard report.
  • If Anthropic overloads again, retries absorb up to 5 minutes of transient 529s with visible stderr notices.
  • If the overload outlasts the budget, the audit exits with code 3 and a partial JSON with everything scored so far — no more lost progress.

Re-run the command from the cycle-2 runbook. Everything from the midway point onward just works.

v0.1.2

Choose a tag to compare

@hallengray hallengray released this 13 Apr 16:18

v0.1.2 — 529 Overloaded retry hotfix

Release date: 2026-04-13
Scope: rag-forge-evaluator (Python). Other packages bumped in lockstep for monorepo version alignment.

Why this release exists

During the PearMedica cycle-2 validation audit on v0.1.1, the run crashed on the first faithfulness judge call with:

anthropic._exceptions.OverloadedError: Error code: 529 - 'Overloaded'

v0.1.1 replaced the cycle-1 manual retry loop with the Anthropic SDK's built-in retry (Anthropic(max_retries=5)), assuming it covered the full set of transient error codes. It does not. The SDK's retry set is 408, 429, 500, 502, 503, 504 — 529 Overloaded is Anthropic-specific and falls outside that set, so it propagates as APIStatusError and crashes long-running audits during any Anthropic capacity event.

Two back-to-back PearMedica cycle-2 runs crashed on 529 within the first 1–2 samples, confirming the issue is deterministic during capacity pressure and blocks all --judge claude audits until it clears.

The fix

packages/evaluator/src/rag_forge_evaluator/judge/claude_judge.py now wraps messages.create() in an explicit 529 retry loop layered on top of the SDK's built-in retry:

  • Catches APIStatusError where status_code == 529.
  • Retries up to max_retries times (default 5, overridable via RAG_FORGE_JUDGE_MAX_RETRIES).
  • Exponential backoff: 2s, 4s, 8s, 16s, 32s.
  • All other APIStatusError codes propagate immediately — non-529 errors remain the SDK's responsibility.

This mirrors the cycle-1 manual patch that had proven the approach in production before v0.1.1 incorrectly assumed the SDK covered 529.

Tests

New test file tests/test_claude_judge_529_retry.py:

  1. Retry then succeed — 529 on first call, success on second, verifies the judge returns the successful payload and called time.sleep once.
  2. Exhaust retries — 529 on every call, verifies APIStatusError is re-raised after max_retries + 1 attempts.
  3. Non-529 errors bypass retry — 400 Bad Request propagates immediately with zero sleeps.

All 23 judge-related tests pass.

Upgrade path

uv pip install --upgrade rag-forge-evaluator==0.1.2 rag-forge-observability==0.1.2

No code or config changes required. If you were on v0.1.1 and hit a 529 crash, v0.1.2 resolves it with no further action.

Package versions

All 6 packages bumped to 0.1.2 for monorepo alignment, even though the only functional change is in rag-forge-evaluator:

  • @rag-forge/cli@0.1.2
  • @rag-forge/mcp@0.1.2
  • @rag-forge/shared@0.1.2
  • rag-forge-core==0.1.2
  • rag-forge-evaluator==0.1.2 ← the fix is here
  • rag-forge-observability==0.1.2

What's not in this release

  • The RAGAS engine still does not honor --judge claude — deferred to a future release.
  • No changes to the marketing site, docs, CLI, or MCP server.

Cycle-2 follow-up

With v0.1.2 installed, the PearMedica cycle-2 validation audit is unblocked. Re-run the command from the cycle-2 runbook; the 529 retry wrapper will absorb transient Anthropic capacity events and the audit will either complete or fail with a fundamentally different error (in which case we investigate fresh).

v0.1.1

Choose a tag to compare

@hallengray hallengray released this 13 Apr 15:46

RAG-Forge v0.1.1 — Post-Audit Hardening Release

Release date: 2026-04-13
Branch: fix/v011-audit-bugs
Trigger: First real-world audit run against PearMedica's clinical RAG pipeline surfaced 12 distinct issues across correctness, packaging, and UX. v0.1.1 fixes all of them.

TL;DR — Why you should upgrade immediately

If you ran rag-forge audit against any production pipeline on v0.1.0, your scorecard is mathematically wrong in a direction that makes your pipeline look ~3-4× worse than reality. v0.1.1 fixes the silent parse-failure-to-zero coercion that polluted aggregate metrics, plus a hardcoded judge token budget that truncated mid-array on long structured responses (clinical, legal, financial). Re-running the same audit against v0.1.1 will give you trustworthy numbers for the first time.

Critical correctness fix

Aggregate scores no longer count parse failures as zero

Bug: Every metric wrapped json.loads() in a try/except that returned score=0.0 on any failure. The LLMJudgeEvaluator then averaged across ALL samples including the failed ones, dragging real averages down by up to 4×.
Real-world impact: PearMedica's reported context_relevance was 0.063 (15/19 samples parse-failed). Real average over the 4 successfully-scored samples was ~0.30. The reported answer_relevance of 0.768 was actually 0.913 once forced zeros were excluded — moving it from "fails threshold" to "passes threshold."
Fix: New shared parse_judge_json helper handles empty strings, code fences, leading prose, trailing prose, and truncated output uniformly. All four metrics emit MetricResult(skipped=True) on parse failure. LLMJudgeEvaluator aggregates over scored_count only and reports skipped_count separately on each metric.

Missing-field scores no longer default to worst-case

Bug: When the judge returned valid JSON but the expected field was absent, hallucination.py defaulted hallucination_rate=1.0 ("100% hallucinated") and faithfulness.py defaulted score=0.0. A judge that returned a refusal or a partial response was recorded as proof of total failure.
Fix: Missing fields now mark the result as skipped=True rather than scoring worst-case.

Generalization fixes (the "different customer" lens)

Configurable judge max_tokens — default bumped 1024 → 4096

Bug: max_tokens=1024 was structurally too small for faithfulness/hallucination metrics, which ask the judge to enumerate every factual claim. On long structured responses (~30-50 claims), the judge truncated mid-array.
Fix: ClaudeJudge and OpenAIJudge now accept max_tokens via constructor argument, RAG_FORGE_JUDGE_MAX_TOKENS env var, or a 4096 default. Cost difference for unused budget is zero — providers only charge for tokens actually generated.

Configurable judge max_retries — default 5 with SDK exponential backoff

Bug: Anthropic SDK default is 2 retries. A single 529 Overloaded killed the entire 19-sample audit mid-run after $0.40 of judge calls.
Fix: Anthropic(max_retries=5) + OpenAI(max_retries=5). Configurable via constructor arg or RAG_FORGE_JUDGE_MAX_RETRIES env var.

--judge-model CLI flag

Bug: The CLI --judge flag only accepted mock | claude | openai. A law firm wanting Claude Opus, or a startup wanting gpt-4o-mini, had no way to override.
Fix: New --judge-model flag passes through any provider-specific model id (claude-opus-4-6, gpt-4-turbo, claude-haiku-4-5, etc.). Falls back to RAG_FORGE_JUDGE_MODEL env var, then the provider default.

--evaluator ragas rejects non-OpenAI judges loudly

Bug: The RAGAS engine ignores --judge and uses its own internal OpenAI judge. Running --evaluator ragas --judge claude silently swapped models and failed mid-run with an opaque auth error if OPENAI_API_KEY wasn't set.
Fix: AuditOrchestrator.__init__ validates the combo at construction time and raises ConfigurationError with concrete remediation steps. Mock and OpenAI judges are still allowed.

RAGAS defensive result access + version pin

Bug: 'EvaluationResult' object has no attribute 'get' on ragas ≥ 0.4 because the result type changed shape between major versions.
Fix: Pin to ragas>=0.2.10,<0.3 for v0.1.1. New _extract_ragas_score helper falls through dict access → __getitem__ → attribute access → numeric cast so future upgrades degrade gracefully.

UX fixes

Pre-run banner with cost + duration estimate, plus per-sample progress streaming

Bug: A 5-15 minute audit gave zero feedback. Users had no idea whether the run was hung, how long it would take, or how much it would cost.
Fix:

  • New ProgressReporter protocol with StderrProgressReporter default
  • Pre-run banner: sample count, judge model, evaluator engine, judge call total, estimated minutes range, estimated USD cost
  • Confirmation prompt before any judge calls (--yes/-y to bypass; required for non-interactive runs)
  • Per-sample stderr line streaming faith=… ctx=… ans=… hall=… OK|WARN N skipped (elapsed)
  • End-of-run summary with elapsed time, scored/skipped totals, RMM level, report path
  • Mock-judge runs (cost = $0) skip the prompt automatically so existing test suites are unaffected

Stdout no longer block-buffers on Windows non-TTY

Bug: python -m rag_forge_evaluator.cli audit ... from a non-TTY shell sat at 0 bytes for 5-10 minutes, then dumped everything at once. This is Python's default block-buffering when stdout isn't a terminal.
Fix: Both subprocess paths covered:

  • Node bridge passes -u to the python interpreter
  • Python CLI calls sys.stdout.reconfigure(line_buffering=True) at startup

--pdf checks Playwright at audit start, not end

Bug: PearMedica audit ran $0.40 of judge calls then crashed at the very end with "Playwright not installed" because the [pdf] extra was undocumented.
Fix: New is_available() helper in report/pdf.py checks both the playwright import AND the Chromium binary. AuditOrchestrator.run() calls it as step 0 when --pdf is set, raises ConfigurationError with the exact install command if either check fails.

Packaging fixes

@rag-forge/cli is actually installable from npm

Bug: v0.1.0 was published with "@rag-forge/shared": "workspace:*" literal, making it uninstallable anywhere outside the source monorepo. Same bug affected @rag-forge/mcp.
Root cause: The publish workflow used npm publish, which does not understand the pnpm workspace protocol and ships package.json verbatim.
Fix: Switched to pnpm publish --no-git-checks for all 3 npm packages. pnpm publish rewrites workspace:* to the actual version range before uploading. Provenance attestations still emitted via --provenance.

uv tool install rag-forge-evaluator works

Bug: Failed with "no executables are provided" because the package had no [project.scripts] entry.
Fix: Added rag-forge-eval = "rag_forge_evaluator.cli:main" console script.

rag-forge-observability declared as a runtime dependency

Bug: cli.py imported rag_forge_observability.tracing at module load but the package was not declared. Installing from PyPI then running the CLI crashed immediately.
Fix: Added rag-forge-observability>=0.1.1 to dependencies.

Documentation

JudgeProvider protocol prominently documented

The protocol is two methods (judge, model_name) and ~20 lines of code to plug in any provider. The evaluator README now includes a working GeminiJudge example, the protocol contract, wiring instructions, and three implementation hints (max_tokens budget, retry config, raw response handling). First-party Gemini, Bedrock, and Ollama judges are still tracked for v0.1.2.

Deferred to v0.1.2

  • Truncation-detection retry loop. When finish_reason='length', retry with doubled max_tokens. Requires upgrading the JudgeProvider interface to return (text, finish_reason) instead of just text.
  • "Retrieval bypass" root-cause classification. When context_relevance < 0.3 AND faithfulness < 0.3 AND answer_relevance > 0.8, the diagnosis is "LLM is ignoring retrieval and answering from training data" — not "generator hallucination." This is the actual RMM-0 pathology PearMedica exhibited. v0.1.2 will surface it as a distinct category in the report.
  • Per-metric max_tokens. Faithfulness/hallucination need bigger budgets than answer_relevance. v0.1.2 will let each MetricEvaluator declare its own token requirement.
  • --judge claude propagation through RAGAS. Needs a LangchainLLMWrapper around ClaudeJudge. Architecturally invasive — better in v0.1.2.
  • First-party Gemini, Bedrock, and Ollama judges alongside Claude and OpenAI.

Test count

v0.1.0: 136 tests · v0.1.1: 177 tests (+41 covering the parser, aggregation, judge config, audit validation, and ragas extractor)

Acknowledgments

This release exists because of a single forensic audit run against PearMedica's clinical RAG pipeline on 2026-04-13. Eight of the twelve issues fixed here would have shipped to every customer otherwise.

v0.1.0 — Initial release

Choose a tag to compare

@hallengray hallengray released this 13 Apr 09:14

Initial release of RAG-Forge: framework-agnostic CLI toolkit for production-grade RAG pipelines with evaluation baked in.