Repository navigation
v0.1.3
v0.1.3 — Audit resilience
Release date: 2026-04-13
Scope: rag-forge-evaluator (Python). Other packages bumped in lockstep for monorepo version alignment.
Why this release exists
The PearMedica cycle-2 validation audit on v0.1.2 surfaced two operational gaps that don't show up until a real long-running audit meets a real Anthropic capacity event:
- Mid-loop crashes vaporized all progress. When the audit loop crashed on sample 5 of 19 (or 9 of 19, depending on the run), there was no partial report file on disk. Every sample scored before the crash was lost from the report writer's perspective — the only reason we have any cycle-2 data at all is that the user reconstructed it from the terminal progress stream.
- The 529 retry budget was too small and silent. v0.1.2's retry wrapper used a hardcoded ~62-second total backoff window. That's enough for transient flaps but inadequate for sustained overload events, which can last 5–15 minutes. There was also no way to see when a retry was in flight; the audit just looked frozen.
v0.1.3 fixes both.
What's new
1. Partial reports on mid-loop abort (resolves #24)
Any mid-loop exit — KeyboardInterrupt, unhandled exception, or retry-budget exhaustion — now writes audit-report.partial.json before propagating. The file contains:
partial: truemarkercompleted_samples: N/total_samples: Maborted_reason:"keyboard_interrupt"|"retry_budget_exhausted"|"unhandled_exception"error_message: the triggering exception- Full per-sample scored results in the same shape as a complete report
metrics: null,rmm_level: null,rmm_name: null,overall_score: nullat the top level
Subset aggregates are intentionally namespaced under partial_metrics.by_metric with an in-band caveat:
"partial_metrics": {
"note": "Subset aggregates over 9/19 samples. NOT comparable to a full-run report. Not thresholded against RMM. Do not screenshot these numbers without this caveat.",
"by_metric": {
"faithfulness": {"score": 0.50, "scored_count": 9, "skipped_count": 0},
...
}
}RMM level is unconditionally omitted from partial reports. Computing it over a subset is wrong (threshold claims apply to the whole pipeline), not partial.
Exit codes:
0— audit completed successfully1— hard failure (unrecoverable error, no report)2— usage / config error (argparse default)3— partial audit;audit-report.partial.jsonwritten
CI scripts can now distinguish partial from hard failure:
rag-forge-eval audit ...
code=$?
if [ $code -eq 3 ]; then
echo "Partial audit — inspect $OUTDIR/audit-report.partial.json"
fi2. Configurable 529 retry budget + visibility (resolves #25)
New environment variable: RAG_FORGE_JUDGE_OVERLOAD_BUDGET_SECONDS (default: 300).
- Retries on 529 fire until either
max_retriesattempts are exhausted or cumulative elapsed time + next backoff would exceed the budget — whichever trips first. - Also exposed as a constructor argument on
ClaudeJudge:overload_budget_seconds=300.0.
Retry notices stream to stderr in real time, so a long-running audit no longer looks frozen during a capacity event:
[judge: 529 overload, retry 1, elapsed 0s / 300s budget]
[judge: 529 overload, retry 2, elapsed 2s / 300s budget]
[judge: 529 overload, retry 3, elapsed 6s / 300s budget]
On budget exhaustion, the CLI emits a structured fallback hint before writing the partial report:
Anthropic was in a sustained 529 overload event. Options:
(a) wait 10-15 minutes and re-run
(b) switch judges with --judge openai
(c) increase the budget: RAG_FORGE_JUDGE_OVERLOAD_BUDGET_SECONDS=600
The retry wrapper raises OverloadBudgetExhaustedError (new, exported from rag_forge_evaluator.judge.claude_judge) which wraps the underlying APIStatusError(529) on .original. Callers that want to branch on the distinction can catch the new type.
Implementation notes
ClaudeJudgegains anon_retrycallback. Kept off theProgressReporterprotocol on purpose: the judge has multiple call sites (tests, programmatic use, MCP server) and shouldn't unilaterally decide stderr is the right output channel. The CLI wires in a stderr writer; other callers get silence by default.- Partial state lives on
LLMJudgeEvaluator. The accumulator lists are promoted from local variables to instance attributes so the orchestrator can reach in after an exception. RAGAS-backed audits don't get partial reports in v0.1.3 — RAGAS batches internally and has no mid-loop hook. PartialAuditErroris the bridge between the orchestrator and the CLI. It wraps the original exception, carries the path to the written partial report, and is what the CLI catches to emit exit code 3.
Upgrade path
uv pip install --upgrade \
rag-forge-evaluator==0.1.3 \
rag-forge-observability==0.1.3No breaking changes for existing consumers:
ClaudeJudgeconstructor gains optionaloverload_budget_secondsandon_retry— both default to None and preserve v0.1.2 behavior minus the crash-on-529 bug.AuditConfiggains optionalon_judge_retry— defaults to None.- The new
OverloadBudgetExhaustedErrorinherits fromRuntimeError; any caller that previously caughtExceptionorRuntimeErrorstill catches it. - Exit code 3 is new. If you had a CI script that treated any non-zero exit as a hard failure, it will now treat partials as failures too — which was already the old behavior. Upgrade your script to branch on
3when you're ready to surface partial artifacts.
Tests
tests/test_claude_judge_529_retry.py— extended with budget-exhausts-before-max-retries, env var override, on_retry callback assertion. 7 tests, all passing.tests/test_partial_report.py— new. Covers: partial JSON on mid-loop exception, PartialAuditError wrapping, KeyboardInterrupt path, and the zero-samples-scored case where no partial report is written and the original exception propagates unchanged. 4 tests, all passing.- Full evaluator suite: 202 passing. Full monorepo suite: 527 passing.
What's not in this release
- HTML rendering for partial reports. v0.1.3 writes partial JSON only. HTML can come later if anyone asks.
- Resume-from-partial. Out of scope — partial reports are handoff artifacts, not resumable checkpoints.
- RAGAS partial reports. RAGAS doesn't expose a mid-loop hook. An LLM-judge run is the only engine that gets the resilience fix in v0.1.3.
- Partial HTML + RMM-level workarounds — intentionally omitted to avoid misleading screenshots.
Cycle-2 follow-up
With v0.1.3 installed, the PearMedica cycle-2 validation audit is fully unblocked:
- If Anthropic stays healthy, the audit runs to completion and writes the standard report.
- If Anthropic overloads again, retries absorb up to 5 minutes of transient 529s with visible stderr notices.
- If the overload outlasts the budget, the audit exits with code 3 and a partial JSON with everything scored so far — no more lost progress.
Re-run the command from the cycle-2 runbook. Everything from the midway point onward just works.