Repository navigation
v0.1.2
v0.1.2 — 529 Overloaded retry hotfix
Release date: 2026-04-13
Scope: rag-forge-evaluator (Python). Other packages bumped in lockstep for monorepo version alignment.
Why this release exists
During the PearMedica cycle-2 validation audit on v0.1.1, the run crashed on the first faithfulness judge call with:
anthropic._exceptions.OverloadedError: Error code: 529 - 'Overloaded'
v0.1.1 replaced the cycle-1 manual retry loop with the Anthropic SDK's built-in retry (Anthropic(max_retries=5)), assuming it covered the full set of transient error codes. It does not. The SDK's retry set is 408, 429, 500, 502, 503, 504 — 529 Overloaded is Anthropic-specific and falls outside that set, so it propagates as APIStatusError and crashes long-running audits during any Anthropic capacity event.
Two back-to-back PearMedica cycle-2 runs crashed on 529 within the first 1–2 samples, confirming the issue is deterministic during capacity pressure and blocks all --judge claude audits until it clears.
The fix
packages/evaluator/src/rag_forge_evaluator/judge/claude_judge.py now wraps messages.create() in an explicit 529 retry loop layered on top of the SDK's built-in retry:
- Catches
APIStatusErrorwherestatus_code == 529. - Retries up to
max_retriestimes (default 5, overridable viaRAG_FORGE_JUDGE_MAX_RETRIES). - Exponential backoff: 2s, 4s, 8s, 16s, 32s.
- All other
APIStatusErrorcodes propagate immediately — non-529 errors remain the SDK's responsibility.
This mirrors the cycle-1 manual patch that had proven the approach in production before v0.1.1 incorrectly assumed the SDK covered 529.
Tests
New test file tests/test_claude_judge_529_retry.py:
- Retry then succeed — 529 on first call, success on second, verifies the judge returns the successful payload and called
time.sleeponce. - Exhaust retries — 529 on every call, verifies
APIStatusErroris re-raised aftermax_retries + 1attempts. - Non-529 errors bypass retry — 400 Bad Request propagates immediately with zero sleeps.
All 23 judge-related tests pass.
Upgrade path
uv pip install --upgrade rag-forge-evaluator==0.1.2 rag-forge-observability==0.1.2No code or config changes required. If you were on v0.1.1 and hit a 529 crash, v0.1.2 resolves it with no further action.
Package versions
All 6 packages bumped to 0.1.2 for monorepo alignment, even though the only functional change is in rag-forge-evaluator:
@rag-forge/cli@0.1.2@rag-forge/mcp@0.1.2@rag-forge/shared@0.1.2rag-forge-core==0.1.2rag-forge-evaluator==0.1.2← the fix is hererag-forge-observability==0.1.2
What's not in this release
- The RAGAS engine still does not honor
--judge claude— deferred to a future release. - No changes to the marketing site, docs, CLI, or MCP server.
Cycle-2 follow-up
With v0.1.2 installed, the PearMedica cycle-2 validation audit is unblocked. Re-run the command from the cycle-2 runbook; the 529 retry wrapper will absorb transient Anthropic capacity events and the audit will either complete or fail with a fundamentally different error (in which case we investigate fresh).