Skip to content

v0.1.2

Choose a tag to compare

@hallengray hallengray released this 13 Apr 16:18
· 32 commits to main since this release

v0.1.2 — 529 Overloaded retry hotfix

Release date: 2026-04-13
Scope: rag-forge-evaluator (Python). Other packages bumped in lockstep for monorepo version alignment.

Why this release exists

During the PearMedica cycle-2 validation audit on v0.1.1, the run crashed on the first faithfulness judge call with:

anthropic._exceptions.OverloadedError: Error code: 529 - 'Overloaded'

v0.1.1 replaced the cycle-1 manual retry loop with the Anthropic SDK's built-in retry (Anthropic(max_retries=5)), assuming it covered the full set of transient error codes. It does not. The SDK's retry set is 408, 429, 500, 502, 503, 504 — 529 Overloaded is Anthropic-specific and falls outside that set, so it propagates as APIStatusError and crashes long-running audits during any Anthropic capacity event.

Two back-to-back PearMedica cycle-2 runs crashed on 529 within the first 1–2 samples, confirming the issue is deterministic during capacity pressure and blocks all --judge claude audits until it clears.

The fix

packages/evaluator/src/rag_forge_evaluator/judge/claude_judge.py now wraps messages.create() in an explicit 529 retry loop layered on top of the SDK's built-in retry:

  • Catches APIStatusError where status_code == 529.
  • Retries up to max_retries times (default 5, overridable via RAG_FORGE_JUDGE_MAX_RETRIES).
  • Exponential backoff: 2s, 4s, 8s, 16s, 32s.
  • All other APIStatusError codes propagate immediately — non-529 errors remain the SDK's responsibility.

This mirrors the cycle-1 manual patch that had proven the approach in production before v0.1.1 incorrectly assumed the SDK covered 529.

Tests

New test file tests/test_claude_judge_529_retry.py:

  1. Retry then succeed — 529 on first call, success on second, verifies the judge returns the successful payload and called time.sleep once.
  2. Exhaust retries — 529 on every call, verifies APIStatusError is re-raised after max_retries + 1 attempts.
  3. Non-529 errors bypass retry — 400 Bad Request propagates immediately with zero sleeps.

All 23 judge-related tests pass.

Upgrade path

uv pip install --upgrade rag-forge-evaluator==0.1.2 rag-forge-observability==0.1.2

No code or config changes required. If you were on v0.1.1 and hit a 529 crash, v0.1.2 resolves it with no further action.

Package versions

All 6 packages bumped to 0.1.2 for monorepo alignment, even though the only functional change is in rag-forge-evaluator:

  • @rag-forge/cli@0.1.2
  • @rag-forge/mcp@0.1.2
  • @rag-forge/shared@0.1.2
  • rag-forge-core==0.1.2
  • rag-forge-evaluator==0.1.2 ← the fix is here
  • rag-forge-observability==0.1.2

What's not in this release

  • The RAGAS engine still does not honor --judge claude — deferred to a future release.
  • No changes to the marketing site, docs, CLI, or MCP server.

Cycle-2 follow-up

With v0.1.2 installed, the PearMedica cycle-2 validation audit is unblocked. Re-run the command from the cycle-2 runbook; the 529 retry wrapper will absorb transient Anthropic capacity events and the audit will either complete or fail with a fundamentally different error (in which case we investigate fresh).