bulkhead

A resilient LLM gateway — retries, circuit breaking, rate limiting, caching, failover, and cost governance in front of any provider, in ~2k readable lines of Python.

bulkhead sits between your application and a model backend and turns a flaky, rate-limited, expensive dependency into a dependable one. It is not a framework and not a proxy server: it is a tight, legible implementation of the reliability control plane every production LLM call needs — the exponential-backoff retry loop, the circuit breaker that protects a struggling backend, per-key token-bucket rate limiting, an exact + semantic response cache, ordered failover across providers/models, and hard per-key USD budget caps — all composed into a single gateway.complete(request) call.

Why this exists / what it demonstrates

Most "LLM gateway" code hides its reliability behavior behind a config file and a vendor. This repo does the opposite: it makes the mechanics the point. Reading it should teach you

• the resilience primitives, done right — full-jitter exponential backoff (not fixed sleeps), a real three-state circuit breaker, continuously-refilling token buckets for RPM and TPM, and read-through caching with correct invalidation rules;
• why they compose — each primitive is a small, independently-testable unit; the Gateway is just the pipeline that threads a request through them in the right order, with clean failover and honest observability fields on every Response (from_cache, attempts, served_by);
• the load-bearing invariants — only clean text completions are cached (a tool-use response is a request to act and must reach the caller live); a cache hit never touches a provider; a terminal 4xx fails over without wasting retries; budget is checked before spending, never after;
• how to test timing without a network or a clock — the keystone design choice is a deterministic injected Clock. Every sleep, recovery window, bucket refill, TTL, and hedge delay reads from it, so a ManualClock drives the entire suite — backoff sequences, a circuit tripping and recovering after exactly recovery_timeout seconds, a bucket refilling over a virtual minute — with zero real sleeping and zero flakiness.

Architecture

                       gateway.complete(request)
                                  │
              ┌───────────────────┼────────────────────┐
              ▼                   ▼                     ▼
        ┌───────────┐      ┌────────────┐        ┌───────────┐
        │  Budget   │      │ RateLimiter│        │   Cache   │  hit ─▶ return
        │ precheck  │─────▶│ RPM + TPM  │───────▶│ exact/sem │  (from_cache=True)
        │ (USD cap) │      │  buckets   │        └─────┬─────┘
        └───────────┘      └────────────┘              │ miss
                                                       ▼
                            ordered failover across Targets
              ┌────────────────────────┬────────────────────────┐
              ▼                        ▼                         ▼
        ┌───────────┐            ┌───────────┐             ┌───────────┐
        │ Target 0  │  fail ───▶ │ Target 1  │  fail ───▶  │   ...     │
        │ Circuit + │            │ Circuit + │             │ NoHealthy │
        │  Retry    │            │  Retry    │             │  Target   │
        └─────┬─────┘            └───────────┘             └───────────┘
              │ Provider.complete()
              ▼
     ┌──────────────────┐        ┌──────────────────┐
     │ AnthropicProvider│  (or)  │   FakeProvider   │  scripted, offline, no creds
     │  Bedrock | direct│        │  (tests + demo)  │
     └────────┬─────────┘        └──────────────────┘
              │ .messages.create(...)
              ▼
        anthropic SDK  ── the ONLY module that imports it, lazily ──

   ▲ every timing decision (backoff, recovery, refill, TTL, hedge) reads the
   │ ONE injected Clock — SystemClock in prod, ManualClock in every test.

The Clock seam is the whole trick: it is to timing what the sibling cogs record/replay is to model output — the single nondeterministic dependency pushed behind an interface so everything above it is pure and reproducible.

Module map

Module	Responsibility
bulkhead/types.py	Normalized frozen dataclasses: Request, Response (+ observability fields), Usage (__add__), ToolCall, and the ProviderName/Outcome enums.
bulkhead/clock.py	The keystone Clock protocol. SystemClock (real monotonic + sleep) and ManualClock (virtual time; sleep fast-forwards; advance for tests).
bulkhead/errors.py	The exception taxonomy — Transient vs Terminal, RateLimited/Overloaded/Timeout, plus gateway control errors — and the is_retryable predicate.
bulkhead/provider.py	Provider protocol, AnthropicProvider (Bedrock/direct, only file importing anthropic, lazily; maps SDK exceptions → taxonomy), and FakeProvider (scripted, offline).
bulkhead/retry.py	RetryPolicy + call_with_retry: exponential backoff with full jitter from a seeded RNG, honoring Retry-After, sleeping on the injected clock.
bulkhead/circuit.py	CircuitBreaker (CLOSED/OPEN/HALF_OPEN) driven entirely by the clock, plus a per-target CircuitRegistry.
bulkhead/ratelimit.py	Deterministic TokenBucket and a per-key RateLimiter enforcing RPM and TPM, in block-or-raise modes.
bulkhead/cache.py	ExactCache (SHA-256 of canonical request JSON, clock TTL) and optional SemanticCache with a dependency-free HashingEmbedder.
bulkhead/budget.py	Price table (anthropic.-prefix-tolerant), CostMeter, and per-key Budget caps enforced as a precheck.
bulkhead/router.py	The Gateway — composes budget → rate limit → cache → circuit/retry/failover → hedge → charge/record into one complete().
bulkhead/metrics.py	In-memory counters and an immutable Snapshot with to_dict() / to_markdown().
bulkhead/cli.py	python -m bulkhead — an offline narrated demo and a --live Bedrock path.

Quickstart

1. Offline demo (no credentials, no network)

The fastest way to see the whole story. It drives a scripted FakeProvider and a ManualClock through a real Gateway and narrates retries with backoff, a cache hit, a circuit trip, and a failover — with zero real time elapsed.

uv venv
uv pip install -e '.[dev]'
uv run python -m bulkhead        # or: uv run python examples/gateway_demo.py

Under the hood the pipeline is just:

from bulkhead import Gateway, Target, Request, FakeProvider, ManualClock, RetryPolicy
from bulkhead.router import GatewayConfig
from bulkhead.errors import TransientError

clock = ManualClock()
primary = FakeProvider([TransientError("503"), TransientError("503"), None], clock=clock)
gateway = Gateway(
    [Target(primary, model="claude-opus-4-8")],
    clock=clock,
    config=GatewayConfig(retry=RetryPolicy(max_attempts=4)),
)
resp = gateway.complete(Request(messages=({"role": "user", "content": "hi"},)))
print(resp.text, resp.attempts, resp.served_by)   # served after 2 retries, no real sleep

2. Live against AWS Bedrock

The only requirement is Bedrock access. Provider and model are configured by environment; the request surface is kept minimal for Opus 4.8 (no temperature/top_p/top_k; adaptive thinking is opt-in).

export BULKHEAD_PROVIDER=bedrock          # default; use "anthropic" for the direct API
export AWS_REGION=us-east-1
export BULKHEAD_MODEL=claude-opus-4-8      # resolves to anthropic.claude-opus-4-8 on Bedrock
uv run python examples/live_bedrock.py "Explain the bulkhead pattern in one sentence."

Variable	Default (bedrock)	Meaning
BULKHEAD_PROVIDER	bedrock	bedrock or anthropic
BULKHEAD_MODEL	claude-opus-4-8	model id (Bedrock adds the anthropic. prefix)
AWS_REGION	us-east-1	Bedrock region

Switching to a direct Anthropic API key later is a one-line change (BULKHEAD_PROVIDER=anthropic) — the gateway, all primitives, and every test are untouched, which is the point of the provider seam.

Development

uv run ruff check .
uv run pytest

Both run fully offline. The suite never imports anthropic, never touches the network, and never really sleeps — a ManualClock drives all timing.

Intentional non-goals

This is a tasteful mini-implementation, scoped deliberately. What's left out is left out on purpose:

• Synchronous, not async. The pipeline is sequential and easy to follow, which is exactly what makes it deterministic under a ManualClock. Real hedging fires concurrent requests and cancels the loser; here hedging is a documented simplified model (try primary; back it up with a second target on failure/slow), honest about the trade-off in router.py. A production build would use asyncio and true request racing.
• In-memory only. The cache, rate-limiter buckets, budget meter, and metrics live in process memory. A multi-instance deployment needs a shared store (Redis for buckets/cache, a ledger for budgets) — the protocols (Cache, Clock) are the seams where that slots in without touching the router.
• Estimate-grade pricing. The PRICES table is Anthropic list price; Bedrock and other resellers charge differently. Budgets are a governance guardrail, not a billing system of record.
• A lexical HashingEmbedder, not a real embedding model. The SemanticCache ships a dependency-free hashing embedder so the mechanics are exercised offline; it captures lexical overlap, not meaning. Swap in a real Embedder for production semantic caching — and note semantic caching can serve one question's answer to a different question, which is why it is off by default.
• No streaming, no server. bulkhead is a library you call, not an HTTP proxy. Streaming and a network front-end are presentation concerns layered on top of this same pipeline.
• One provider family. Only Anthropic (Bedrock/direct) is wired. The Provider protocol is the seam for a second backend; nothing above it would change.

Each of these is a place where a production system does more — and where this repo deliberately stops, so the core stays legible.

The platform

bulkhead is one repo in a five-part agent platform. Each owns a single concern, stands alone, and shares the same spine: a normalized, Bedrock-default provider seam and a single injected nondeterministic dependency (record/replay, scripted solver, Clock) that makes the whole thing testable offline.

Repo	Concern
cogs	the agent runtime — the loop, tool protocol, provider seam, record/replay
bulkhead	reliable serving — a gateway (retries, circuit breaking, rate limits, caching, failover, budgets) in front of any provider ← this repo
loom	context engineering — retrieve, compact, and assemble what goes in the window
sonar	observability — reconstruct a run as a cost/latency timeline
gauntlet	evaluation — hermetic tool-use tasks scored with pass@k + confidence intervals

How work flows through them:

loom ──assemble context──▶ cogs ──model calls──▶ bulkhead ──▶ provider
                            │
              run cassette ─┴──▶ sonar (timeline, cost)
              eval result ─────▶ gauntlet (pass@k)

The seams are real, not aspirational: bulkhead speaks the same normalized Provider/message types as cogs and loom, so it drops into cogs's provider seam as the transport for every model call; and sonar ingests cogs cassettes and gauntlet results directly. The sonar README has the one-command combined demo.

License

MIT © 2026 Deepak