Skip to content

Scaling And State

Fabrizio Salmi edited this page Sep 6, 2026 · 1 revision

Scaling and state

What one process gives you

Measured on Apple Silicon, dev mode, proxy stack only with no upstream call:

Endpoint Req/s p50 p99
/health, cold 1,313 7 ms 28 ms
/health, saturated at 100 connections 1,176 82 ms 149 ms
/api/v1/registry, light DB read 1,158 81 ms 188 ms

These measure the proxy's own overhead: auth middleware, firewall, routing, serialisation. They do not include the LLM call, which in real traffic dominates everything else by an order of magnitude or two.

The project calls roughly 1.2k req/s on one process a moderate-load number, which is the honest reading. For most LLM workloads it is also irrelevant: if each request waits seconds on a provider, you run out of concurrency long before you run out of proxy throughput.

Reproduce with wrk -t4 -c100 -d30s --latency http://localhost:8090/health.

What is shared and what is not

This is the part that changes behaviour when you scale out.

State Where it lives Scaling out
Endpoints, budgets, audit ledger SQLite by default, swappable for Postgres Move to Postgres and it is shared correctly
Rate limits In memory, per process Each worker enforces its own counter
Circuit breaker In memory, per process Each worker trips independently

The consequence is worth stating plainly: with N workers, a per-process rate limit of R is an effective limit of N times R, because a client's requests are spread across workers that do not compare notes. A limit of 100 requests per minute across four workers admits up to 400.

The circuit breaker has the mirror-image behaviour. One worker having tripped on a failing provider does not stop the other three from continuing to send traffic to it, so a provider outage is noticed independently by each worker rather than once.

Neither is a bug: the README states both are per process by design. It is a property to plan around.

Planning around it

  • Divide your intended limit by the worker count, and accept that the enforcement is approximate at the boundary
  • Where the limit is a hard requirement, cost control against a paid provider being the usual case, enforce it somewhere shared as well: budgets in the database, or a limiter in front of the proxy
  • Move to Postgres before scaling out, not after. SQLite plus several processes is the setup that works in testing and produces lock contention under real load
  • One process is a legitimate answer. For an LLM workload dominated by upstream latency, a single well-monitored process is often enough, and it makes both the rate limit and the circuit breaker mean what they say

Restarts

The proxy is on the critical path: when it is down, LLM traffic stops. Give it a restart policy, mount /app/data so the ledger and budgets survive, and alert on the health endpoint rather than on user reports.

Clone this wiki locally