We stress-tested LiteLLM spend limits: what held, what slipped, and 4 related issues

[petreli12]

Hi all, first time posting here.

I spent October testing how well LiteLLM’s per-customer budget limits hold when things get messy: parallel calls, oversized responses, backend outages, restarts, and 250 customers at once. Full write-up: We Stress-Tested AI Spend Limits. Here’s Where They Held and Where They Slipped.
In the interest of disclosure, we’re exploring a product in this space, but this post is about LiteLLM’s behavior and how people run it.
Setup: LiteLLM v1.106.6 as a self-hosted proxy with Postgres, per-customer max_budget, gpt-4o-mini, and a $0.05 budget per test customer. The full test set cost under $2 in OpenAI usage.

What held up
Configured well, LiteLLM held every budget to within about 2.5%, even through a database outage. With Redis (REDIS_HOST/REDIS_PORT, use_redis_transaction_buffer: true) and fail_closed_budget_enforcement: true, the gaps below closed: the oversized call was rejected before it reached OpenAI, and nothing was lost in an outage. It also handled a 20-call burst better than we expected. Only 4 of 20 got through, because it reserves worst-case cost before forwarding.

Where it slipped on the default setup
An oversized call (16,000 tokens at 95% of budget) ended 15.6% over.
A Postgres outage lost spend for good: only $0.0196 of the $0.0512 actually spent was recorded once the database came back.
A proxy restart handed budget back, because it reloaded the lower database figure.
With 250 customers active, 98 of them got one extra call past budget (more on this below).

Related issues
The 250-customer result is consistent with #40221: Spend counter cache evicts active budget counters after 200 entries. The 98 affected customers were the oldest entries in the cache, and none were affected with Redis enabled.
We didn’t reproduce these three, and I’m posting partly to ask whether others have:
#35529: End-user budget checks fail open on database lookup errors. A failed end-user lookup returns None, treating an unavailable lookup like a missing record.
#34238: End-user budget can be bypassed via stale per-pod fallback spend after Redis counter clean-miss. When the Redis end-user counter expires, enforcement can fall back to a stale per-pod value that’s lower than persisted spend.
#43732: Key over max_budget is admitted again after 60s idle, until the batch writer flushes spend. A blocked key can be admitted again after about a minute idle, until spend is flushed to the database.

My question for the community
If you run LiteLLM with per-customer budgets in production, especially multi-pod with Redis, have you seen any of these three in practice, and what settings did you land on? Redis plus the fail-closed setting closed every gap for us, but I’d like to know whether that matches what others run. And if any of this behaves differently in newer releases, corrections are very welcome.
We also turned the checks into a free tool, stipend-check, which runs the same checks against your own proxy in about two minutes. The repo includes the test harness, and I’m happy to share the raw results on request.