Expected RPS of single LiteLLM proxy instance

[rhalaly]

What is the expected RPS for a single LiteLLM proxy instance (single pod, single worker, single core)?

[Ganesh-403]

@rhalaly
There isn't a single expected RPS number for a 1 worker / 1 core LiteLLM instance, because it depends heavily on what's enabled (database writes, callbacks, logging, Redis, upstream latency, streaming, etc.).

From the official benchmarks, LiteLLM has recently been tested at:

• ~1,000 QPS with no failures on a single 4 vCPU / 8 GB instance.
• Scaling up to ~5,000 QPS on that same 4 vCPU instance in stress testing.
• The published benchmark also recommends configuring the number of workers to match the CPU count. :contentReference[oaicite:0]{index=0}

Given those numbers, I wouldn't simply divide by four and conclude that a single core = ~250 RPS. The relationship isn't linear because of:

• Python runtime overhead,
• async I/O,
• thread pools,
• database/callback work,
• and the characteristics of the upstream provider. :contentReference[oaicite:1]{index=1}

So my expectation would be:

• If the proxy is doing very little beyond forwarding requests (minimal callbacks, logging disabled, fast upstream), a single worker on one core should comfortably handle hundreds of RPS.
• If you're enabling spend tracking, database writes, custom callbacks, guardrails, etc., the achievable throughput will be correspondingly lower.

A couple of clarifying questions:

1. Are you asking about pure proxy overhead (using a mock endpoint), or end-to-end throughput against a real LLM provider?
2. Is this with the default production configuration (Postgres, logging, spend tracking), or a minimal proxy?

Those two scenarios can have very different throughput characteristics.

[AzafTechnologies]

Hey @rhalaly!

If we are talking about a strict 1 core / 1 worker / 1 pod setup, here is the realistic breakdown you can expect:

Bare-bones / Mock setup (no DB, no callbacks, fast upstream): You're looking at roughly 100 to 250 RPS. Because Python/FastAPI handles async I/O well for simple passthroughs, it won't totally choke, but the single event loop will start queuing requests pretty quickly.

Real-world production setup (Postgres spend tracking, logging, active callbacks): That number is going to plummet down to 20 to 50 RPS. The reason is that synchronous database writes and heavy middleware run on that exact same single core, creating a massive bottleneck due to Python's GIL and I/O blocking.

You shouldn't run a single-core production proxy if you expect high throughput. If your pod has more vCPUs available, definitely bump your workers up to match (--num_workers 4 or whatever your core count is) to actually get close to those 1,000+ QPS benchmark numbers.