Multi-user pipeline benchmark: 200 requests, 5ms proxy overhead, 76Mi memory

[yossiovadia]

Summary

Ran a multi-user load test through a full praxis-ai filter pipeline deployed on OpenShift (IBM Cloud, OCP 4.21). 20 concurrent users, 10 multi-turn conversations each, realistic payloads with growing context.

Backend: llm-katan echo server with simulated latency (--ttft-ms=800 --itl-ms=15) — zero provider cost.

Pipeline

Every request goes through the complete filter chain:

api_key_auth → identity_header_guard → router → external_metering → token_count → token_usage_headers → headers → load_balancer

• api_key_auth: validates API key via HTTP callout to key management service
• identity_header_guard: captures identity headers to filter_metadata, strips before upstream
• external_metering: sends CloudEvents usage reports to metering service
• token_count: extracts token usage from response body
• credential_injection: swaps client key for provider key

Results

Metric	Value
Total requests	200
Success rate	100%
Concurrent users	20
Turns per user	10
Duration	27.4s
Throughput	7.3 req/s

Latency

Percentile	Value
avg	1,768ms
p50	1,778ms
p95	2,246ms
p99	2,435ms
min	1,164ms
max	2,626ms

Note: llm-katan adds 800ms simulated TTFT per request. Subtract that for proxy-only latency.

Proxy overhead (from Prometheus metrics)

Metric	Value
Praxis avg total latency	805ms
llm-katan TTFT	800ms
Proxy pipeline overhead	~5ms

The full 8-filter pipeline — including an HTTP callout for auth validation and an async metering report — adds approximately 5 milliseconds of overhead per request.

Resource usage under load

Component	CPU	Memory
Praxis proxy	9m	76Mi
llm-katan (backend)	15m	36Mi
maas-api (key mgmt)	2m	17Mi
metering-service	7m	14Mi
PostgreSQL	11m	47Mi

Memory is stable — 74Mi at idle, 76Mi under 20 concurrent users. No growth, no leaks.

Scaling (concurrent users vs latency)

Concurrent	Requests	Success	p50	p95	Throughput	Praxis CPU	Praxis Mem
5	25	100%	1,304ms	1,415ms	3.0 req/s	1m	70Mi
10	50	100%	1,388ms	1,607ms	5.1 req/s	2m	72Mi
20	100	100%	1,689ms	2,130ms	7.4 req/s	3m	72Mi
30	150	100%	2,277ms	2,990ms	7.8 req/s	3m	74Mi
50	250	100%	3,394ms	4,721ms	7.6 req/s	9m	75Mi

Throughput plateaus at ~7.5 req/s — this is the llm-katan backend limit (single-threaded Python with 800ms TTFT), not Praxis. Praxis CPU scales linearly, memory stays flat.

Setup

• Cluster: IBM Cloud ROKS, OCP 4.21, 3x bxf.4x16 workers
• Praxis image: built from feat/dogfood-gateway branch on yossiovadia/ai
• Backend: llm-katan v0.21.0 (PyPI) deployed on-cluster
• Payloads: Multi-turn Anthropic Messages API format, ~600-2400 tokens per request (growing conversation history)
• Benchmark script: scripts/benchmark.sh in the dogfood branch

Reproduction

The benchmark script and deployment manifests are on the feat/dogfood-gateway branch. llm-katan is pip install llm-katan — starts in milliseconds, zero GPU required.

Notes

• llm-katan is a format-faithful multi-provider test backend (OpenAI, Anthropic, Vertex, Azure, Bedrock) designed for exactly this kind of pipeline testing
• The TTFT/ITL simulation flags make latency numbers comparable to real provider deployments
• The metering dashboard shows per-user token attribution in real time during the benchmark

[yossiovadia]