Add LLM Gateway to the AI gateway benchmark

[smakosh]

What this adds

LLM Gateway as a fifth participant in the AI gateway benchmark:

• src/ai-gateway/providers.ts — provider entry: OpenAI wire format, api.llmgateway.io, model anthropic/claude-haiku-4-5. The anthropic/ prefix is LLM Gateway's provider-pinning syntax (provider/model), so requests route to Anthropic itself — same underlying model and deployment as the other participants, satisfying the "every gateway is hit directly via its own native routing" fairness rule.
• src/ai-gateway/generate-svg.ts — display label.
• env.example — LLMGATEWAY_API_KEY. Missing key → reported as SKIPPED, as with the other gateways.
• package.json — bench:ai-gateway:llmgateway filter script.
• AI_GATEWAYS.md — participant list, model naming, wire formats, fairness section, round-robin example, and env var docs updated.

No committed results are touched — happy to leave publishing numbers to a run from your usual vantage point once LLMGATEWAY_API_KEY is added to the runner env.

Token-extraction fix (second commit)

While validating, LLM Gateway scored ~1 tok/s: extractOutputTokens on the openai path takes the first "usage":{..."completion_tokens":N} match in the SSE buffer, but some OpenAI-compatible gateways stream cumulative usage on early events (mirroring Anthropic's message_start) rather than only on the final chunk. The fix takes the last match — consistent with the anthropic path directly below it and with the function's own docstring ("latest known output-token count"). No behavior change for streams that only emit usage on the final chunk (OpenRouter, Vercel).

Local validation

npm run bench:ai-gateway (10 cold + 10 warm, other gateways skipped for lack of keys), from a residential vantage point:

llmgateway:
  Cold E2E TTFT: 916ms (median), Warm TTFT: 812ms (median), 84.4 tok/s
  Score: 89.4 (20/20 OK)

Phase medians: DNS 6ms, TCP 20ms, TLS 39ms, cold TTFB 271ms, warm TTFB 197ms.

Disclosure: I work on LLM Gateway.

https://claude.ai/code/session_01DLdncZM43JVsEEhYkEhby2

[open-cla]

Contributor License Agreement

All contributors are covered by a CLA.

[github-actions]

Storage Benchmark Results

1MB Files

#	Provider	Score	Download	Throughput	Upload	Status
1	Azure Blob Storage	96.3	0.03s	313.7 Mbps	0.08s	10/10
2	Tigris	95.2	0.04s	206.1 Mbps	0.47s	10/10
3	Vercel Blob	95.1	0.06s	134.9 Mbps	0.27s	10/10
4	AWS S3	94.3	0.18s	47.5 Mbps	0.32s	10/10
5	Cloudflare R2	94.3	0.16s	54.6 Mbps	0.38s	10/10

---

View full run · SVGs available as build artifacts

[github-actions]

Browser Throughput Benchmark Results

#	Provider	Score	APS (med)	Task (med)	Task (p95)	Screenshot	Status
1	Kernel	83.2	6.45/s	1.55s	1.73s	232ms	3/3
2	Browseruse	70.2	3.76/s	2.66s	3.79s	282ms	100/100
3	Hyperbrowser	68.1	3.25/s	3.08s	3.21s	467ms	3/3
4	Browserbase	61.4	2.29/s	4.36s	5.87s	427ms	100/100
5	Notte	50.7	1.22/s	8.19s	10.22s	1143ms	100/100
6	Steel	50.4	1.32/s	7.59s	12.34s	727ms	100/100

---

View full run · SVG available as build artifact

[github-actions]

Browser Benchmark Results

#	Provider	Score	Create	Connect	Navigate	Release	Total	Status
1	Kernel	98.0	0.02s	0.06s	0.11s	0.04s	0.23s	10/10
2	Browserbase	94.0	0.24s	0.13s	0.13s	0.14s	0.65s	10/10
3	Hyperbrowser	93.6	0.33s	0.13s	0.16s	0.09s	0.74s	10/10
4	Notte	90.9	0.19s	0.51s	0.21s	0.09s	1.08s	10/10
5	Browseruse	88.7	0.87s	0.16s	0.07s	0.06s	1.20s	10/10
6	Steel	84.7	0.29s	0.69s	0.15s	0.16s	1.47s	10/10

---

View full run · SVG available as build artifact

[github-actions]

Snapshot/Fork Benchmark Results

small dataset

#	Provider	Score	Snapshot create	Fork (snapshot)	Fork (live)	First read	Status
1	Tigris	99.6	0.12s	0.36s	0.16s	0.20s	1/1
2	Azure-blob	99.0	0.66s	0.60s	0.65s	0.02s	1/1
3	Cloudflare R2	95.3	3.33s	2.94s	2.92s	0.13s	1/1
4	AWS S3	0.0	0.00s	0.00s	0.00s	0.00s	0/1

---

View full run

[github-actions]

Sandbox Benchmark Results

Sequential

#	Provider	Score	Median TTI	P95	P99	Status
1	createos	99.3	0.05s	0.11s	0.11s	10/10
2	declaw	99.2	0.04s	0.13s	0.13s	10/10
3	isorun	99.0	0.01s	0.23s	0.23s	10/10
4	northflank	98.6	0.10s	0.20s	0.20s	10/10
5	archil	98.0	0.15s	0.29s	0.29s	10/10
6	lightning	97.7	0.19s	0.30s	0.30s	10/10
7	blaxel	96.9	0.23s	0.42s	0.42s	10/10
8	superserve	96.7	0.29s	0.39s	0.39s	10/10
9	upstash	95.1	0.42s	0.60s	0.60s	10/10
10	beam	94.0	0.16s	1.27s	1.27s	10/10
11	runloop	93.2	0.57s	0.84s	0.84s	10/10
12	e2b	90.3	0.82s	1.19s	1.19s	10/10
13	vercel	90.1	0.42s	1.86s	1.86s	10/10
14	tensorlake	89.7	0.63s	1.63s	1.63s	10/10
15	tenki	87.3	1.02s	1.65s	1.65s	10/10
16	cloudflare	83.4	1.59s	1.78s	1.78s	10/10
17	cloud-run	83.1	0.54s	3.42s	3.42s	10/10
18	daytona	79.0	0.94s	3.85s	3.85s	10/10
19	modal	77.0	0.49s	5.00s	5.00s	10/10
20	codesandbox	46.7	2.21s	22.02s	22.02s	10/10
21	hopx	0.0	0.00s	0.00s	0.00s	0/10

Staggered

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.8	0.02s	0.03s	0.03s	10/10
2	createos	99.5	0.05s	0.05s	0.05s	10/10
3	declaw	99.4	0.05s	0.07s	0.07s	10/10
4	northflank	99.0	0.07s	0.14s	0.14s	10/10
5	archil	98.5	0.11s	0.20s	0.20s	10/10
6	lightning	98.1	0.18s	0.21s	0.21s	10/10
7	beam	97.9	0.16s	0.29s	0.29s	10/10
8	blaxel	97.8	0.21s	0.24s	0.24s	10/10
9	superserve	97.3	0.25s	0.29s	0.29s	10/10
10	upstash	95.7	0.42s	0.45s	0.45s	10/10
11	tensorlake	94.1	0.56s	0.63s	0.63s	10/10
12	vercel	94.1	0.37s	0.92s	0.92s	10/10
13	modal	93.8	0.49s	0.82s	0.82s	10/10
14	runloop	93.7	0.58s	0.70s	0.70s	10/10
15	e2b	89.9	0.95s	1.10s	1.10s	10/10
16	tenki	89.8	1.00s	1.05s	1.05s	10/10
17	cloud-run	83.3	0.55s	3.34s	3.34s	10/10
18	daytona	81.4	1.08s	3.02s	3.02s	10/10
19	cloudflare	79.4	1.69s	2.62s	2.62s	10/10
20	codesandbox	72.8	2.42s	3.17s	3.17s	10/10
21	hopx	0.0	0.00s	0.00s	0.00s	0/10

Burst

#	Provider	Score	Median TTI	P95	P99	Status
1	isorun	99.8	0.02s	0.02s	0.02s	10/10
2	createos	99.3	0.07s	0.08s	0.08s	10/10
3	declaw	99.3	0.07s	0.08s	0.08s	10/10
4	northflank	98.8	0.11s	0.12s	0.12s	10/10
5	archil	98.3	0.15s	0.21s	0.21s	10/10
6	beam	98.1	0.18s	0.20s	0.20s	10/10
7	lightning	98.0	0.19s	0.20s	0.20s	10/10
8	blaxel	97.5	0.22s	0.28s	0.28s	10/10
9	superserve	96.6	0.32s	0.37s	0.37s	10/10
10	upstash	94.6	0.52s	0.57s	0.57s	10/10
11	runloop	94.0	0.59s	0.63s	0.63s	10/10
12	modal	93.8	0.55s	0.73s	0.73s	10/10
13	tensorlake	93.0	0.67s	0.75s	0.75s	10/10
14	cloud-run	91.4	0.61s	1.24s	1.24s	10/10
15	e2b	91.1	0.82s	1.00s	1.00s	10/10
16	vercel	91.0	0.37s	1.69s	1.69s	10/10
17	daytona	81.6	1.05s	3.03s	3.03s	10/10
18	codesandbox	69.0	2.91s	3.37s	3.37s	10/10
19	cloudflare	67.0	3.15s	3.52s	3.52s	10/10
20	tenki	60.0	3.99s	4.02s	4.02s	10/10
21	hopx	0.0	0.00s	0.00s	0.00s	0/10

---

View full run · SVGs available as build artifacts

[github-actions]

Sandbox Dax Benchmark Results

Provider	Phases	Total	Prepare	Clone	Install	Typecheck	Status
archil	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
beam	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
blaxel	6/7	54.11s	2.04s	2.12s	11.53s	0.00s	0/1 OK
cloud-run	2/7	1.80s	0.45s	0.00s	0.00s	0.00s	0/1 OK
cloudflare	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
codesandbox	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
createos	7/7	48.78s	2.52s	2.57s	12.14s	26.30s	1/1 OK
daytona	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
declaw	7/7	92.53s	8.78s	6.88s	19.41s	39.41s	1/1 OK
e2b	7/7	72.43s	7.71s	5.27s	9.86s	40.91s	1/1 OK
hopx	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
isorun	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
lightning	0/7	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
modal	7/7	98.74s	4.47s	2.40s	15.44s	67.76s	1/1 OK
namespace	7/7	52.92s	5.35s	1.56s	11.97s	27.33s	1/1 OK
northflank	--	0.00s	0.00s	0.00s	0.00s	0.00s	0/1 OK
runloop	2/7	124.08s	0.00s	0.00s	27.24s	53.33s	1/1 OK
superserve	6/7	29.53s	20.44s	3.20s	3.54s	0.00s	0/1 OK
tenki	6/7	25.35s	5.93s	3.96s	11.51s	0.00s	0/1 OK
tensorlake	7/7	83.16s	12.17s	1.70s	14.96s	47.34s	1/1 OK
upstash	7/7	53.55s	3.82s	2.87s	10.72s	25.81s	1/1 OK
vercel	7/7	73.07s	16.30s	2.19s	12.08s	35.85s	1/1 OK

---

View full run