v0.6.32 — Muse Glimmer 8K c16 351 tok/s, 2.5 s first token; leads llama.cpp on every Muse axis

Muse Glimmer at 16 concurrent 8K-token prompts now runs at 351 tok/s with a 2.5 s first token, up from 281 tok/s and 9.7 s; llama.cpp gives 76 tok/s and 7.9 s. sparkinfer now beats llama.cpp on every Muse Glimmer test we run, including 32K–120K-token prompts, where its first token arrives 3.6–4.2x sooner.

The server sizes its KV cache (the memory for active requests) at startup and keeps 6 GiB of GPU memory free as headroom. On Muse Glimmer, most of that headroom was later taken by extra copies of some weights that speed up prompt processing, and they speed it up very little. Muse now keeps 4 GiB free instead (#1328):

• More concurrent long requests: the KV cache holds 151K tokens instead of 72K (--ctx 32768), enough for all 16 requests in the 8K test at once.
• Prompt processing just as fast: 196 tok/s with 8K prompts at 4 requests, unchanged.
• Scope: other models keep 6 GiB, and SPARKINFER_KV_HEADROOM_GIB still sets it for any model.

Measured

RTX 5090, AIPerf streaming chat, reasoning tokens counted. Output tok/s:

Muse Glimmer 30B	chat, 4 requests	16	32	8K prompts, 4	8K prompts, 16
0.6.31	274	663	838	196	281
0.6.32	274	665	847	196	351
llama.cpp 436f6f8, same GGUF	189	363	411	89	76

Long prompts, one request at a time, --ctx 131072. Time to first token at 32K / 64K / 120K tokens: 2.15 / 4.84 / 10.45 s against llama.cpp's 8.99 / 18.9 / 38.1 s. Time between output tokens: 9.5–9.9 ms against 12.8–13.7 ms.

Benchmarks: all three scripts in bench/competitors/serving/ now take AIP_EXTRA, and longctx.sh can run llama.cpp as the competitor.

What's Changed

• perf(server): Muse Glimmer's KV pool grows into 4 GiB of headroom instead of 6 (8K c16 281 -> 351 tok/s, TTFT 9.7 -> 2.5 s) by @skyrocket2026 in #1328
• docs: 0.6.32 — Muse Glimmer 8K c16 351 tok/s, 2.5 s first token; leads llama.cpp on every Muse axis by @skyrocket2026 in #1329

Full Changelog: v0.6.31...v0.6.32