v0.6.4 — time to first token at 16-32 concurrent chats matches vLLM

Time to first token at 16-32 concurrent chats now matches vLLM.

Until this release, the first token at 16 and 32 concurrent chats arrived later than vLLM's (p50 591 / 1,183 ms against 360 / 356). Waiting prompts queued behind each other while the decode batch ran. Now prefill and decode share one forward pass by default, and one pass carries several waiting prompts.

Measured

AIPerf, chat (1024-token prompts, 256-token answers), RTX 5090, Qwen3.8-27B NVFP4 as the release container ships it (DFlash2 drafter loaded). vLLM 0.30 runs without a drafter.

	TTFT p50 / p90 / p99 (ms)	output tok/s
16 chats: 0.6.3 → 0.6.4	591 / 1,255 / 1,889 → 355 / 1,240 / 1,845	861 → 858
16 chats: vLLM 0.30	360 / 1,287 / 1,916	753
32 chats: 0.6.3 → 0.6.4	1,183 / 2,712 / 3,372 → 372 / 2,602 / 3,386	1,239 → 1,266
32 chats: vLLM 0.30	356 / 2,811 / 3,815	1,031

• Other benchmark cells: the other 9 of the 12 AIPerf cells are within run-to-run variance of 0.6.3.
• Per-token latency: at 16 / 32 chats it rises from 15.4 / 18.9 to 17.8 / 22.3 ms, still below vLLM's 19.8 / 29.2.
• Poisson arrivals (eval/spec_open_arrivals.py): mean latency 2.91 → 2.82 s at 1 request/s and 4.67 → 4.58 s at 2.

What changed

• One forward pass carries several waiting prompts (#1251). It fills a budget of 1024 prompt tokens (SPARKINFER_MIXED_CHUNK) across up to 8 waiting prompts, oldest first, beside the requests already decoding.
• A prompt that ends in that pass takes its first token from it. Before, every prompt ended with a separate forward of its own.
• Where mixing does not apply:
  • Prompts with more than 2048 tokens left keep their own prefill pass, which is faster for them.
  • Loads of 8 requests or fewer are left to speculation.
• Turning it off: SPARKINFER_MIXED_CHUNK=0 restores 0.6.3's behaviour.

Full notes: CHANGELOG.md.

Container: ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:0.6.4 (attested).

What's Changed

• perf(runtime): mixed steps carry several prompts and are on by default (chat c32 TTFT p50 1183 -> 372 ms; vLLM 356) by @skyrocket2026 in #1251
• docs: 0.6.4 — time to first token at 16-32 concurrent chats matches vLLM (chat c32 p50 1183 -> 372 ms) by @skyrocket2026 in #1253

Full Changelog: v0.6.3...v0.6.4