v0.6.20 — Qwen3.6 serving +12% / +13% at 16 / 32 requests

Qwen3.6 serving at 16 / 32 concurrent requests: +12% / +13%.

Three changes to how a busy Qwen3.6-35B-A3B server spends its GPU time:

• Bigger mixed steps for MoE models. While requests decode, waiting prompts ride along in the same forward pass. On an MoE model those prompt tokens touch nearly every expert, so each such step reads the whole expert set (~19.5 GB) whatever its size. MoE models now take up to 4,096 prompt tokens per step instead of 1,024, so that read is paid a quarter as often. Dense models keep 1,024.
• The shared expert runs on the int8 tensor cores. At 32 concurrent requests, Qwen3.6's shared expert took about 110 µs of every layer, a third of a decode step's GPU time. It now uses int8 tensor-core instructions (exact per block, deterministic).
• The prompt scratch memory is kept. Large mixed steps used to free and re-allocate up to ~1.8 GB of scratch on every pass. Each free stalls the whole GPU. It is now kept up to 2 GB.

Measured

RTX 5090, Qwen3.6-35B-A3B UD-Q4_K_M. AIPerf streaming chat with 1,024-token prompts, 256 output tokens with ignore_eos, and a distinct prompt set per cell. Output tok/s:

requests	0.6.19	0.6.20	vLLM 0.30.0 (nvidia NVFP4)	llama-server
4	902	906	666	339
16	1,355	1,517	1,686	372
32	1,523	1,716	2,367	404

vLLM still leads at 16 and 32 concurrent requests. sparkinfer leads at 4 requests, and with 8K-token prompts at 4 requests (494 vs 450 tok/s). Closing the gap at high concurrency is the current work.

What's Changed

• perf(runtime): an MoE model's mixed steps carry up to 4096 prompt tokens by @skyrocket2026 in #1294
• perf(kernels): the Qwen3.6 Q8_0 shared expert runs a packed step's rows on the int8 tensor cores by @skyrocket2026 in #1295
• perf(runtime): the prefill arena is kept up to 2 GB by @skyrocket2026 in #1296
• docs: 0.6.20 — Qwen3.6 serving +12% / +13% at c16 / c32 by @skyrocket2026 in #1297

Full Changelog: v0.6.19...v0.6.20