Repository navigation
v0.62.0: Gemma 4 writes 11 % faster, long prompts up to 7 % faster, Kimi-Linear runs
Highlights
- Gemma 4 writes 11.5 % faster: Gemma-4-26B-A4B NVFP4 tg128 271.74 -> 302.99 tok/s vs v0.61.0 (median of 3, RTX 5090), from a fused FP32-residual norm chain (#3012), a weight-streaming small-M GEMM (#3011) and GeGLU + quantize in one kernel (#3022).
- Many users at once got faster: Qwen3-8B FP8 at 16 streams 1759.9-1764.0 -> 2595.3-2599.4 tok/s (#3004), NVFP4 MoE at 16 streams Gemma-4-26B 1840 -> 2065, Qwen3-30B-A3B 1804 -> 2073 tok/s (#3008).
- Long prompts read up to 6.7 % faster: pp4096 vs v0.61.0 Qwen3-14B NVFP4 +6.74 %, Qwen3-8B NVFP4 +6.22 %, Qwen3.8-27B NVFP4 +4.71 % (#3024, #2985).
- Better NVFP4 answers: W4A4 activations use the checkpoint's static input scale, Qwen3-30B-A3B PPL 11.3565 -> 10.9678 (#3015, #3019). Measured against vLLM 0.31.0 on the same checkpoints: Qwen3 within 0.24 %, Gemma-4-26B 1.75 % lower (#3021).
- Kimi-Linear (
kimi_linear) runs from the Firworks NVFP4 checkpoint, experts in host RAM, pp512 2000 / tg128 316 tok/s; vLLM 0.31.0 runs out of memory on it (#2984).
Also in here
- First request answers sooner: Gemma-4-26B first-request TTFT 12.4 -> 6.6 ms, KV pool growth off the admission path (#3017, #3018).
- Fixed: Nemotron-3-Nano at 2+ concurrent requests replied "one,,,,,," (#3009); NVFP4 MoE and FP8 prefix-cache resends answer bit-equal to the first send (#3002, #3005, #3006).
- New opt-in FP8 KV scales:
kv_cache.fp8_scaleandkv_cache.fp8_scale_split(Gemma-4 FP8 KV PPL over FP16 +1.92 % -> +0.69 %) (#3016, #3023). - Measured and refuted, off by default: GPTAQ and residual rotation for NVFP4 exports, TriAttention KV eviction (#2986-#2993).
- Slower: Qwen3-Coder-30B-A3B FP4 pp512 33618.23 -> 32747.95 tok/s (-2.59 %, cause not bisected); its pp4096 +0.82 %.
- Dependency bumps (#2994-#2997).
Gate
make verify-fast on the tagged tree, Qwen3-8B-NVFP4-cortecs, RTX 5090 (32 GB):
decode tg128 = 283.30 tok/s (baseline 282.84, delta 0.16%)
prefill pp512 = 50402.83 tok/s (baseline 50374.39, delta 0.06%)
prefill pp4096 = 50754.14 tok/s (baseline 50318.99, delta 0.86%, FA2)
own peak VRAM = 6140 MiB (baseline 6140, delta 0.00%)
PPL ppl_corpus_45k = 11.2352 (baseline 11.2352, delta 0.000%)
graphs ON tg256 = 287.00 tok/s (1.51x, threshold 1.3x)
server c8 = 1855.07 tok/s, slowest request 543 ms (baseline 1899.81, delta -2.35%)
=== verify fast: OK ===
scripts/check-release.sh: all gates passed; release bar 2: 7/9 heroes contested vs vLLM 0.31.0, all above +5 %.
No breaking changes.
Full detail: CHANGELOG. 42 PRs since v0.61.0.