Skip to content

v0.62.0: Gemma 4 writes 11 % faster, long prompts up to 7 % faster, Kimi-Linear runs

Choose a tag to compare

@kekzl kekzl released this 03 Nov 09:14
· 142 commits to main since this release
1cc8e45

Highlights

  • Gemma 4 writes 11.5 % faster: Gemma-4-26B-A4B NVFP4 tg128 271.74 -> 302.99 tok/s vs v0.61.0 (median of 3, RTX 5090), from a fused FP32-residual norm chain (#3012), a weight-streaming small-M GEMM (#3011) and GeGLU + quantize in one kernel (#3022).
  • Many users at once got faster: Qwen3-8B FP8 at 16 streams 1759.9-1764.0 -> 2595.3-2599.4 tok/s (#3004), NVFP4 MoE at 16 streams Gemma-4-26B 1840 -> 2065, Qwen3-30B-A3B 1804 -> 2073 tok/s (#3008).
  • Long prompts read up to 6.7 % faster: pp4096 vs v0.61.0 Qwen3-14B NVFP4 +6.74 %, Qwen3-8B NVFP4 +6.22 %, Qwen3.8-27B NVFP4 +4.71 % (#3024, #2985).
  • Better NVFP4 answers: W4A4 activations use the checkpoint's static input scale, Qwen3-30B-A3B PPL 11.3565 -> 10.9678 (#3015, #3019). Measured against vLLM 0.31.0 on the same checkpoints: Qwen3 within 0.24 %, Gemma-4-26B 1.75 % lower (#3021).
  • Kimi-Linear (kimi_linear) runs from the Firworks NVFP4 checkpoint, experts in host RAM, pp512 2000 / tg128 316 tok/s; vLLM 0.31.0 runs out of memory on it (#2984).

Also in here

  • First request answers sooner: Gemma-4-26B first-request TTFT 12.4 -> 6.6 ms, KV pool growth off the admission path (#3017, #3018).
  • Fixed: Nemotron-3-Nano at 2+ concurrent requests replied "one,,,,,," (#3009); NVFP4 MoE and FP8 prefix-cache resends answer bit-equal to the first send (#3002, #3005, #3006).
  • New opt-in FP8 KV scales: kv_cache.fp8_scale and kv_cache.fp8_scale_split (Gemma-4 FP8 KV PPL over FP16 +1.92 % -> +0.69 %) (#3016, #3023).
  • Measured and refuted, off by default: GPTAQ and residual rotation for NVFP4 exports, TriAttention KV eviction (#2986-#2993).
  • Slower: Qwen3-Coder-30B-A3B FP4 pp512 33618.23 -> 32747.95 tok/s (-2.59 %, cause not bisected); its pp4096 +0.82 %.
  • Dependency bumps (#2994-#2997).

Gate

make verify-fast on the tagged tree, Qwen3-8B-NVFP4-cortecs, RTX 5090 (32 GB):

decode  tg128 =  283.30 tok/s  (baseline  282.84, delta 0.16%)
prefill pp512 = 50402.83 tok/s  (baseline 50374.39, delta 0.06%)
prefill pp4096 = 50754.14 tok/s  (baseline 50318.99, delta 0.86%, FA2)
own peak VRAM = 6140 MiB  (baseline 6140, delta 0.00%)
PPL ppl_corpus_45k = 11.2352  (baseline 11.2352, delta 0.000%)
graphs ON  tg256 =  287.00 tok/s   (1.51x, threshold 1.3x)
server c8 = 1855.07 tok/s, slowest request 543 ms  (baseline 1899.81, delta -2.35%)
=== verify fast: OK ===

scripts/check-release.sh: all gates passed; release bar 2: 7/9 heroes contested vs vLLM 0.31.0, all above +5 %.

No breaking changes.

Full detail: CHANGELOG. 42 PRs since v0.61.0.