Skip to content

v0.60.0: Gemma 4, Qwen3-Coder and Nemotron read long prompts up to 6 % faster

Choose a tag to compare

@kekzl kekzl released this 31 Oct 16:39
· 243 commits to main since this release
ed4422b

Highlights

  • imp now uses its own math kernels on these models. Long prompts and pasted documents are processed faster, answers come out at the same speed as before. Measured v0.59.2 vs v0.60.0, RTX 5090, NVFP4, 3 alternating pairs each (tools/analysis/release_ab.sh):

    Model long prompt (4096 tokens) short prompt (512) answer speed
    Gemma-4-26B-A4B +6.0 % +5.2 % +0.3 %
    Qwen3-Coder-30B-A3B +1.0 % +5.4 % -0.7 %
    Nemotron-3-Nano-30B-A3B +0.6 % +3.8 % +0.3 %
    Qwen3-14B +2.2 % +1.6 % +0.1 %
    Qwen3-8B (NVFP4 / FP8) +1.4 % / +2.2 % +1.9 % / -0.1 % -0.2 % / -0.1 %
    Qwen3.8-27B +1.8 % +1.1 % 0.0 %
  • Qwen3.8-Flash-Next (123.6 GiB on one 32 GB card) answers at the same speed as before: 91.8 tokens/s on the server (v0.59.0: 92.1); direct runs on 3 prompts +4.8 % / +2.1 % / -0.8 % (tools/analysis/flashnext_tg_ab.sh).

  • CUTLASS, cuSOLVER and the cuBLAS v2 API are out of the build; imp links cuBLASLt only (#2938, #2949, #2950).

  • Answer speed vs vLLM 0.31.0 (one stream, speculation off on both): imp leads on all 6 models vLLM serves, +27.6 % (Qwen3-14B) to +70.0 % (Qwen3-Coder-30B-A3B); vLLM cannot start Nemotron-3-Nano.

  • /v1/embeddings on Qwen3-Embedding returns correct vectors: cosine vs Hugging Face 0.23-0.57 -> 0.999993+ (#2925).

Also in here

  • NVFP4 MoE with experts in host RAM: KV pool no longer stuck at 16 blocks, Qwen3-30B-A3B-NVFP4 with 48 host layers tg128 0 -> 304.5 tok/s (#2909).
  • MTP drafts on GGUF models with a Q6_K head no longer crash (#2917).
  • GGUF is legacy: it keeps loading, new speedups go to SafeTensors NVFP4 / FP8; vLLM is the reference engine.
  • Hero set: Qwen3.8-Flash-Next in, Qwen3.6-35B-A3B out (#2954).

Gate

make verify-fast, Qwen3-8B-NVFP4-cortecs, RTX 5090 32 GB, same src/ as the tag: tg128 279.98 tok/s (pin 281.35, -0.49 %), pp512 45393.58 (-1.00 %), pp4096 45001.70 (+0.15 %), own peak 6140 MiB (0.00 %), PPL 11.2949 (0.000 %), server c8 1848.38 tok/s (-0.36 %). check-release.sh: all gates PASS (check-alloc-interpose re-run on port 8199, 8099 held by an unrelated container).

No breaking change with default settings: the Docker image and a plain imp-server --model ... run as before. Breaking only if you set one of these yourself:

  • --mxfp4-prefill, or --set with attention.mxfp4*, gemm.moe_tile, gemm.mxfp8_tile: startup now stops with "unknown"; drop the option. In imp.conf the same keys only log a warning and are ignored.
  • C API: ImpConfig::use_mxfp4_prefill is gone (ABI change); programs linking the C API rebuild against the new header.

Full detail: CHANGELOG. 66 PRs since v0.59.2.