Skip to content

v0.10.0 - GLM-4, faster decode, native quantizer

Choose a tag to compare

@srvsngh99 srvsngh99 released this 18 Jun 20:36
· 277 commits to 4cc41951f5de5c9aa1a2c526c5c85f672c01c23c since this release

KrillLM v0.10.0 brings the GLM-4 family, faster decode across the board, and a fully native (Python-free) quantizer that now covers MoE, vision, and Gemma.

Added

  • Native Swift+MLX GLM-4-0414 / GLM-Z1 runtime (glm-4-9b-0414, glm-z1-9b, glm-4-32b-0414), gated argmax-exact against an mlx-lm logit oracle.
  • krillm quantize is now 100% Swift+MLX - the python3/mlx_lm.convert shell-out is gone, so the shipped binary has no Python anywhere. Dense quantization is byte-identical to the canonical MLX op; --mode affine|nvfp4|mxfp4|mxfp8, --dtype default fp16.
  • krillm quantize --reference <a 4-bit build> extends the quantizer to MoE / vision / Gemma. It learns the exact quantized-module set from the reference checkpoint, so MoE (experts affine at the top-level group, float router gate preserved), vision (float vision tower preserved), and Gemma (PLE / tied-head) all work with no per-family code. --protect raises chosen modules to a higher precision (the Gemma vision/audio projectors auto-protect at 8-bit affine for color fidelity). gemma-4-12b nvfp4 + 8-bit projectors verifies 1012/1012 byte-identical.

Changed

  • Single-stream decode overlaps CPU graph-build with GPU compute: +13-23% tok/s across dense families, byte-exact, default-on (KRILL_DECODE_PIPELINE=0 to disable).
  • The concurrent batcher overlaps CPU/GPU the same way: +9-11% throughput at N=8, byte-exact.
  • Adaptive n-gram (prompt-lookup) speculative decode is on by default on both the single stream and the batcher, with a stall monitor that hands off to the byte-exact overlap path so non-echo workloads are not penalized (KRILL_NGRAM_SPEC=0 to disable).

Fixed

  • Gemma-4 media markers (<start_of_image> ... <end_of_image>, audio) no longer leak into visible output on any server or CLI path.

Install: brew tap srvsngh99/krillm && brew install krillm (or brew upgrade krillm)
Asset: krillm-0.10.0-arm64-apple-macos.tar.gz (Apple Silicon). Full changelog: the [0.10.0] section in CHANGELOG.md.