Repository navigation
v0.10.0 - GLM-4, faster decode, native quantizer
·
277 commits
to 4cc41951f5de5c9aa1a2c526c5c85f672c01c23c
since this release
KrillLM v0.10.0 brings the GLM-4 family, faster decode across the board, and a fully native (Python-free) quantizer that now covers MoE, vision, and Gemma.
Added
- Native Swift+MLX GLM-4-0414 / GLM-Z1 runtime (
glm-4-9b-0414,glm-z1-9b,glm-4-32b-0414), gated argmax-exact against an mlx-lm logit oracle. krillm quantizeis now 100% Swift+MLX - the python3/mlx_lm.convert shell-out is gone, so the shipped binary has no Python anywhere. Dense quantization is byte-identical to the canonical MLX op;--mode affine|nvfp4|mxfp4|mxfp8,--dtypedefault fp16.krillm quantize --reference <a 4-bit build>extends the quantizer to MoE / vision / Gemma. It learns the exact quantized-module set from the reference checkpoint, so MoE (experts affine at the top-level group, float router gate preserved), vision (float vision tower preserved), and Gemma (PLE / tied-head) all work with no per-family code.--protectraises chosen modules to a higher precision (the Gemma vision/audio projectors auto-protect at 8-bit affine for color fidelity). gemma-4-12b nvfp4 + 8-bit projectors verifies 1012/1012 byte-identical.
Changed
- Single-stream decode overlaps CPU graph-build with GPU compute: +13-23% tok/s across dense families, byte-exact, default-on (
KRILL_DECODE_PIPELINE=0to disable). - The concurrent batcher overlaps CPU/GPU the same way: +9-11% throughput at N=8, byte-exact.
- Adaptive n-gram (prompt-lookup) speculative decode is on by default on both the single stream and the batcher, with a stall monitor that hands off to the byte-exact overlap path so non-echo workloads are not penalized (
KRILL_NGRAM_SPEC=0to disable).
Fixed
- Gemma-4 media markers (
<start_of_image>...<end_of_image>, audio) no longer leak into visible output on any server or CLI path.
Install: brew tap srvsngh99/krillm && brew install krillm (or brew upgrade krillm)
Asset: krillm-0.10.0-arm64-apple-macos.tar.gz (Apple Silicon). Full changelog: the [0.10.0] section in CHANGELOG.md.