Skip to content

v0.6.9 — fast odd-bpw E81B decode + vLLM 0.25

Choose a tag to compare

@cnygaard cnygaard released this 12 Jul 11:53
d5ac29f

Fast odd-bpw (3/5/7) decode

The trailing E81B residual stage now uses SM-saturating split-K GEMM/GEMV kernels (+ M-parallel grouped-MoE) with a smem-resident codebook, replacing the 1-warp lookup-matmul — odd-bpw decode was ~7–8× slower than even-bpw, now near parity. Bit-exact + deterministic; existing odd-bpw checkpoints work unchanged.

vLLM 0.25 support

glq_vllm detects MoE layers via RoutedExperts (0.25 split FusedMoE into a factory function), version-guarded for ≤0.23. Serves coherent across 26B-A4B MoE, SmolLM2/3, and Gemma-4 E4B/12B/31B.

Fixes

Mixed-precision (--min-bpw/--max-bpw) profiling KeyError on the e8p codebook.

pip install glq