Skip to content

Lumen v0.4.0 — shape-aware VNNI/fp32 hybrid dispatch

Choose a tag to compare

View all tags
@redchupa redchupa released this 18 May 00:13
· 22 commits to main since this release
v0.4.0
a00970e

v0.4.0 ships the result of four measurement-led phases (7.M → 7.P): three attempts to push int-dot product wins through, each disproved by bench, and a final hybrid dispatch that combines them all into the actual win. Tokens stay bit-identical to the naive Rust reference across every commit.

tg32 (Qwen2.5-0.5B-Q8_0, Windows, 8 worker threads on Zen 4)

Path tok/s vs ggml -t 8
Lumen v0.3.0 (Q8×F32 4-acc, custom pool) ~60 1.51× slower
Lumen v0.4.0 (per-shape VNNI/fp32 hybrid) ~65 1.39× slower
ggml -t 8 90.90 1.0×

Sanity: Lumen 8t (65) is 1.58× faster than ggml -t 1 (41.32).

v0.1.0 → v0.4.0 spread: 4.43 → 65.3 tok/s = 14.7× decode in ~5 days.

What landed (and why the path zig-zagged)

Phase 7.M — Q8×Q8 AVX2 int dot

Built the full Q8×Q8 fused matmul infrastructure: 7 new AVX2 integer encoders (vpsignb, vpmaddubsw, vpmaddwd, vmovdqu, vpaddd, vpcmpeqd, vpsrlw_imm8), the IR pattern (Param Q8 + Param Q8 + 2× Dequantize + MatMul), MatmulJitCache::get_or_compile_q8q8, model.rs activation quantization pipeline. Measured net-neutral on Zen 4 — gate_up/lm_head/qkv/wo all improved, down_matmul regressed +27%. Default-off, infrastructure kept.

Phase 7.N — VNNI vpdpbusd encoders + 3-way dispatch

Added VEX-256 (AVX-VNNI client CPUs) and EVEX-256 (AVX-512 VNNI servers) forms of vpdpbusd plus runtime CPU detection. EVEX prefix encoded by hand (4 bytes: 0x62 + P0/P1/P2). The Q8×Q8 unit test confirmed EVEX runs on Zen 4 even though avxvnni CPUID bit is clear. Measured -3.6% on Zen 4 with single-accumulator design. Default-off, infrastructure kept.

Phase 7.O — 4-accumulator VNNI kernel

Applied Phase 7.G's 4-acc trick to VNNI: unroll the kb loop by 4 and feed each block into its own fp32 sub-accumulator. Win64-safe via ymm6 save/restore on the stack. Measured still -2.7% on Zen 4 — gate_up/qkv/wo/lm_head won as predicted (-9% to -28%) but down_matmul still regressed +27%, exactly canceling out. Default-off, infrastructure kept.

Phase 7.P — per-shape hybrid dispatch (the real win)

Three phases of data made the answer obvious: VNNI wins on short-K matmuls (K_blocks=28: -9% to -28%) and loses on long-K (K_blocks=152: +27%) because of CPU OoO window saturation in the longer inner loop. Solution: dispatch per matmul shape.

fn use_vnni_for_matmul(d_in: usize) -> bool {
    has_vnni() && (d_in / 32) <= 64
}

Routes Qwen's 5 short-K matmuls (qkv/wo/gate/up/lm_head) to VNNI and FFN-down to fp32 4-acc. +4.5% bench (62.5 → 65.3 tok/s), -13.3% per-step profile. Default-on.

v0.4.0 architectural promise

Every piece of code from 7.M, 7.N, 7.O is now on the production hot path via 7.P — no dead code. The 3-way CPU dispatch (avx512vnni > avxvnni > AVX2 fallback) and 3-way codegen dispatch (4-acc / single-acc / AVX2 chain) all light up depending on host and shape. External runtime deps still just thiserror.

v0.4 → v0.5

Single clear next move: AVX-512 ZMM fp32 + ZMM VNNI. The remaining 1.39× gap to ggml is almost entirely lane width — ggml uses ZMM (512-bit), we use YMM (256-bit). Extending the EVEX encoder family to ZMM ops and writing new ZMM kernels is the v0.5 work.

Read more

Assets 2
Loading