Lumen v0.2.0 — Q8-native decode, 4× faster than v0.1.0

Five days after v0.1.0, Lumen's single-thread Qwen2.5-0.5B decode runs 4.05× faster while still producing bit-identical tokens to the naive Rust reference.

tg32 (single thread, Windows AVX2)

Path	tok/s	vs llama.cpp
Lumen naive Rust matmul	2.91	14.2× slower
Lumen JIT v0.1.0	4.43	9.3× slower
Lumen JIT v0.1.0 + 1×N 4-acc decode (Phase 7.C)	5.08	8.1× slower
Lumen JIT v0.2.0 (+ Q8-native fused matmul)	17.97	2.30× slower
llama.cpp (ggml b9174 AVX2)	41.32	1.0×

What landed

Phase 7.C — 1×N 4-accumulator decode kernel (+15%)

The 1×8 AVX2 path used a single ymm accumulator, so each kk step issued one vfmadd231ps into a live dependency chain. New dispatcher branch routes M=1, N % 32 == 0 shapes — every weight matmul in autoregressive decode — to a kernel that runs 4 independent FMA chains over 32 output columns. Decode matmul throughput up; full forward up 15% (4.43 → 5.08 tok/s) because matmul wasn't the only bottleneck.

Phase 7.D — Q8-native fused matmul (+254%)

The real win. v0.1.0 dequantized Q8_0 weights to F32 at load time (~640MB → ~2.4GB in memory) and ran the matmul on the fp32 buffer. v0.2.0 keeps weights in their native Q8_0 ggml block layout end-to-end and feeds them into a new fused kernel (emit_quant_matmul_q8_n1_body) that:

• broadcasts the per-block fp16 scale d
• vectorizes K-direction over each 32-element block (8-wide × 4 unrolled)
• horizontally reduces ymm0 to a scalar with vhaddps + vextractf128 + vaddss
• writes one output row per pass

No load-time dequant pass. No per-call fp32 materialization. transpose_for_jit drops from 1.94s to 700ns because the F32 weights that needed transposing are now Q8 weights that don't.

End-to-end: 5.08 → 17.97 tok/s (+3.54×).

Phase 7.E — release plumbing

Cargo workspace version bumped to 0.2.0. README perf table and roadmap refreshed. New retrospective blog post (KR).

Correctness

qwen_generate_jit_matches_naive_and_speed continues to pass — the JIT path produces the same token sequence as the pure Rust reference on the same prompt ("안녕" → "안녕하세요, 저는").

What's next (toward v0.3)

• Flash-style attention (matters most on long context)
• Multi-thread prefill (near-linear on physical cores)
• AVX-512 dispatch on supporting CPUs
• Q8 token embeddings (the last big fp32 buffer in the model)

Read more

• BENCHMARK.md — full breakdown
• docs/blog/phase7d-v0.2.0-q8-native-ko.md — retrospective (KR)
• docs/blog/phase7-v0.1.0-end-to-end-ko.md — v0.1.0 end-to-end story (KR)