Skip to content

v0.5.0 — Measure-driven cycle (gap 1.376× → 1.304×)

Latest
Latest

Choose a tag to compare

View all tags
@redchupa redchupa released this 20 May 21:57
· 3 commits to main since this release
v0.5.0
7513c72

Headline

Metric v0.4.0 v0.5.0 Change
tg32 @ 1 thread (Lumen vs ggml) 41.85 vs 40.40 45.64 vs 40.40 Lumen +13%
tg32 @ 8 threads (Lumen vs ggml) 65.5 vs 90.9 67.4 vs 87.9 gap 1.376× → 1.304×
pp32 (LUMEN_PREFILL=1, opt-in) n/a 54.15 tok/s new path

Tokens remain bit-identical to v0.4.0. Same Qwen2.5-0.5B-Q8_0 model, same AMD Zen 4 host.

What shipped

  • Phase 8.A — ThreadPool rewrite (mpsc + per-task allocation + mutex receiver → atomic next_task counter + generation-based wake). LUMEN_THREADS=N env var added for scaling experiments.
  • Phase 8.E.1 — Q8 prefill via N=1 fan-out dispatcher. Sidesteps the slow N>1 codegen body (2.3-2.6× slower than N=1×N at every Qwen projection shape, confirmed by Phase 8.D.5 micro-bench). LUMEN_PREFILL=1 opt-in.

What didn't ship but stays in tree (default-off)

These are the measure-driven negative results from this cycle. Infrastructure preserved; activation waits for either a better host (ZMM on native 512-bit silicon) or the kernel rework (Phase 8.E.2).

  • AVX-512 ZMM kernels (Phase 7.R/S/T). Zen 4 implements AVX-512 as double-pumped 256-bit, so ZMM ops don't deliver lane-width win on this host.
  • chunk_rows L2-fit cap (8.B), software prefetch encoders (8.C), forward_layer_prefill_jit + Model::forward_prefill_jit (8.D.1-3). All measured net-negative on the current N>1 codegen.

11-cycle pattern (v0.5 measure-driven decisions)

Phase Hypothesis Result
7.M Q8 native int dot net-neutral
7.N VNNI single-acc -3.6%
7.O VNNI 4-acc -2.7% (revived in 7.P)
7.R/S/T AVX-512 ZMM lane width -4.5%
8.A atomic ThreadPool +9% 1t, +3% 8t ✓
8.B chunk L2-fit cap -3.5% (reverted)
8.C software prefetch 8t -49% (reverted)
8.D.3 prefill batching -2.9× pp32
8.D.3 retro attention is the cause 1% only (false)
8.D.5 N>1 codegen is slow confirmed ✓
8.E.1 N=1 fan-out fixes plumbing +2.39× pp32 ✓

Net: 1 merged, 1 diagnostic, 1 partial fix vs 7 reverts. Gap closed 1.376× → 1.304×.

What's next (see ROADMAP.md)

  1. Phase 8.E.2 — Q8 N>1 codegen rewrite (port the 7.G 4-accumulator YMM pattern). Expected to close most of the 8t gap.
  2. ARM64 backend — Apple Silicon / Graviton / Raspberry Pi support.
  3. Q4 native matmul — enables Qwen2.5-1.5B and 3B at the same memory footprint.

Retrospective blog series

16 posts across the v0.1-v0.5 build, indexed at docs/blog/INDEX.md. The v0.5-cycle retros:

Assets 2
Loading