Skip to content

Lumen v0.3.0 — profile, parallelize, 4× faster than v0.2.0

Choose a tag to compare

View all tags
@redchupa redchupa released this 17 May 06:36
· 29 commits to main since this release
v0.3.0
c1c3bf8

A day after v0.2.0, Lumen's Qwen2.5-0.5B decode runs 3.1× faster when allowed to use multiple threads, and +72% even on a single thread. Tokens remain bit-identical to the naive Rust reference.

tg32 (Qwen2.5-0.5B-Q8_0, Windows AVX2)

Single thread

Path tok/s vs ggml
Lumen v0.2.0 17.97 2.30× slower
Lumen v0.3.0 ~31 1.32× slower
ggml -t 1 41.32 1.0×

8 threads (default on the test machine)

Path tok/s vs ggml -t 8
Lumen v0.3.0 ~56 1.62× slower
ggml -t 8 90.90 1.0×

Sanity check: Lumen 8t (56) is 1.36× faster than ggml 1t (41.32). Multi-thread scaling: Lumen 1.81×, ggml 2.20× — narrowing that gap is v0.4.

What landed

Phase 7.E.0 — profile decode time distribution

StepTimer accumulator + generate_greedy_jit_timed + an #[ignore] test that runs 32 decode tokens and prints a sorted breakdown. Showed:

layer/gate_up_matmul  692ms  36.1%
      lm_head_matmul  446ms  23.2%
   layer/down_matmul  414ms  21.6%
      layer/silu_mul  155ms   8.1%
    layer/qkv_matmul   83ms   4.3%
     layer/wo_matmul   65ms   3.4%
          layer/rope   43ms   2.2%
     layer/attention   17ms   0.9%   ← killed the planned flash-attn phase
       (rest)              <1%

88.6% matmul, 0.9% attention compute. The originally planned Phase 7.E (flash-style attention) would have saved at most fractions of a percent — deprioritized on the spot.

Phase 7.G — 4-accumulator Q8 N=1 decode kernel

Same dependency-chain fix from Phase 7.C, now applied to the Q8 kernel. 4 independent ymm accumulators replace a single ymm0 chain. Q8 matmuls landed 1.87–1.95× faster essentially across the board — bounded by memory bandwidth, not FMA throughput. Caught the Win64 ABI xmm6-xmm15 callee-saved trap (unit test passed; Qwen e2e produced all-NaN logits because RMSNorm's xmm6 state got clobbered).

Phase 7.H — vectorize SiLU and elementwise mul

std::arch::x86_64 only, no extra deps. SiLU uses an inlined polynomial expf:

  • range-reduce: x = n*ln2 + r via FMA
  • exp(r) ≈ degree-5 Horner polynomial
  • 2^n packed into integer exponent bits, reinterpreted as f32

SiLU 87× faster (153ms → 1.76ms / 32 tokens). SiLU+mul went from 13.9% to 0.4% of decode.

Phase 7.I — RoPE inv_freq + sin_cos hoist

The original inner loop recomputed base.powf(...) and theta.sin_cos() per (t, h, i). For Qwen2 decode that's 14× redundant on Q and 2× on K. Hoist them out: 3.7× faster on RoPE (43ms → 12ms).

Phase 7.J — multi-thread Q8 matmul with rayon

Splits the M (output row) dimension of large Q8 matmuls across rayon worker threads. Per-call work-based threshold (M × K_blocks ≥ 100K) avoids regressions on small matmuls that don't have enough compute per row to outrun rayon's dispatch overhead. Single-thread ~31 → 8-thread ~56 tok/s (+1.81×).

v0.3 → v0.4 candidate moves

  1. Custom thread pool to replace rayon for tighter per-call overhead (likely closes the 1.81×→2.20× scaling gap).
  2. Q8-native int dot product (vpmaddubsw/vpdpbusd) instead of the dequant→fp32 FMA path. The biggest single-thread move left.
  3. Q8 token embeddings (545MB → 144MB; modest decode impact).
  4. Prefill batch path for long prompts.

Read more

Assets 2
Loading