Skip to content

Lumen v0.1.0 — end-to-end LLM inference compiler in Rust

Choose a tag to compare

@redchupa redchupa released this 16 May 05:36
· 38 commits to main since this release

First public milestone of Lumen — a from-scratch LLM inference compiler + runtime in Rust.

What's in this release

  • Self-built x86_64 instruction encoder (REX/VEX, ModR/M, SIB, AVX2+F16C+FMA)
  • Register-tile (4×8) matmul: 57 GFLOPS / 19× scalar in standalone bench
  • IR-level auto-synthesis of dequant×matmul fused kernels
  • GGUF v3 reader + GPT-2 byte-level BPE tokenizer
  • Full Qwen2.5-0.5B (24 layers, GQA 14Q/2KV, RoPE, SiLU FFN, KV cache) forward pass
  • Shape-keyed JIT cache (MatmulJitCache) feeding every weight matmul
  • Bit-identical tokens between naive Rust path and JIT path

Production runtime dependencies: thiserror — that's the entire list.

Benchmark (Qwen2.5-0.5B-Instruct-Q8_0, single thread, Windows AVX2)

Path tg32 tok/s vs llama.cpp
Lumen naive Rust matmul 2.91 14.2× slower
Lumen JIT (4×8 tile / 1×8 AVX2) 4.43 9.3× slower
llama.cpp (ggml, b9174 AVX2) 41.32 1.0×

Honest gap breakdown and next-10× roadmap in BENCHMARK.md.

What's next (toward v0.2)

  • Cache-blocked matmul (outer block loop around the 4×8 tile)
  • Q8×F32 fused matmul on the model forward path (no dequant pass)
  • Flash-style attention
  • Multi-thread prefill

Read the story