Repository navigation
Lumen v0.1.0 — end-to-end LLM inference compiler in Rust
First public milestone of Lumen — a from-scratch LLM inference compiler + runtime in Rust.
What's in this release
- Self-built x86_64 instruction encoder (REX/VEX, ModR/M, SIB, AVX2+F16C+FMA)
- Register-tile (4×8) matmul: 57 GFLOPS / 19× scalar in standalone bench
- IR-level auto-synthesis of dequant×matmul fused kernels
- GGUF v3 reader + GPT-2 byte-level BPE tokenizer
- Full Qwen2.5-0.5B (24 layers, GQA 14Q/2KV, RoPE, SiLU FFN, KV cache) forward pass
- Shape-keyed JIT cache (
MatmulJitCache) feeding every weight matmul - Bit-identical tokens between naive Rust path and JIT path
Production runtime dependencies: thiserror — that's the entire list.
Benchmark (Qwen2.5-0.5B-Instruct-Q8_0, single thread, Windows AVX2)
| Path | tg32 tok/s | vs llama.cpp |
|---|---|---|
| Lumen naive Rust matmul | 2.91 | 14.2× slower |
| Lumen JIT (4×8 tile / 1×8 AVX2) | 4.43 | 9.3× slower |
| llama.cpp (ggml, b9174 AVX2) | 41.32 | 1.0× |
Honest gap breakdown and next-10× roadmap in BENCHMARK.md.
What's next (toward v0.2)
- Cache-blocked matmul (outer block loop around the 4×8 tile)
- Q8×F32 fused matmul on the model forward path (no dequant pass)
- Flash-style attention
- Multi-thread prefill
Read the story
- End-to-end retrospective (KR): docs/blog/phase7-v0.1.0-end-to-end-ko.md
- Architecture: docs/ARCHITECTURE.md
- Plan: PLAN.md