lumen - v1.0

Highlights

Lumen is a lightweight, high-performance LLM inference framework built from scratch with OpenAI Triton
kernels. It achieves up to 4x speedup over HuggingFace Transformers through custom Triton operators,
efficient memory management, and operator fusion techniques.

Supported Models

• Llama 3.x (1B, 3B)
• Qwen2.5
• Qwen3 (including Qwen3-0.6B)
• Llava1.5 (multimodal)

Key Features

• FlashAttention v1/v2 — efficient prefill attention
• FlashDecoding — optimized decode stage with high GPU parallelism
• Nopad Attention — eliminates padding-related computation waste
• Dynamic KV Cache — TokenAttention-style on-demand allocation
• Operator Fusion: skip_rmsnorm, swiglu (gate+up fusion), KV linear fusion
• GQA via KV Head Index — replaces repeat_kv for grouped-query attention
• Streaming generation with top-p sampling
• CLI interface with interactive chat and one-shot modes

Performance

On Llama-3.2-3B (prompt_len=25, batch_size=12, max_gen_len=1900):

Metric	Lumen	Transformers	Speedup
Throughput	730 tok/s	184 tok/s	~4x
Per-token Latency	1.37 ms	5.44 ms	~4x

What's New in v1.0.0

• Project refactored from lite_llama/ to lumen/ with cleaner architecture
• Support for Qwen3 model family with explicit head_dim configuration
• FlashDecoding integration for decode stage acceleration
• Dynamic KV cache management
• Nopad attention support
• Multi-operator fusion (skip_rmsnorm, swiglu, KV linear)
• Weight conversion tool from HuggingFace format
• Comprehensive test suite and benchmark scripts