A minimal LLaMA-style inference engine for MiniCPM5-1B, built purely in Rust.
I built this project to study the fundamentals of large language models. Every understanding is rephrased directly in the code — dense, but organized. My hope is that it helps you learn too.
Inspired by tiny-vllm.
- CPU-only inference — no GPU required
- No KV-cache — focuses on the core model mechanics
- Single-file implementation —
src/main.rs(~1800 lines) - TUI visualization — real-time view of how the model "thinks"
| Component | Details |
|---|---|
| Embedding | Token ID lookup via embedding table |
| RoPE | Rotary position embeddings |
| Attention | Multi-head attention with GQA (Grouped Query Attention) |
| MLP | SwiGLU activation function |
| Normalization | RMSNorm |
| Residual connections | Standard skip connections after attention and MLP |
# Download MiniCPM5-1B from HuggingFace
# Put the model files in a directory, e.g. ./models/minicpm5-1b/
# The directory should contain:
# - config.json
# - model-00000-of-00001.safetensors
# - tokenizer.jsoncargo run --release -- "<model_dir>" "<your prompt>"Example:
cargo run --release -- ./models/minicpm5-1b "What is artificial intelligence?"Press q to quit early.
The best way to read this code is top-to-bottom in src/main.rs:
- Tensor ops — A minimal
TinyTensorwrapper over candle tensors with matrix multiply, reshape, transpose, softmax, etc. I will try to write the Maths operations by hand. - Model loading —
ModelConfigurations,LlamaModel, andTransformerBlockstructs that map directly to the safetensors keys. predict_next_token— The inference loop through every transformer layer. Each operation is annotated with shape comments and timing.- TUI — Ratatui widgets render attention maps, tensor shapes, and logits in real time.
MIT — see LICENSE