Repository navigation
KrillLM v0.2.0 - First Release
KrillLM v0.2.0
Fast local LLM inference for Apple Silicon. Built on MLX.
Performance (llama3.2:1b, M4 Pro)
| Metric | KrillLM | Ollama | Delta |
|---|---|---|---|
| Decode (32 tok) | 252 tok/s | 161 tok/s | 1.57x faster |
| TTFT | 17ms | 136ms | 8x faster |
| Memory | 704 MB | 1,685 MB | 58% less |
Install
git clone https://github.com/srvsngh99/KrillLM.git
cd KrillLM
make release && make installPrerequisites: macOS 14+, Apple Silicon, Xcode with Metal Toolchain (xcodebuild -downloadComponent MetalToolchain)
Quick Start
krillm pull llama-3.2-3b
krillm run llama-3.2-3b "Hello!"
krillm serve --model llama-3.2-3b # OpenAI + Ollama compatible APIWhat's Included
- 5 model families: Llama, Qwen, Mistral, Gemma, Phi
- 14 curated models via
krillm pull - OpenAI + Ollama compatible HTTP server
- Prefix cache for sub-20ms TTFT on repeated prompts
- Speculative decoding (1.5-3x decode with draft models)
- Quantized KV cache (2x memory savings at long context)
- Custom fused Metal kernels (SwiGLU)
- Benchmark harness (
krillm bench)