Skip to content

KrillLM v0.2.0 - First Release

Choose a tag to compare

@srvsngh99 srvsngh99 released this 03 May 19:46
· 439 commits to main since this release

KrillLM v0.2.0

Fast local LLM inference for Apple Silicon. Built on MLX.

Performance (llama3.2:1b, M4 Pro)

Metric KrillLM Ollama Delta
Decode (32 tok) 252 tok/s 161 tok/s 1.57x faster
TTFT 17ms 136ms 8x faster
Memory 704 MB 1,685 MB 58% less

Install

git clone https://github.com/srvsngh99/KrillLM.git
cd KrillLM
make release && make install

Prerequisites: macOS 14+, Apple Silicon, Xcode with Metal Toolchain (xcodebuild -downloadComponent MetalToolchain)

Quick Start

krillm pull llama-3.2-3b
krillm run llama-3.2-3b "Hello!"
krillm serve --model llama-3.2-3b  # OpenAI + Ollama compatible API

What's Included

  • 5 model families: Llama, Qwen, Mistral, Gemma, Phi
  • 14 curated models via krillm pull
  • OpenAI + Ollama compatible HTTP server
  • Prefix cache for sub-20ms TTFT on repeated prompts
  • Speculative decoding (1.5-3x decode with draft models)
  • Quantized KV cache (2x memory savings at long context)
  • Custom fused Metal kernels (SwiGLU)
  • Benchmark harness (krillm bench)