Repository navigation
v0.5.0 - Gemma 4 12B, natively
Headline: Gemma 4 12B, natively. A native, encoder-free Gemma 4 12B "unified" multimodal runtime (text + vision + audio on one dense backbone) in 4-bit-float nvfp4, plus one-command coding-agent launch.
Highlights:
- Native Gemma 4 12B unified multimodal runtime (#171); nvfp4/mxfp4 quant, shipped checkpoint scores 77.6% MMLU-500 at ~27.7 tok/s (#173/#176).
krillm launch <agent>wires Claude Code, Codex, OpenCode, Copilot, Droid, Hermes, Pi to your local server, incl. a native OpenAI Responses API and Anthropic streaming tool_use, with the model pinned for the session (#161-#164, #185).- tool_choice + grammar-constrained / auto-constrained tool calls (#179, #184); native vision: LLaVA-1.5, Llama-3.2-Vision, Qwen2.5-VL; DeepSeek-V3 absorbed-MLA; n-gram speculative decode in the batcher; shared-prefix KV reuse for agentic/RAG.
Where KrillLM leads Ollama on Mac: capability (multimodal, native voice, structured output, constrained tools), concurrency (~2x via the batcher), cold start, and agentic/RAG latency. Single-stream text decode is at parity with Ollama's MLX backend (bandwidth roof). The server defaults to port 57455 and coexists with Ollama on 11434.
Install: brew tap srvsngh99/krillm && brew install krillm
Full notes: see CHANGELOG.md (0.5.0).