Skip to content

v0.5.0 - Gemma 4 12B, natively

Choose a tag to compare

@srvsngh99 srvsngh99 released this 09 Jun 07:08
· 225 commits to main since this release

Headline: Gemma 4 12B, natively. A native, encoder-free Gemma 4 12B "unified" multimodal runtime (text + vision + audio on one dense backbone) in 4-bit-float nvfp4, plus one-command coding-agent launch.

Highlights:

  • Native Gemma 4 12B unified multimodal runtime (#171); nvfp4/mxfp4 quant, shipped checkpoint scores 77.6% MMLU-500 at ~27.7 tok/s (#173/#176).
  • krillm launch <agent> wires Claude Code, Codex, OpenCode, Copilot, Droid, Hermes, Pi to your local server, incl. a native OpenAI Responses API and Anthropic streaming tool_use, with the model pinned for the session (#161-#164, #185).
  • tool_choice + grammar-constrained / auto-constrained tool calls (#179, #184); native vision: LLaVA-1.5, Llama-3.2-Vision, Qwen2.5-VL; DeepSeek-V3 absorbed-MLA; n-gram speculative decode in the batcher; shared-prefix KV reuse for agentic/RAG.

Where KrillLM leads Ollama on Mac: capability (multimodal, native voice, structured output, constrained tools), concurrency (~2x via the batcher), cold start, and agentic/RAG latency. Single-stream text decode is at parity with Ollama's MLX backend (bandwidth roof). The server defaults to port 57455 and coexists with Ollama on 11434.

Install: brew tap srvsngh99/krillm && brew install krillm

Full notes: see CHANGELOG.md (0.5.0).