Repository navigation
v0.3.0 - beats Ollama on warm VL (2.75x) and text (12-15%)
v0.3.0 - 2026-05-23
Headline: native Swift+MLX Qwen 2.5-VL runtime now beats Ollama
on the WS5 handoff benchmark (28 ms vs 77 ms wall on warm same-image
requests, 2.75x faster). The same last-token-only prefill
optimization generalizes to every dense family so Llama 3.2 3B and
Qwen 2.5 3B beat Ollama on text-only too.
Performance vs Ollama (median of 5 warm runs)
| family | KrillLM | Ollama | ratio |
|---|---|---|---|
| Qwen 2.5-VL 3B (224x224 image) | 28 ms | 77 ms | 0.36x (2.75x faster) |
| Llama 3.2 3B (text) | 343 ms | 399 ms | 0.86x (14 percent faster) |
| Qwen 2.5 3B (text) | 177 ms | 202 ms | 0.88x (12 percent faster) |
Cold-path Qwen 2.5-VL (first request on a distinct image) is still
~265 ms - tracked as follow-up work in the WS5 plan's "next
hotspots" list (custom Q4_K matmul Metal kernels, windowed-attention
compute reduction).
Highlights
- Native Swift+MLX runtimes for Qwen 2.5-VL (#32 #35 #37 #46),
Qwen 3 MoE (#33 #34 #36 #45), and Gemma 4 native audio
(#22). All Python sidecars retired. - WS5 perf chain: PR #48 (Phase 1 + Phase 2 optimizations), PR #49
(vision-embed + prefix KV cache - beats Ollama on warm). - Dense-families perf: PR #50 (last-token-only prefill across every
text family) + PR #51 (same for speculative-decode draft prefill). - WS5 telemetry fix (#47).
- Remote model catalog (#39), native tool calling at Ollama parity
(#23), WS3 server-side ModelAdapter routing (#38). - Speculative decoding verification batched (#41); strict 1.5x
decode gate honestly demoted to advisory under strict on M-series
(#42, #43). - WS7 reranker forward batched (#44); WS7 specialized model types
(ASR / TTS / diffusion / video / OCR) explicitly rejected (#40).
See CHANGELOG.md
for the full list.
Verification
swift build -c releaseclean;krillm versionprints0.3.0.swift test -c release: 407 / 407 pass, no failures (22 skipped
on missing gated checkpoints; all green when env vars set).- Includes per-position decode-equivalence for Qwen 2.5-VL
(Qwen25VLRuntimeTests) and oracle rubric
(Qwen25VLSmokeTests.testMatchesOracleRubric) - both green.
Install
Homebrew formula not yet bumped for 0.3.0 (formula update is a
separate small PR; build from source until that lands):
git clone https://github.com/srvsngh99/KrillLM
cd KrillLM
make release