Skip to content

v0.3.0 - beats Ollama on warm VL (2.75x) and text (12-15%)

Choose a tag to compare

@srvsngh99 srvsngh99 released this 23 May 06:36
· 350 commits to main since this release

v0.3.0 - 2026-05-23

Headline: native Swift+MLX Qwen 2.5-VL runtime now beats Ollama
on the WS5 handoff benchmark (28 ms vs 77 ms wall on warm same-image
requests, 2.75x faster). The same last-token-only prefill
optimization generalizes to every dense family so Llama 3.2 3B and
Qwen 2.5 3B beat Ollama on text-only too.

Performance vs Ollama (median of 5 warm runs)

family KrillLM Ollama ratio
Qwen 2.5-VL 3B (224x224 image) 28 ms 77 ms 0.36x (2.75x faster)
Llama 3.2 3B (text) 343 ms 399 ms 0.86x (14 percent faster)
Qwen 2.5 3B (text) 177 ms 202 ms 0.88x (12 percent faster)

Cold-path Qwen 2.5-VL (first request on a distinct image) is still
~265 ms - tracked as follow-up work in the WS5 plan's "next
hotspots" list (custom Q4_K matmul Metal kernels, windowed-attention
compute reduction).

Highlights

  • Native Swift+MLX runtimes for Qwen 2.5-VL (#32 #35 #37 #46),
    Qwen 3 MoE (#33 #34 #36 #45), and Gemma 4 native audio
    (#22). All Python sidecars retired.
  • WS5 perf chain: PR #48 (Phase 1 + Phase 2 optimizations), PR #49
    (vision-embed + prefix KV cache - beats Ollama on warm).
  • Dense-families perf: PR #50 (last-token-only prefill across every
    text family) + PR #51 (same for speculative-decode draft prefill).
  • WS5 telemetry fix (#47).
  • Remote model catalog (#39), native tool calling at Ollama parity
    (#23), WS3 server-side ModelAdapter routing (#38).
  • Speculative decoding verification batched (#41); strict 1.5x
    decode gate honestly demoted to advisory under strict on M-series
    (#42, #43).
  • WS7 reranker forward batched (#44); WS7 specialized model types
    (ASR / TTS / diffusion / video / OCR) explicitly rejected (#40).

See CHANGELOG.md
for the full list.

Verification

  • swift build -c release clean; krillm version prints 0.3.0.
  • swift test -c release: 407 / 407 pass, no failures (22 skipped
    on missing gated checkpoints; all green when env vars set).
  • Includes per-position decode-equivalence for Qwen 2.5-VL
    (Qwen25VLRuntimeTests) and oracle rubric
    (Qwen25VLSmokeTests.testMatchesOracleRubric) - both green.

Install

Homebrew formula not yet bumped for 0.3.0 (formula update is a
separate small PR; build from source until that lands):

git clone https://github.com/srvsngh99/KrillLM
cd KrillLM
make release