Repository navigation
v0.3.1
Headline: cold-path multimodal prefill on Gemma 4 drops the per-position
vocab matmul over a 262144-vocab head to a single position, and a
family-aware engine warmup pass eliminates the first-request MLX
compile / Metal JIT spike. Plus a Homebrew install fix that was
already broken on v0.2.0 (binary alone, no metallib), and the
project's first Swift CI workflow.
Added
LoadedModel.multimodalPrefillForward(#53): an optional
last-token-only variant ofmultimodalForward. Gemma 4 wires it.
The engine prefers it on multimodal prefill and falls back when
the family does not. Bit exact for the sampled token. Closes the
cold-path gap on Gemma 4's262144 x L -> 262144 x 1matmul that
PR #50 had only addressed for the text path.- Family-aware engine warmup (#54):
InferenceEngine.warmup()
runs a tiny dry forward afterload()andswap(). Vision-capable
models include a 224x224 synthetic gray PNG so the 32 vision-block
MLX.compile slots (from PR #48) get populated. Behind
KRILL_SKIP_WARMUP=1for CI / cold-start-sensitive use. Best
effort: warmup errors never block accepting real requests. - Swift CI workflow (#60):
.github/workflows/swift-tests.yml
builds the Swift package and runsswift teston every PR and on
main. Singlemacos-15job, SwiftPM cache keyed on
Package.resolved, metallib step gated byREQUIRE_METALLIB=1.
The repo previously only ran the Python tools-tests workflow; the
entire Swift core had no automated coverage. KrillLMVersionconstant inSources/KLMRegistry/KrillLMVersion.swift
(#56) replaces four hardcoded "0.3.0" string literals across CLI,
server, and Ollama-compat.KrillLMVersionMatchesVersionFiletest
asserts agreement with the repo-rootVERSIONfile at build time.- Per-family
lastTokenOnlyslice-equivalence tests (#57): pin
the bit-exact property of PRs #50 and #53 against a future
refactor that moves a non-elementwise op after the slice. - Both-paths vision-stage profile (#59): the Qwen 2.5-VL profile
test now times the pre-PR-58 additive-mask windowed-attention
path and the batched-per-window path side by side, keeping the
delta auditable in CI.
Changed
- Qwen 2.5-VL vision tower (#58): replaces the additive
-1e4
inter-window mask with per-window batched SDPA for the 28
windowed vision blocks. SDPA cost drops fromO(L^2)to
O(L * windowSize)per block (16x on the canonical
224x224 / 8x8 LLM grid). Wall-time impact is small (~1 percent)
because MLP + Linears dominate the vision-block cost; the SDPA
reduction is the right code shape regardless. - Homebrew formula (#55): bumped to v0.3.0 (URL + sha256) and
fixed the install layout to putmlx.metalliband the
mlx-swiftCmlxbundle next to the binary via libexec/+symlink.
Fixed
- Homebrew install was broken on v0.2.0 (#55): the formula
installed only the barekrillmbinary, but the MLX runtime
needsmlx.metallibadjacent to the executable to initialize
Metal. No user hit it because source builds dominate on a small
project; now correct from v0.3.0 onward.
Performance vs Ollama
- First request after engine load is ~20 ms faster on
Qwen 2.5-VL 3B (317 ms -> 287 ms median across 5 cold-cold-cold
back-to-back distinct-image requests) thanks to the warmup pass
populating MLX.compile slots and the Metal kernel JIT before the
first user request lands (#54). - Cold-path Gemma 4 image prefill drops the
262144 x L
vocab matmul to262144 x 1. Wall-time delta scales with the
number of prompt tokens; bit exact for the sampled token (#53).