Skip to content

v0.3.1

Choose a tag to compare

@srvsngh99 srvsngh99 released this 23 May 21:08
· 341 commits to main since this release

Headline: cold-path multimodal prefill on Gemma 4 drops the per-position
vocab matmul over a 262144-vocab head to a single position, and a
family-aware engine warmup pass eliminates the first-request MLX
compile / Metal JIT spike. Plus a Homebrew install fix that was
already broken on v0.2.0 (binary alone, no metallib), and the
project's first Swift CI workflow.

Added

  • LoadedModel.multimodalPrefillForward (#53): an optional
    last-token-only variant of multimodalForward. Gemma 4 wires it.
    The engine prefers it on multimodal prefill and falls back when
    the family does not. Bit exact for the sampled token. Closes the
    cold-path gap on Gemma 4's 262144 x L -> 262144 x 1 matmul that
    PR #50 had only addressed for the text path.
  • Family-aware engine warmup (#54): InferenceEngine.warmup()
    runs a tiny dry forward after load() and swap(). Vision-capable
    models include a 224x224 synthetic gray PNG so the 32 vision-block
    MLX.compile slots (from PR #48) get populated. Behind
    KRILL_SKIP_WARMUP=1 for CI / cold-start-sensitive use. Best
    effort: warmup errors never block accepting real requests.
  • Swift CI workflow (#60): .github/workflows/swift-tests.yml
    builds the Swift package and runs swift test on every PR and on
    main. Single macos-15 job, SwiftPM cache keyed on
    Package.resolved, metallib step gated by REQUIRE_METALLIB=1.
    The repo previously only ran the Python tools-tests workflow; the
    entire Swift core had no automated coverage.
  • KrillLMVersion constant in Sources/KLMRegistry/KrillLMVersion.swift
    (#56) replaces four hardcoded "0.3.0" string literals across CLI,
    server, and Ollama-compat. KrillLMVersionMatchesVersionFile test
    asserts agreement with the repo-root VERSION file at build time.
  • Per-family lastTokenOnly slice-equivalence tests (#57): pin
    the bit-exact property of PRs #50 and #53 against a future
    refactor that moves a non-elementwise op after the slice.
  • Both-paths vision-stage profile (#59): the Qwen 2.5-VL profile
    test now times the pre-PR-58 additive-mask windowed-attention
    path and the batched-per-window path side by side, keeping the
    delta auditable in CI.

Changed

  • Qwen 2.5-VL vision tower (#58): replaces the additive -1e4
    inter-window mask with per-window batched SDPA for the 28
    windowed vision blocks. SDPA cost drops from O(L^2) to
    O(L * windowSize) per block (16x on the canonical
    224x224 / 8x8 LLM grid). Wall-time impact is small (~1 percent)
    because MLP + Linears dominate the vision-block cost; the SDPA
    reduction is the right code shape regardless.
  • Homebrew formula (#55): bumped to v0.3.0 (URL + sha256) and
    fixed the install layout to put mlx.metallib and the
    mlx-swift Cmlx bundle next to the binary via libexec/+symlink.

Fixed

  • Homebrew install was broken on v0.2.0 (#55): the formula
    installed only the bare krillm binary, but the MLX runtime
    needs mlx.metallib adjacent to the executable to initialize
    Metal. No user hit it because source builds dominate on a small
    project; now correct from v0.3.0 onward.

Performance vs Ollama

  • First request after engine load is ~20 ms faster on
    Qwen 2.5-VL 3B (317 ms -> 287 ms median across 5 cold-cold-cold
    back-to-back distinct-image requests) thanks to the warmup pass
    populating MLX.compile slots and the Metal kernel JIT before the
    first user request lands (#54).
  • Cold-path Gemma 4 image prefill drops the 262144 x L
    vocab matmul to 262144 x 1. Wall-time delta scales with the
    number of prompt tokens; bit exact for the sampled token (#53).