Skip to content

v0.2.8

Choose a tag to compare

View all tags
@waybarrios waybarrios released this 12 Apr 16:32
· 357 commits to main since this release
v0.2.8
d2e7f88
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Compatibility & Bug Fixes

  • mlx-lm 0.31.x BatchGenerator API compatibility (#294, closes #293)
  • Gemma 4 BatchKVCache, attention and RotatingKVCache patches (#268, #256)
  • Qwen3.5 ArraysCache hybrid model batching (#160)
  • mlx-lm 0.31.x prompt_checkpoints tuple compatibility
  • RotatingKVCache support in MLLM batching
  • Streaming UTF-8 safe detokenizer (#109)
  • Platform module rename to avoid stdlib shadowing (#185)
  • Base64 image hash no longer truncated (#206)
  • Specprefill: avoid dense tail expansion within cache window (#291)
  • MTP optimistic RNN snapshot leak (#196)
  • Normalize messages before chat template application (#240)
  • Import Path in tokenizer utils (#283)
  • Streaming tool call XML suppression (#129)
  • Clean up detokenizer pool in abort/reset/error recovery
  • Preserve prompt checkpoints in chunked prefill resume (#221)
  • SimpleEngine serialized across cancellation (#220)
  • Preserve hybrid recurrent state across prefix cache blocks (#217)

New Features

  • Full sampling parameter support: top_k, min_p, presence_penalty, repetition_penalty (#213)
  • Gemma 4 multimodal model support (#268)
  • Gemma 4 tool call parser (#269)
  • MiniMax tool call parser (#231)
  • Harmony tool parser exposed in serve CLI (#284)
  • Qwen tool parser <function=name> format support (#281)
  • Reasoning parser rewritten as O(1) state machine (#234)
  • Anthropic thinking content blocks via ThinkRouter
  • SimpleEngine tool-enabled chat on streaming path (#285)
  • Resumable model download with retry, timeout, offline mode (#77)
  • Production backport: MTP, tool parsers, sampling, prefill (#278)
  • Full sampling params support for MLLM continuous batching (#258)

Infrastructure

  • Apache 2.0 license file added (#282)
  • Bump mlx-lm minimum to 0.31.0 for hybrid model batching
  • Upgrade mlx-vlm and torchvision for Qwen3.5 multimodal

Benchmark (Llama-3.2-1B-Instruct-4bit, mlx-lm 0.31.2)

Total time: 2.38s
Prompts: 10
Prompts/second: 4.19
Total prompt tokens: 80
Total completion tokens: 960
Total tokens: 1040
Tokens/second: 402.52
Throughput: 436.06 tok/s

Full changelog: v0.2.7...v0.2.8

Assets 2
Loading