0.5.8.dev3

This development release adds Meta Muse Glimmer 30B with DFlash speculative decoding, makes DeepSeek V4 prefill 23-28% faster across context lengths, persists GDN recurrent state as bounded SSD sidecars, and fixes Ling accuracy, grammar decoding overhead, and tool-call parsing edge cases.

• Added Meta Muse Glimmer 30B VLM support with DFlash speculative decoding. The model implementation (hybrid SWA/NoPE text backbone, ViT-G/14 encoder) is vendored from the upstream mlx-vlm port by @Blaizzy (Blaizzy/mlx-vlm#1838), plus the quantized embedding-norm fix by @johntdavies (Blaizzy/mlx-vlm#1839). Thank you both for the quick upstream turnaround. On the oMLX side this adds channel-scoped output parsing with ATEM tool calls, prefix and SSD cache integration, and oQ quantization (#2586), and DFlash speculative decoding is wired in #2587, reaching about 1.4x decode on the oQ4 pair.
• Faster DeepSeek V4 prefill, 23-28% across context lengths. Layered across four changes following the ablation in #2558: ratio-128 layers now route through the native attention kernel (#2559), the MXFP4 MoE block threshold matches 2048-token chunks (#2562), and redundant all-pooled indexer scoring is skipped (#2563), all by @DiscoStew6082. DeepSeek V4 cache blocks now target 2048 tokens so those kernels engage with the prompt cache enabled. On top of that, a fused windowed+pooled prefill kernel, skipping the discarded lm_head projection on intermediate chunks, in-place pooled cache appends, and a fused indexer mask, by @jonathan308 in #2568. Measured on an M3 Ultra (V4-Flash 4-bit, default server config, cold prompts, prompt processing tok/s):

stage	pp4096	pp16384	pp32768	pp65536	pp131072
0.5.8.dev2	496.5	481.2	–	453.3	398.4
+ #2559 #2562 #2563	491.0	475.2	–	442.4	405.4
+ 2048-token cache blocks	549.4	558.7	549.5	524.2	461.6
+ #2568 (= 0.5.8.dev3)	631.1	604.9	589.3	559.1	508.5

The kernel PRs need 2048-token prefill chunks to pay off, which is why the middle row is flat: the block-alignment change is what unlocks them under the default cache-enabled config. Decode speed is unchanged at every context length. Note that temperature-0 outputs can differ from 0.5.8.dev2 for identical prompts: the new kernels change floating-point reduction order, which only flips tokens where the model is effectively indifferent, and quality is unaffected.

• Reduced long-context memory for GDN models when the SSD cache is enabled. GDN recurrent state is now persisted as bounded SSD sidecars instead of retaining one full in-memory state per historical boundary, making long-context Qwen3.6 serving practical on unified-memory Macs. Opt-in and backward-compatible. By @hojin12312 in #2569.
• Fixed Ling-3.0-flash code accuracy, +17pp on HumanEval. Its late layers were running without the trained SwiGLU clamp shipped in config.json; applying it raises HumanEval from 71.3% to 88.4% at no runtime cost. By @True2456 in #2571.
• Removed a per-token host sync from grammar-constrained decoding. Token acceptance is deferred to the top of the next step with bit-identical output, so the forward dispatch never blocks on host work. By @fxd0h in #2561.
• Improved adaptive prefill memory estimates at high context. The estimator now remembers process footprint released after a chunk and accounts for it when sizing the next one, closing an over-admission gap. By @mvdbos in #2573.
• Fixed tool calls whose arguments contain a literal close marker. These were dropped with raw markup leaking into the response; payload boundaries are now located by span scanning with JSON-aware decoding, in both the batch parser and the streaming filter. By @richgoodson in #2544, with follow-up fixes for prose around malformed envelopes at end of stream and for JSON array payloads.

New Contributors

• @DiscoStew6082 made their first contribution in #2559.
• @fxd0h made their first contribution in #2561.
• @mvdbos made their first contribution in #2573.