0.6.3

oMLX 0.6.3

oMLX 0.6.3 adds first-class support for Qwen3.8-Flash-Next and GLM-5.3-Flash, strengthens the experimental Qwen ANE prefill path, and rolls the performance, cache, API, and distributed-serving improvements from the three release candidates into the final release.

Qwen3.8-Flash-Next and GLM-5.3-Flash

• Added first-class Qwen3.8-Flash-Next support. oMLX now serves the qwen4_exp architecture for text and image requests, including structured tool calls, continuous batching, prefix-cache state preservation, oQ conversion, and Lightning MTP. The large external N-gram PLE table can remain resident or use SSD mmap, including affine oQ2-oQ8 layouts. Initial support by @jonathan308 in #3169, followed by production hardening in #3174.

• Added GLM-5.3-Flash support. Both text-only and multimodal glm5_next checkpoints now integrate with model discovery, oQ/oQe conversion, continuous batching, vision feature caching, memory estimation, and cluster planning. oMLX reuses its existing sparse-attention, MoE, vector-gated delta, fused KDA, and affine prefill kernels where GLM-5.3 has compatible tensor contracts. #3184.

• Improved Qwen checkpoint loading. Both RMSNorm checkpoint layouts are supported, and chunked loading prevents Metal watchdog failures on very large BF16 models. Video input remains unsupported.

M3 Ultra 512 GB Benchmarks

Qwen3.8-Flash-Next

Measured with Qwen3.8-Flash-Next-oQ4e-mtp. Each row used a fresh process, SSD-mapped PLE storage, no prefix-cache reuse, the same code-Python prompt and seed, and 128 generated tokens at temperature 1.0 and top-p 1.0. Lightning MTP used adaptive maximum depth 3.

Context	Lightning MTP	PP	TG	TG speedup	MLX peak	Process peak	Acceptance
4K	Off	1,062.5 tok/s	22.2 tok/s	baseline	70.88 GiB	73.39 GiB	-
4K	On	1,035.9 tok/s	58.1 tok/s	2.62x	72.43 GiB	75.00 GiB	96.8%
16K	Off	964.6 tok/s	21.9 tok/s	baseline	72.32 GiB	75.55 GiB	-
16K	On	934.1 tok/s	53.7 tok/s	2.45x	73.95 GiB	77.07 GiB	97.9%
32K	Off	840.8 tok/s	21.0 tok/s	baseline	74.44 GiB	79.08 GiB	-
32K	On	809.4 tok/s	49.0 tok/s	2.33x	76.18 GiB	80.77 GiB	97.8%

Lightning MTP increased generation throughput by 2.33-2.62x, with a 2.5-3.7% prompt-processing cost and approximately 1.6-1.8 GiB of additional MLX peak memory.

GLM-5.3-Flash

Measured with a strict oQ4e conversion of the official GLM-5.3-Flash checkpoint. The resulting model contains 34 safetensors shards and occupies 169 GiB. Each run used a full uncached prefill followed by 128 generated tokens.

Prompt	Generation	PP	TG	TTFT	Peak footprint
4,096	128	482.3 tok/s	24.1 tok/s	8.49 s	179.50 GiB
16,384	128	443.2 tok/s	23.7 tok/s	36.97 s	179.49 GiB
32,768	128	449.6 tok/s	23.5 tok/s	72.89 s	179.77 GiB

A separate 12,362-token cache test correctly reported 0, 8,192, and 12,288 reused tokens for cold, partial-hit, and full reusable-block runs. All three produced identical reasoning and final responses, while SSD persistence completed without cache errors or write drops.

The source FP8 checkpoint was not benchmarked because the current sanitize path expands it to an approximately 598.6 GiB BF16 representation, exceeding both physical memory and the Metal working-set limit on this machine.

ANE Stability and Long-Prompt Quality

• Made Qwen GDN prefill offload recurrent-safe. Approximate INT8 ANE execution is now limited to token-local z projection rows, while recurrent qkv rows remain on the checkpoint-precision GPU path. This prevents approximation error from accumulating through long prompts and restores deterministic 32K retrieval, coding, ordering, and summarization quality. #3133.

• Retained ANE prefill performance. On the M3 Ultra test system, the recurrent-safe path remained 26.3% faster than GPU-only prefill at 4K and 22.7% faster at 32K, without reducing decode throughput.

• Restored ANE compile-cache compatibility on macOS 27. The rc3 compiler assigned a staging URL that the newer ANE bundle validator rejected with Code=10. oMLX now leaves the framework-derived model URL intact, retains content-hash cache reuse, and removes failed cached artifacts before retrying.

Other Changes

• Activated prompt priming for legacy Lightning MTP models under BatchGenerator. DeepSeek-V4-Flash testing improved draft acceptance and increased measured decode throughput by 19.4%. By @avlp12 in #3138.
• Made the audio upload limit configurable. The previous fixed 100 MB ceiling can now be changed through server settings, the admin UI, the native app, OMLX_MAX_AUDIO_UPLOAD_SIZE, or --max-audio-upload-size. By @aywengo in #3160.
• Prevented large checkpoint loads from tripping the Metal GPU watchdog. Lazy parameter trees are evaluated in bounded chunks rather than one multi-hundred-gigabyte command buffer. By @scubamount in #3180.
• Fixed custom VLM batch-cache conversion. Model-owned cache types can now preserve their auxiliary state during the continuous-batching handoff. GLM batch convolution state is also retained correctly. By @DiscoStew6082 in #3188.
• Added an HTTP fallback for stalled Hugging Face Xet downloads. Known startup stalls and CAS transport failures are retried once in an isolated non-Xet process while preserving completed files. #3186.
• Preserved model settings configured before the first chat message. Lazy chat-session creation no longer replaces edited temperature, token, sampling, or thinking-budget values with model defaults.
• Improved Hy-MT2 decode performance. HyV3 models now use the existing bit-exact routed MoE gate/up fusion, reducing projection launches and improving measured decode throughput by 5.3-8.3%. By @chenqianhe in #3055.
• Cleaned up mounted update images before relaunch. The macOS updater now detaches its DMG and removes temporary download files before the application exits for an update. By @chenqianhe in #3190.

Changes Included from rc1-rc3

• Qwen ANE performance: Added Tuner V2, Q5/Q6/Q8 support, wide prompt tiling, CPU sharing, fused MLP/down offload, and calibrated tail padding.
• ANE memory and reliability: Reduced compilation memory, warmed procedures at load, added bounded waits, and improved failure recovery.
• DFlash 2: Added end-to-end runtime support with updated draft settings, sampling parity, and SSD prefix-cache compatibility.
• DeepSeek-V4-Flash: Added an M2 Ultra-tuned sparse-indexer kernel and improved Lightning MTP behavior.
• Prefix and SSD caching: Added optional hot-cache write-through, stronger recurrent-state validation, and crash-safe SSD writes.
• Lightning MTP and sampling: Restored MTP after temporary parking, improved long-prefix priming, and added configurable repetition context.
• API compatibility: Corrected Responses API truncation, prefill-memory errors, and VLM tool-result image handling.
• Agent workflows: Restored prefix-cache reuse in long Claude Code sessions and fixed forwarded CLI arguments.
• Distributed serving: Improved cluster planning, activation, teardown, SSH pairing, CUDA enrollment, and incident reporting.
• Admin and settings: Added atomic settings writes, model-setting fixes, benchmark controls, and clearer ANE status reporting.
• Laguna performance: Added bit-exact compiled decode optimizations for Laguna XS 2.1.

Upgrade Notes

• Qwen ANE prefill, CPU sharing, fused MLP/down offload, and tail padding remain experimental and depend on private Apple runtime interfaces.
• Qwen3.8-Flash-Next and GLM-5.3-Flash support text and image input. Video input remains unsupported.
• Lightning MTP is not yet enabled for GLM-5.3-Flash.
• The ANE compile cache and hot-cache write-through remain disabled by default.
• Distributed inference remains experimental and disabled by default.
• No manual settings migration is required.

Contributors

Thanks to @liang2kl, @GordoAR, @onthehub97, @beaglemoo, @xunlinkx, @alytaphoenix, @williamxie1989, @Pietroski, @fparrav, @mvdbos, @0xClandestine, @finaltv971, @popfido, @nathanodle, @saichowdary007, @alexferrao, @ashhart, @qdaszx, @joshuaswarren, @jonathan308, @chenqianhe, @hesara, @monroewilliams, @stellarsailor, @avlp12, @aywengo, @scubamount, and @DiscoStew6082 for their contributions to oMLX 0.6.3.

Thanks also to @taozhiyuai, @quochuy, @panuhallfors, @Collinw24, @waterliu1981, @cropduster, @fxd0h, @idallasj, @bunnyfu, @hellobao-yu, @pyros-projects, and @SneakyBerry for detailed reports, hardware testing, and release-candidate validation.

Full Changelog: v0.6.2...v0.6.3