0.6.3rc3

oMLX 0.6.3rc3

Thank you for the time and care behind the recent wave of contributions to oMLX. The current PR volume is more than I can review at once as the sole maintainer, so reviews may take longer than usual. I still welcome useful contributions. For now, I plan to review cluster and distributed inference PRs after the final 0.6.3 release.

For stacked work, please keep dependent follow-up PRs as drafts until the parent PR is merged. If you have several open PRs, I may ask which one or two you want me to review first. AI-assisted contributions are welcome, but authors are expected to understand, validate, and maintain the changes they submit.

This release adds fused Qwen ANE MLP/down-projection offload, tuner-calibrated tail padding, and an optional persistent ANE compile cache. It also hardens ANE execution and memory handling, restores Lightning MTP after temporary performance parking, improves prefix and SSD cache integrity, and reports prefill-memory failures correctly across the OpenAI, Anthropic, and Responses APIs.

Qwen ANE Prefill Performance

• Added fused MLP and down-projection offload. Compatible Qwen3.5/3.6/3.8 Q4 models on dual-ANE systems can now split the MLP hidden dimension across both ANEs, an optional CPU branch, and the GPU. Each branch performs its matching down projection before the results are combined, reducing GPU synchronization and intermediate transfers. The hardware-adaptive tuner now calibrates the fused split, CPU share, worker count, and GDN balance using real native dispatches and full-model verification. By @onthehub97 in #2935.

On the reported M3 Ultra measurement with a Qwen3.8 27B checkpoint:

Configuration	Prompt throughput	Improvement
GPU-only baseline	349.4 tok/s	baseline
Fused ANE/CPU/GPU prefill	527.4 tok/s	+50.9%

Decode remains on the normal GPU path. Results depend on the Mac, checkpoint, quantization, prompt shape, and system load.

• Added tuner-calibrated intermediate tail padding. Residual prompt tails that are large enough to benefit can now run through a fixed-shape ANE tile instead of always returning to the GPU. Only intermediate MLP and GDN activations are zero-padded, and synthetic rows are removed before recurrence or later model stages. Prompt tokens, positions, KV entries, and recurrent state are not extended. By @onthehub97 in #2966.

For a reported 4,095-token prefill with a 2,047-token residual tail:

Tail path	Prompt throughput	Improvement
GPU residual	415.3 tok/s	baseline
Padded hybrid tile	535.8 tok/s	+29.0%

• Added an optional persistent ANE compile cache. Previously compiled Qwen ANE programs can be reused across fresh oMLX processes through Apple's AOT cache. Reported fresh-process model-load reductions ranged from 53% to 66% across M1 Ultra, M1 Max, and M2 Max systems, with byte-identical output. The feature can be enabled in Advanced Cache settings or with OMLX_QWEN35_ANE_COMPILE_CACHE=1, and takes effect on the next restart. By @joshuaswarren in #2975.

ANE Reliability and Memory Management

• Prevented ANE driver stalls from hanging the server indefinitely. Evaluation waits now have a bounded timeout, failed programs are latched, and affected modules permanently fall back to GPU execution for the current model load. The default timeout is 30 seconds and can be configured with OMLX_ANE_WAIT_TIMEOUT_S. By @beaglemoo in #3002.

• Stopped procedure-bank retry ladders before process memory reaches jetsam territory. ANE compilation now observes total process footprint, allows failed driver allocations time to settle, and falls back to per-layer compilation when retrying would risk exhausting system memory. By @alytaphoenix in #2999.

• Reduced fused-down compilation memory. The compiler now dequantizes only the columns assigned to ANE and CPU branches instead of materializing the complete down-projection matrix and discarding the GPU suffix. The resulting staged weights remain bit-identical to the previous path. By @alytaphoenix in #3110.

• Made ANE dispatch exception-safe. Single, dual, and fused dispatch paths now retire command buffers and outstanding tickets when an exception occurs between submission and worker-thread ownership. This prevents a single failed dispatch from leaving the ANE program stuck in an overlapping-evaluation state until restart. Initial fix by @alytaphoenix in #3105, with follow-up coverage for the fused path.

• Recovered memory from ANE banks during long-context pressure. When ordinary buffer reclamation cannot restore enough prefill headroom, oMLX can release the requesting model's ANE procedure banks and continue serving with GPU prefill. /api/status reports the shed state, and ANE prefill is rebuilt on the next model load. By @beaglemoo in #3103.

• Improved tuner behavior on unsupported and mixed configurations. Machines without the private ANE compiler now receive a completed GPU-only recommendation instead of a failed tuning run. SpecPrefill is disabled during dense ANE calibration so sparse prompt compression cannot invalidate the measurement. By @beaglemoo and @monroewilliams in #3067 and #3080.

Prefix and SSD Cache Reliability

• Added optional hot-cache write-through. KV blocks can now remain in RAM for fast same-process reuse while also being written to SSD for restart durability. Clearing the hot cache flushes dirty blocks instead of silently discarding them. The feature is disabled by default and can be enabled through settings, --hot-cache-write-through, or OMLX_HOT_CACHE_WRITE_THROUGH. By @jonathan308 in #3015.

• Aligned Qwen GDN cache boundaries with wide prefill. On validated classic-Metal Qwen3.8 configurations, prefix caching no longer splits a 4,096-token model forward into two numerically different 2,048-token forwards. This fixes deterministic coding-agent sessions that repeated completed tool calls or forgot recent progress only when prefix caching was enabled. NAX-capable and smaller-memory systems retain the validated 2,048-token geometry. By @hesara in #3066.

• Hardened recurrent-state and shared-prefix correctness. Cache storage now avoids attaching end-of-prompt recurrent state to an earlier full block when a partial tail was skipped, restores pre-call GDN state before stock fallback, verifies shared-prefix hashes during acquisition, and rejects blocks whose stored KV lengths disagree with their declared token count. By @alytaphoenix in #3085, #3086, #3087, and #3094.

• Improved SSD cache crash safety and preload stability. SSD blocks, GDN sidecars, boundary snapshots, and vision feature entries are flushed before atomic rename, with parent directories synchronized afterward. SSD preload also keeps mx.load() on the caller thread to avoid known MLX worker-thread deadlocks. By @alytaphoenix in #3091 and #3095.

Lightning MTP and Sampling

• Made Lightning MTP performance parking reversible. A request that temporarily falls back to standard decoding can retry Lightning MTP after a bounded cooldown. Successful probes resume MTP, while repeated failures use exponential backoff up to 4,096 tokens. Existing aligned-batch and late-join safety checks remain in place. #3004.

• Restored MTP prompt priming after long cached prefixes. The prime-window memory limit now applies to the newly folded prompt span rather than the absolute context offset. Long sessions with a small uncached remainder can therefore retain the high initial acceptance rate of prompt priming without removing the configured memory bound. By @beaglemoo in #3005.

• Added repetition_context_size to OpenAI completion requests. Clients can extend the lookback window used by repetition_penalty, allowing penalties to cover loops longer than mlx-lm's 20-token default. The setting is forwarded through local, VLM, distributed, and DFlash engines while remaining omitted when unset for compatibility. By @xunlinkx in #3012.

API and Model Compatibility

• Reported prefill-memory failures correctly. Fast non-streaming rejections now return the real HTTP error status instead of a successful HTTP 200 with an error body. Streaming OpenAI, Anthropic, and Responses API paths preserve structured memory-guard details instead of flattening them into generic server errors. By @alytaphoenix and @beaglemoo in #2992 and #3060.

• Fixed Responses API tool-result images for VLMs. Images in function_call_output lists now travel through the normal multimodal path instead of having their base64 payload tokenized as prompt text. Text-only engines receive a placeholder, and ordinary JSON tool results retain their previous behavior. Addresses #2989.

• Restored oQ calibration for legacy Hy-MT2 checkpoints. Root-level rope_theta values a...

0.6.3rc2

oMLX 0.6.3rc2

This is the second release candidate for oMLX 0.6.3. It focuses on performance, memory use, and reliability issues found during rc1 testing. Please report any remaining regressions before the final 0.6.3 release.

This release extends experimental Qwen ANE prefill with optional CPU sharing, sharply reduces ANE compilation memory spikes, and improves first-request latency and tuner reliability. It also adds an M2 Ultra-optimized DeepSeek-V4-Flash indexer kernel, corrects hybrid-model KV memory accounting, makes settings writes resilient to concurrent processes, and hardens SSH and CUDA cluster enrollment.

Qwen ANE Prefill: CPU Sharing and Reliability

CPU Sharing requirement: CPU Sharing requires a separately prepared checkpoint clone whose floating-point tensors use the FP16 dtype, such as Qwen3.8-27B-oQ4e-fp16-mtp. This remains an oQ4e quantized checkpoint: packed integer weights are unchanged. On the stock BF16 checkpoint, CPU Sharing stays disabled and prefill continues on the ANE/GPU path.

• Added optional CPU sharing to Qwen ANE/GPU prefill. Independent slices of MLP gate/up, MLP down projection, and residual GDN qkv work can now run on the CPU alongside the ANE and GPU branches. The in-app tuner jointly calibrates five workload controls and supports both single- and dual-ANE configurations. By @onthehub97 in #2892.

A fully warmed A/B on an M3 Ultra used 4K and 16K prefills followed by 128 generated tokens. Each configuration ran twice after an ane_2048 warm-up, with the second pass reported:

Configuration	Model	4K prefill	16K prefill	Versus baseline
ANE off	Qwen3.8-27B-oQ4e-mtp	458 tok/s	436 tok/s	baseline
ANE/GPU split	Qwen3.8-27B-oQ4e-mtp	588 tok/s	555 tok/s	+28% / +27%
ANE/CPU/GPU split	Qwen3.8-27B-oQ4e-fp16-mtp	625 tok/s	579 tok/s	+36% / +33%

CPU Sharing added approximately another 4–6% prefill throughput over the ANE/GPU split on this workload. Decode throughput was unchanged, and the measured peak-memory cost was approximately 7 GB.

• Reduced ANE bank compilation memory use. Weight slices are converted and staged incrementally instead of retaining every layer in FP32 until both banks are built. On Qwen3.8-27B-oQ4e-mtp, the measured enable-time memory spike fell from 35.8 GB to 4.7 GB, while retained MLX buffer-cache memory fell from 32.1 GB to 0.2 GB with unchanged compile time and parity. This addresses the memory pressure reported in #2781.

• Moved ANE first-evaluation overhead into model loading. Compiled procedures are now warmed immediately after creation, so the first matching user prompt measures inference rather than private-runtime initialization. On an M5 Pro with a 16K prompt, first-request time improved from 46.1–48.3 seconds to 41.1–41.5 seconds, with approximately 2.9 seconds added to model loading. By @beaglemoo in #2898.

• Made ANE no-ops visible. Ineligible models, failed dispatch installation, and zero-procedure compilation now produce actionable warnings. /api/status also reports whether ANE prefill was attempted, whether it was successfully configured, and the number of MLP, GDN, dual-ANE, and resident procedures for each loaded model. By @finaltv971 in #2904.

• Hardened the ANE tuner and memory admission path. Tuning now respects each checkpoint's structural GDN floor, avoids recommending GDN when no GDN operations ran, stages candidates without DFlash wrappers, and releases calibration engines before verification reloads. The memory guard now includes fixed ANE I/O surfaces and CPU-sharing allocations, preventing long prefills from repeatedly crossing the hard watermark on smaller-memory systems. CPU-sharing failures also drain committed GPU work before propagating the error.

• Preserved model profiles while editing ANE settings. The native app now stages ANE controls and tuner recommendations in the working profile until the user explicitly saves or updates it, instead of clearing the active profile association. By @popfido in #2932.

DeepSeek-V4-Flash Performance on M2 Ultra

• Added an M2 Ultra-tuned MMA indexer score kernel for DeepSeek-V4-Flash. The new simdgroup_matrix path keeps the K tile resident in threadgroup memory and reduces synchronization overhead. On the reported M2 Ultra measurements, the score operation improved by 1.37–1.38x from 32K through 1M context while remaining bit-exact with the existing Steel kernel. Activation is restricted to the validated model and hardware fingerprint; unsupported configurations continue using the existing path. By @nathanodle in #2802.

Memory and Settings Reliability

• Corrected KV-cache sizing for hybrid models. Memory estimates now use only layers that retain per-token KV state rather than all transformer layers. This removes the 4x overestimate seen on affected Qwen3.5/3.6/3.8 models, preventing adaptive prefill throttling and HTTP 507 rejections caused by phantom KV growth. Rotating-only and fixed-state cache layouts also retain a conservative SSD writer-queue bound. By @saichowdary007 in #2912, addressing #2725.

• Made settings writes safe across concurrent server processes. Global and per-model settings are written through per-process temporary files, flushed, and atomically replaced instead of allowing two servers to interleave writes. Existing malformed settings.json files are moved to a timestamped .corrupt-* backup and logged clearly before defaults are loaded, preserving evidence for recovery. By @alexferrao in #2926.

Cluster Enrollment and Recovery

• Improved recovery from SSH pairing failures. Authentication errors now reach the correct pairing guidance, Bonjour .local names and internal FQDNs can resolve to the same local Mac, and managed SSH keys rotate only after an explicit confirmed request. Replacement keys are generated off-path so a failed rotation leaves the working key pair intact. By @ashhart in #2930.

• Hardened CUDA worker enrollment and activation. One-time join credentials are claimed before slow provisioning, controller key restrictions are updated safely during re-enrollment, and symlinked authorized_keys paths are rejected. CUDA workers rerun bounded NCCL direct-link verification before every activation instead of trusting stale browser state. By @ashhart in #2929.

App and Documentation

• Reorganized reasoning-effort controls in the native app. Custom mode, value selection, and force-override controls now appear in task order while retaining a readable two-line fallback at narrow widths and accessibility text sizes. By @popfido in #2922.

• Updated installation documentation for M5 Macs. Apple Silicon requirements now consistently list M1 through M5 across the translated READMEs. By @qdaszx in #2925.

Upgrade Notes

• This is a release candidate. Please report regressions before the final 0.6.3 release.

• Qwen ANE/GPU prefill and CPU sharing remain experimental, opt-in, and dependent on private Apple runtime interfaces and the oMLX native custom kernels.

• CPU sharing requires a separately preprocessed FP16 clone created with tools/clone_mlx_model_fp16.py. The source checkpoint is not modified or dequantized in place.

• ANE prefill remains an approximate acceleration path because selected weights are requantized to per-output-channel INT8. Use the built-in tuner for the specific Mac, checkpoint, quantization, and prompt shape.

• The DeepSeek-V4-Flash MMA score kernel activates by default only for the validated M2 Ultra/model pairing. Other hardware and model layouts retain the existing Steel path.

• If an existing settings.json is malformed, oMLX now moves it aside as settings.json.corrupt-<timestamp> and starts with defaults. Review the log and restore authentication or other settings from the preserved file when applicable.

• Distributed inference remains experimental and disabled by default.

• No manual settings migration is otherwise required.

Thanks to @onthehub97, @beaglemoo, @finaltv971, @popfido, @nathanodle, @saichowdary007, @alexferrao, @ashhart, and @qdaszx for their contributions, and to @taozhiyuai, @quochuy, @panuhallfors, @Collinw24, and @waterliu1981 for detailed reports and validation.

Full Changelog: v0.6.3rc1...v0.6.3rc2

0.6.3rc1

oMLX 0.6.3rc1

This is the release candidate for oMLX 0.6.3. After a short testing period, this release will be followed by either rc2 or the final release. Thank you, as always, to everyone who has patiently waited for this release!

This release introduces a faster Qwen ANE split tuner, extends ANE prefill to additional quantization formats, and adds DFlash 2 runtime support. It also substantially improves distributed-cluster reliability, restores prefix-cache reuse in long Claude Code sessions, reports Responses API truncation correctly, and adds bit-exact Laguna decode optimizations.

Qwen ANE Tuner V2 and Wider Quantization Support

• Made the ANE tuner much faster. Instead of preparing and testing many complete model configurations, the tuner first measures a few representative layers, predicts the best ANE/GPU split, and then verifies it with the full model. This reduces full-model setup runs from 11 to at most 3 while still basing the final recommendation on real prompt performance. By @onthehub97 in #2891.

• Added affine Q5, Q6, and Q8 ANE prefill support. Eligible Qwen3.5/3.6/3.8 MLP and GDN projections with group sizes 64 or 128 can now use the hybrid ANE/GPU path. Q4 behavior remains unchanged. By @GordoAR and @onthehub97 in #2833 and #2889.

• Improved ANE prefill for longer prompts. Large prompt chunks can now be divided into blocks that fit the compiled ANE program, while any remaining tokens continue on the GPU. This allows the server to keep efficient prompt chunk sizes instead of shrinking every chunk to the ANE block size. #2890.

• Made ANE tuning results easier to trust and diagnose. The tuner no longer recommends a configuration when the ANE was compiled but never actually used. Benchmark logs also show how much work ran on the ANE and GPU, making it easier to understand why a candidate succeeded or failed. By @beaglemoo in #2829.

Local 2,048-token Qwen3.8-27B measurements on M3 Ultra reported:

Model	GPU prefill	ANE MLP + GDN	Improvement
Q6 oQ6e	446.0 tok/s	560.8 tok/s	+25.7%
Q8 oQ8e	432.5 tok/s	557.4 tok/s	+28.9%

These are local serial measurements. Performance and the best split depend on the Mac, checkpoint, quantization, and prompt shape.

DFlash 2 Support

• Added end-to-end DFlash 2 support. Compatible checkpoints can use checkpoint-derived sliding windows and per-model block-size overrides. DFlash now matches the regular batched engine's sampling behavior, including min_p and seeded requests. The default sink size is now 0, SSD prefix-cache writes remain functional with that setting, and versioned DFlash2 checkpoints appear in the draft-model picker. Initial support by @liang2kl in #2840, finalized in #2850, with picker support by @williamxie1989 in #2879.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, the z-lab/Qwen3.8-27B-DFlash2 draft, temperature 0.7, and a 128-token generation tail:

Context	Baseline decode	DFlash 2 decode	Speedup	Acceptance
4K	33.8 tok/s	45.0 tok/s	1.33x	60.9%
16K	31.3 tok/s	44.8 tok/s	1.43x	65.6%
32K	29.3 tok/s	38.6 tok/s	1.32x	68.8%

Exact repeated prompts continue to use the in-memory prefix cache. SSD snapshot writes also work with the new sink-size default.

Distributed Serving and Cluster Reliability

• Added a persistent cluster incident feed. Activation, staging, and peer-health failures are recorded by the server and remain visible after dashboard refreshes. Incidents have stable sequence numbers, machine-readable guidance codes, severity, source, and explicit dismissal state. By @alytaphoenix in #2866.

• Improved two-Mac activation and planning reliability. Dashboard polling no longer clears a valid plan or changes tensor topology during activation. Cross-user clusters resolve model paths in each peer's own home directory, and unsupported VLM pipeline plans are no longer offered. By @alytaphoenix in #2819.

• Stopped remote rank processes more reliably. Teardown now validates each peer's deployment marker and process identity, sends SIGTERM, escalates to SIGKILL when necessary, and reports whether the rank was actually reaped. Failure markers remain available as diagnostic evidence. By @xunlinkx in #2722.

• Reduced repeated planner work. Cluster autoconfiguration caches model layouts and shard metadata until the relevant files change, avoiding repeated safetensors scans and monkey-patch installation during dashboard polling. By @alytaphoenix in #2883.

• Added tensor-parallel support for quantized Nemotron-H at TP=2. Uneven but group-aligned routed-expert shards, corrected quantized Mamba slicing, and stricter planner checks allow compatible Nemotron-H checkpoints to load and generate across two Macs. By @alytaphoenix in #2844.

• Improved distributed request compatibility and failure handling. Unsupported reasoning_effort values are retried through the same bounded fallback used by local engines. Requests also check rank health before streaming begins, allowing a dead or partially failed cluster to return HTTP 503 instead of an empty HTTP 200 response. By @alytaphoenix in #2869.

• Preserved configured memory-guard settings during remote admission. Cluster probes now initialize local settings before falling back to defaults, so custom memory tiers and ceilings are honored. By @xunlinkx in #2835.

APIs, Agent Workflows, and Admin Fixes

• Restored prefix-cache reuse in long Claude Code sessions. The Anthropic adapter now strips changing <total_tokens> budget markers from system and developer content while preserving user-quoted markers. On the reported 112K-token workload, reused tokens increased from 4,096 to 112,640 and turn latency fell from minutes to seconds. By @Pietroski in #2882.

• Reported Responses API truncation correctly. Requests stopped by max_output_tokens now return status: "incomplete" with incomplete_details.reason: "max_output_tokens". Streaming responses terminate with response.incomplete instead of response.completed, allowing clients to continue truncated reasoning or generation. By @mvdbos in #2817.

• Fixed forwarded arguments in omlx launch. The oMLX forwarding separator is removed without discarding a second separator intended for the launched tool, so flags passed to Claude Code and similar CLIs remain flags instead of becoming prompt text. By @fparrav in #2818.

• Added a token-budget override for external accuracy benchmarks. The optional value acts as a floor over each benchmark's default, preventing thinking models from being truncated before producing an answer without reducing larger benchmark-native budgets. By @williamxie1989 in #2863.

• Fixed default-model state after unsetting a model. Removing the default flag now clears the server-side pointer and updates the dashboard immediately instead of leaving requests routed to the old default until restart. By @alytaphoenix in #2884.

Laguna Decode Performance

• Added default-on, bit-exact compiled fusions for Laguna XS 2.1. Compiled router renormalization, expert combination, per-head softplus gating, and SiLU gate/up work improved single-token decode by 3.96% on an M4 Max, from 66.4 to 69.1 tok/s. The verified greedy trajectory remained token-identical. By @0xClandestine in #2827.

• Experimental routed and shared-expert fusion banks remain opt-in because they were neutral or slower on the tested MLX runtime.

Upgrade Notes

• This is a release candidate. Please report regressions before the final 0.6.3 release.

• DFlash draft sink size now defaults to 0 when unset. Existing explicit non-negative values are preserved.

• Qwen ANE/GPU prefill remains experimental, opt-in, and dependent on private Apple runtime interfaces and the oMLX native custom kernels.

• Keep the configured ANE sequence_length at or below the scheduler's delivered prefill chunk width. With prefix caching enabled, 2,048 remains the safe default because cache boundaries split delivered chunks at 2,048 tokens.

• Streaming Responses API clients should handle response.incomplete as the terminal event when generation reaches max_output_tokens.

• Distributed inference remains experimental and disabled by default.

• No manual settings migration is otherwise required.

Thanks to @liang2kl, @GordoAR, @onthehub97, @beaglemoo, @xunlinkx, @alytaphoenix, @williamxie1989, @Pietroski, @fparrav, @mvdbos, and @0xClandestine for their contributions.

Full Changelog: v0.6.2...v0.6.3rc1

0.6.2

oMLX 0.6.2

oMLX 0.6.2 combines focused hotfixes for regressions and edge cases uncovered in 0.6.1 with a new built-in ANE/GPU split tuner. It fixes the TurboQuant KV + Lightning MTP verification crash, restores exact GDN SSD-cache snapshots as the default, and addresses several distributed, cluster, MCP, memory, and admin issues.

Thanks to an incredible contribution from @onthehub97, oMLX can now benchmark ANE/GPU splits directly on each Mac instead of relying on ratios tuned for another machine. Dedicated NAX GPU suffix kernels also allow M5-family Macs to find and retain measurable ANE/GPU prefill gains.

Built-in ANE Split Tuner and M5 NAX Support

• Added a built-in ANE/GPU split tuner. The macOS app and web dashboard can now benchmark a GPU-only baseline, several MLP-only splits, and combined MLP/GDN splits directly on the current Mac. Candidate settings remain temporary until the user explicitly applies the result. By @onthehub97 in #2814.

• Added NAX GPU suffix kernels for M5-family Macs. Supported Qwen3.5/3.6/3.8 quantized layers use dedicated NAX QMM kernels for the GPU portion of hybrid ANE/GPU prefill. This resolves the regression seen when early M5 testing used split ratios and GPU kernels tuned for previous generations.

• Removed the blanket M5 ANE disable. Experimental ANE prefill remains disabled by default and must still be enabled per model, but M5 systems can now use the tuner to determine whether a local ANE/GPU split outperforms GPU-only execution.

• Kept safe runtime fallbacks. If NAX support or its metallib is unavailable, oMLX falls back to the existing Metal path. OMLX_QWEN35_QMM_NAX=0 remains available as a NAX kill switch.

Community testing on M5 reported approximately a 5% improvement in a practical benchmark, with the best tuner trial approaching 1,000 prompt tokens/s. Results depend on the chip, model, quantization, and prompt shape, so the tuner recommends GPU-only execution when the best ANE candidate is less than 1% faster.

ANE program loading was also hardened for single-die chips by retrying with smaller program banks when the preferred layout exceeds the device window. Benchmark traces now report the actual compiled MLP and GDN layer counts for easier validation.

Qwen and Cache Hotfixes

• Fixed Lightning MTP generation crashing with TurboQuant KV enabled. The optimized Qwen verification-attention path now safely rejects TurboQuant proxy objects it cannot process and falls back to the compatible attention path. This fixes the regression reported in #2778 and #2780. By @aaramos in #2782.

• Restored exact GDN SSD-cache snapshots as the default. GDN sidecar state now uses FP32 unless reduced precision is explicitly selected. Existing settings carrying the v0.6.0 lossy default are migrated back to FP32, preventing reduced-precision cache restoration from unexpectedly changing greedy outputs or breaking bit-level reproducibility. Reported in #2775.

• Fixed boundary-snapshot arrays remaining alive through a Python closure cycle. The scheduler now traverses snapshot state without creating a recursive closure that could retain large Metal-backed arrays. This prevents latent memory growth for reallocating cache implementations, including affected TurboQuant workflows. By @hojin12312 in #2809.

• Fixed VLM MTP profile conflicts during profile application. Applying output-shaping settings such as presence_penalty now preserves the requested settings and disables the incompatible speed-only VLM MTP toggle, matching request-time behavior. Remaining invalid combinations return an actionable HTTP 400 instead of an internal server error. By @jakeuj in #2806.

Distributed Serving and Cluster Management

• Added distributed thinking_budget enforcement. Rank workers now apply the same per-request thinking-budget processor used by single-machine inference, allowing distributed Qwen deployments to place a hard bound on reasoning without disabling it entirely. By @xunlinkx in #2731.

• Made the distributed request read timeout configurable. Long prefills and cold model reloads can use OMLX_DISTRIBUTED_REQUEST_READ_TIMEOUT to override the existing 300-second default. Invalid, non-finite, or non-positive values now fail with a clear configuration error. By @xunlinkx in #2714.

• Stopped idle cluster polling from repeatedly running the planner. The cluster dashboard still creates its initial plan and responds to explicit model or control changes, but no longer posts /plan every polling cycle. By @xunlinkx in #2721.

• Preserved the selected role of the local coordinator. Dedicated coordinator Macs can now remain configured as headless instead of being reset to the workstation role and unnecessarily reserving 32 GiB. By @xunlinkx in #2723.

• Improved compatibility reporting for older or uninitialized workers. Cluster preflight now handles peers predating omlx._version, distinguishes a missing worker runtime from an actual version mismatch, and keeps unverified workers non-ready until compatibility has been established. By @hellodk in #2737 and #2738.

MCP, Memory, and Admin Fixes

• Restored remote MCP Streamable HTTP connections with MCP SDK 2.x. The client now follows the SDK's reader/writer return contract, and package constraints consistently require MCP 2.x. By @nnhanndakara in #2776, addressing #2520.

• Fixed wired-memory limit recommendations on 128 GiB Macs. Backend warnings, the admin API, and the displayed iogpu.wired_limit_mb command now use the same safe whole-MiB value, eliminating an impossible rounding loop. By @jakeuj in #2799.

Upgrade Notes

• Qwen ANE/GPU prefill remains experimental, opt-in, and dependent on private Apple runtime interfaces. It requires the oMLX native custom kernels and may increase model loading time and memory use.

• The ANE split tuner takes several minutes because it performs full prewarm and end-to-end measurements across multiple temporary configurations. Saved model settings do not change until the recommended result is explicitly applied.

• Existing GDN SSD-cache settings using the legacy gdn_sidecar_state_dtype key are reset to exact FP32 storage. Users who intentionally accept reduced precision for smaller SSD snapshots can select it again in the updated settings UI.

• The distributed request timeout remains 300 seconds unless OMLX_DISTRIBUTED_REQUEST_READ_TIMEOUT is configured.

Thanks to @onthehub97, @xunlinkx, @hellodk, @nnhanndakara, @jakeuj, @hojin12312, and @aaramos for their contributions, and to everyone who reported and validated the 0.6.1 issues.

Full Changelog: v0.6.1...v0.6.2

0.6.1

oMLX 0.6.1

oMLX 0.6.1 focuses on Qwen3.8 performance and fixes several compatibility regressions found after 0.6.0. It adds opt-in dual-ANE/GPU prefill on M3 Ultra (+18.9% throughput at 32K context), accelerates Lightning MTP generation (+34.0% decode throughput at 16K context), and restores reasoning-effort compatibility, Qwen vision loading, model discovery, and tool-heavy prefix-cache reuse.

Experimental Qwen3.8 Dual-ANE/GPU Prefill

• Added experimental prompt processing across both ANEs and the GPU. Eligible fixed-size Qwen3.8 MLP and GDN prefill blocks split work across two physical ANEs and Metal, while decode, verification, residual chunks, and unsupported layers continue using the existing GPU path. The feature supports compatible AWQ and oQ4e layouts and is disabled by default. By @onthehub97 in #2756, extended and hardened in #2760.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, Lightning MTP disabled, a 2,048-token ANE block, and a 128-token generation tail:

Prompt	GPU PP	ANE/GPU PP	Improvement
4K	452.8 tok/s	458.5 tok/s	+1.3%
16K	439.1 tok/s	517.0 tok/s	+17.7%
32K	408.9 tok/s	486.0 tok/s	+18.9%

Longer prompts benefit more because they contain more complete fixed-size blocks. The 4K case contains only one 2,048-token ANE block followed by a 2,047-token GPU tail.

This path uses private Apple runtime interfaces and approximate INT8 weights. On the reference system, enabling it increased peak memory by about 4.15 GB and model loading from 3.35 seconds to approximately 27–29 seconds. Decode remains on the GPU, so this is a prompt-processing optimization rather than a token-generation acceleration.

Qwen3.8 Lightning MTP and Serving Performance

• Sped up Lightning MTP verification and Qwen3.8 prefill. Fused target-verification attention and GDN prework reduce dispatch overhead, while high-memory systems use larger prefill chunks when memory permits. #2751.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, Lightning MTP enabled, and a 128-token generation tail:

Prompt	Decode before	Decode after	Improvement
8K	80.7 tok/s	90.6 tok/s	+12%
16K	56.1 tok/s	75.2 tok/s	+34%
32K	51.9 tok/s	64.6 tok/s	+25%
64K	43.8 tok/s	53.4 tok/s	+22%

• Improved concurrent Qwen3.8 VLM throughput. When several requests are ready together, oMLX now prefers ordinary batching instead of mixing Lightning MTP and batched decoding on the same model. Batch-4 throughput improved from 69.34 to 108.14 tok/s for greedy decoding and from 64.68 to 103.79 tok/s for sampled decoding, while uncontended single-request Lightning MTP remains available. By @DiscoStew6082 in #2752.

• Fused Qwen MoE router top-k selection. The optimized Metal path improved Qwen3.6-35B-A3B decode throughput by 3.9% at 4K context and 2.5% at 16K in the reported workload. #2755.

Qwen3.8 VLM and Model Loading

• Fixed affected Qwen3.8 checkpoints falling back to text-only inference. Channels-first vision patch-embedding weights are normalized during MLX checkpoint loading, preserving full vision support and Lightning MTP. By @frank-beans in #2754.

• Fixed Qwen VLM checkpoints with root mtp.* weights. These weights are now mapped to the attached language-model MTP module instead of failing strict VLM loading and silently disabling vision. Reported in #2732.

• Restored intentionally filtered Hugging Face cache models to discovery. Auxiliary safetensors omitted by filtered downloads no longer cause otherwise loadable models to disappear from the model list. Reported in #2742.

Reasoning and Agent Compatibility

• Normalized reasoning_effort across model chat templates. Common client values are retried with compatible aliases or the model's native default instead of returning HTTP 400 when a template uses a different effort vocabulary. This fixes Qwen3.8 clients sending values such as high. Reported in #2740.

• Added explicit DeepSeek V4 reasoning-effort aliases. OpenAI- and Hermes-style values such as minimal, medium, xhigh, and ultra now map to the closest supported DeepSeek level. By @jonathan308 in #2724.

• Preserved prefix-cache reuse across tool-adjacent system messages. Claude Code system reminders that follow tool results can remain in place when the model template safely supports them, avoiding repeated large prefills in long tool-heavy sessions. Template-specific probing keeps the fallback safe for incompatible templates. By @q-p in #2753, addressing #2483.

Distributed Serving and Benchmarks

• Fixed false distributed-version mismatches. Cluster probes now compare local MLX and MLX-LM versions using the same source reported by the peer, so matching editable or nightly installations no longer fail enrollment because module and package metadata differ. By @hellodk in #2758.

• Improved community intelligence benchmark identity. Uploads now carry the organization-qualified model repository separately from the display name, allowing omlx.ai to distinguish checkpoints with the same leaf name without cluttering the model-name column.

Upgrade Notes

• Qwen ANE prefill remains experimental and disabled by default. It requires compatible native custom kernels and has currently been validated on M3 Ultra.

• Enabling or changing Qwen ANE settings reloads the model because fixed-shape ANE programs are compiled eagerly.

• Existing configurations remain compatible, and no manual settings migration is required.

Thanks to @onthehub97, @DiscoStew6082, @frank-beans, @q-p, @hellodk, and @jonathan308 for their contributions.

Full Changelog: v0.6.0...v0.6.1

0.6.0

oMLX 0.6.0

Compared with 0.6.0rc1, the final release adds community intelligence benchmark publishing and broader Qwen3.8 and Gemma 4 support while improving reasoning controls, embeddings, model discovery, and serving stability.

oMLX 0.6.0 introduces experimental distributed serving, more responsive concurrent inference, lower long-context memory use, new model families, built-in web search, speech-to-text, and community intelligence benchmark publishing.

Distributed Serving

• Added experimental distributed serving across Macs. Models can use tensor or pipeline parallelism with capability-aware planning, selective shard staging, memory guards, and one-click activation. Qwen3.6-27B reached 28.6 tok/s across two Macs versus 16.1 tok/s on one, while a 225 GB MiniMax-M3 checkpoint loaded across 128 GB and 256 GB Macs. By @ashhart in #2423.

• Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA workers can contribute to one logical pool with memory-aware layer placement, worker enrollment, ConnectX discovery, and NCCL verification. By @ashhart in #2591.

• Added SSD-backed prompt reuse for distributed ranks. Ranks restore only cache prefixes available across the cluster, with incremental storage that stays linear as context grows. #2620.

Distributed serving is included in the DMG but remains experimental and disabled by default. Enable it from Web Dashboard > Global Settings > Advanced > Distributed Inference, then save and restart oMLX.

Community Intelligence Benchmarks

• Intelligence benchmark results can now be published to omlx.ai. Eligible local runs upload a score summary and compressed per-question answers without uploading question text. Small or external-endpoint runs remain private, and both the web dashboard and macOS app show upload status and link to My Submissions. #2693.

Serving Responsiveness and Long-Context Memory

• Decode throughput during concurrent prefill improved by 1.6x to 43x. Prefill now yields GPU time to active decodes and adapts chunk size to a target stall time, while solo prefill performance remains unchanged. #2633.

• Mixed CacheList prefix storage is now linear instead of quadratic. The affected Inkling workload dropped from 282.7 GB at 84K tokens to 15.3 GB at 94K tokens while preserving byte-identical restores. By @Pietroski in #2550.

• GDN recurrent state now uses bounded SSD sidecars by default. The new policy keeps recurrent state separate from ordinary KV storage, with RHT-INT16 reducing storage by 1.93x versus FP32 while restoring in FP32. By @hojin12312 in #2569 and #2644.

• Memory pressure now reclaims MLX buffers before shrinking the hot cache. This avoids repeatedly discarding reusable prompt state while retaining immediate emergency-pressure enforcement. Reported in #2581, fixed by @PhilipJohnBasile in #2635.

Model Support and Performance

• Expanded Qwen3.8 support. oMLX now loads blockwise FP8, embedded MTP, and the mixed ModelOpt NVFP4 unsloth/Qwen3.8-27B-NVFP4 checkpoint directly. NAX prompt throughput for common group-size-128 checkpoints improved from 513 to 925 tok/s in the reported workload. By @scaryrawr, @LKyaha, and @True2456 in #2653, #2659, and #2657.

• DeepSeek V4 prefill is 23–28% faster across tested context lengths. Optimized attention, cache, indexer, and scheduler paths improve prompt throughput, while native handling of unaligned tails reduced peak memory from 142.255 GiB to 105.059 GiB in a 347K-token test. By @DiscoStew6082 and @jonathan308 in #2559, #2562, #2563, #2568, and #2674.

• Added Lightning MTP support for unified Gemma 4 checkpoints. Embedded assistant configurations are preserved, and depth-8 verification now restores rotating caches correctly after rejected drafts. By @djr747 and @adamw3455 in #2683 and #2690.

• Added Ling 3.0 Flash support. FP8, mixed FP4/FP8, oQ/oQe conversion, batching, and SSD prefix-cache restores are supported. The trained SwiGLU clamp improved measured HumanEval accuracy from 71.3% to 88.4%. By @scaryrawr in #2526, #2534, and #2571.

• Added Meta Muse Glimmer 30B VLM support with DFlash. Text, vision, reasoning, ATEM tool calls, prefix caching, and oQ quantization are supported, with about 1.4x decode speedup on the tested oQ4 pair. #2586, #2587, and #2643.

• Added complete Jina Reranker v3.5 support. Projector loading, sliding-window attention, dual matching, block fusion, and reference-score parity are included. By @damascoemi in #2449 and #2494.

• Sentence-transformers checkpoints now honor declared pooling metadata. CLS, last-token, and mean pooling are selected from the checkpoint instead of silently returning incompatible embeddings. By @cedricamram in #2594.

Chat, APIs, Reasoning, and Tool Calling

• Added built-in web search to admin chat. Models can use web_search and fetch_url through DDGS, DuckDuckGo, Brave, or SearXNG, with source cards and redirect-safe URL fetching. External API clients are unchanged. #2596.

• Added file and realtime speech-to-text. Compatible Whisper, Voxtral, and Qwen3-ASR models support file transcription and live microphone input with correct segmentation, language detection, and CJK streaming. #2603.

• Improved reasoning output and controls. Responses API streams prompt-opened thinking through dedicated reasoning events, while the API, web dashboard, and macOS app now preserve free-form or numeric reasoning-effort values such as xhigh and 0.9. #2623, #2675.

macOS App and Updates

• Reworked the self-updater to make bundle replacement crash-safe. Atomic bundle exchange prevents interrupted updates from leaving an incomplete or damaged app. This addresses the macOS 27 failures reported in #2522.

Affected macOS 27 users upgrading from a build older than 0.6.0rc1 should install 0.6.0 manually from the DMG once. The new updater handles subsequent releases safely.

Miscellaneous

• Improved pressure handling, DeepSeek V4 prefill memory estimates, cache cleanup, hot-cache statistics, repeated aborts, and active-engine unload safety. #2512, #2639.

• Fixed DeepSeek V4 generation anchors, tool-turn round trips, parser edge cases, and thinking-mode benchmark extraction. #2510, #2513, #2661.

• Fixed NVFP4 oQ sensitivity analysis, incomplete sharded-model discovery, default macOS cluster node names, and Homebrew installation guidance. #2433, #2432, #2679, #2694.

• Improved accessibility, Chinese typography, localization, network binding, Dock behavior, and Codex/Claude launcher integration. #2542, #2671, #2480.

• Reorganized Global Settings, refreshed model presets, added MCP exposure and chat tool-round controls, honored web-search result limits, and surfaced unenforced Responses API formats. #2560, #2699, #2696, #2702.

• Improved cluster runtime discovery, mixed Python minor-version support, version status reporting, and macOS server restart recovery. #2697, #2704, #2705.

Upgrade Notes

• Distributed inference remains experimental and disabled by default. Enable it under Global Settings > Advanced, save, and restart oMLX.

• Decode fairness is enabled by default. Set decode_fairness=false to restore the previous concurrent-prefill behavior.

• GDN storage defaults to auto with RHT-INT16. Existing explicit storage and FP32 settings are preserved.

• Legacy mixed-CacheList SSD blocks are invalidated automatically because the new linear format is incompatible.

• DeepSeek V4 temperature-0 output may differ at near-tied token choices because optimized kernels change floating-point reduction order.

• Use...

0.6.0rc1

oMLX 0.6.0rc1

This is the first release candidate for oMLX 0.6.0. It rolls up every change since 0.5.7, including the 0.5.8 development releases and 0.6.0.dev1.

This release introduces experimental distributed serving, keeps active generation responsive during concurrent prefills, substantially improves long-context cache and memory behavior, adds several new model families, and expands the admin chat with built-in web search and speech-to-text.

Distributed Serving

• Added experimental distributed serving across Macs. Models can be split with tensor or pipeline parallelism, with capability-aware planning, selective shard staging, per-rank memory guards, liveness supervision, and one-click activation from the Cluster dashboard. Qwen3.6-27B reached 28.6 tok/s across two Macs versus 16.1 tok/s on one Mac with byte-identical output. A 225 GB MiniMax-M3 checkpoint that could not fit either machine alone also loaded across 128 GB and 256 GB Macs and passed its distributed canary. By @ashhart in #2423.

• Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA CUDA workers can contribute to one logical model-memory pool, with memory-aware contiguous-layer placement, short-lived worker enrollment, ConnectX discovery, and NCCL verification. The current compatibility path keeps physical workers in the outer MLX Ring; the hierarchical Ring-to-NCCL gateway remains future work. By @ashhart in #2591.

• Added SSD-backed prompt reuse for distributed ranks. Each rank stores a process-lifetime chain of cache-boundary snapshots and restores only prefixes available on every rank. Incremental segments keep storage linear: a 12K-token GLM-5.2 chain used 1.17 GB instead of roughly 6.5 GB with cumulative copies. #2620.

Distributed serving is included in the DMG but remains experimental and disabled by default. Enable it from Web Dashboard > Global Settings > Advanced > Distributed Inference, then save the settings and restart oMLX.

Serving Responsiveness and Long-Context Memory

• Active decodes now remain responsive during concurrent prefill. Prefill yields GPU time to running decodes, including streams in another engine, and sizes contended chunks by a target stall time. The feature is enabled by default and can be changed live. Across the tested workloads, overlap decode improved by about 1.6x to 43x; contended prefill took 1.15–2.0x longer while solo prefill remained unchanged. #2633.

• Mixed CacheList prefix storage is now linear instead of quadratic. Sliceable KV members are stored per block while small boundary states remain attached to their checkpoints. On the affected Inkling workload, the legacy path used 282.7 GB for an 84K-token session; the new layout used 15.3 GB for 94K tokens and restored byte-identical output. Incompatible legacy blocks are invalidated automatically on first load. By @Pietroski in #2550.

• GDN recurrent state now uses bounded SSD sidecars by default when the SSD cache is available. The new auto | ssd_sidecar | embedded policy separates recurrent state from ordinary KV storage. RHT-INT16 is the new default sidecar codec, measuring 1.93x less storage than FP32 with 0.002477% mean relative L2 error; restored state still runs in FP32. FP32, BF16, INT8, and RHT-INT8 remain selectable. Existing explicit legacy settings and FP32 deployments are preserved. By @hojin12312 in #2569 and #2644.

• Memory pressure now reclaims the MLX buffer pool before shrinking the hot cache. Reclaimable Metal buffers no longer cause reusable prompt state to be repeatedly discarded. Emergency pressure still enforces immediately, while ordinary hard pressure receives a bounded reclaim grace period. Reported in #2581, fixed by @PhilipJohnBasile in #2635.

• Adaptive prefill admission now accounts for recently reclaimed memory, waits briefly for asynchronous cache cleanup before rejecting long requests, and uses DeepSeek V4’s actual hybrid cache layout instead of the generic KV estimate. Hot-cache promotion failures are also exposed through runtime cache statistics.

Model Support and Performance

• Added Qwen3.8-27B FP8 support and official reasoning levels. Blockwise FP8 checkpoints and embedded MTP tensors load through the Qwen3.5-family compatibility path. reasoning_effort in Chat Completions and reasoning.effort in Responses now reach the model without overriding explicit template arguments. By @scaryrawr in #2653.

• Restored Qwen3.8/Qwen3.5 group-size-128 prefill performance on NAX hardware. Common four-bit checkpoints no longer route through a slower classic Metal fallback when stock MLX can use NAX. On the reported Qwen3.8-27B workload, prompt throughput improved from 513 to 925 tok/s and end-to-end time fell from 10.4 to 6.8 seconds. Pre-NAX Macs retain the existing optimized path. By @True2456 in #2657.

• Added Ling 3.0 Flash support. This includes FP8, official mixed FP4/FP8 checkpoints, oQ/oQe conversion, variable-length batching, and SSD prefix-cache restores. Applying the trained per-layer SwiGLU clamp improved measured HumanEval accuracy from 71.3% to 88.4%. Original implementation and FP4 support by @scaryrawr in #2526, #2534, and #2571.

• Added Meta Muse Glimmer 30B VLM support with DFlash speculative decoding. Text, vision, channel-scoped reasoning, ATEM tool calls, prefix/SSD cache, and oQ quantization are supported. DFlash reached about 1.4x decode on the tested oQ4 pair; image requests continue through the VLM fallback. The vendored implementation was also synchronized with upstream reasoning and FP32 numerical behavior. #2586, #2587, and #2643.

• Added complete Jina Reranker v3.5 support. Projector loading, sliding-window attention, dual matching, block fusion, and reference-score parity are included. By @damascoemi in #2449 and #2494.

• DeepSeek V4 prefill is 23–28% faster across tested context lengths. Native attention routing, 2048-token cache blocks, fused windowed and pooled attention, skipped intermediate lm_head work, in-place pooled-cache appends, and a fused indexer mask improve prompt throughput without changing decode speed. By @DiscoStew6082 and @jonathan308 in #2559, #2562, #2563, and #2568.

• DeepSeek V4 unaligned final prefill tails stay on the native indexer path. A 347,929-token validation reduced full-model MLX peak memory from 142.255 GiB to 105.059 GiB by avoiding a large FP32 fallback workspace. The fix handles partial Metal tiles directly rather than globally splitting scheduler chunks. Reported in #2627, fixed in #2674.

• Added MTPLX sidecar MTP import for compatible Qwen3.6 checkpoints, fixed DeepSeek V4 affine AWQ expert bias/zero-point loading, and removed a per-token host synchronization from grammar-constrained decoding.

Chat, APIs, Reasoning, and Tool Calling

• Added built-in web search to admin chat. Models can use web_search and fetch_url with keyless DDGS metasearch or optional DuckDuckGo, Brave, and SearXNG providers. Results remain visible as source cards across tool rounds. URL fetching rechecks private and loopback destinations after every redirect and caps content before conversion. External API clients are unchanged because only the chat UI advertises these tools. #2596.

• Added file and realtime speech-to-text to chat. Audio STT models switch the composer into transcription mode, while compatible Whisper and Voxtral models expose live microphone input over WebSocket. Whisper file streaming now uses bounded segments with the full anti-hallucination path and per-segment language detection. Qwen3-ASR uses cumulative decoding so multi-token CJK characters remain intact. #2603.

• Fixed streaming Responses reasoning for prompts that already open <think>. DeepSeek V4 reasoning now streams through dedicated reasoning events, creates a separate reasoning item, and reports reasoning token usage instead of leaking thought text through response.output_text.delta. Confirmed with a real checkpoint on Apple Silicon. Reported in #2584, fixed by @leepokai in #2623.

• Hardened tool-call parsing. Tool arguments containing literal close markers are preserved, deeply nested model output now fails closed instead of escaping as RecursionError or SyntaxError, and a malformed call no longer drops a valid neighboring call. #2544, #2593.

• Fixed VLM and Muse Glimmer tool-call paths. VLM requests now carry tool schemas through to protocol parser sessions, preserving JSON-looking values declared as strings. Muse Glimmer also keeps tool call...

0.6.0.dev1

This development release introduces opt-in distributed model serving across Macs and mixed Metal/CUDA nodes, adds built-in web search and streaming speech-to-text to chat, keeps active generation responsive during concurrent prefills, and brings SSD-backed prompt reuse to distributed ranks.

• Added experimental distributed serving across Macs. One model can now be split across multiple Apple Silicon machines using tensor or pipeline parallelism, with capability-aware planning, selective shard staging, per-rank memory guards, liveness supervision, and one-click activation from the Cluster dashboard. Qwen3.6-27B tensor parallel reached 28.6 tok/s across two Macs versus 16.1 tok/s on one Mac, with byte-identical output; a 225 GB MiniMax-M3 checkpoint that could not fit either machine alone also loaded across 128 GB and 256 GB Macs and passed its distributed canary. The feature is disabled by default under Settings > Advanced while it remains an experimental source-build preview. By @ashhart in #2423, with follow-up fixes for SSH failure handling, clock-safe liveness, MLA/MTP memory accounting, and load admission.

• Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA CUDA workers can contribute to one logical model-memory pool, with automatic contiguous-layer placement based on usable memory, short-lived GUI-generated worker enrollment, ConnectX discovery and NCCL verification, and cluster diagnostics in the dashboard. The current compatibility path keeps every physical worker in the outer MLX Ring; verified CUDA pairs are placed together, but the future hierarchical Ring-to-NCCL gateway is not implemented yet. By @ashhart in #2591.

• Added decode fairness during concurrent prefill. Prefill now yields GPU time to active decodes, including streams running in another engine, and sizes contended chunks by a target stall time instead of a fixed token count. The feature is enabled by default and can be changed live from the dashboard. On an M3 Ultra, overlap decode improved by about 1.6x to 43x across the tested model pairings, while contended prefill took 1.15-2.0x longer and solo prefill remained unchanged. By #2633.

decoding + prefilling	solo	before	after
Qwen3.6-27B + Qwen3.5-0.8B prefilling 21k	50 tok/s	11-18 tok/s	23-24 tok/s
Qwen3.5-0.8B + Qwen3.6-27B prefilling 8.4k	203 tok/s	1.7-1.8 tok/s	76-78 tok/s
DeepSeek-V4-Flash + Qwen3.6-27B prefilling 8.4k	34 tok/s	0.3 tok/s	11-12 tok/s
Qwen3.6-27B + DeepSeek-V4-Flash prefilling 7.7k	47 tok/s	2.2-2.4 tok/s	15-21 tok/s

• Added process-lifetime SSD prompt caching for distributed ranks. Each rank now stores a chain of cache-boundary snapshots and restores the longest prefix held by every rank when the in-memory prompt cache misses. Incremental KV segments keep storage growth linear instead of copying the full prefix at every boundary; a 12k-token GLM-5.2 chain used 1.17 GB instead of about 6.5 GB. Snapshots are scoped to the current rank process and are cleaned up on restart or teardown. By #2620.

• Added built-in web search to the admin chat. The chat composer can expose web_search and fetch_url tools to the selected model, with keyless DDGS metasearch by default and optional DuckDuckGo, Brave Search API, or SearXNG providers. Results stay visible as source cards across tool rounds, while fetched URLs are checked against private and loopback destinations on every redirect and capped before MarkItDown conversion. External API clients are unchanged because only the chat UI advertises the built-in tools. By #2596.

• Added file and realtime speech-to-text to chat. Selecting an audio STT model switches the composer into ASR mode for streamed audio or video transcription, and compatible models also expose live microphone input over WebSocket. Whisper file streaming now uses bounded segments with the full anti-hallucination path and per-segment language detection, while Qwen3-ASR uses cumulative token decoding so multi-token CJK characters no longer turn into replacement characters. Multipart request logging also skips binary bodies instead of buffering and printing them. By #2603.

• Improved Meta Muse Glimmer and DFlash. Long Muse Glimmer oQ4 prefill now routes quantized projections through the native QMM tile, measuring about 4.5% higher prompt throughput on a cold 51k-token prompt with bit-exact output. DFlash responses now preserve parser-emitted tool calls instead of dropping them during speculative generation.

• Fixed MTP performance state after cross-engine prefill contention. The adaptive loop-tax probe could sample a handoff while another engine was prefilling, latch the resulting slowdown onto the model, and suppress MTP performance until restart. Contaminated measurements are now skipped or discarded, and stale high measurements decay back toward the default margin.

• Fixed DeepSeek-V4 affine AWQ checkpoint loading. Per-expert bias and zero-point tensors are now stacked with weights and scales during sanitize, so externally quantized affine checkpoints no longer fail strict weight loading with thousands of leftover *.biases keys. By @True2456 in #2598.

• Improved oQ discovery for Hugging Face cache models. Original, unconverted checkpoints inside the Hugging Face snapshot cache now appear as quantization sources with their canonical repository identity instead of a snapshot hash. By @monroewilliams in #2605, with a follow-up identity-preservation fix.

• Exposed model-management state through the API. /v1/models/status now includes is_favorite and is_hidden, allowing API clients to reproduce dashboard filtering without an additional model-list request. By @monroewilliams in #2602.

New Contributors

• @ashhart made their first contribution in #2423.

0.5.8.dev3

This development release adds Meta Muse Glimmer 30B with DFlash speculative decoding, makes DeepSeek V4 prefill 23-28% faster across context lengths, persists GDN recurrent state as bounded SSD sidecars, and fixes Ling accuracy, grammar decoding overhead, and tool-call parsing edge cases.

• Added Meta Muse Glimmer 30B VLM support with DFlash speculative decoding. The model implementation (hybrid SWA/NoPE text backbone, ViT-G/14 encoder) is vendored from the upstream mlx-vlm port by @Blaizzy (Blaizzy/mlx-vlm#1838), plus the quantized embedding-norm fix by @johntdavies (Blaizzy/mlx-vlm#1839). Thank you both for the quick upstream turnaround. On the oMLX side this adds channel-scoped output parsing with ATEM tool calls, prefix and SSD cache integration, and oQ quantization (#2586), and DFlash speculative decoding is wired in #2587, reaching about 1.4x decode on the oQ4 pair.
• Faster DeepSeek V4 prefill, 23-28% across context lengths. Layered across four changes following the ablation in #2558: ratio-128 layers now route through the native attention kernel (#2559), the MXFP4 MoE block threshold matches 2048-token chunks (#2562), and redundant all-pooled indexer scoring is skipped (#2563), all by @DiscoStew6082. DeepSeek V4 cache blocks now target 2048 tokens so those kernels engage with the prompt cache enabled. On top of that, a fused windowed+pooled prefill kernel, skipping the discarded lm_head projection on intermediate chunks, in-place pooled cache appends, and a fused indexer mask, by @jonathan308 in #2568. Measured on an M3 Ultra (V4-Flash 4-bit, default server config, cold prompts, prompt processing tok/s):

stage	pp4096	pp16384	pp32768	pp65536	pp131072
0.5.8.dev2	496.5	481.2	–	453.3	398.4
+ #2559 #2562 #2563	491.0	475.2	–	442.4	405.4
+ 2048-token cache blocks	549.4	558.7	549.5	524.2	461.6
+ #2568 (= 0.5.8.dev3)	631.1	604.9	589.3	559.1	508.5

The kernel PRs need 2048-token prefill chunks to pay off, which is why the middle row is flat: the block-alignment change is what unlocks them under the default cache-enabled config. Decode speed is unchanged at every context length. Note that temperature-0 outputs can differ from 0.5.8.dev2 for identical prompts: the new kernels change floating-point reduction order, which only flips tokens where the model is effectively indifferent, and quality is unaffected.

• Reduced long-context memory for GDN models when the SSD cache is enabled. GDN recurrent state is now persisted as bounded SSD sidecars instead of retaining one full in-memory state per historical boundary, making long-context Qwen3.6 serving practical on unified-memory Macs. Opt-in and backward-compatible. By @hojin12312 in #2569.
• Fixed Ling-3.0-flash code accuracy, +17pp on HumanEval. Its late layers were running without the trained SwiGLU clamp shipped in config.json; applying it raises HumanEval from 71.3% to 88.4% at no runtime cost. By @True2456 in #2571.
• Removed a per-token host sync from grammar-constrained decoding. Token acceptance is deferred to the top of the next step with bit-identical output, so the forward dispatch never blocks on host work. By @fxd0h in #2561.
• Improved adaptive prefill memory estimates at high context. The estimator now remembers process footprint released after a chunk and accounts for it when sizing the next one, closing an over-admission gap. By @mvdbos in #2573.
• Fixed tool calls whose arguments contain a literal close marker. These were dropped with raw markup leaking into the response; payload boundaries are now located by span scanning with JSON-aware decoding, in both the batch parser and the streaming filter. By @richgoodson in #2544, with follow-up fixes for prose around malformed envelopes at end of stream and for JSON array payloads.

New Contributors

• @DiscoStew6082 made their first contribution in #2559.
• @fxd0h made their first contribution in #2561.
• @mvdbos made their first contribution in #2573.

0.5.8.dev2

This development release fixes quadratic mixed CacheList storage and MTP request head-of-line blocking, hardens Ling and Inkling tool-call parsing, and improves Codex launch isolation, dual-stack networking, model compatibility, and localization.

• Fixed quadratic SSD and memory growth when storing long-context prefixes for models with mixed CacheList layers. oMLX now stores sliceable KV members per block while preserving small boundary states, with validation and upgrade handling for legacy cache entries. Original implementation by @Pietroski in #2550.
• Fixed head-of-line blocking when requests arrive during single-stream MTP decoding. Late arrivals can now join promptly through a safe drain handoff instead of waiting for the active generation to finish. Reported by @anicaise-ai in #2515.
• Fixed Ling role and XML tool-protocol markers leaking into visible output, while preserving structured tool-call parsing and request-scoped tool schema validation. By @scaryrawr in #2528.
• Fixed Inkling tool calls using OpenAI-style arguments payloads or truncated nested JSON. The parser now accepts supported payload variants and repairs incomplete braces only after strict JSON parsing fails. By @studioburnside in #2555.
• Fixed omlx launch codex modifying persistent Codex configuration. Provider settings and the model context window are now passed only to the launched process. By @wolfyy970 in #2519.
• The macOS app's “All networks” listen mode now uses the dual-stack IPv6 wildcard, allowing both IPv4 and IPv6 clients while retaining an explicit IPv4-only option. By @tannerdsilva in #2552.
• Localized the Runtime Cache Observability panel in the admin dashboard and added matching strings to every supported locale catalog. By @LXD-8 in #2540.
• Added MTPLX side-car MTP import for compatible Qwen3.6 checkpoints, including contract validation, normalization to model-mtp.safetensors, and a one-click dashboard import path. Original implementation by @rsnow in #2497.
• Added complete Simplified Chinese localization for the macOS app and filled the remaining Chinese translations in the admin dashboard. By @luziyi in #2554.
• Updated MarkItDown to 0.1.7, bringing upstream PPTX conversion performance and compatibility fixes. #2466.

New Contributors

• @Pietroski made their first contribution in #2550.
• @studioburnside made their first contribution in #2555.
• @tannerdsilva made their first contribution in #2552.
• @LXD-8 made their first contribution in #2540.
• @luziyi made their first contribution in #2554.