Repository navigation
oMLX 0.8.0
oMLX 0.8.0 adds the Prefill Accelerator, which lets a PC with NVIDIA GPUs or a second Mac speed up prefill for a big model on your Mac, adds oQh for more accurate low-bit quants at the same size, and makes DeepSeek V4.1, Qwen and long-context decode faster.
Download: macOS 26 / 27 | macOS 15 Sequoia
Prefill Accelerator with NVIDIA GPUs or a Mac
A PC with NVIDIA GPUs, a DGX Spark or a second Mac with a faster GPU runs the first layers of each prompt chunk, while your Mac runs the rest, decodes and keeps the KV cache. Auto picks how many layers the helper takes from the measured speed of both machines and the link, and keeps the prompt on the Mac when a split would be slower (#5065, #5163, #5195). oQh models and Inkling 2 Small run on both kinds of helper (#5391, #5234).
The CUDA helper builds on the great CUDA kernels of TensorFold by @ashhart and mlx-serve by @ddalcu, and would not have been possible without them. Thank you both for sharing them in the open. The adapted kernels and their licenses are in third_party/.
Prefill tok/s on a 64K prompt, oQ4e models, M3 Ultra 512GB with a CUDA helper over 2.5 GbE. Decode speed stays about the same.
| Model | M3 Ultra alone | + DGX Spark | + 2x RTX A6000 |
|---|---|---|---|
| Qwen3.8 27B | 352 | 1,062 (+202%) | 1,879 (+434%) |
| Qwen3.8 Flash-Next | 1,370 | 2,715 (+98%) | 3,928 (+187%) |
| DeepSeek V4.1 Flash | 1,085 | 2,412 (+122%) | 1,914 (+76%) |
| GLM-5.3 Flash | 555 | 1,076 (+94%) | 978 (+76%) |
| MiniMax M3.1 Flash | 312 | 504 (+62%) | 468 (+50%) |
A second Mac works the same way. With an M5 Max 128GB as the helper, DeepSeek V4.1 Flash prefills a 64K prompt at 3,123 tok/s instead of 1,060 (+195%), and Thunderbolt or 5GbE is enough.
Turn on Global Settings > Advanced > Enable Distributed Inference, pair the helper in the Cluster tab, then pick it under Prefill Accelerator in the model's settings (guide). An NVIDIA PC needs only the command below and the code from docker logs omlx-helper, with no SSH or oMLX install. It needs driver 580 or newer and an Ampere or newer GPU or a GB10, and has not been tested on Windows yet.
docker run -d --name omlx-helper --restart unless-stopped --gpus all --network host -v omlx-helper:/data jundot/omlx-cuda-helper
oQh and More Accurate oQe
oQh is a new oQe option. At the same file size it stays closer to the original model, most of all at 2 and 3 bits, and runs at about the same speed (#5317). It supports Qwen3.5 to 3.8, Qwen3.8 Flash-Next, GLM-5.3 and DeepSeek V4.1. oQe itself is also more accurate at the same size (#5292), and Qwen3.5 and 3.6 models larger than your Mac's memory can now be quantized (@keefar in #5332).
FLORES+ KL against the original model in 7 languages, lower is better. All three builds in a row are the same size.
| Model | Level | Size | oQe before | oQe now | oQh |
|---|---|---|---|---|---|
| Qwen3.5 9B | oQ2 | 3.3 GiB | 0.54 | 0.48 (-11%) | 0.25 (-54%) |
| Qwen3.6 35B-A3B | oQ2 | 11.8 GiB | 0.23 | 0.22 (-4%) | 0.11 (-52%) |
| Qwen3.8 27B | oQ3 | 12.2 GiB | 0.066 | 0.059 (-11%) | 0.040 (-39%) |
| Qwen3.8 Flash-Next | oQ3 | 84.9 GiB | 0.082 | 0.073 (-11%) | 0.060 (-27%) |
Turn on Rotated codes (oQh) in the oQe section, and the output name ends in h, for example Qwen3.5-9B-oQ2.5h (guide).
Faster DeepSeek V4.1, Qwen and Long Context
Since 0.8.0rc1, Qwen models decode faster with many requests, with two long requests at once and on MoE models, and Flash-Next keeps its speed with its tables on SSD. Before that, the development and rc1 releases made DeepSeek V4.1 faster with many requests and on clusters, and Affine KV compression much faster at long context.
Since 0.8.0rc1
| Model and workload | Change | Hardware | PR |
|---|---|---|---|
| Qwen3.8 Flash-Next, SSD tables not yet cached, 8 requests, MTP off | Total decode 133.7 -> 200.9 tok/s (+50%) | M5 Max | @yohann-bearzi in #5225 |
| Qwen3.8 27B oQ8 with MTP, 25K + 100K requests at once | Total decode 27.6 -> 40.8 tok/s (+48%) | M5 Max | @StefVanG in #5167 |
| Qwen3.8 27B oQ4e-mtp, 8 requests | Total decode 222.6 -> 307.1 tok/s (+38%) | M5 Ultra | @Li-Lee in #5309 |
| Qwen3.6 35B-A3B oQ4 | Decode 100.7 -> 112.2 tok/s (+11%) | M3 Ultra | #5370 |
| Qwen3.6 35B-A3B 4-bit, 4 requests, DFlash2 | Total decode 331 -> 365 tok/s (+10%) | M5 Max | #5344 |
| DeepSeek V4.1 Flash oQ2e-mtp, 32K prompt | Prefill 3,802 -> 3,948 tok/s (+4%) | M5 Ultra | @junmo-kim in #5376 |
- Affine KV decode with DFlash is 4% faster at 128K (@avifenesh in #5248).
- Loops of short prompts keep their prefix cache hits, batched steps read padding offsets once, and top-p MTP verify picks from the top 512 candidates (@masterfasa in #5324, #5314, #5313).
- Image prefixes hash in one pass, with about 4x less hashing work per request (@LXD-8 in #5202).
From 0.7.4.dev1 and 0.8.0rc1
| Model and workload | Change | Hardware | PR |
|---|---|---|---|
| DeepSeek V4.1 Flash oQ3e-mtp, DSpark, 8 requests | Total decode 73.2 -> 107.9 tok/s (+47%) | M3 Ultra | #5154 |
| DeepSeek V4.1 Flash oQ3e-mtp, DSpark, 2 requests, pipeline | Total decode 46.7 -> 61.7 tok/s (+32%) | M3 Ultra + M5 Max | #5154 |
| DeepSeek V4.1 Flash oQ4e, tensor parallel (estimated) | Prefill 742 -> 1,284 tok/s (+73%), decode 19.8 -> 28.8 tok/s (+45%) | M3 Ultra + M5 Max, Thunderbolt RDMA | #5086 |
| DeepSeek V4.1 Flash oQ2e-mtp, 32K prompt | Prefill 2,631 -> 3,779 tok/s (+44%) | M5 Ultra | @junmo-kim in #5031 |
| Qwen3.8 Flash-Next oQ2.5e FP16, 32K | Decode 38.9 -> 63.0 tok/s (+62%) | M3 Ultra | #5028 with @qoli |
| Flint 40B-A2.5B oQ6e, Affine8 KV | Decode 22.8 -> 36.1 tok/s (+58%) | M5 Pro | @avifenesh in #5095 |
- DeepSeek V4.1: the first decode on a cold shader cache goes 12.4 -> 45.8 tok/s (+269%) (@junmo-kim in #5025), and MoE expert offload runs the experts already in memory first, for 17% lower TTFT (@cropduster in #4959).
- Qwen: two long requests at once decode up to 30% faster (@StefVanG in #5090), and 6-request 8-bit decode is 16% faster (b5c7793 with @yohann-bearzi).
- On 128 GB Macs, back-to-back long Flash-Next prompts stop slowing down, 90.7 -> 13.1 s for the slowest, and decode after long prompts is 3-5% faster on M5 Max (@yohann-bearzi in #4902, #4901).
- Attention: models with attention sinks like gpt-oss prefill about 4x faster with Affine KV (@avifenesh in #5088, #5101).
- MiniMax M3.1: prefill on M5 runs on the tensor units, up to 43% faster per MoE layer at 64K (#5084, #5157, #5083).
Changes since 0.8.0rc1
If you run 0.8.0rc1, these are the changes in this release.
- New:
- oQh and more accurate oQe builds, described above (#5317, #5292, @keefar in #5332).
- Inkling 2 Small loads with text and image input and Lightning MTP on by default, and runs on Mac and NVIDIA Prefill Accelerators (#5229, #5234).
- oQh models run on the Prefill Accelerator with CUDA helper 0.1.6 (#5391).
- Integrations launch from one Applications card in the web UI and the macOS app, and the Claude tile holds the Claude Code settings (@LXD-8 in #5238, #5243, #5244).
- Splash gets
--activations, with bf16 attention on M5 and later by default (@Woody-Hu in #5337). - Benchmark runs with a generation length other than 128 skip the leaderboard upload, and the banner says why (@OmVidana in #5338, ed045bb).
- Faster: see "Since 0.8.0rc1" in the speed highlight above.
- Fixed:
- The Cluster tab finds a DGX Spark or NVIDIA PC from the macOS app and Homebrew installs (bedde0c, f265b65).
- CUDA helpers keep prefix cache hits and stop growing memory between jobs, and a DGX Spark stays within the memory it planned with (#5316, @GumbiiDigital in #5271).
- MoE expert offload keeps its resident experts when a prefill runs short of memory, releasing only the slots it needs (@JarJarFran in #4842).
- Tool calls with array parameters split across JSON fragments are merged, and gpt-oss replies keep their completed messages when the last header is malformed (@Woody-Hu in #5345, #5224).
- Thinking defaults are read by rendering the chat template, so GLM-4.6V and GLM-OCR report thinking on (44b0525).
- Cluster replies that arrive only as reasoning come back as content, and peer model lists and free space read correctly (@joshuaswarren in #5328, 0d13e98, f897a00 with @digitalfactory-ako).
- Model lists survive bad files, skipping an unreadable safetensors header or Hugging Face refs file, and a
~DFlash drafter path is expanded (@phaedrus1992 in #5379, @joshuaswarren in #5329, @avifenesh in #5212). - Qwen3-Reranker scores no longer tie at 1.0 for confident documents (@Jason-Barbour in #5207).
- More setups take a working path, with MiniMax M3 prompts past 4K tokens on M2 Ultra and Qwen prefill on M5 source installs without the native kernels (@begcdn in #5239, f6782ad).
- Benchmarks report real numbers, as batch tests wider than Max Concurrent Requests no longer inflate tg TPS and code benchmarks no longer hang (@zviratko in #5312, bf28cd6, @Maheidem in #5241).
- The dashboard shows live numbers, with per-request tok/s over the last 3 seconds, Splash requests counted in
/api/statusand update checks that look past other channels' releases (#5075, @Woody-Hu in #5335, @isnaenihidayat in #5347). - Settings and apps behave better, with the profile repair on older settings files, cleared aliases in the macOS app, the camera notch named when the menu bar icon is hidden, and
omlx launch claudekeeping the auto-compact window in range (#5216, @ngutech21 in #5236, @fabriziosalmi in #5208, @masterfasa in #5319). - Tests are cleaner, with one pytest config and no state leaks between files, and the macOS app tests pass on macOS 27 (@shaanchander in #5218, @ngutech21 in #5303, @contrapuntal in #5277, e35c2a2 with @yanzhaohui1999, @digitalfactory-ako in #5363).
New Features and Improvements
- Concurrent requests: the new Concurrent Request Priority setting decides what runs first when a prompt arrives while others stream. Its Balanced default keeps running streams at 10 tok/s or more and cuts the new prompt's time to first token by 26% (#5164).
- A short request that arrives during a long prefill starts after the running chunk, so its first token comes in 4.4 s instead of 13.4 s (@arnavprabhu in #5160, #5175).
- Each row on the dashboard has a stop button that ends one request without unloading the model (#5156).
- Integrations:
omlx launch codex,codex_appandcopilothand the agent every chat model on the server, so/modelswitches between them (#5026 with @scaryrawr, @scaryrawr in #5066). - macOS app:
- Settings get a sidebar search, plus Web Search and MarkItDown in Integrations (@ngutech21 in #5111, #5172, #5107).
- The model library list is simpler and shows aliases, which the copy button now copies (@ngutech21 in #5091).
- The app speaks Italian (@fabriziosalmi in #4506, #5198), and window counts can show exact digits (@LXD-8 in #4641).
- Web UI: a new Theme Settings editor sets the dashboard's light and dark colors, checks contrast, drafts dark colors from light ones, and saves themes as JSON or images (#5037, #5038, @LXD-8 in #5097, #5098, #5102, #5113, #5114, #5115, #5117).
- GPU keep-warm: Keep GPU Awake After Requests in Advanced settings sets how long the GPU stays warm after the last request, 5 minutes by default (@parasew in #5085).
- Server: a long text prompt no longer stalls other streams while it is prepared, and plain streamed text skips the tool-marker scan (@Woody-Hu in #5140, @LXD-8 in #4945).
- Benchmarks: local accuracy runs take a Max Tokens Override so thinking models are not cut off (#5189), labels spell PP and TG (@LXD-8 in #4619), and pp result logs include
gen_tps(@yanzhaohui1999 in #5126). - Dependencies and builds: transformers up to 5.22 (#5131), dev tools as a PEP 735 dependency group (@ngutech21 in #5188), and CI builds and tests the macOS app (@ngutech21 in #5120, #5119, #5174).
Bug Fixes
- INT8 Activation Prefill stays off unless a model turns it on, with a "May reduce quality" note in Model Settings and its own SSD cache blocks (#5108, f61b29e).
- Unloading a model no longer affects the models still loaded, as the Metal wired limit stays in place (#5165).
reasoning_effort: "none"and"off"turn thinking off, instead of mapping tolow(@syntax922 in #5151).- Tool calls are parsed more reliably, with tool names like
readcounted only in call position and schema keys in their usual order (@Flo5k5 in #4433, #5161). - Tool call errors are clearer, with a coded error for an empty
tool_callsfinish and escaped model text in harmony logs (@joshuaswarren in #5139, @yanzhaohui1999 in #5125). - Memory no longer grows with many distinct JSON schemas, as the compiled-grammar cache is capped at 1 GiB (#4321 with @yanzhaohui1999).
- Prefill Accelerator helpers recover instead of failing, with OOM step-down, layers handed back for long prompts, and retries when offline (#5180, d83d1d4, @yanzhaohui1999 in #5169).
- Clusters pass DeepSeek V4.1 tool arguments correctly, and the link gate names the slow link when it blocks tensor parallel (ca70d2b with @twiedenbein, @hellodk in #3024 with @HamedShams).
- Speculative decoding stays on its fast paths, with wide TurboQuant verify windows on the fused kernel and DFlash drafter context kept on exact prefix hits (#4866, 54f6797, 22f2c43 with @avifenesh).
- Qwen3.8 Flash-Next runs cleanly on M1 and M2, taking the MLX path instead of failing the fused kernel probes (@begcdn in #5089).
- VLM and audio requests no longer fail silently, with Nemotron Omni audio read from
sound_clips, Florence-2 skipped at discovery, and realtime STT fixes (@shaanchander in #5162, @Woody-Hu in #5142, @Aminark in #5092, #5043). - oQ handles more models, with oQe on Spark-X and loadable LFM3-Omni output (@Woody-Hu in #5184, @arnavprabhu in #5153).
- Model Settings saves work again after a draft is deleted, with profiles checking their drafts and rejected saves naming the field (@LXD-8 in #4401, #5105, #4513).
- Splash follows oMLX more closely, with its SSD cache off when oMLX's is off, MTP shown as unused, a clean stop and fixed bench runs (#5082, @yanzhaohui1999 in #5168, #5078, #4938 with @OmVidana).
- Integrations connect more reliably, with configured headers on MCP SSE servers and
omlx launch codexfinding newer ChatGPT app bundles without a slow DNS lookup (#5080, #5079, 9715639). - Supervised logs stay small, as the console log under
brew servicesand the macOS app keeps only warnings and errors (@zviratko in #5179). - Web UI polish, with SSD totals that include unloaded models, paired stat tiles on phones, rounded panel corners and a heatmap legend that reads right in dark mode (#5063, @LXD-8 in #5123, @CamJN in #5159, @yanzhaohui1999 in #5144).
Smaller features and fixes from the development release are in the 0.7.4.dev1 release notes.
Upgrade Notes
- Source installs: run
make installafter updating to build the new native kernels, including oQh's.make devneeds pip 25.1 or newer (#5317, #5188). - oQh: oQh models run only in oMLX. mlx-lm, mlx-vlm, Splash, A8 and ANE prefill, and MoE expert offload cannot run them (#5317).
- oQe: existing oQe models stay as they are. Quantize again to get the better rounding, which adds 5 to 20 minutes to a build on an M3 Ultra (#5292).
- Concurrent requests: existing installs move to the new Balanced default, and Prefill Priority is renamed Long Prompt Handling (#5164).
- Prefill Accelerator: both Macs need the same oMLX version. For an NVIDIA helper, pull
jundot/omlx-cuda-helperagain to update it (#5065, #5163). - Logging: under
brew servicesand the macOS app, the console log keeps only warnings and errors.~/.omlx/logs/server.logstill has everything (#5179). - API:
reasoning_effort: "none"and"off"now turn thinking off (#5151), and a trailing assistant message on/v1/messagesis continued instead of starting a new turn (#4995). - Affine KV: greedy output with Affine4 or Affine8 KV compression can differ from 0.7.3 (#5095).
- Integrations:
omlx launch codexandcodex_appneed Codex 0.148 or newer, andomlx launch copilotneeds Copilot CLI 1.0.83 or newer (#5026, #5066).
New Contributors
@fabriziosalmi in #4506, @alexandergreil in #4673, @christianschmidt in #5014, @begcdn in #5018, @ianderse in #5050, @ska2704 in #5057, @StefVanG in #5090, @Aminark in #5092, @avifenesh in #5101, @syntax922 in #5151, @CamJN in #5159, @Li-Lee in #5309, @GumbiiDigital in #5271, @Maheidem in #5241, @keefar in #5332, @OmVidana in #5338, @isnaenihidayat in #5347.

