Repository navigation
oMLX 0.8.3
oMLX 0.8.3 is an optimization release for Inkling 2 Mini and the Hy4 family, which got day-0 support in 0.8.1. Inkling 2 Mini prefills about 20% faster and decodes 10-13% faster, Hy4 Flash starts answering sooner and needs less memory, and Claude Code's web search now works with local models.
Download: macOS 26 / 27 | macOS 15 Sequoia
Faster Inkling 2 Mini
Inkling 2 Mini prefills about 20% faster and decodes 10-13% faster than in 0.8.2 (#5538). Rewriting a file or other text from the context is more than twice as fast, as Lightning MTP can now copy up to 15 tokens at a time from it (#5528). A long context also needs about a sixth of the KV memory, 18 -> 3 GB at 98K tokens (-83%) (@masterfasa in #5505).
Inkling-2-Mini-oQ4e-mtp, Lightning MTP on, temperature 1.0, on M5 Max. The file rewrite row is on M3 Ultra and compares before and after #5528.
| Workload | 0.8.2 | 0.8.3 |
|---|---|---|
| Rewriting a 242-line file, decode | 62.7 tok/s | 158.5 tok/s (+153%) |
| Zero-shot coding, decode | 86.0 tok/s | 96.8 tok/s (+13%) |
| 4K prompt, prefill | 1,869 tok/s | 2,255 tok/s (+21%) |
| 16K prompt, prefill | 1,764 tok/s | 2,125 tok/s (+20%) |
| 16K prompt, decode | 52.9 tok/s | 58.0 tok/s (+10%) |
Also faster: Inkling 2 prefill with Float16 Activations on M1 and M2, 155 -> 171 tok/s at 2K (+10%) (@j3k0 in #5526), and the attention projections of the looped layers (@BUKOWSKIREAL in #5495).
Hy4 Family Optimizations
Hy4 Flash gives its first token about 20% sooner with the SSD cache on (@0xbrianh-dev in #5535) and prefills faster on M5 (@0xbrianh-dev in #5536). Turn on SSD N-gram Offload in a Hy4 Flash model's settings to keep its Engram tables on SSD, which takes memory from 124.7 to 87.7 GiB (-30%) (@0xbrianh-dev in #5475, tested by @apetersson). A long prompt also writes 60% less to the SSD cache, 1.24 -> 0.50 GiB (@0xbrianh-dev in #5533).
Hy4 Flash with the SSD cache on. Each row compares before and after its PR.
| Workload | Before | After |
|---|---|---|
| First token, cold 5.8K prompt | 9.75 s | 7.78 s (-20%) |
| First token, next turn | 6.57 s | 5.37 s (-18%) |
| 17K prompt, prefill on M5 Max | 660 tok/s | 743 tok/s (+13%) |
Multi-turn chats and agent sessions on Hy4 Flash no longer lose earlier turns after a cache hit (@SolNine-Git in #5534), and its streamed reasoning no longer mixes into the answer (@0xbrianh-dev in #5476).
New Features and Improvements
- Web search: Claude Code's WebSearch tool works with local models, answered by the search provider set in oMLX (@mufeng05 in #5342). Page fetches also work behind fake-IP proxies such as Clash and sing-box (@mufeng05 in #5343).
- Speed:
- DeepSeek V4.1 2-bit builds decode faster with several requests, 148 -> 165 tok/s in total with 8 on M5 Ultra (+11%) (@bleugreen in #5525).
- DeepSeek V4.1 Lightning MTP drafts up to 5 tokens by default, so one request at 4K context decodes 79.7 -> 88.4 tok/s (+11%) (@bleugreen in #5527).
- Prefill Accelerator: a 24 GB RTX 4090 helper holds 43 layers of 8-bit Qwen3.8-27B instead of 22, as it keeps Qwen3.5-family weights packed (@davesecops in #5513, #5522).
- 32K prefill goes 720 -> 1,095 tok/s with 39 layers (+52%), and 1,118 -> 1,373 tok/s when the Mac asks for all 43 instead of 37 (+23%).
- Inkling 2: checkpoints set to one loop pass, such as the varloop research release, now load (@mlubbad in #5514).
Bug Fixes
- Inkling 2 answers stay the same with Lightning MTP on, as greedy output now matches decoding without it, also on an SSD cache hit after a restart (#5531).
- Inkling 2 runs in more setups, as float16 oQ packs no longer return NaN and run to the token limit (@CharlesJS in #5523), and KV cache quantization no longer crashes the first forward (@masterfasa in #5505).
- Inkling 2 stops thinking for
enable_thinking: falseandreasoning_effort: "off", instead of thinking anyway or falling back to low (@masterfasa in #5506). - Inkling 2 DFlash drafts with pre-drafting match those without it, as the next draft starts from the newest accepted token (@masterfasa in #5508).
- Long prompts on a 32 GB Mac are refused instead of running out of Metal memory, with compressed memory and Inkling 2's prefill mask now counted (#5516).
- Saving model settings keeps each model's own Lightning MTP depth, instead of writing 3 over DeepSeek V4.1's 5 and Gemma 4's and Inkling 2's 8 (#5541).
- Images sent as Hugging Face-style
{"type": "image", "image": ...}parts are no longer dropped (@nurlanmnn in #5530).
Upgrade Notes
- Lightning MTP depth: a depth of 3 saved before 0.8.3 now reads as Model default. Pick 3 again in Model Settings to keep it (#5541).
- Prefill Accelerator: pull
jundot/omlx-cuda-helperagain to update an NVIDIA helper to 0.1.8. The accelerator checks warn until it is updated (#5513, #5522). - Hy4 Flash: its SSD cache now uses 2048-token blocks, so blocks saved by 0.8.2 are not reused, including any that #5534 saved wrong (#5535).
New Contributors
@BUKOWSKIREAL in #5495, @CharlesJS in #5523, @j3k0 in #5526, @mufeng05 in #5343, @mlubbad in #5514, @0xbrianh-dev in #5476, @SolNine-Git in #5534, @nurlanmnn in #5530.