Skip to content

0.8.3

Latest

Choose a tag to compare

@jundot jundot released this 22 Nov 18:23
e72868d

oMLX 0.8.3

oMLX 0.8.3 is an optimization release for Inkling 2 Mini and the Hy4 family, which got day-0 support in 0.8.1. Inkling 2 Mini prefills about 20% faster and decodes 10-13% faster, Hy4 Flash starts answering sooner and needs less memory, and Claude Code's web search now works with local models.

Download: macOS 26 / 27 | macOS 15 Sequoia

Faster Inkling 2 Mini

Inkling 2 Mini prefills about 20% faster and decodes 10-13% faster than in 0.8.2 (#5538). Rewriting a file or other text from the context is more than twice as fast, as Lightning MTP can now copy up to 15 tokens at a time from it (#5528). A long context also needs about a sixth of the KV memory, 18 -> 3 GB at 98K tokens (-83%) (@masterfasa in #5505).

Inkling-2-Mini-oQ4e-mtp, Lightning MTP on, temperature 1.0, on M5 Max. The file rewrite row is on M3 Ultra and compares before and after #5528.

Workload 0.8.2 0.8.3
Rewriting a 242-line file, decode 62.7 tok/s 158.5 tok/s (+153%)
Zero-shot coding, decode 86.0 tok/s 96.8 tok/s (+13%)
4K prompt, prefill 1,869 tok/s 2,255 tok/s (+21%)
16K prompt, prefill 1,764 tok/s 2,125 tok/s (+20%)
16K prompt, decode 52.9 tok/s 58.0 tok/s (+10%)

Also faster: Inkling 2 prefill with Float16 Activations on M1 and M2, 155 -> 171 tok/s at 2K (+10%) (@j3k0 in #5526), and the attention projections of the looped layers (@BUKOWSKIREAL in #5495).

Hy4 Family Optimizations

Hy4 Flash gives its first token about 20% sooner with the SSD cache on (@0xbrianh-dev in #5535) and prefills faster on M5 (@0xbrianh-dev in #5536). Turn on SSD N-gram Offload in a Hy4 Flash model's settings to keep its Engram tables on SSD, which takes memory from 124.7 to 87.7 GiB (-30%) (@0xbrianh-dev in #5475, tested by @apetersson). A long prompt also writes 60% less to the SSD cache, 1.24 -> 0.50 GiB (@0xbrianh-dev in #5533).

Hy4 Flash with the SSD cache on. Each row compares before and after its PR.

Workload Before After
First token, cold 5.8K prompt 9.75 s 7.78 s (-20%)
First token, next turn 6.57 s 5.37 s (-18%)
17K prompt, prefill on M5 Max 660 tok/s 743 tok/s (+13%)

Multi-turn chats and agent sessions on Hy4 Flash no longer lose earlier turns after a cache hit (@SolNine-Git in #5534), and its streamed reasoning no longer mixes into the answer (@0xbrianh-dev in #5476).

New Features and Improvements

  • Web search: Claude Code's WebSearch tool works with local models, answered by the search provider set in oMLX (@mufeng05 in #5342). Page fetches also work behind fake-IP proxies such as Clash and sing-box (@mufeng05 in #5343).
  • Speed:
    • DeepSeek V4.1 2-bit builds decode faster with several requests, 148 -> 165 tok/s in total with 8 on M5 Ultra (+11%) (@bleugreen in #5525).
    • DeepSeek V4.1 Lightning MTP drafts up to 5 tokens by default, so one request at 4K context decodes 79.7 -> 88.4 tok/s (+11%) (@bleugreen in #5527).
  • Prefill Accelerator: a 24 GB RTX 4090 helper holds 43 layers of 8-bit Qwen3.8-27B instead of 22, as it keeps Qwen3.5-family weights packed (@davesecops in #5513, #5522).
    • 32K prefill goes 720 -> 1,095 tok/s with 39 layers (+52%), and 1,118 -> 1,373 tok/s when the Mac asks for all 43 instead of 37 (+23%).
  • Inkling 2: checkpoints set to one loop pass, such as the varloop research release, now load (@mlubbad in #5514).

Bug Fixes

  • Inkling 2 answers stay the same with Lightning MTP on, as greedy output now matches decoding without it, also on an SSD cache hit after a restart (#5531).
  • Inkling 2 runs in more setups, as float16 oQ packs no longer return NaN and run to the token limit (@CharlesJS in #5523), and KV cache quantization no longer crashes the first forward (@masterfasa in #5505).
  • Inkling 2 stops thinking for enable_thinking: false and reasoning_effort: "off", instead of thinking anyway or falling back to low (@masterfasa in #5506).
  • Inkling 2 DFlash drafts with pre-drafting match those without it, as the next draft starts from the newest accepted token (@masterfasa in #5508).
  • Long prompts on a 32 GB Mac are refused instead of running out of Metal memory, with compressed memory and Inkling 2's prefill mask now counted (#5516).
  • Saving model settings keeps each model's own Lightning MTP depth, instead of writing 3 over DeepSeek V4.1's 5 and Gemma 4's and Inkling 2's 8 (#5541).
  • Images sent as Hugging Face-style {"type": "image", "image": ...} parts are no longer dropped (@nurlanmnn in #5530).

Upgrade Notes

  • Lightning MTP depth: a depth of 3 saved before 0.8.3 now reads as Model default. Pick 3 again in Model Settings to keep it (#5541).
  • Prefill Accelerator: pull jundot/omlx-cuda-helper again to update an NVIDIA helper to 0.1.8. The accelerator checks warn until it is updated (#5513, #5522).
  • Hy4 Flash: its SSD cache now uses 2048-token blocks, so blocks saved by 0.8.2 are not reused, including any that #5534 saved wrong (#5535).

New Contributors

@BUKOWSKIREAL in #5495, @CharlesJS in #5523, @j3k0 in #5526, @mufeng05 in #5343, @mlubbad in #5514, @0xbrianh-dev in #5476, @SolNine-Git in #5534, @nurlanmnn in #5530.

Full changelog