v26.9.2 — Per-model settings, chat providers, faster Flash Next

[ddalcu]

Highlights

• Qwen 3.8 Flash Next is faster across the board. Speculative decoding costs less per step, reaches full speed on the first request instead of the fifteenth, and picks its draft from a small shortlist instead of reading the whole vocabulary. Long prompts process faster too. On an M4 Max the headline goes 83 to 93 tokens per second, with +50% at short prompts and +30% at 64k and 128k. Generated text is unchanged. (thanks @beamivalice)
• Every model can have its own settings. Right-click a model in My Models > Model Settings to give it its own context size, KV cache precision and speculative-decoding default. They apply every time that model loads, and a model that is already running picks them up on the spot. Headless: ~/.mlx-serve/model-settings.json.
• Chat with other servers from the same picker. Settings > Providers takes any OpenAI-compatible chat server (a cloud API, another Mac, a local runtime) with its key, and its models appear in the model picker as <model>@<name>. Chat only for now; provider models are never shared over the LAN. Headless: ~/.mlx-serve/providers.json.
• Long Flash Next conversations can live on SSD. With --prefix-cache-disk, memory holds the model and the conversation you are in; every other conversation is written to disk in the background and comes back in seconds instead of minutes when you return to it. (thanks @beamivalice)
• Speculative decoding knows when to stop helping. Past some conversation length a speculative step costs more than it saves; Flash Next now measures both and switches speculation off there and back on when it pays again. --max-mtp-ctx <n> sets a hard cutoff if you want one. (thanks @beamivalice)
• Structured (JSON schema) output at full speed. Constrained decoding used to crawl at about one token per second on Flash Next; it now runs at the model's normal speed. (JSON-schema constrained decoding costs ~60x on decode for Qwen3.8-Flash-Next (0.9 vs 53.9 tok/s) #380)
• The chat reads the way you want. Pick Narrow, Medium or Wide from Settings or F1 to F3. Your own messages get the same hover actions as replies, photos lay out in a grid, the reasoning block collapses out of the way and shows how long the model thought, tables no longer squeeze their headers, and numbered lists, quotes and inline code render properly. (Let the reader choose the chat column, and lay a turn out around it #339, thanks @lojza3d)
• My Models shows how much disk space is left. (ui: add remaining space to model browser view in my models #328, thanks @justinluque)

Fixes

• Claude Code no longer loses its SessionStart hook output, CLAUDE.md or any other context a client puts in a system message inside messages on /v1/messages. It was discarded without a warning, so the reply looked plausible on a third less prompt. (Anthropic /v1/messages silently drops role:"system" entries inside messages (Claude Code loses SessionStart hook content) #365, thanks @nikolai-vysotskyi)
• A developer message is read as the system turn instead of being dropped for an unknown role.
• Mage-Flow Edit loads again. It had been refused for a missing vision tower, which the loader was dropping before the backend saw it.
• Gemma 4, LFM2.5-8B-A1B and Muse-Glimmer no longer show their thinking as the answer when streaming. When the model opened its own thought rather than the prompt template, the streamed reply carried the whole chain of thought as text while the same request unstreamed split it correctly.
• A reply cut off in the middle of a character (an emoji, an accented letter) no longer makes the whole response unreadable to the client.
• Pasting binary data into a chat (for example grep -a output) no longer silently cuts the conversation short at that point, and no longer knocks the prompt back to a generic format. The model used to answer with nothing and the agent's turn ended empty.
• Tool calls survive tricky arguments: a value that happens to contain the tool format's own closing tags, a call cut off mid-way, or a number-like value such as 0755 is now passed through as written instead of being emptied, dropped, renamed or turned into a number.
• Streaming and non-streaming replies agree in more places: a spent reasoning budget no longer leaks the rest of the thinking into the answer, a tool-calling reply keeps the sentence the model said before the call, stop sequences cut at the exact match, a client that hangs up is reported as a disconnect instead of a token limit, and seed: -1 means unseeded instead of crashing the server.
• A long hybrid-model conversation no longer suddenly re-reads its whole history from scratch (nine minutes at 390k tokens). Two causes: a cached turn did not inherit the restore points of the entry it grew from, and a text turn after a run of image turns could not find any of them. Both fixed, and a miss with a matching prompt is now logged.
• Running out of GPU memory during a very long prompt no longer kills the server: that request gets an error and the next one is served. Requests are also admitted more accurately, so a long prompt that fits is no longer refused for memory it would never use, and the cache is evicted to make room instead of the request being turned away. (Cold prefill past ~400k tokens is admitted by the memory preflight and dies on an uncatchable Metal OOM (buffer growth copy and SSM checkpoints unbilled) #353)
• Raising iogpu.wired_limit_mb is honoured: a 448k-token Flash Next conversation used to be refused because the guard looked at RAM other apps left free rather than the limit you set.
• The prefix cache budget follows what is actually loaded. It used to be fixed at startup against every model on the machine, so a model loaded next to a large one could keep a near-zero cache for the whole session. (Hot prefix-cache budget is frozen at init, so a model loaded beside another keeps a ~0 budget for life #364)
• A batch of unrelated requests no longer evicts your live conversation from the cache; each workload evicts its own entries first. (Prefix cache: a batch sweep evicts the live conversation when one model serves two workloads #378)
• Speculative decoding on Flash Next no longer pays for a full draft round when the model's very first token ends the reply, and that prompt still lands in the cache for the next turn.
• Very long prompts no longer process at the narrowest width on a 1M-context server; the width is chosen per request.
• Long Flash Next sessions no longer end in "Failed to create Metal shared event": a small buffer per generated token was never released.
• Models with padding rows past the end of their vocabulary (Flash Next has 243) can no longer pick one; a reply used to lose a step when that happened.
• A model shipping both a drafter and an MTP head now uses the drafter, as intended; --no-drafter hands the round back to MTP. The drafter also samples 8 rounds before giving up on a request instead of 3.
• A stale speculative-decoding cost table from an older build is cleaned up at load instead of steering the planner away from the fastest width. (round-cost: a poisoned width-1 cell survives the load-time sweep (dropImplausibleCells starts at w=2) #382)
• A malformed Flash Next checkpoint is refused at load instead of served.
• --max-tokens N in serve mode sets the reply budget for clients that do not send one, and mlx-serve launch claude passes the server's real context size instead of assuming 200k.
• The server log says auto instead of a billion tokens when a client omits max_tokens.
• Show log in the image, video, audio and 3D panes opens the Server Log window, and a recommended model that ships a draft head is rated at the speed it actually runs.

---

This discussion was created from the release v26.9.2 — Per-model settings, chat providers, faster Flash Next.

[Andrei-cloud]

That is amazing perfomance increase on M5Max 40GPU 128GB:

In real coding session with Hermes orchestrating OpenCode on TUI revamp, 60K lines of code. 90-115 tok/s on Qwen3.8 Flash Next - Mixed 4-8bit

that is HUGE!

[beamivalice]

More coming actually. lol.