Sushi Engine Integration

[justanut]

Problem

Sushi packs (e.g. beamster/Qwen3.8-Flash-Next-Sushi-4bpw) store routed experts as EXL3 trellis tensors, which mlx-lm/mlx-vlm can't load. oMLX fails with Received N parameters not in model on them.

Proposal

Add Sushi as a second external engine next to Splash, reusing the same pattern: oMLX launches sushi serve on a loopback port and forwards chat requests to its OpenAI-compatible API.

• SushiEngine subclasses SplashEngine (streaming, process groups and stats are shared).
• A pack with expert_quant.format == "exl3" is routed to Sushi through external_engine_id(), alongside the existing GGUF → Splash fallback. Other models can opt in with external_engine: "sushi".
• No effect on any model that doesn't need Sushi.

Tested to be more memory efficient: on an M5 Max 128 GB, Qwen3.8 Flash Next 4bpw runs with a 500k context at >40 tok/s using about 70% of memory.

Questions for you

1. Is a second external engine welcome, or would you rather keep Splash the only one?
2. The PR also adds optional conveniences (download Sushi on first use, scan ~/.sushi/models, raise iogpu.wired_limit_mb via the macOS admin dialog). Keep them, make them default-off, or split them into a follow-up PR?

Implementation: #4572

[justanut]

Updated for 0.7.3

[jundot]

Because of my maintenance capacity, a Sushi external engine sits a bit lower on my priority list right now. Each external engine is another path I have to keep working across oMLX releases, and Splash already takes a fair amount of that. It's not about the implementation itself.

I'll keep #4572 open and see how people respond to this discussion before moving it forward. If more people ask for Sushi support here, I'll take that into account and move it up in priority.

Feel free to pause rebasing #4572 for now. I don't want you spending your time keeping it up to date on every release while it's waiting, and it can be rebased once it moves forward.

In the meantime Sushi has its own OpenAI compatible server, so these packs can be used with it directly.

I appreciate you opening the discussion first and laying out the questions clearly, and the time you put into the integration.

[justanut]

I’ll continue to maintain the branch~ mostly for my own use cos I really love the oMLX backend for my other models / model switching, but Qwen Flash Next just runs better with less overheads on Sushi.

[jc9]

Each external engine is another path I have to keep working across oMLX releases, and Splash already takes a fair amount of that.

Another option could be not supporting engines directly, but implementing adding models in a router mode - something along the lines of what llama-swap does.
Maybe not the first class solution, as this would push wiring engine/APIs to end user, but it would allow using basically any engine available with oMLX front.

[justanut]

Yeah I think that is even more work for Jun. I thought that by using the external engine he has built it would be the least intrusive way for now.

[publicExcess]

I would like to propose oMLX gets a single ExternalEngine base (process launch and watchdog, OpenAI streaming proxy, stats — the parts Splash already has) which allows any OpenAI-compatible server to be added with a small JSON file in ~/.omlx/external_engines/, either launched per model or attached to a server that's already running. So Splash keeps its own code on top of the shared base; engines like Sushi or my engine Ijah (EXL3 based) would then just become config files, with no engine-specific code in oMLX to maintain.

I have this working locally on 0.7.3, with tests. Happy to open a small PR for just the base and the config files if you'd consider it.

[justanut]

But does that mean that your engine needs to run it's own server alongside oMLX? The issue I found with Sushi is that it hoards the 70gb of RAM for QFN if I run it in the background, but if I launch it through oMLX, OMLX can manage the memory and free it up when unused.

[publicExcess]

In launch mode oMLX starts the engine's server when the model is first used and stops it again on unload (idle timeout, memory pressure, or manually), counting its memory from a memory_gb value in the JSON file. Stopping the server frees all of it, KV cache included, so Sushi wouldn't hold RAM in the background. The trade off is a cold start (model reload) on the first request after an unload. Attach mode is just an option for people who'd rather keep a server running themselves.

[eusip]

Linking this comment here for visibility.

Any updates on merging Sushi engine support? The quantization recipe of Sushi packs is quite robust; it is a substantial zero-cost quality gain.

Let me know if there is any #4572 testing that I can help out with to push forward this merge.

[justanut]

I've updated my branch with the latest from 0.7.4.dev1