You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
v1.7.0 — Sushi Cooler by default, faster DeepSeek, a lighter prompt cache
Qwen3.8 and DeepSeek-V4.1 run on Sushi Cooler, sushi's own Metal engine, by default: every Qwen3.8 STQ pack and
its BF16 checkpoint, and every DeepSeek-V4.1-Flash pack including its own release checkpoint, prefill, decode, batch
concurrent requests and read images on it, on M1 through M5 Macs, with the tokens MLX gives. A repeat start skips the
engine's warm-up, and nothing compiles mid-request. --qwen-mlx-engine, --ds41-mlx-engine or the model's Cooler
checkbox keeps a model on MLX.
DeepSeek-V4.1-Flash is faster and steadier: about 3% faster decode, faster long-prompt prefill, about a third less
CPU while generating, cached turns that answer about 120 ms sooner, and agent turns that no longer re-read their
previous reply. Sampled replies no longer slip in a random token (1.6.1-dev2 and 1.7.0-dev1), and greedy answers no
longer depend on drafting or chunk size. On a Mac that streams its experts from the SSD, a new opt-in setting, "Faster
SSD streaming (not exact)" in the web UI or --expert-cache-bias, trades a little answer quality for speed: the
router favours an expert already in memory over a slightly better one it would read from the SSD (about 2x faster
decode at 0.2 on a 64 GB Mac, about 30% at 0.5 on a 32 GB Mac whose cache holds far fewer experts; off by default, so
answers stay exact). Thanks @odokododok for it.
A lighter prompt cache: a Qwen3.8 conversation writes about three quarters less to the SSD (six turns on the STQ
4bpw pack: 370 MB instead of 1,409 MB, same replies), holding its newest restore points in up to 250 MB of RAM so a
regenerated answer restores at once, even after a restart. The SSD cache is back to full speed when saving and
restoring, and stays within its budget on disk.
Less CPU, sooner answers: about 20% less CPU over a full benchmark on Qwen3.8-Flash-Next with decode about 5%
faster, and a 400-turn tool-calling conversation builds its prompt in 2 to 4 ms instead of up to 0.6 s.
Every request runs inside its memory budget: a wide prefill no longer runs a 32 GB Mac out of memory, a 1536²
image on Qwen3.8 takes 1.9 GB instead of 22.8 GB, and what does not fit is refused by name instead of crashing.
Exact prompts and tool calls on every API: chat, Anthropic Messages and Responses render one prompt and say one
reply; tool arguments arrive spelled as the model wrote them; GLM's forced tool calls are enforced as Qwen's are; a
strict JSON schema is enforced as written or refused with a reason; min_p sampling joins every API and the chat
page.
Removed: --mtp-typical, --mtp-tokenv3 and --mtp-greedy-tail, with their settings: speculative decoding
accepts drafts by the exact rule only, so a seeded request repeats its text. A settings file that still names them
loads with a warning.