Skip to content

sushi v1.7.1

Latest

Choose a tag to compare

@github-actions github-actions released this 19 Nov 16:50
· 48 commits to main since this release

v1.7.1 — Sushi Cooler, made right: memory, refusals, long conversations

  • Every request stays inside the memory it was admitted with: Qwen3.8's speculative requests are now billed for what
    their draft rounds hold (a short one needs up to about 560 MB more free; the advertised context does not move),
    DeepSeek counts its memory margin once, and a raised --wired-margin-gib applies once again, under the wired limit,
    as on 1.6.0. Leaks after a failed speculative request, after joined requests with a logit bias, and after every reload
    of an EXL3 or STQ pack are closed; concurrent and sampled requests decode together on Sushi Cooler again, and a
    failure fails only the requests it touched. Where free RAM rather than the GPU limit binds, the advertised context is
    now the one a request at that length is admitted at, so it can come out smaller.
  • Long Qwen3.8 conversations keep going: when a turn shares the last turn's cached entry and nothing else frees
    enough room, the turn takes that entry over instead of being refused, whether it adds to the conversation, is shorter
    or regenerates the last reply (on a 32 GB Mac, a 131k-token turn that was refused is now served); turns that share
    part of an entry keep their restore points and prefill less.
  • DeepSeek-V4.1 is steadier: a long prompt no longer re-reads decoder layers from the SSD while other requests
    decode or after a failed memory borrow, regenerating a reply reuses the conversation, a cancelled turn no longer
    costs its retry the cache (or hands it another conversation's), and an image request is no longer refused for memory
    a prompt of the same size gets. A greedy or seeded DSpark request on the release checkpoint repeats its text: on an
    M5 Ultra 256 GB with its experts streamed at --ssd-budget-gb 200, 26.6 tok/s decode cold and 38.6 tok/s repeated.
  • Refusals say what to do: a request or load refused for memory names the max_tokens, context and SSD streaming
    memory that fit, offers streaming when a pack only fits streamed, names the Settings
    label beside each flag, prints one unit with the need rounded up and what is available rounded down, and never
    suggests a wired limit past the Mac's RAM. Sushibar shows the whole reason, the web page keeps it on screen until
    dismissed, every request to a refused model gets it, and a load the GPU limit refuses answers at once.
  • APIs and server: errors on /v1/messages come in Anthropic's shape; a Responses WebSocket turn refused before it
    runs ends with an error frame, and store: false is echoed as sent; a client that stops reading no longer blocks an
    unload (a busy unload answers 409 within 30 s); a large SSD prompt cache
    no longer stops new connections; the web page's Thinking panel stays where you scroll it. The HTTP request-size limit
    is a setting, server.max_request_mb (Settings > Server, 1-1,024 MiB, 128 by default): thanks @Cianidos.
  • On the MLX route (--qwen-mlx-engine, GLM-5.3-Flash, MiMo): a kv8, kv6 or kv4 cache grows on MLX's page grid, so
    its appends are never copied past the bill; a bill moves by a few MB, and on a tight Mac the advertised context can
    drop by a few hundred tokens. Sushi Cooler is unchanged by this.
  • Known issue: DeepSeek-V4.1-Flash's FP8 release checkpoint does not load at its default SSD streaming memory on a
    256 GB Mac; set --ssd-budget-gb 215 (the refusal names it). Thanks @nikolay-bratanov, @younesehb, @NikolaKrstic94
    and @cyrilmhansen for the reports behind this release.