You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Qwen3.8-Flash-Next EXL3 quantizer: sushi convert exl3 writes a Sushi pack from the BF16 checkpoint at the rate you pick, with the trunk in the format the packs serve; runs resume where they stopped, keep finished layers and packs, and fit their Metal threadgroups to any Apple GPU. Check the result with sushi kld. Thanks to @cosmic-quokka for the M2 Pro report (#57).
Adaptive DFlash2 for Qwen3.8-Flash-Next: point --drafter at the DFlash2 drafter and Qwen verifies 2 to 8 draft rows per round as the drafter's confidence pays, up to 20% faster than MTP on Sushi-2.6bpw and level with it on Sushi-4bpw (--drafter-min-depth / --drafter-max-depth); MTP stays the default. Thanks to @1RAFAL for the request (#59).
--kv-quant 6 on all three models: a 6-bit KV cache between kv4 and kv8 for Qwen, MiMo and GLM, also as kv_quant 6 in model-settings.json and per request; GLM stores its MLA latent at 6 bits and keeps DFlash2 and prefix reuse.
Web UI Dashboard: a Chat | Dashboard switch in the top bar with live throughput, requests, GPU and memory charts plus sessions and models tables (needs --metrics); switching never interrupts a chat. Thanks to @cosmic-quokka for the dashboard (#55).
Requests that used to fall back now run as asked: MTP stays on with presence_penalty and repeat_penalty; Qwen serves high and max effort as xhigh, GLM serves medium as high and xhigh as max, each with a notice in the log and the response.