Repository navigation
dstar now runs Tencent Hy4 Flash on a 64 GB Mac: a 220B mixture of experts with 11B active per token, from peasantsmith's Maya-XS IQ2_XXS GGUF.
Hy4 Flash, text only.
dstar pull hy4
dstar serve hy4
dstar chat hy4
dstar pull hy4 downloads 109 GB: the 45 GB model and a 64 GB n-gram (Engram) table that stays on the SSD and is read a few rows per token, plus the 0.7 GB DFly2 drafter. If the two shards are already in your Hugging Face cache, only the drafter is fetched. The model maps 42.7 GB, so it needs a 64 GB Mac. serve hy4 defaults to a 32K context; --ctx takes more.
M1 Max 64 GB:
| tokens/s | |
|---|---|
| prompt, 3K tokens | 128 |
prompt, 3K tokens, --ane |
140 |
prompt, 14K tokens, --ane |
132 (121 without) |
| generation | 24 |
generation on code, --dfly2 |
27 |
What went into it:
- Prompt processing in 512-token chunks with the attention head projections batched over the chunk: 49.5 -> 128 tokens/s at 3K.
- Decode kernels for one token at a time (experts, head projections, split attention): 22.1 -> 24 tokens/s.
--aneruns part of the prompt projections on the Neural Engine while the GPU works on the next chunk. It reads the weights in place, with about 300 MB of extra buffers.--dfly2drafts two tokens at a time with the DFly2 drafter and verifies them in one pass. It works with greedy decoding only (temperature 0;chat hy4 --dfly2sets that). On code it is about 12% faster; on prose about 2% slower, and it pauses itself when drafts keep missing.- The server keeps the KV cache between chat turns: a follow-up turn on a 1.4K-token conversation starts in 0.65 s instead of processing the whole prompt again.
reasoning_effort: "no_think"is accepted asnone.
Greedy output is the same token for token with and without --dfly2. A needle test (seven names and numbers spread through a 14K-token story) finds all seven with and without --ane.
ISTA's Coder model: --quant coder. Qwen3.8 Flash-Next with half of the experts removed (256 of 512 per layer), tuned for code. It needs 36.5 GiB of GPU memory at 262K context instead of about 45 for q2, so it leaves the most room on 64 GB and may fit a 48 GB Mac (untested). It is slower than q2: its experts use quants with slower kernels here, and the MTP block was trained on the full model.
dstar pull --quant coder # 30 GB next to another quant, 59 GB alone
dstar serve --quant coder
| M1 Max 64 GB | coder | q2 |
|---|---|---|
| GPU memory at 262K | 36.5 GiB | 45 GiB |
| perplexity on C code | 1.637 | 1.605 |
| prompt, 3K tokens | 287 tok/s | 384 |
| prompt, 43K tokens | 365 | 389 |
prompt, 43K tokens, --ane |
465 | 491 |
| generation on code, MTP | 37 | 52 |
generation on code, --dflash |
47 | 70 |
The Neural Engine turns itself off when it slows the prompt down. With --ane, after two chunk pairs on a long prompt dstar runs one pair on the GPU alone and keeps whichever is faster for the rest of the server's life. The log says which: Neural Engine prefill check: X tok/s with it, Y without. On a simulated slow Neural Engine this takes the prompt from 234 to 335 tok/s (376 on the GPU alone); when the Neural Engine is fast the check costs about 1% of the first long prompt. DS4_ANE_GUARD=0 turns the check off. Hy4 gets the same check.
Qwen3.8 Flash-Next is otherwise unchanged. Same output and speed as 1.3. Each quant other than q2 now keeps its own KV disk cache directory (~/.ds4/server-kv-i8-iq3 and so on), so one quant never loads another's saved KV.
Install or update (the models you already have stay where they are):
curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash
Built from dcbc852. Check the download against the attached .sha256.