Repository navigation
dstar 1.3
Better answers from Qwen3.8 Flash-Next, faster decoding, a DFlash2 drafter for the server, SSD streaming for Macs with less memory, and far fewer SSD writes from the KV disk cache.
Sampling now follows Qwen's model card. When a client sends no sampling settings (OpenCode sends none), ds4 used the DeepSeek defaults: temperature 1, top_p 1, no top_k, min_p 0.05. That samples from a much wider tail than Qwen recommends. 1.3 uses the model card, which the GGUF also carries:
| temperature | top_p | top_k | min_p | |
|---|---|---|---|---|
| thinking (default) | 1.0 | 0.95 | 20 | 0 |
| thinking off | 0.7 | 0.8 | 20 | 0 |
Values a request sets still win. This applies to dstar serve, dstar chat and ds4-agent.
MTP no longer picks most tokens greedily. The MTP draft was accepted only when it matched the most likely token, so up to two of every three tokens were chosen greedily instead of sampled, which Qwen advises against while thinking. dstar serve and dstar chat now run MTP with exact sampling: the output follows the sampling distribution above. That costs some speed, which the decode work below more than makes up for. dstar serve --fast-mtp brings back the old, faster behaviour.
Decoding about 10% faster. dstar serve on IQ3_XXS, M1 Max, same prompts: 34.5 -> 38.2 tokens/s with the new default sampling.
- The hyper-connection weights are converted from BF16 to 8-bit when the model loads, in place, so memory use does not grow. That is 568 MiB less to read per token. Perplexity moves by less than 0.02%.
DS4_QWEN4_HC_Q8=0keeps BF16. - The MTP draft is now sampled from its own distribution and accepted with probability min(1, p/q), so 69-74% of drafts are accepted instead of 59-66%, with the same output distribution.
- The sampler no longer calls a library function per logit to check for NaN: 1.0 -> 0.2 ms per token.
- Decode kernels from upstream ds4 (antirez#1238): attention with two query heads per simdgroup, a faster attention merge, and the hyper-connection norm loaded ahead.
Experimental: DFlash2 drafter. dstar serve --dflash and dstar chat --dflash draft up to 7 tokens at a time with incoai/Qwen3.8-Flash-Next-DFlash2 instead of one with MTP, greedy or sampled. On code it runs at 41.8 tokens/s against 38.9 for MTP; on prose and edits it is level with MTP or a little slower. It needs 2.1 GB more memory; fetch it with dstar pull --dflash.
Experimental: SSD streaming for the ISTA quants. --ssd-streaming used to refuse the IQ3_XXS and Q2_0 files. Now the experts can stay on the SSD with only a cache of them in memory, so the model runs on Macs with less memory. With a 10 GiB expert cache the whole process plans about 23 GiB at 8K context:
dstar serve --quant iq3 -- --ssd-streaming --ssd-streaming-cache-experts 10GB
M1 Max, 4K context, 10 GiB cache: IQ3_XXS 16.8 tokens/s, Q2_0 19.0 (fully in memory: 33.4 and 38.5). Output is identical to the in-memory model. This was measured on a 64 GB Mac; with less memory macOS keeps less of the file in its own cache, so expect slower decoding until the expert cache warms up.
Upstream ds4 merged. The 116 commits on antirez/ds4 since the last sync are in, with fixes so that batched and one-token decoding give the same results.
KV disk cache: about 6 times fewer writes (#4). Every checkpoint written while a prompt is processed holds the whole prefix, and one was written every 10K tokens, so the bytes written grew with the square of the context: about 63 GiB to fill 262K tokens. A checkpoint now also waits until the context has grown by half since the last one (10K, 20K, 30K, 50K, 80K, 120K, 180K), about 9.6 GiB in all for 262K. After a restart or a switch between conversations, the server re-processes at most a third of the context.
A retried turn resumes instead of starting over. When a reply was cut short and the client sent the same prompt again (OpenCode does this after a reply that ends inside the thinking block), a rejected MTP draft could make the server forget the state it saved at the start of the turn. It then fell back to an older disk checkpoint, or processed the whole prompt again. That state is now kept.
Install or update (the models you already have stay where they are):
curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash
Built from 9da1288. Check the download against the attached .sha256.