Skip to content

Releases

dstar 1.4

Choose a tag to compare

@paperniuk paperniuk released this 21 Nov 20:16

dstar now runs Tencent Hy4 Flash on a 64 GB Mac: a 220B mixture of experts with 11B active per token, from peasantsmith's Maya-XS IQ2_XXS GGUF.

Hy4 Flash, text only.

dstar pull hy4
dstar serve hy4
dstar chat hy4

dstar pull hy4 downloads 109 GB: the 45 GB model and a 64 GB n-gram (Engram) table that stays on the SSD and is read a few rows per token, plus the 0.7 GB DFly2 drafter. If the two shards are already in your Hugging Face cache, only the drafter is fetched. The model maps 42.7 GB, so it needs a 64 GB Mac. serve hy4 defaults to a 32K context; --ctx takes more.

M1 Max 64 GB:

tokens/s
prompt, 3K tokens 128
prompt, 3K tokens, --ane 140
prompt, 14K tokens, --ane 132 (121 without)
generation 24
generation on code, --dfly2 27

What went into it:

  • Prompt processing in 512-token chunks with the attention head projections batched over the chunk: 49.5 -> 128 tokens/s at 3K.
  • Decode kernels for one token at a time (experts, head projections, split attention): 22.1 -> 24 tokens/s.
  • --ane runs part of the prompt projections on the Neural Engine while the GPU works on the next chunk. It reads the weights in place, with about 300 MB of extra buffers.
  • --dfly2 drafts two tokens at a time with the DFly2 drafter and verifies them in one pass. It works with greedy decoding only (temperature 0; chat hy4 --dfly2 sets that). On code it is about 12% faster; on prose about 2% slower, and it pauses itself when drafts keep missing.
  • The server keeps the KV cache between chat turns: a follow-up turn on a 1.4K-token conversation starts in 0.65 s instead of processing the whole prompt again.
  • reasoning_effort: "no_think" is accepted as none.

Greedy output is the same token for token with and without --dfly2. A needle test (seven names and numbers spread through a 14K-token story) finds all seven with and without --ane.

ISTA's Coder model: --quant coder. Qwen3.8 Flash-Next with half of the experts removed (256 of 512 per layer), tuned for code. It needs 36.5 GiB of GPU memory at 262K context instead of about 45 for q2, so it leaves the most room on 64 GB and may fit a 48 GB Mac (untested). It is slower than q2: its experts use quants with slower kernels here, and the MTP block was trained on the full model.

dstar pull --quant coder     # 30 GB next to another quant, 59 GB alone
dstar serve --quant coder
M1 Max 64 GB coder q2
GPU memory at 262K 36.5 GiB 45 GiB
perplexity on C code 1.637 1.605
prompt, 3K tokens 287 tok/s 384
prompt, 43K tokens 365 389
prompt, 43K tokens, --ane 465 491
generation on code, MTP 37 52
generation on code, --dflash 47 70

The Neural Engine turns itself off when it slows the prompt down. With --ane, after two chunk pairs on a long prompt dstar runs one pair on the GPU alone and keeps whichever is faster for the rest of the server's life. The log says which: Neural Engine prefill check: X tok/s with it, Y without. On a simulated slow Neural Engine this takes the prompt from 234 to 335 tok/s (376 on the GPU alone); when the Neural Engine is fast the check costs about 1% of the first long prompt. DS4_ANE_GUARD=0 turns the check off. Hy4 gets the same check.

Qwen3.8 Flash-Next is otherwise unchanged. Same output and speed as 1.3. Each quant other than q2 now keeps its own KV disk cache directory (~/.ds4/server-kv-i8-iq3 and so on), so one quant never loads another's saved KV.

Install or update (the models you already have stay where they are):

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash

Built from dcbc852. Check the download against the attached .sha256.

dstar 1.3

Choose a tag to compare

@paperniuk paperniuk released this 13 Nov 13:14

Better answers from Qwen3.8 Flash-Next, faster decoding, a DFlash2 drafter for the server, SSD streaming for Macs with less memory, and far fewer SSD writes from the KV disk cache.

Sampling now follows Qwen's model card. When a client sends no sampling settings (OpenCode sends none), ds4 used the DeepSeek defaults: temperature 1, top_p 1, no top_k, min_p 0.05. That samples from a much wider tail than Qwen recommends. 1.3 uses the model card, which the GGUF also carries:

temperature top_p top_k min_p
thinking (default) 1.0 0.95 20 0
thinking off 0.7 0.8 20 0

Values a request sets still win. This applies to dstar serve, dstar chat and ds4-agent.

MTP no longer picks most tokens greedily. The MTP draft was accepted only when it matched the most likely token, so up to two of every three tokens were chosen greedily instead of sampled, which Qwen advises against while thinking. dstar serve and dstar chat now run MTP with exact sampling: the output follows the sampling distribution above. That costs some speed, which the decode work below more than makes up for. dstar serve --fast-mtp brings back the old, faster behaviour.

Decoding about 10% faster. dstar serve on IQ3_XXS, M1 Max, same prompts: 34.5 -> 38.2 tokens/s with the new default sampling.

  • The hyper-connection weights are converted from BF16 to 8-bit when the model loads, in place, so memory use does not grow. That is 568 MiB less to read per token. Perplexity moves by less than 0.02%. DS4_QWEN4_HC_Q8=0 keeps BF16.
  • The MTP draft is now sampled from its own distribution and accepted with probability min(1, p/q), so 69-74% of drafts are accepted instead of 59-66%, with the same output distribution.
  • The sampler no longer calls a library function per logit to check for NaN: 1.0 -> 0.2 ms per token.
  • Decode kernels from upstream ds4 (antirez#1238): attention with two query heads per simdgroup, a faster attention merge, and the hyper-connection norm loaded ahead.

Experimental: DFlash2 drafter. dstar serve --dflash and dstar chat --dflash draft up to 7 tokens at a time with incoai/Qwen3.8-Flash-Next-DFlash2 instead of one with MTP, greedy or sampled. On code it runs at 41.8 tokens/s against 38.9 for MTP; on prose and edits it is level with MTP or a little slower. It needs 2.1 GB more memory; fetch it with dstar pull --dflash.

Experimental: SSD streaming for the ISTA quants. --ssd-streaming used to refuse the IQ3_XXS and Q2_0 files. Now the experts can stay on the SSD with only a cache of them in memory, so the model runs on Macs with less memory. With a 10 GiB expert cache the whole process plans about 23 GiB at 8K context:

dstar serve --quant iq3 -- --ssd-streaming --ssd-streaming-cache-experts 10GB

M1 Max, 4K context, 10 GiB cache: IQ3_XXS 16.8 tokens/s, Q2_0 19.0 (fully in memory: 33.4 and 38.5). Output is identical to the in-memory model. This was measured on a 64 GB Mac; with less memory macOS keeps less of the file in its own cache, so expect slower decoding until the expert cache warms up.

Upstream ds4 merged. The 116 commits on antirez/ds4 since the last sync are in, with fixes so that batched and one-token decoding give the same results.

KV disk cache: about 6 times fewer writes (#4). Every checkpoint written while a prompt is processed holds the whole prefix, and one was written every 10K tokens, so the bytes written grew with the square of the context: about 63 GiB to fill 262K tokens. A checkpoint now also waits until the context has grown by half since the last one (10K, 20K, 30K, 50K, 80K, 120K, 180K), about 9.6 GiB in all for 262K. After a restart or a switch between conversations, the server re-processes at most a third of the context.

A retried turn resumes instead of starting over. When a reply was cut short and the client sent the same prompt again (OpenCode does this after a reply that ends inside the thinking block), a rejected MTP draft could make the server forget the state it saved at the start of the turn. It then fell back to an older disk checkpoint, or processed the whole prompt again. That state is now kept.

Install or update (the models you already have stay where they are):

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash

Built from 9da1288. Check the download against the attached .sha256.

dstar 1.1: 262K on a 64 GB Mac

Choose a tag to compare

@paperniuk paperniuk released this 02 Nov 18:10
5c63fe2

Both 1.1 betas plus a smaller KV cache: the full 262K context now fits on a 64 GB Mac with the IQ3_XXS model, and agents get faster prompts.

The KV cache of Qwen3.8-Flash-Next is now stored in int8, with one scale per 32 values. At 262K tokens it takes 4.5 GB instead of 7.8 GB.

On an M1 Max 64 GB:

1.0.1 1.1
IQ3_XXS default context 131K 262K
IQ3_XXS 262K with --ane does not fit fits with --ctx 262k
KV cache at 262K 7.8 GB 4.5 GB

A 231K-token prompt on IQ3_XXS with --ane prefills at 355 tokens per second, then decodes at 25 tokens per second with MTP.

Prefill on Q2_0, tokens per second:

prompt GPU alone --ane
8K 397 530
32K 380 501

What changed since 1.0.1:

  • int8 KV cache, on by default on Metal. Over 16K and 64K-token texts the next token is the same as with the f16 cache in 128 of 128 positions. Decode is about 2% slower, prefill the same. DS4_QWEN4_KV_F16=1 goes back to f16.
  • dstar serve --ane lets the Neural Engine run part of the prompt prefill next to the GPU: about 30% faster prompts on Q2_0. Off by default.
  • Faster prompts for agents: the expert tiles use half-precision activations, the disk KV cache is written straight into its file, and short final chunks no longer wait for the Neural Engine.
  • An image added to a running conversation keeps its vision start and end tokens, so the cached prompt still matches on the next turn (#2).
  • 160 commits merged from upstream ds4.

Thanks to @hackintosh-J for #2: they found why the prompt cache missed after an image and sent the fix with a regression test.

The disk cache moves to ~/.ds4/server-kv-i8, because old f16 checkpoints don't fit the new format and are not loaded. The old ~/.ds4/server-kv folder can be deleted.

Tested on an M1 Max. If something is slower or breaks on your Mac, please open an issue with your Mac model, macOS version, the command and the stderr lines.

Install or update (the models you already have stay where they are):

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash

Built from 5c63fe2. Check the download against the attached .sha256.

dstar 1.1 beta 2: faster prompts for agents

Choose a tag to compare

@paperniuk paperniuk released this 14 Oct 00:59

Beta: install it only if you want to try it. install.sh without a tag still installs 1.0.1.

Faster prompts on top of beta 1, mostly for agents, where every turn sends a short new piece of prompt and the server saves its cache.

Prefill on an M1 Max 64 GB, Q2_0, tokens per second:

beta 1 beta 2
8K prompt, GPU alone 379 397
8K prompt, --ane 499 530
32K prompt, GPU alone 363 380
32K prompt, --ane 473 501

Time to process a prompt in dstar serve --ane, 262K context:

prompt beta 1 beta 2
755 tokens 2.42 s 1.82 s
3774 tokens 8.96 s 8.18 s
15279 tokens 33.40 s 30.80 s

What changed:

  • The Q2_0 expert tiles stage their activations in half precision: about 5% faster prefill, on the GPU alone too. DS4_QWEN4_MOE_REG_HALF=0 goes back. The teacher-forced NLL stays within the noise.
  • The disk KV cache is written once, straight into the cache file. The server used to write it to a temporary file and copy it while the prompt waited: a 1.3 GB checkpoint at 41K tokens now takes 0.3 s instead of 1.7 s.
  • With --ane, a last prompt chunk of 512 to 1023 tokens no longer waits for the Neural Engine: about 10% faster for those.
  • With --ane, the KV buffers stay mapped for the GPU between Neural Engine runs. The short tail the server prefills after the prompt anchor took about 85 ms longer before. Memory use at a short context grows by about 1 GB; once the context fills it is the same as before.

Tested only on an M1 Max. If prefill with --ane stalls or leaves the GPU idle on your machine, please open an issue with your macOS version, the command, and the stderr lines around the stall.

Install the beta (the models you already have stay where they are):

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | DSTAR_TAG=dstar-1.1-beta2 bash

Built from 6ea5981. Check the download against the attached .sha256.

dstar 1.1 beta 1: the Neural Engine helps with the prompt

Choose a tag to compare

@paperniuk paperniuk released this 13 Oct 07:31

Beta: install it only if you want to try it. install.sh without a tag still installs 1.0.1.

New, off by default: dstar serve --ane runs part of the prompt processing on the Neural Engine while the GPU computes the rest. Two prompt chunks go through the layers together, so the Neural Engine works on one while the GPU runs the experts of the other. Generation speed does not change; it helps where you wait for a long prompt or a big tool output.

Prefill on an M1 Max 64 GB, tokens per second:

GPU alone --ane
Q2_0, 8K prompt 375 495
Q2_0, 32K prompt 359 467
IQ3_XXS, 8K prompt 300 374

It costs 0.9 GB more memory. The teacher-forced NLL stays within the noise (8K prompt: 0.1010 on the GPU alone, 0.1027 with the Neural Engine). If a Neural Engine evaluation fails, hangs or returns non-finite values, the chunks are computed again on the GPU and the Neural Engine stays off for the session.

Also:

  • The Q2_0 expert down projection uses a register tile with two row fragments per simdgroup: about 3% faster prefill, on the GPU alone too.
  • The prompt anchor is kept for image-conditioned sessions (thanks @angrytako) and when the new one would follow hidden reasoning.

Tested only on an M1 Max, and the split between the GPU and the Neural Engine is tuned for it. Other chips should work but may gain less; reports welcome.

Install the beta (the models you already have stay where they are):

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | DSTAR_TAG=dstar-1.1-beta1 bash

Built from 85f7a74. Check the download against the attached .sha256.

dstar 1.0.1: the quant follows the memory

Choose a tag to compare

@paperniuk paperniuk released this 03 Oct 19:53

Same engine as 1.0, launcher fixes:

  • With no model on disk, dstar now picks the quant by the memory: Q2_0 below 96 GB, IQ3_S (ISTA's best file) from 96 GB. Before, a 128 GB Mac was pointed at Q2_0.
  • On 128 GB or more, dstar doctor points at ds4's own Q4 file, which answers better still: dstar pull qwen38-q4k, then dstar serve Q4 -- --mtp.
  • The download size of dstar pull is now given right: 69 GB with the MTP block and the vision encoder.

Install or update (the models you already have stay where they are):

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash

Built from 37a5f4a. Check the download against the attached .sha256.

dstar 1.0: Qwen3.8-Flash-Next and DeepSeek V4 Flash on Apple Silicon

Choose a tag to compare

@paperniuk paperniuk released this 03 Oct 17:57

Prebuilt ds4 for Apple Silicon Macs (macOS 15 or newer) with the dstar launcher. Nothing to compile.

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash
dstar doctor     # what your Mac fits
dstar pull       # Qwen3.8-Flash-Next Q2_0, the MTP block, the vision encoder (67 GB)
dstar serve      # OpenAI/Anthropic compatible server on http://127.0.0.1:8010/v1
dstar opencode   # provider block for OpenCode

Other quants and models: dstar pull --quant iq3, dstar models, dstar serve deepseek or dstar serve /path/to/model.gguf. A model larger than the RAM streams from the SSD.

Measured on an M1 Max 64 GB with MTP: Qwen3.8-Flash-Next Q2_0 decodes at 44 tok/s at 4K, 38.8 at 256K and 35.4 at 398K, prefill 330 to 290 tok/s. IQ3_XXS: 35.2 to 31.7 tok/s up to 259K. DeepSeek V4 Flash Q2 streamed from the SSD: 12 to 9 tok/s up to 127K.

Only the M1 Max has been tested. M2 to M5 run the same code; reports from other chips are welcome (dstar doctor plus one dstar chat run).

Contents: ds4, ds4-server, dstar, download_model.sh, the Metal shaders, LICENSE. Built from 4997d51. Check the download against the attached .sha256.

Unofficial fork of antirez/ds4 for Apple Silicon. The quants are by ISTA-DASLab.