Skip to content

dstar 1.1: 262K on a 64 GB Mac

Choose a tag to compare

@paperniuk paperniuk released this 02 Nov 18:10
5c63fe2

Both 1.1 betas plus a smaller KV cache: the full 262K context now fits on a 64 GB Mac with the IQ3_XXS model, and agents get faster prompts.

The KV cache of Qwen3.8-Flash-Next is now stored in int8, with one scale per 32 values. At 262K tokens it takes 4.5 GB instead of 7.8 GB.

On an M1 Max 64 GB:

1.0.1 1.1
IQ3_XXS default context 131K 262K
IQ3_XXS 262K with --ane does not fit fits with --ctx 262k
KV cache at 262K 7.8 GB 4.5 GB

A 231K-token prompt on IQ3_XXS with --ane prefills at 355 tokens per second, then decodes at 25 tokens per second with MTP.

Prefill on Q2_0, tokens per second:

prompt GPU alone --ane
8K 397 530
32K 380 501

What changed since 1.0.1:

  • int8 KV cache, on by default on Metal. Over 16K and 64K-token texts the next token is the same as with the f16 cache in 128 of 128 positions. Decode is about 2% slower, prefill the same. DS4_QWEN4_KV_F16=1 goes back to f16.
  • dstar serve --ane lets the Neural Engine run part of the prompt prefill next to the GPU: about 30% faster prompts on Q2_0. Off by default.
  • Faster prompts for agents: the expert tiles use half-precision activations, the disk KV cache is written straight into its file, and short final chunks no longer wait for the Neural Engine.
  • An image added to a running conversation keeps its vision start and end tokens, so the cached prompt still matches on the next turn (#2).
  • 160 commits merged from upstream ds4.

Thanks to @hackintosh-J for #2: they found why the prompt cache missed after an image and sent the fix with a regression test.

The disk cache moves to ~/.ds4/server-kv-i8, because old f16 checkpoints don't fit the new format and are not loaded. The old ~/.ds4/server-kv folder can be deleted.

Tested on an M1 Max. If something is slower or breaks on your Mac, please open an issue with your Mac model, macOS version, the command and the stderr lines.

Install or update (the models you already have stay where they are):

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash

Built from 5c63fe2. Check the download against the attached .sha256.