Datapoint: Flash q2-imatrix SSD streaming on a 24 GB base M5 — 2.4-2.6 t/s gen

[tinhnguyen0110]

Datapoint: Flash q2-imatrix SSD streaming works on a 24 GB base M5 — 2.4–2.6 t/s gen

Sharing numbers from what may be the smallest-RAM machine reported so far (the README's smallest streaming example is a 64 GB MacBook, and #406 suggested 96 GB as the practical floor). Short version: it just works, first try, no OOM, and generation lands at 2.4–2.6 t/s.

Setup

• Hardware: Apple M5 (base, 10 cores), 24 GB unified memory, internal SSD (Mac mini class), macOS 26.5.2
• Build: 54b36ed, stock make (Metal), no code changes
• Model: DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf (80.76 GiB)
• Command: ./ds4 -m ds4flash.gguf --ssd-streaming [--ssd-streaming-cache-experts N] --ctx 4096 --nothink --temp 0 -n 64|256

Results

config	prefill t/s	gen t/s	planned memory (startup log)
auto budget (399 dyn experts + 3.38 GiB prefill reserve)	2.98	2.42	7.39 GiB
--ssd-streaming-cache-experts 700 (4.61 GiB)	3.36	2.61	6.00 GiB
--ssd-streaming-cache-experts 1000 (6.59 GiB)	3.35	2.54	7.98 GiB

Short prompt, greedy, -n 256 for the manual-cache rows (-n 64 for the auto row). Output quality: coherent multi-paragraph answers, byte-identical across the 700/1000 runs as expected under greedy.

Observations

• The auto budget handled 24 GB gracefully. 80% of the 17.76 GiB recommended working set minus 8.20 GiB non-routed left a 399-expert dynamic cache; startup printed the expected thrashing warning (cache < 2× the 258 experts/token working set) but ran fine.
• Gen is SSD-bandwidth-bound, and cache size barely matters at this scale. 700 → 1000 experts changed nothing (2.61 → 2.54, likely run noise). Back-of-envelope: ~258 routed experts/token × 6.75 MiB ≈ 1.7 GiB/token; at 2.6 t/s that's ~4.4 GB/s of reads — about what this SSD sustains. With 43×256 = 11,008 routed experts total, a ≤1000-expert cache can't lift the hit rate enough to move the needle, so the practical knee on this class of machine is simply "enough cache to clear the thrashing threshold" (~516), then the disk sets the ceiling.
• Memory stayed comfortable throughout (6–8 GiB planned vs ~20 GiB free); no compressor/swap pressure observed.

Happy to run other configs (larger -n, --ctx, specific budgets) if useful — this machine class (24 GB Macs) seems worth a line in the README's streaming examples, since the answer to "can it run at all?" is a clear yes.

[GiorgioOppo]

@tinhnguyen0110 do you want to try this implementation #618? It could speed up the performance a bit without taking away too much quality from the model. The gguf would be this: https://huggingface.co/antirez/deepseek-v4-gguf/resolve/5ca1abfe6d0c518ffcd731eae01bc0c88a2e59db/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix.gguf?download=true