PRO Q4 on two M3 Ultra 512GB: decode ~3.6 s/token on BOTH halves (~0.14 t/s); a single decode token is slower than a 512-token prefill

[treemanz]

Summary

Running the split DeepSeek V4 PRO Q4 GGUF across two Mac Studio M3 Ultra 512GB machines (coordinator 0:30, worker 31:output), the distributed route builds fine and text is coherent, but generation runs at 0.14 t/s (gen_tps=0.14, ~7.1 s/token) versus the README's 11.47 t/s smoke test — ~80x slower.

Both halves are symmetrically slow. The remote worker (layers 31:60) reports ~3.64 s/token, and by subtraction the coordinator's local layers 0:30 is ~3.5 s/token (7.1 s total − 3.64 s remote). Since the coordinator's local half never touches the network, this rules out the interconnect and the distributed protocol entirely — the problem is PRO Q4 per-token decode compute on the M3 Ultra.

The smoking gun: decoding 1 token (~7.1 s) takes longer than prefilling 512 tokens (~6.3 s, prefill_tps=80.71). For a 6-of-384 routed MoE this is impossible unless the sparse-expert fast path (the README's "queue-resident exact expert tables") is inactive during decode and each token re-binds the broad multi-GiB routed tensors. Each machine maps ~420 GB resident (426 / 411 GB) on a 512 GB box, leaving little headroom for the resident expert tables — consistent with the README's warning about hitting "Metal memory accounting limits."

Environment

• Hardware: 2x Mac Studio Mac15,14, Apple M3 Ultra (32-core CPU: 24P+8E / 80-core GPU), 512 GB unified memory (one 2TB SSD, one 4TB SSD — no perf difference)
• OS: macOS 26.4.1 (build 25E253), system firmware 18000.101.7; built with Command Line Tools (Xcode.app not installed)
• Commit: 80ebbc3 (current main HEAD, 2026-06-17) — confirmed identical on both machines (coordinator and worker both report 80ebbc3)
• Model: DeepSeek-V4-Pro-Q4K-Layers00-30.gguf (coordinator) + DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf (worker)
• Link: direct Thunderbolt bridge, 192.168.2.1 to 192.168.2.2 (static IPs), ping 0.5–0.9 ms — matches the README's Thunderbolt reference (~0.45 ms), so the interconnect is not the bottleneck

Repro

Worker (machine B):

./ds4 \
  -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf \
  --role worker \
  --layers 31:output \
  --coordinator 192.168.2.1 1234 \
  --debug

Coordinator (machine A):

./ds4-bench \
  -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf \
  --prompt-file /tmp/smoke_512.txt \
  --ctx-start 512 --ctx-max 512 --step-incr 512 \
  --gen-tokens 8 \
  --role coordinator \
  --layers 0:30 \
  --listen 192.168.2.1 1234 \
  --dist-activation-bits 16 \
  --debug

Coordinator telemetry (the smoking gun)

ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes
request=1  layers=31:60 pos=0   tokens=512 eval=3903.008ms downstream_wait=0.000ms forward_send=0.000ms input=28.00MiB
request=2  layers=31:60 pos=512 tokens=1   eval=3660.819ms downstream_wait=0.000ms forward_send=0.000ms input=0.05MiB
request=3  layers=31:60 pos=513 tokens=1   eval=3665.961ms downstream_wait=0.000ms forward_send=0.000ms input=0.05MiB
request=9  layers=31:60 pos=519 tokens=1   eval=3597.725ms downstream_wait=0.000ms forward_send=0.000ms input=0.05MiB
512,512,80.71,8,0.14,0

Decomposition of the ~7.1 s/token (from gen_tps=0.14):

segment	per-token	where measured
remote worker layers 31:60	~3.64 s	reported in telemetry
local coordinator layers 0:30	~3.5 s	= 7.1 s total − 3.64 s remote
network (downstream_wait + forward_send)	~0 ms	telemetry

Key observations:

• Both halves are ~3.5 s/token, symmetrically. The local half runs entirely on the coordinator with no network involved, so this is a compute problem, not a distributed/interconnect problem.
• Decode is slower than prefill of 512 tokens: 1 decode token ~7.1 s vs 512-token prefill ~6.3 s (prefill_tps=80.71). This indicates the sparse-MoE fast path is inactive during decode and each token re-binds/reads the broad routed tensors. Per half, ~420 GB / 3.5 s ≈ 120 GB/s effective, ~1/7 of the M3 Ultra's ~819 GB/s.

What I've ruled out

Hypothesis	Evidence	Verdict
Network latency	Thunderbolt bridge, ping 0.5–0.9 ms; downstream_wait=0, forward_send=0	ruled out
Network bandwidth	--dist-activation-bits 16 halved bytes, no change in 3.6 s/token	ruled out
CPU fallback / Metal not initialized	worker init log shows Metal residency + warmup + metal backend	ruled out
This machine is slow / memory pressure	single-machine Flash Q4 on same box is healthy (33 t/s)	ruled out
SSD size	2TB and 4TB boxes benchmark within 0.2–0.3%	ruled out

Single-machine Flash Q4 on the same hardware is normal:

ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes
512,512,396.00,64,33.55,31042956
32768,32768,474.16,256,25.11,475014540
65536,32768,395.96,256,23.98,926033292

Hypothesis

PRO Q4 decode isn't using the "queue-resident exact expert tables" path and instead re-binds/reads the broad routed tensors every token (the README's known slow/early failure mode). This happens on both halves, so it is not distributed-specific. Most likely trigger: each machine maps ~420 GB resident on a 512 GB box, leaving too little headroom for the resident expert tables, so Metal memory-accounting forces the fallback. Prefill (which reads broadly anyway, once per chunk) is only mildly affected; decode — where the sparse-gather fast path should dominate — pays the broad-binding cost per token and ends up slower than a 512-token prefill.

Questions

• Is the "queue-resident exact expert tables" decode path expected to be active for split PRO Q4 on 2x 512 GB (each half ~420 GB resident), or does that headroom already trip the "Metal memory accounting limits" fallback the README mentions?
• Is there a memory-headroom threshold for the fast path? Would freeing more RAM, or a more headroom-preserving split, restore it?
• Any flag to force/verify the fast path, or extra telemetry (e.g. a per-token line for the coordinator's local layers) to confirm which decode path each half takes?

Diagnostics (note: --trace does not apply here)

--trace is not the right diagnostic for this bug. --trace is a ds4-eval / ds4-server flag (defined in ds4_eval.c). ds4-bench rejects it, and plain ds4 produces no trace file for this run path. It also logs agent-session prompts / cache decisions / tool-parser events, not decode timing or Metal residency. The evidence for this performance bug is already inline above: the --debug per-hop telemetry, the ds4-bench CSV (gen_tps=0.14 / prefill_tps=80.71), and the memory_pressure wired-residency snapshots.

Optional extra evidence, easy to capture: the per-token Metal graph timing that --debug prints (same shape as #465, e.g. ds4: metal graph token pos=N encode=... execute=... total=...), and a memory_pressure snapshot taken DURING the slow decode on both machines to show the wired collapse with Swapouts=0.

# run on BOTH machines mid-decode to show wired collapse + zero swap:
memory_pressure
# per-token Metal timing already emitted by --debug (grep the run log):
#   ds4: metal graph token pos=N encode=... execute=... total=...

---

UPDATE — Root cause confirmed: wired residency collapses every token, on BOTH halves

Captured memory_pressure during an actual slow run on both machines. The wired (pinned/GPU-resident) footprint is NOT stable across the generation loop: it is wired up for each request's compute and released again — on BOTH the coordinator and the worker. This is Metal residency churn in physical RAM, not OS paging: Swapins/Swapouts stay 0 and Pageins increase by only ~34–44 pages between phases, so the hundreds of GiB that leave "wired" stay resident in RAM (as "inactive") and never touch SSD.

machine / phase	wired	free %	swapouts
coordinator — prefill (request=1)	~433 GiB	15%	0
coordinator — decode (request>1)	~38 GiB	92%	0
worker — compute phase	~232 GiB	54%	0
worker — released phase	~5.5 GiB	98%	0

The coordinator wires ~433 GiB (nearly its full 426 GiB local half) during compute, then drops to ~38 GiB; the worker peaks at only ~232 GiB (well below its 411 GiB half) and drops to ~5.5 GiB. Both oscillate per request. This directly explains: (a) decode being slower than a 512-token prefill (prefill wires once for 512 tokens; decode re-wires per token), and (b) both halves being symmetrically slow.

Upgraded hypothesis (supersedes the section above): during decode the PRO Q4 path does not keep the routed expert tables queue-resident; it releases their GPU/Metal residency after each request and re-binds the broad multi-GiB routed tensors for every token — the exact slow path the README's "queue-resident exact expert tables" was meant to avoid. The cost is residency re-establishment, not disk I/O (confirmed: Swapouts=0, Pageins flat). Whatever keeps the expert tables resident across the prefill call is not being kept alive across the per-token decode calls.

Added ruled-out row: OS memory pressure / swap — memory_pressure shows Swapins/Swapouts = 0 throughout and compressor ~8 MB (coordinator); the slowdown is not OS paging.

---

Related issues / prior art (checked before filing)

I searched the README and existing issues. This scenario is not a duplicate — the closest reports are either single-machine, SSD-streaming, CUDA, or a different trigger. Mapping:

#	relation	why it differs from this report
#465 Unexpected slow decode when using --power	closest symptom: multi-second per-token Metal execute, ~0.24 t/s	single machine, triggered by --power, reporter suspects macOS 27 Beta GPU ramping, warmup mitigates
#455 Metal SSD streaming: exact path misses routed expert views	same subsystem: "Metal cannot wrap routed-expert model ranges"	SSD-streaming test path, not distributed in-RAM residency
#491 Expert bundle sidecar for contiguous MoE expert reads	same pain point (routed expert reads)	targets SSD-streaming cold-miss disk I/O; here there is no disk I/O
#437 / #439 Metal SSD-streaming benchmarks / cache pressure	same area (Metal routed-expert residency)	SSD streaming (model does not fit)
#293 Distributed worker host-registers the whole GGUF	distributed + whole-model registration concept	CUDA / DGX Spark, not Metal
#477 page-cache residency when RAM ≥ model (decode +52%)	residency optimization	CUDA, not Metal
README (queue-resident exact expert tables)	names the exact mechanism and the "Metal memory accounting limits" failure mode; flags distributed as not release-stable	documents it as a known hard area, but not this fits-in-RAM per-token residency collapse

What makes this report distinct (and, I think, a clean repro):

• The model fully fits and stays in RAM — this is NOT SSD streaming. Wired peaks at ~433 GiB (coordinator) / ~232 GiB (worker), Swapouts stay 0, Pageins are flat across the prefill→decode transition. No disk I/O is involved.
• The wired residency of the routed expert tables collapses in RAM between prefill and decode and is re-established per token, on BOTH halves symmetrically.
• macOS 26.4.1 (not the 27 Beta suspected in Unexpected slow decode and prefill(?) when using --power  #465); no --power flag; the slowdown is persistent across all decode tokens, not a first-token/warmup artifact.

---

Additional evidence: the slowdown is sustained (rules out warmup; distinguishes from #465)

A gen_tokens=256 run shows 24 consecutive decode tokens all at ~3.6 s (eval 3565–3774 ms) — steady state, not a first-token/warmup artifact. This is the key difference from #465, where a warmup period mitigated the slowness. Two independent gen_tokens=8 runs reproduce the CSV: 80.71/0.14 and 81.40/0.14. Precise ping on the Thunderbolt bridge: 0.209 ms avg to 192.168.2.1, 0.762 ms avg to 192.168.2.2.

Full raw logs (ping, per-token telemetry for the 256- and 8-token runs, and both memory_pressure captures): https://gist.github.com/treemanz/271e62fe85dd3f9cfd815e997a917f63