ROCm: restore DeepSeek V4 Flash decode performance and Q4 SSD streaming by kyuz0 · Pull Request #623 · antirez/ds4

kyuz0 · 2026-07-28T18:38:06Z

Summary

This restores the ROCm DeepSeek V4 Flash paths that regressed when the backend
moved Q8 decode activations to FP32:

restores prequantized Q8 decode for regular, paired, HC-expand, and
low-rank attention-output projections;
fixes Q4 SSD streaming by staging the compact selected-expert table before
the pointer-backed routed-MoE launch;
keeps both changes out of GLM. The Q8 selector checks the model family, and
the SSD fix explicitly excludes DS4_MODEL_FAMILY_GLM_DSA;
keeps --quality on the existing FP32 activation path;
provides DS4_ROCM_DSV4_PREQUANT_DECODE=0 as a diagnostic rollback.

Tested models

Single-node resident inference:
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
Two-node distributed inference and single-node SSD streaming:
DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf

Performance

AMD Radeon 8060S / gfx1151, ROCm 7.2.4.

Resident IQ2, 64-token decode:

Context	FP32 activation path	Restored Q8 path
512	9.19 t/s	16.12 t/s
1024	9.13 t/s	15.89 t/s
2048	9.01 t/s	15.51 t/s
4096	8.76 t/s	14.82 t/s

Prefill stayed effectively unchanged. Decode improved by 70-76%.

Two-node Q4 decode, layers 0:21 / 22:output, compared with the current
upstream base (antirez/main at 54b36ed):

Context	Upstream	This PR	Improvement
512	8.32 t/s	13.94 t/s	+67.5%
1024	8.25 t/s	13.73 t/s	+66.4%
2048	8.17 t/s	13.48 t/s	+65.0%
4096	7.97 t/s	12.93 t/s	+62.2%

Profiling localized the regression to Q8 attention and shared-expert
projections. Routed Q4 MoE performance was unchanged.

Distributed prefill also benefits from the existing pipeline when DeepSeek
uses smaller chunks. For a 4096-token prompt:

`--dist-prefill-chunk`	Prefill
4096	62.10 t/s
2048	78.72 t/s
1024	89.49 t/s
512	94.23 t/s
256	91.96 t/s

The recommended two-node DeepSeek setting is:

--dist-prefill-chunk 512 --dist-prefill-window 2

In the normal frontier benchmark this improved prefill by 24.5% at context
2048 and 41.9% at context 4096. Decode performance was unchanged.

Correctness

The DeepSeek Flash 100-case fixture scored 2,289 continuation tokens.

Resident IQ2, restored Q8 path versus FP32 activation path:

average NLL delta       +0.000444815
perplexity change       +0.044491%
bootstrap 95% CI        [-0.003072755, +0.003961388]

Distributed Q4, exact PR candidate against the same hosted-reference fixture:

average NLL             0.171642570
first-token matches     87 / 100
API top-1 rate          0.942332896
API pair rate           0.996669904

The resident A/B found no measurable quality regression. The distributed
candidate completed all 100 cases, but no clean distributed A/B against the
current upstream base was collected.

Q4 SSD streaming previously failed on its first decode with:

ROCm SSD streaming routed MoE missing compact selected experts

With this PR it decoded coherently through the 512, 1024, and 2048 frontiers
under repeated expert-cache replacement. No missing-expert failure, OOM,
non-finite output, or incorrect decode was observed.

Two-node example

The same Q4 model and context size must be available on both nodes.

Worker:

ds4-server \
  -m DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf \
  --ctx 126000 \
  --role worker \
  --layers 22:output \
  --coordinator COORDINATOR_IP 8081

Coordinator:

ds4-server \
  -m DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf \
  --ctx 126000 \
  --host 0.0.0.0 \
  --port 8000 \
  --role coordinator \
  --layers 0:21 \
  --listen COORDINATOR_IP 8081 \
  --dist-prefill-chunk 512 \
  --dist-prefill-window 2

VBart · 2026-07-28T20:45:41Z

Works for me, thanks!

Gen speed increased from ~9.16 t/s to ~15.6 t/s with DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf.

Before:

Jul 28 22:05:10 faex1 ds4-server[19423]: 0728 22:05:10 ds4-server: chat ctx=0..317:317 prompt start
Jul 28 22:05:14 faex1 ds4-server[19423]: 0728 22:05:14 ds4-server: chat ctx=0..317:317 prompt done 3.514s
Jul 28 22:05:14 faex1 ds4-server[19423]: ds4: ROCm Q8 one-token shared-input kernel enabled through 64 KiB LDS (in_dim=12288)
Jul 28 22:05:19 faex1 ds4-server[19423]: 0728 22:05:19 ds4-server: chat ctx=317..367:50 gen=50 THINKING decoding chunk=9.14 t/s avg=9.14 t/s 5.470s
Jul 28 22:05:25 faex1 ds4-server[19423]: 0728 22:05:25 ds4-server: chat ctx=367..417:50 gen=100 THINKING decoding chunk=9.19 t/s avg=9.16 t/s 10.911s
Jul 28 22:05:30 faex1 ds4-server[19423]: 0728 22:05:30 ds4-server: chat ctx=417..467:50 gen=150 THINKING decoding chunk=9.19 t/s avg=9.17 t/s 16.350s
Jul 28 22:05:36 faex1 ds4-server[19423]: 0728 22:05:36 ds4-server: chat ctx=467..517:50 gen=200 THINKING decoding chunk=9.19 t/s avg=9.18 t/s 21.792s
Jul 28 22:05:41 faex1 ds4-server[19423]: 0728 22:05:41 ds4-server: chat ctx=517..567:50 gen=250 THINKING decoding chunk=9.19 t/s avg=9.18 t/s 27.232s
..

After:

Jul 28 23:37:24 faex1 ds4-server[28356]: 0728 23:37:24 ds4-server: chat ctx=0..317:317 prompt start
Jul 28 23:37:28 faex1 ds4-server[28356]: 0728 23:37:28 ds4-server: chat ctx=0..317:317 prompt done 3.433s
Jul 28 23:37:31 faex1 ds4-server[28356]: 0728 23:37:31 ds4-server: chat ctx=317..367:50 gen=50 THINKING decoding chunk=15.69 t/s avg=15.69 t/s 3.186s
Jul 28 23:37:34 faex1 ds4-server[28356]: 0728 23:37:34 ds4-server: chat ctx=367..417:50 gen=100 THINKING decoding chunk=15.73 t/s avg=15.71 t/s 6.365s
Jul 28 23:37:37 faex1 ds4-server[28356]: 0728 23:37:37 ds4-server: chat ctx=417..467:50 gen=150 THINKING decoding chunk=15.75 t/s avg=15.72 t/s 9.540s
Jul 28 23:37:40 faex1 ds4-server[28356]: 0728 23:37:40 ds4-server: chat ctx=467..517:50 gen=200 THINKING decoding chunk=15.73 t/s avg=15.72 t/s 12.719s
Jul 28 23:37:44 faex1 ds4-server[28356]: 0728 23:37:44 ds4-server: chat ctx=517..567:50 gen=250 THINKING decoding chunk=15.75 t/s avg=15.73 t/s 15.894s
..

jhohertz · 2026-07-28T22:02:15Z

Looks good so far here, one thing to note, not sure this is even meant to be supported, but using the IQ2 quant + the ssd-streaming flag causes a hang after prefill. (Q4K is fine). Trying distributed in a moment.

VBart · 2026-07-29T17:43:38Z

With these settings:

DS4_GLM_MEMORY_GUARD=0 /opt/ds4/src/ds4-server --rocm \
  -m ../models/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf \
  --ssd-streaming \
  --ssd-streaming-cache-experts 115GB \
  --ctx 16384 \
  --host 0.0.0.0 \
  --port 8080

It constantly very soon gets into loops in thinking.

kyuz0 · 2026-07-29T17:48:39Z

It constantly very soon gets into loops in thinking.

Did you have that also before? Just trying to establish if this is a regression, I didn't do much testing on SSD streaming as I don't use it.

VBart · 2026-07-29T18:01:46Z

It constantly very soon gets into loops in thinking.

Did you have that also before? Just trying to establish if this is a regression, I didn't do much testing on SSD streaming as I don't use it.

Unfortunately I haven't tried DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf before. I'll try to test on current main.

That looks very suspicious because it starts looping very early on both temp 0 and temp 1.0 on the same prompt that DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf running without SSD streaming never went into loop.

VBart · 2026-07-29T19:01:46Z

With these settings I definitely see a regression:

DS4_GLM_MEMORY_GUARD=0 /opt/ds4/src/ds4-server --rocm \
  -m ../models/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf \
  --ssd-streaming \
  --ssd-streaming-cache-experts 90GB \
  --ctx 16384 \
  --host 0.0.0.0 \
  --port 8080

Current main: 54b36ed

Jul 29 21:51:06 faex1 flock[88573]: ds4: SSD streaming mixed-precision model: 6/43 routed layers off the slab size class will bypass the expert cache and read experts via mapped model views
Jul 29 21:51:06 faex1 flock[88573]: ds4: SSD streaming initial rocm model map restricted to token embedding (1 spans, 0.99 GiB tensor span)
Jul 29 21:51:06 faex1 flock[88573]: ds4: ROCm loading model tensors into device cache
Jul 29 21:51:06 faex1 flock[88573]: ds4: ROCm preparing model tensor mappings
Jul 29 21:51:06 faex1 flock[88573]: ds4: ROCm startup model preparation covered 0.99 GiB of tensor spans in 0.000s
Jul 29 21:51:06 faex1 flock[88573]: ds4: rocm backend initialized for graph diagnostics
Jul 29 21:51:06 faex1 flock[88573]: ds4: memory: KV 0.57 GiB (raw 0.36 + compressed 0.21) + buffers 0.13 GiB + resident model 0.99 GiB + expert cache 62.44 GiB + prefill expert reserve 3.38 GiB = 67.49 GiB planned
Jul 29 21:51:06 faex1 flock[88573]: ds4: memory detail: ctx=16384 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=4098 backend=rocm
Jul 29 21:51:06 faex1 flock[88573]: 0729 21:51:06 ds4-server: context buffers 710.75 MiB (ctx=16384, backend=rocm, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=4098)
Jul 29 21:51:07 faex1 flock[88573]: 0729 21:51:07 ds4-server: listening on http://0.0.0.0:8080
Jul 29 21:51:16 faex1 flock[88573]: 0729 21:51:16 ds4-server: chat ctx=0..355:355 prompt start
Jul 29 21:51:17 faex1 flock[88573]: ds4: ROCm loading model tensors 0.99 GiB cached
Jul 29 21:51:21 faex1 flock[88573]: ds4: ROCm loading model tensors 16.04 GiB cached
Jul 29 21:51:26 faex1 flock[88573]: ds4: ROCm loading model tensors 32.06 GiB cached
Jul 29 21:51:36 faex1 flock[88573]: ds4: ROCm loading model tensors 22.18 GiB cached
Jul 29 21:51:43 faex1 flock[88573]: 0729 21:51:43 ds4-server: chat ctx=0..355:355 prompt done 26.622s
Jul 29 21:51:52 faex1 flock[88573]: 0729 21:51:52 ds4-server: chat ctx=355..405:50 gen=50 THINKING decoding chunk=5.59 t/s avg=5.59 t/s 8.951s
Jul 29 21:51:58 faex1 flock[88573]: 0729 21:51:58 ds4-server: chat ctx=405..455:50 gen=100 THINKING decoding chunk=8.05 t/s avg=6.60 t/s 15.159s
Jul 29 21:52:04 faex1 flock[88573]: 0729 21:52:04 ds4-server: chat ctx=455..505:50 gen=150 THINKING decoding chunk=8.02 t/s avg=7.01 t/s 21.397s
Jul 29 21:52:11 faex1 flock[88573]: 0729 21:52:11 ds4-server: chat ctx=505..555:50 gen=200 THINKING decoding chunk=8.06 t/s avg=7.25 t/s 27.600s
Jul 29 21:52:17 faex1 flock[88573]: 0729 21:52:17 ds4-server: chat ctx=555..605:50 gen=250 THINKING decoding chunk=8.00 t/s avg=7.39 t/s 33.850s
..

... and it goes into loops.

Current PR 623:

Jul 29 21:56:25 faex1 flock[91017]: ds4: SSD streaming mixed-precision model: 6/43 routed layers off the slab size class will bypass the expert cache and read experts via mapped model views
Jul 29 21:56:25 faex1 flock[91017]: ds4: SSD streaming initial rocm model map restricted to token embedding (1 spans, 0.99 GiB tensor span)
Jul 29 21:56:25 faex1 flock[91017]: ds4: ROCm loading model tensors into device cache
Jul 29 21:56:25 faex1 flock[91017]: ds4: ROCm preparing model tensor mappings
Jul 29 21:56:25 faex1 flock[91017]: ds4: ROCm startup model preparation covered 0.99 GiB of tensor spans in 0.000s
Jul 29 21:56:25 faex1 flock[91017]: ds4: rocm backend initialized for graph diagnostics
Jul 29 21:56:25 faex1 flock[91017]: ds4: memory: KV 0.57 GiB (raw 0.36 + compressed 0.21) + buffers 0.13 GiB + resident model 0.99 GiB + expert cache 62.44 GiB + prefill expert reserve 3.38 GiB = 67.49 GiB planned
Jul 29 21:56:25 faex1 flock[91017]: ds4: memory detail: ctx=16384 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=4098 backend=rocm
Jul 29 21:56:25 faex1 flock[91017]: 0729 21:56:25 ds4-server: context buffers 710.75 MiB (ctx=16384, backend=rocm, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=4098)
Jul 29 21:56:25 faex1 flock[91017]: 0729 21:56:25 ds4-server: listening on http://0.0.0.0:8080
Jul 29 21:56:35 faex1 flock[91017]: 0729 21:56:35 ds4-server: chat ctx=0..355:355 prompt start
Jul 29 21:56:35 faex1 flock[91017]: ds4: ROCm loading model tensors 0.99 GiB cached
Jul 29 21:56:40 faex1 flock[91017]: ds4: ROCm loading model tensors 16.04 GiB cached
Jul 29 21:56:45 faex1 flock[91017]: ds4: ROCm loading model tensors 32.06 GiB cached
Jul 29 21:56:55 faex1 flock[91017]: ds4: ROCm loading model tensors 22.18 GiB cached
Jul 29 21:57:01 faex1 flock[91017]: 0729 21:57:01 ds4-server: chat ctx=0..355:355 prompt done 26.434s
Jul 29 21:57:04 faex1 flock[91017]: ds4: ROCm streaming selected expert id -1 outside 0..256 (layer=38 slot=0 selected=[-1,-1,-1,-1,-1,-1,-1,-1])
Jul 29 21:57:04 faex1 flock[91017]: 0729 21:57:04 ds4-server: chat ctx=355..356:1 gen=1 THINKING decoding chunk=0.33 t/s avg=0.33 t/s 3.006s
Jul 29 21:57:04 faex1 flock[91017]: ds4-server: thinking not closed, ignoring DSML in reasoning
Jul 29 21:57:04 faex1 flock[91017]: 0729 21:57:04 ds4-server: chat ctx=0..355:355 gen=1 THINKING finish=error error="rocm decode failed" 29.439s

VBart · 2026-07-29T19:06:23Z

With these settings:

DS4_GLM_MEMORY_GUARD=0 /opt/ds4/src/ds4-server --rocm \
  -m ../models/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf \
  --ssd-streaming \
  --ssd-streaming-cache-experts 115GB \
  --ctx 16384 \
  --host 0.0.0.0 \
  --port 8080

It constantly very soon gets into loops in thinking.

With these settings on current main it doesn't even run:

Jul 29 22:04:10 faex1 flock[93814]: ds4: rocm SSD streaming total expert budget 111.00 GiB = 6.75 GiB prefill headroom + 104.24 GiB dynamic cache (7907 experts, 13.50 MiB each)
Jul 29 22:04:10 faex1 flock[93814]: ds4: SSD streaming initial rocm model map restricted to token embedding (1 spans, 0.99 GiB tensor span)
Jul 29 22:04:10 faex1 flock[93814]: ds4: ROCm loading model tensors into device cache
Jul 29 22:04:10 faex1 flock[93814]: ds4: ROCm preparing model tensor mappings
Jul 29 22:04:10 faex1 flock[93814]: ds4: ROCm startup model preparation covered 0.99 GiB of tensor spans in 0.000s
Jul 29 22:04:10 faex1 flock[93814]: ds4: rocm backend initialized for graph diagnostics
Jul 29 22:04:10 faex1 flock[93814]: ds4: memory: KV 0.57 GiB (raw 0.36 + compressed 0.21) + buffers 0.13 GiB + resident model 0.99 GiB + expert cache 104.24 GiB + prefill expert reserve 6.75 GiB = 112.67 GiB planned
Jul 29 22:04:10 faex1 flock[93814]: ds4: memory detail: ctx=16384 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=4098 backend=rocm
Jul 29 22:04:10 faex1 flock[93814]: 0729 22:04:10 ds4-server: context buffers 710.75 MiB (ctx=16384, backend=rocm, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=4098)
Jul 29 22:04:11 faex1 flock[93814]: 0729 22:04:11 ds4-server: listening on http://0.0.0.0:8080
Jul 29 22:04:24 faex1 flock[93814]: 0729 22:04:24 ds4-server: chat ctx=0..355:355 prompt start
Jul 29 22:04:25 faex1 flock[93814]: ds4: ROCm loading model tensors 0.99 GiB cached
Jul 29 22:04:28 faex1 flock[93814]: ds4: ROCm loading model tensors 16.04 GiB cached
Jul 29 22:04:32 faex1 flock[93814]: ds4: ROCm loading model tensors 32.06 GiB cached
Jul 29 22:04:42 faex1 flock[93814]: ds4: ROCm loading model tensors 31.83 GiB cached
Jul 29 22:04:53 faex1 flock[93814]: ds4: ROCm loading model tensors 31.83 GiB cached
Jul 29 22:05:02 faex1 flock[93814]: 0729 22:05:02 ds4-server: chat ctx=0..355:355 prompt done 37.626s
Jul 29 22:05:02 faex1 flock[93814]: ds4: ROCm SSD streaming routed MoE missing compact selected experts (layer=3 tokens=1 total_experts=256 selected=6); full expert table is not mapped
Jul 29 22:05:02 faex1 flock[93814]: ds4-server: thinking not closed, ignoring DSML in reasoning
Jul 29 22:05:02 faex1 flock[93814]: 0729 22:05:02 ds4-server: chat ctx=0..355:355 gen=0 THINKING finish=error error="rocm decode failed" 37.873s

VBart · 2026-07-29T19:50:35Z

So, summarizing my observations... that's probably not a regression but just behavior change in already broken implementation of SSD streaming on ROCm with DeepSeek.

With DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf

main: looping
pr623: error on first token

With DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf

main: error on first token
pr623: looping

No issues with GLM.

kyuz0 · 2026-07-29T20:36:59Z

So, summarizing my observations... that's probably not a regression but just behavior change in already broken implementation of SSD streaming on ROCm with DeepSeek.

If you do SSD streaming with the smaller model, does it have the same behaviour? And I imagine with the small model totally resident in memory you do not observe the looping behaviour, right?

VBart · 2026-07-29T21:40:34Z

So, summarizing my observations... that's probably not a regression but just behavior change in already broken implementation of SSD streaming on ROCm with DeepSeek.

If you do SSD streaming with the smaller model, does it have the same behaviour? And I imagine with the small model totally resident in memory you do not observe the looping behaviour, right?

I'll try to verify, but from my current observation --ssd-streaming cause looping regardless of whether a model fits into memory or not.

VBart · 2026-07-29T22:43:10Z

Now I've tested with DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf.

Here's with PR 623 build.

With srteaming:

/opt/ds4/src/ds4-server --rocm \
  -m ../models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  --ssd-streaming \
  --ssd-streaming-cache-experts 115GB \
  --ctx 16384 \
  --host 0.0.0.0 \
  --port 8080


Jul 30 01:01:14 faex1 flock[133444]: ds4: rocm SSD streaming total expert budget 111.00 GiB = 3.38 GiB prefill headroom + 72.56 GiB dynamic cache (11008 experts, 6.75 MiB each)
Jul 30 01:01:14 faex1 flock[133444]: ds4: SSD streaming initial rocm model map restricted to token embedding (1 spans, 0.99 GiB tensor span)
Jul 30 01:01:14 faex1 flock[133444]: ds4: ROCm loading model tensors into device cache
Jul 30 01:01:14 faex1 flock[133444]: ds4: ROCm preparing model tensor mappings
Jul 30 01:01:14 faex1 flock[133444]: ds4: ROCm startup model preparation covered 0.99 GiB of tensor spans in 0.000s
Jul 30 01:01:14 faex1 flock[133444]: ds4: rocm backend initialized for graph diagnostics
Jul 30 01:01:14 faex1 flock[133444]: ds4: memory: KV 0.57 GiB (raw 0.36 + compressed 0.21) + buffers 0.13 GiB + resident model 0.99 GiB + expert cache 72.56 GiB + prefill expert reserve 3.38 GiB = 77.62 GiB planned
Jul 30 01:01:14 faex1 flock[133444]: ds4: memory detail: ctx=16384 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=4098 backend=rocm
Jul 30 01:01:14 faex1 flock[133444]: 0730 01:01:14 ds4-server: context buffers 710.75 MiB (ctx=16384, backend=rocm, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=4098)
Jul 30 01:01:15 faex1 flock[133444]: 0730 01:01:15 ds4-server: listening on http://0.0.0.0:8080
Jul 30 01:01:37 faex1 flock[133444]: 0730 01:01:37 ds4-server: chat ctx=0..355:355 prompt start
Jul 30 01:01:37 faex1 flock[133444]: ds4: ROCm loading model tensors 0.99 GiB cached
Jul 30 01:01:41 faex1 flock[133444]: ds4: ROCm loading model tensors 16.04 GiB cached
Jul 30 01:01:46 faex1 flock[133444]: ds4: ROCm loading model tensors 32.06 GiB cached
Jul 30 01:01:56 faex1 flock[133444]: ds4: ROCm loading model tensors 24.04 GiB cached
Jul 30 01:01:59 faex1 flock[133444]: 0730 01:01:59 ds4-server: chat ctx=0..355:355 prompt done 22.475s
Jul 30 01:02:05 faex1 flock[133444]: 0730 01:02:05 ds4-server: chat ctx=355..405:50 gen=50 THINKING decoding chunk=8.70 t/s avg=8.70 t/s 5.750s
Jul 30 01:02:09 faex1 flock[133444]: 0730 01:02:09 ds4-server: chat ctx=405..455:50 gen=100 THINKING decoding chunk=12.47 t/s avg=10.25 t/s 9.758s
Jul 30 01:02:13 faex1 flock[133444]: 0730 01:02:13 ds4-server: chat ctx=455..505:50 gen=150 THINKING decoding chunk=12.50 t/s avg=10.90 t/s 13.757s
Jul 30 01:02:17 faex1 flock[133444]: 0730 01:02:17 ds4-server: chat ctx=505..555:50 gen=200 THINKING decoding chunk=12.39 t/s avg=11.24 t/s 17.792s
...
Jul 30 01:24:28 faex1 flock[133444]: 0730 01:24:28 ds4-server: chat ctx=16255..16305:50 gen=15950 THINKING decoding chunk=11.66 t/s avg=11.82 t/s 1349.246s
Jul 30 01:24:33 faex1 flock[133444]: 0730 01:24:33 ds4-server: chat ctx=16305..16355:50 gen=16000 THINKING decoding chunk=11.70 t/s avg=11.82 t/s 1353.520s
Jul 30 01:24:35 faex1 flock[133444]: 0730 01:24:35 ds4-server: chat ctx=16355..16384:29 gen=16029 THINKING decoding chunk=11.67 t/s avg=11.82 t/s 1356.004s
Jul 30 01:24:35 faex1 flock[133444]: ds4-server: thinking not closed, ignoring DSML in reasoning
Jul 30 01:24:35 faex1 flock[133444]: 0730 01:24:35 ds4-server: chat ctx=0..355:355 gen=16029 THINKING finish=length 1378.480s

It ended up looping over the same few phrases pretty soon and exhausted the whole context.

Then I removed SSD streaming:

/opt/ds4/src/ds4-server --rocm \
  -m ../models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  --ctx 16384 \
  --host 0.0.0.0 \
  --port 8080


Jul 30 01:26:47 faex1 flock[141160]: ds4: Linux rocm backend set oom_score_adj=1000
Jul 30 01:26:47 faex1 flock[141160]: ds4: ROCm backend initialized on AMD Radeon 8060S Graphics (sm_115)
Jul 30 01:26:47 faex1 flock[141160]: ds4: ROCm preparing model tensor mappings
Jul 30 01:26:47 faex1 flock[141160]: ds4: ROCm loading model tensors into device cache
Jul 30 01:26:49 faex1 flock[141160]: ds4: ROCm loading model tensors 16.02 GiB cached
Jul 30 01:26:49 faex1 flock[141160]: ds4: ROCm prepared model tensor mappings 16.37 GiB
Jul 30 01:26:51 faex1 flock[141160]: ds4: ROCm loading model tensors 32.06 GiB cached
Jul 30 01:26:51 faex1 flock[141160]: ds4: ROCm prepared model tensor mappings 32.07 GiB
Jul 30 01:26:53 faex1 flock[141160]: ds4: ROCm loading model tensors 48.02 GiB cached
Jul 30 01:26:53 faex1 flock[141160]: ds4: ROCm prepared model tensor mappings 48.43 GiB
Jul 30 01:26:55 faex1 flock[141160]: ds4: ROCm loading model tensors 64.06 GiB cached
Jul 30 01:26:55 faex1 flock[141160]: ds4: ROCm prepared model tensor mappings 64.13 GiB
Jul 30 01:26:57 faex1 flock[141160]: ds4: ROCm loading model tensors 80.04 GiB cached
Jul 30 01:26:57 faex1 flock[141160]: ds4: ROCm prepared model tensor mappings 80.24 GiB
Jul 30 01:26:58 faex1 flock[141160]: ds4: ROCm startup model preparation covered 80.76 GiB of tensor spans in 11.357s
Jul 30 01:26:58 faex1 flock[141160]: ds4: rocm backend initialized for graph diagnostics
Jul 30 01:26:58 faex1 flock[141160]: ds4: memory: KV 0.57 GiB (raw 0.36 + compressed 0.21) + buffers 0.13 GiB + resident model 80.76 GiB = 81.45 GiB planned
Jul 30 01:26:58 faex1 flock[141160]: ds4: memory detail: ctx=16384 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=4098 backend=rocm
Jul 30 01:26:58 faex1 flock[141160]: 0730 01:26:58 ds4-server: context buffers 710.75 MiB (ctx=16384, backend=rocm, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=4098)
Jul 30 01:26:59 faex1 flock[141160]: 0730 01:26:59 ds4-server: listening on http://0.0.0.0:8080
Jul 30 01:27:09 faex1 flock[141160]: 0730 01:27:09 ds4-server: chat ctx=0..355:355 prompt start
Jul 30 01:27:11 faex1 flock[141160]: 0730 01:27:11 ds4-server: chat ctx=0..355:355 prompt done 2.407s
Jul 30 01:27:15 faex1 flock[141160]: 0730 01:27:15 ds4-server: chat ctx=355..405:50 gen=50 THINKING decoding chunk=16.06 t/s avg=16.06 t/s 3.114s
Jul 30 01:27:18 faex1 flock[141160]: 0730 01:27:18 ds4-server: chat ctx=405..455:50 gen=100 THINKING decoding chunk=15.96 t/s avg=16.01 t/s 6.248s
Jul 30 01:27:21 faex1 flock[141160]: 0730 01:27:21 ds4-server: chat ctx=455..505:50 gen=150 THINKING decoding chunk=15.93 t/s avg=15.98 t/s 9.385s
...
Jul 30 01:35:48 faex1 flock[141160]: 0730 01:35:48 ds4-server: chat ctx=8105..8155:50 gen=7800 THINKING decoding chunk=14.55 t/s avg=15.10 t/s 516.539s
Jul 30 01:35:51 faex1 flock[141160]: 0730 01:35:51 ds4-server: chat ctx=8155..8205:50 gen=7850 THINKING decoding chunk=14.51 t/s avg=15.10 t/s 519.985s
Jul 30 01:35:55 faex1 flock[141160]: 0730 01:35:55 ds4-server: chat ctx=8205..8255:50 gen=7900 THINKING decoding chunk=14.46 t/s avg=15.09 t/s 523.444s
Jul 30 01:35:58 faex1 flock[141160]: 0730 01:35:58 ds4-server: chat ctx=8255..8305:50 gen=7950 decoding chunk=14.44 t/s avg=15.09 t/s 526.907s
Jul 30 01:36:02 faex1 flock[141160]: 0730 01:36:02 ds4-server: chat ctx=8305..8355:50 gen=8000 decoding chunk=14.40 t/s avg=15.08 t/s 530.378s
Jul 30 01:36:05 faex1 flock[141160]: 0730 01:36:05 ds4-server: chat ctx=8355..8405:50 gen=8050 decoding chunk=14.42 t/s avg=15.08 t/s 533.845s
Jul 30 01:36:09 faex1 flock[141160]: 0730 01:36:09 ds4-server: chat ctx=8405..8455:50 gen=8100 decoding chunk=14.38 t/s avg=15.07 t/s 537.320s
Jul 30 01:36:12 faex1 flock[141160]: 0730 01:36:12 ds4-server: chat ctx=8455..8505:50 gen=8150 decoding chunk=14.41 t/s avg=15.07 t/s 540.790s
Jul 30 01:36:16 faex1 flock[141160]: 0730 01:36:16 ds4-server: chat ctx=8505..8555:50 gen=8200 decoding chunk=14.44 t/s avg=15.07 t/s 544.254s
Jul 30 01:36:19 faex1 flock[141160]: 0730 01:36:19 ds4-server: chat ctx=8555..8605:50 gen=8250 decoding chunk=14.41 t/s avg=15.06 t/s 547.723s
Jul 30 01:36:23 faex1 flock[141160]: 0730 01:36:23 ds4-server: chat ctx=8605..8655:50 gen=8300 decoding chunk=14.41 t/s avg=15.06 t/s 551.192s
Jul 30 01:36:26 faex1 flock[141160]: 0730 01:36:26 ds4-server: chat ctx=8655..8705:50 gen=8350 decoding chunk=14.43 t/s avg=15.05 t/s 554.658s
Jul 30 01:36:30 faex1 flock[141160]: 0730 01:36:30 ds4-server: chat ctx=8705..8755:50 gen=8400 decoding chunk=14.43 t/s avg=15.05 t/s 558.122s
Jul 30 01:36:33 faex1 flock[141160]: 0730 01:36:33 ds4-server: chat ctx=8755..8805:50 gen=8450 decoding chunk=14.42 t/s avg=15.05 t/s 561.590s
Jul 30 01:36:37 faex1 flock[141160]: 0730 01:36:36 ds4-server: chat ctx=8805..8855:50 gen=8500 decoding chunk=14.52 t/s avg=15.04 t/s 565.033s
Jul 30 01:36:40 faex1 flock[141160]: 0730 01:36:40 ds4-server: chat ctx=8855..8905:50 gen=8550 decoding chunk=14.52 t/s avg=15.04 t/s 568.477s
Jul 30 01:36:43 faex1 flock[141160]: 0730 01:36:43 ds4-server: chat ctx=8905..8955:50 gen=8600 decoding chunk=14.54 t/s avg=15.04 t/s 571.915s
Jul 30 01:36:47 faex1 flock[141160]: 0730 01:36:47 ds4-server: chat ctx=8955..9005:50 gen=8650 decoding chunk=14.52 t/s avg=15.03 t/s 575.360s
Jul 30 01:36:50 faex1 flock[141160]: 0730 01:36:50 ds4-server: chat ctx=9005..9055:50 gen=8700 decoding chunk=14.51 t/s avg=15.03 t/s 578.805s
Jul 30 01:36:54 faex1 flock[141160]: 0730 01:36:54 ds4-server: chat ctx=9055..9105:50 gen=8750 decoding chunk=14.51 t/s avg=15.03 t/s 582.251s
Jul 30 01:36:57 faex1 flock[141160]: 0730 01:36:57 ds4-server: chat ctx=9105..9155:50 gen=8800 decoding chunk=14.52 t/s avg=15.02 t/s 585.695s
Jul 30 01:37:01 faex1 flock[141160]: 0730 01:37:01 ds4-server: chat ctx=9155..9205:50 gen=8850 decoding chunk=14.51 t/s avg=15.02 t/s 589.142s
Jul 30 01:37:04 faex1 flock[141160]: 0730 01:37:04 ds4-server: chat ctx=9205..9255:50 gen=8900 decoding chunk=14.50 t/s avg=15.02 t/s 592.589s
Jul 30 01:37:08 faex1 flock[141160]: 0730 01:37:08 ds4-server: chat ctx=9255..9305:50 gen=8950 decoding chunk=14.51 t/s avg=15.02 t/s 596.035s
Jul 30 01:37:11 faex1 flock[141160]: 0730 01:37:11 ds4-server: chat ctx=9305..9355:50 gen=9000 decoding chunk=14.49 t/s avg=15.01 t/s 599.486s
Jul 30 01:37:12 faex1 flock[141160]: 0730 01:37:12 ds4-server: chat ctx=9355..9365:10 gen=9010 decoding chunk=14.50 t/s avg=15.01 t/s 600.176s
Jul 30 01:37:12 faex1 flock[141160]: 0730 01:37:12 ds4-server: thinking live checkpoint remembered ctx=0..355:355 live=9365 visible=6810
Jul 30 01:37:12 faex1 flock[141160]: 0730 01:37:12 ds4-server: chat ctx=0..355:355 gen=9010 finish=stop 602.584s

It finished well and gen speed was higher.

kyuz0 · 2026-07-31T08:18:09Z

For people following this, I have completed SWE bench for the Q4 distributed one and the quality seems to have gone down a bit, so I'm trying to see if I can recover the original quality from a month ago.

Scorp1o117 · 2026-08-01T08:35:27Z

Independent cross-validation (from #640, which I'm closing in favor of this PR):

I root-caused the same regression independently via git bisect over the 51 commits between 80ebbc3 (6/17) and 54b36ed (7/28), on a Strix Halo gfx1151 / ROCm 7.14 box with the IQ2XXS ...-imatrix-0731.gguf (80.8 GB, full residency). decode t/s via ds4-bench (ctx 2048, gen 64–128):

commit	date	decode t/s
`80ebbc3` (baseline)	6/17	14.97
`519c4d8` (#51)	7/12	15.08
`ef8d923` ("Add ROCm GLM 5.2 support")	7/18	8.43 ← first bad
`646db66` (#39)	7/20	8.42
`fa8b0b9` (#26)	7/22	8.43
`54b36ed` (main)	7/28	8.44

Restoring the four preq decode call sites (single matmul / paired / HC-expand / attention-output) recovers 15.03 t/s (real chat 16.63 t/s, first-token 120 ms → 68 ms) — matching this PR's numbers (9.0 → 15.5 t/s at ctx 2048).

Notes from my side:

My first patch restored the paths without a model-family guard; the DS4_MODEL_FAMILY != GLM_DSA guard here is a good addition I'd have missed.
Commits Validate HTTP Content-Length strictly #45–River City Drive: Elevating West Michigan’s Sports and Performance Gear #50 between 7/12 and 7/18 currently fail to compile on the ROCm path (no ROCm CI coverage) — worth adding a ROCm compile check to CI.
I kept the fp32 paths as fallback on tmp-allocation/launch failure, same as here.

wayfars · 2026-08-01T18:51:56Z

Tested this PR on Strix Halo (gfx1151, Radeon 8060S, ROCm 7.2.3, 128 GB unified), DeepSeek-V4-Flash IQ2XXS q2-imatrix-0731, full 80.76 GiB residency. It fixes the regression completely:

ctx	`main` (`54b36ed`)	this PR	`80ebbc3` (pre-regression)
2048	9.18	15.70	15.69
4096	8.93	14.98	14.98
8192	8.85	14.74	14.77
16384	8.76	14.50	14.52
32768	8.55	13.76	13.89

DS4_ROCM_DSV4_PREQUANT_DECODE=0 gives 9.17 t/s and =1 gives 15.68, so the rollback knob works as documented. I had independently restored the same four call sites locally before finding this PR; logits from the two builds are bit-identical (max abs diff = 0, top-20 unchanged), which is a useful check that the DS4_MODEL_FAMILY_GLM_DSA guard doesn't perturb the DeepSeek path.

Adding some per-stage data that may be useful, since it pins the cause to the dispatch rather than the kernels. I wrapped every non-GLM extern "C" ds4_gpu_* ROCm entry point in HIP-event timers and ran the identical workload on 80ebbc3 and 54b36ed (688 decode calls per stage, ctx 512):

stage	`80ebbc3`	`54b36ed`	ratio
`matmul_q8_0_tensor`	0.3345	0.7255	2.17x
`attention_output_low_q8_tensor`	0.1676	0.4314	2.57x
`matmul_q8_0_hc_expand_tensor`	0.1848	0.4410	2.39x
`matmul_q8_0_pair_tensor`	0.0689	0.1843	2.68x
`shared_down_hc_expand_q8_0_tensor`	0.0565	0.1087	1.92x
`matmul_f16_tensor` (control)	0.1219	0.1257	1.03x
`routed_moe_one_tensor` (control)	0.2974	0.3017	1.01x
`attention_decode_heads_tensor` (control)	0.0623	0.0626	1.00x

Every Q8_0 path regressed 1.9-2.7x while F16 matmuls, the IQ2/Q2_K routed MoE and attention decode were untouched — consistent with single-token decode falling through to the f32 shared-x kernels, which run a 256-thread block per output row instead of a warp per row. matmul_q8_0_preq_rows_w32_kernel and matmul_q8_0_hc_expand_preq_rows_w32_kernel were left defined but uncalled in rocm/ds4_rocm_q8.cuh, which is the giveaway.

Two side notes:

Also worth flagging that git bisect can't cross this range: the commits between 519c4d8 and ef8d923 fail to compile on the ROCm path, so ef8d923 is the first testable bad commit. A ROCm compile check in CI would have caught both that and this regression.
Separately from this PR, 507fd7c disabling the resident IQ2 sorted/expert-tile MoE batch path costs ~10% prefill at ctx >= 4096 here (234 -> 202 t/s at ctx 8192). That one is a correctness fix so it should stay, but the fast path may be worth revisiting once the corruption is understood.

Donato Capitella added 3 commits July 28, 2026 13:33

Restore ROCm DeepSeek prequant decode kernels

1e8f16c

Merge ROCm DeepSeek prequant decode restoration

f23f13d

Fix ROCm DeepSeek Q4 SSD expert staging

9e1988b

kyuz0 mentioned this pull request Jul 28, 2026

AMD GPU (ROCm/HIP) and especially Strix Halo discussion group #16

Open

Tighten ROCm DeepSeek large model arenas

6882a0f

Fix ROCm IQ2 SSD prefill deadlock

2f49b27

Optimize ROCm two-token MTP verification

d250a7c

kyuz0 mentioned this pull request Aug 1, 2026

fix: restore ROCm Q8 prequantized decode kernels #640

Closed

wayfars mentioned this pull request Aug 1, 2026

DeepSeek-V4-Flash-0731 support #635

Open

Conversation

kyuz0 commented Jul 28, 2026

Summary

Tested models

Performance

Correctness

Two-node example

Uh oh!

VBart commented Jul 28, 2026

Uh oh!

jhohertz commented Jul 28, 2026

Uh oh!

VBart commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kyuz0 commented Jul 29, 2026

Uh oh!

VBart commented Jul 29, 2026

Uh oh!

VBart commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VBart commented Jul 29, 2026

Uh oh!

VBart commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kyuz0 commented Jul 29, 2026

Uh oh!

VBart commented Jul 29, 2026

Uh oh!

VBart commented Jul 29, 2026

Uh oh!

kyuz0 commented Jul 31, 2026

Uh oh!

Scorp1o117 commented Aug 1, 2026

Uh oh!

wayfars commented Aug 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

VBart commented Jul 29, 2026 •

edited

Loading

VBart commented Jul 29, 2026 •

edited

Loading

VBart commented Jul 29, 2026 •

edited

Loading