ROCm decode throughput capped at ~58% of theoretical bandwidth on RX 6800/6800XT with Qwen3.8-27B (qwen35 hybrid, Gated DeltaNet) — expected for this arch/backend?

[aikenmilk21]

Setup

CPU: Ryzen 5 3500 (Zen2) / B550 / 32GB RAM. GPU: RX 6800 XT 16GB + RX 6800 16GB (both gfx1030), layer split across both (model doesn't fit on one card). Backend: ROCm (rocm_v7_2, via Ollama 0.33.2 which bundles the current llama.cpp). Model: Qwen3.8-27B (arch qwen35, ~3:1 hybrid of Gated DeltaNet linear-attention layers and regular GQA/full-attention), Q4_K_M, num_ctx 65536. Weight tensors resident on GPU: 15,338 MiB total (ROCm0 7,252 + ROCm1 8,086).

Numbers

Decode: 18.5–19.4 tok/s (19.4 with MTP speculative decoding, --spec-draft-n-max 2). Prefill: ~230 tok/s on a fresh 15k-token prompt (prefix cache busted by prepending a timestamp — first runs without this showed 3,600–13,000 tok/s, clearly just cache hits). Effective bandwidth at 18.5 tok/s: 18.5 × 16.08GB ≈ 297 GB/s, vs. 512 GB/s theoretical (each card is ~512GB/s GDDR6; bandwidth doesn't add across layer-split GPUs, so 512 is the right denominator, not 1024) → ~58%.

What I ruled out

CPU: llama-server pegged at under 1 of 6 cores, loadavg ~1.1 — not it.

PCIe: both GPUs sit behind chipset x1 links (one Gen3, one Gen2 via an ASM1184e splitter), but measured PCIe traffic during decode is ~0.07% of link capacity (only the boundary hidden-state crosses GPUs per token, ~20KB). Only cold-load time is affected (~56s), not decode. Fixed with keep_alive=-1.

Power/clock/thermal: both cards at top memory-clock DPM state during decode, 66–86W (well under 271W/208W caps), 54–58°C. No throttling.

--flash-attn 1 (forced on, vs. the default auto): decode dropped 18.5→16.4 tok/s (-11%). Reverted.

KV cache quantization: irrelevant here — decode speed is identical at 25-token and 15,330-token context (18.5 tok/s either way), consistent with most layers being linear-attention with O(1) state instead of a growing KV cache.

-sm row: fails to load — device ROCm0 does not support split buffers. Checked the source: ggml_backend_split_buffer_type is only implemented in ggml-sycl, not registered for CUDA/HIP.

Vulkan (RADV) backend, built fresh from master: loads and reports both GPUs correctly (AMD Radeon RX 6800 XT (RADV NAVI21) / RX 6800 (RADV NAVI21)), but crashes mid-decode with vk::DeviceLostError / vk::Queue::submit: ErrorDeviceLost. GPUs recovered fine afterward (confirmed by reloading on ROCm, 13s cold load, normal output) — so this looks like a Vulkan-backend/op issue with this hybrid architecture rather than a hardware fault.

draft_num_predict sweep (via Ollama's options, same warm runner each time): 0 → 17.5 tok/s, 2 → 19.4 tok/s, 4 → 14.3 tok/s. 2 is optimal; higher just increases wasted speculation.

Question

Given all of the above is ruled out, is ~58% of theoretical bandwidth (mid-50s to low-60s range, per some community benchmarks I've seen for RDNA2) the expected ceiling for qwen35-style hybrid linear-attention models on the ROCm backend right now? Or is there a known gap specifically for Gated DeltaNet decode kernels on HIP that's being worked on?

Happy to share full llama-bench output, ollama ps/log excerpts, or run additional comparisons if useful.

[ax2211]

I am currently using the same QWEN 3.8 27B dense Q4_M model in a Windows environment with a dual RX 6800 XT setup, a B450M motherboard, a Ryzen 5700X3D, and 32GB RAM. Using LM Studio (non-bionic), I recorded a maximum of about 31 tokens/s.
My test results also showed that even with less-than-ideal PCIe bandwidth, the transfer speed remained stable without significant bottlenecks. Because I am using a B450M motherboard, GPU 1 is connected to PCIe 3.0 x16 and GPU 2 is connected to PCIe 2.0 x4, but it still delivered a sufficient token generation rate.
Have you tried looking into kernel efficiency or downgrading the ROCm version?
Please excuse my poor English.