Eval bug: [ROCm/HIP] Severe Performance and Stability Regression for MoE Models since 4146d6a1a

[earthquake02]

Name and Version

since b6902

Operating systems

Linux

GGML backends

HIP

Hardware

System Information

• Hardware: AI MAX 395 (Strix Halo) / AMD Radeon Graphics (gfx1151)
• OS: Ubuntu 24.04.3 LTS
• Kernel: 6.14.0-1015-oem
• ROCm Version: 7.1.0 (Installed from official .deb package)
• Build Command:

  HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
      cmake -S . -B build -DGGML_HIP=ON -DGGML_HIP_ROCWMMA_FATTN=OFF -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release \
      && cmake --build build --config Release -- -j 32

Models

Qwen3-30B-A3B-Q8_0，gpt-oss-120b-MXFP4

Problem description & steps to reproduce

Problem Description

I have identified a significant performance and stability regression on the ROCm/HIP backend when running Mixture-of-Experts (MoE) models. The issue was introduced in commit 4146d6a and persists to the latest master branch (e.g., a5c07dc / b6949 at the time of this report).

The regression manifests in two ways depending on the model:

1. Performance Degradation: For Qwen3-30B-A3B-Q8_0, the Prompt Processing (pp) speed drops by approximately 15-20%.
2. System Crash: For a larger MoE model like gpt-oss-120b-MXFP4, running llama-bench causes an immediate system freeze (black screen), requiring a hard reboot.

Steps to Reproduce

1. Set up the environment with the system specifications listed above.
2. Test the last known "good" commit: git checkout 8da3c0e20, compile, and run llama-bench. The benchmarks will complete successfully with high performance.
3. Test the first "bad" commit: git checkout 4146d6a1a, clean and re-compile.
4. Run llama-bench on Qwen3-30B-A3B-Q8_0 to observe the performance drop.
5. (Caution!) Run llama-bench on gpt-oss-120b-MXFP4 to observe the system crash.

Analysis & Conclusion

Through git log and targeted testing, the regression was pinpointed to a single commit:

• 4146d6a1a - CUDA: add expert reduce kernel (#16857)

The title of the commit strongly suggests the cause. The key observation is that the performance degradation primarily affects Prompt Processing (pp) speed, while Token Generation (tg) speed remains largely unchanged.

This indicates that the new "expert reduce kernel," while implemented for CUDA, has a detrimental effect on the HIP backend for MoE models. The issue scales with model size, leading from performance loss to a catastrophic system failure.

Full logs are provided in the section below.

First Bad Commit

4146d6a

Relevant log output

## Expected Behavior Log (on commit 8da3c0e20 / b6901)

### Log for Qwen3-30B-A3B-Instruct-2507-Q8_0 (Good Performance):
nklar@nklar-GTR-Pro:~/llama.cpp$ ~/llama.cpp/build/bin/llama-bench -m ~/models/Qwen3-30B-A3B-Instruct-2507-Q8_0-GGUF/qwen3-30b-a3b-instruct-2507-q8_0.gguf -fa 1 -p 2048 -n 32 -ub 2048 -d 0,4096,8192
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |          pp2048 |        909.15 ± 1.89 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |            tg32 |         53.09 ± 0.02 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d4096 |        675.32 ± 0.75 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d4096 |         46.15 ± 0.14 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d8192 |        537.88 ± 0.92 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d8192 |         39.33 ± 0.14 |

build: 8da3c0e20 (6901)

### Log for gpt-oss-120b-MXFP4 (Stable Execution):
nklar@nklar-GTR-Pro:~/llama.cpp$ ~/llama.cpp/build/bin/llama-bench -m ~/models/gpt-oss-120b-GGUF/gpt-oss-120b-mxfp4-00001-of-00003.gguf -fa 1 -p 2048 -n 32 -ub 2048 -d 0,4096,8192
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |          pp2048 |        969.49 ± 2.14 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |            tg32 |         51.60 ± 0.02 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d4096 |        787.09 ± 1.48 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d4096 |         44.98 ± 0.08 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d8192 |        672.62 ± 2.01 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d8192 |         41.45 ± 0.06 |

build: 8da3c0e20 (6901)

## Actual Behavior Logs

### Log on First Bad Commit (4146d6a1a / b6902):
nklar@nklar-GTR-Pro:~/llama.cpp$ ~/llama.cpp/build/bin/llama-bench -m ~/models/Qwen3-30B-A3B-Instruct-2507-Q8_0-GGUF/qwen3-30b-a3b-instruct-2507-q8_0.gguf -fa 1 -p 2048 -n 32 -ub 2048 -d 0,4096,8192
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |          pp2048 |        758.15 ± 1.66 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |            tg32 |         54.12 ± 0.02 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d4096 |        594.29 ± 1.42 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d4096 |         46.60 ± 0.08 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d8192 |        482.33 ± 0.18 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d8192 |         39.68 ± 0.09 |

build: 4146d6a1a (6902)

### Log Confirming Issue Persists on Latest Master (a5c07dcd7 / b6949):
nklar@nklar-GTR-Pro:~/llama.cpp$ ~/llama.cpp/build/bin/llama-bench -m ~/models/Qwen3-30B-A3B-Instruct-2507-Q8_0-GGUF/qwen3-30b-a3b-instruct-2507-q8_0.gguf -fa 1 -p 2048 -n 32 -ub 2048 -d 0,4096,8192
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |          pp2048 |        771.78 ± 1.28 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |            tg32 |         53.74 ± 0.06 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d4096 |        591.12 ± 2.10 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d4096 |         46.62 ± 0.17 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d8192 |        483.90 ± 0.62 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d8192 |         39.52 ± 0.10 |

build: a5c07dcd7 (6949)

### Result for gpt-oss-120b-MXFP4 (System Crash):
Running the `llama-bench` command on this model results in a complete system freeze on commit `4146d6a1a` and all subsequent commits. The display goes black, and the machine becomes unresponsive, necessitating a forced shutdown via the power button. There is no log output because of the immediate crash.

[am17an]

@IMbackK can you reproduce this? I'm not sure why AMD would be affected by a simple mul + add kernel. But in any case we can disable this fusion in case of GGML_USE_HIP

[am17an]

Cc @JohannesGaessler also

[earthquake02]

@IMbackK can you reproduce this? I'm not sure why AMD would be affected by a simple mul + add kernel. But in any case we can disable this fusion in case of GGML_USE_HIP

Hi, thank you for the quick response! I really appreciate you looking into this.

Your proposed solution to disable this kernel fusion for the HIP backend sounds like a direct and effective way to resolve the regression.

To help you and @IMbackK investigate this further, I've run additional tests on standard dense (non-MoE) models to confirm if the issue is specific to the new expert kernel.

The results are conclusive: the performance regression does not occur on dense models. Their performance is virtually identical between the "good" commit (8da3c0e20) and the "bad" commit (4146d6a1a). This strongly confirms that the issue is isolated specifically to the new expert reduce kernel and its interaction with the HIP backend when running MoE models.

Click to see benchmark logs for dense models

Test on Hunyuan-MT-7B (Dense Model)

On "good" commit 8da3c0e20 (b6901):

nklar@nklar-GTR-Pro:~/llama.cpp$ ~/llama.cpp/build/bin/llama-bench -m ~/models/other/hunyuan-MT/Hunyuan-MT-7B.i1-Q4_K_M.gguf -fa 1 -p 2048 -n 32 -ub 2048 -d 0,4096,8192
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |          pp2048 |       1111.97 ± 1.27 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |            tg32 |         41.21 ± 0.01 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d4096 |        844.47 ± 1.42 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d4096 |         33.93 ± 0.02 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d8192 |        680.13 ± 1.40 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d8192 |         28.50 ± 0.02 |

On "bad" commit 4146d6a1a (b6902):

nklar@nklar-GTR-Pro:~/llama.cpp$ ~/llama.cpp/build/bin/llama-bench -m ~/models/other/hunyuan-MT/Hunyuan-MT-7B.i1-Q4_K_M.gguf -fa 1 -p 2048 -n 32 -ub 2048 -d 0,4096,8192
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |          pp2048 |       1110.63 ± 2.74 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |            tg32 |         41.15 ± 0.02 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d4096 |        844.38 ± 1.24 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d4096 |         33.92 ± 0.03 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |  pp2048 @ d8192 |        678.14 ± 0.92 |
| hunyuan-dense 7B Q4_K - Medium |   4.30 GiB |     7.50 B | ROCm       |  99 |     2048 |  1 |    tg32 @ d8192 |         28.47 ± 0.02 |

Test on Gemma-3-1B-IT (Dense Model)

On "good" commit 8da3c0e20 (b6901):

nklar@nklar-GTR-Pro:~/llama.cpp$ ~/llama.cpp/build/bin/llama-bench -m ~/models/gemma-3-1b-it-Q8_0.gguf -fa 1 -p 2048 -n 32 -ub 2048 -d 0,4096,8192
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |          pp2048 |      9744.03 ± 28.92 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |            tg32 |        145.51 ± 0.03 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |  pp2048 @ d4096 |      7770.84 ± 52.69 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |    tg32 @ d4096 |        137.47 ± 0.08 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |  pp2048 @ d8192 |       6807.39 ± 8.55 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |    tg32 @ d8192 |        133.77 ± 0.30 |

On "bad" commit 4146d6a1a (b6902):

nklar@nklar-GTR-Pro:~/llama.cpp$ ~/llama.cpp/build/bin/llama-bench -m ~/models/gemma-3-1b-it-Q8_0.gguf -fa 1 -p 2048 -n 32 -ub 2048 -d 0,4096,8192
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |          pp2048 |      9688.22 ± 40.73 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |            tg32 |        145.55 ± 0.14 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |  pp2048 @ d4096 |      7783.78 ± 38.34 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |    tg32 @ d4096 |        137.59 ± 0.06 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |  pp2048 @ d8192 |      6730.43 ± 12.30 |
| gemma3 1B Q8_0                 | 1013.54 MiB |   999.89 M | ROCm       |  99 |     2048 |  1 |    tg32 @ d8192 |        134.23 ± 0.31 |

As I mentioned before, I am fully available to test any patches or experimental branches on my gfx1151 hardware to help verify the fix.

Thanks again for your work on this!

[earthquake02]

Oops, my apologies. I accidentally closed this issue with a misclick. Since I don't have the permissions to reopen it, could a maintainer please reopen it? Sorry for the noise!

[IMbackK]

I cant reproduce this problem, maybe its specific Qwen3-30B, ill download this model and try or its specific to gfx115x. 115x has a larger register file than cdna or rdna2 so maybe we just random into a worse cu occupancy on that uarch.

GPU	Model	Microbatch size	Test	t/s b6901	t/s master	Speedup
2x MI100	gpt-oss 120B MXFP4 MoE	1	pp512	172.06	176.17	1.02
2x MI100	gpt-oss 120B MXFP4 MoE	4	pp512	190.03	191.34	1.01
2x MI100	gpt-oss 120B MXFP4 MoE	8	pp512	298.39	302.06	1.01
2x MI100	gpt-oss 120B MXFP4 MoE	64	pp512	649.13	651.95	1.00
2x MI100	gpt-oss 120B MXFP4 MoE	256	pp512	764.17	766.09	1.00
2x MI100	gpt-oss 120B MXFP4 MoE	512	pp512	1032.45	1028.87	1.00
RX 6800 XT	gpt-oss 20B MXFP4 MoE	1	pp512	123.92	125.53	1.01
RX 6800 XT	gpt-oss 20B MXFP4 MoE	4	pp512	250.08	251.77	1.01
RX 6800 XT	gpt-oss 20B MXFP4 MoE	8	pp512	407.23	407.75	1.00
RX 6800 XT	gpt-oss 20B MXFP4 MoE	64	pp512	878.30	869.24	0.99
RX 6800 XT	gpt-oss 20B MXFP4 MoE	256	pp512	1857.60	1850.45	1.00
RX 6800 XT	gpt-oss 20B MXFP4 MoE	512	pp512	2103.47	2108.98	1.00

The system crash is obviously a bug in amdgpu.ko that should be tracked separately.

[am17an]

Yeah from what I know Qwen3 has 128 experts and 8 used whereas gpt-oss-20b (32/4) and gpt-oss-120b (128/4), and during warmup we do a warmup run with n_expert_used == n_experts and perhaps this unrolls trips up something w.r.t register allocation, I find that hard to believe though.

llama.cpp/ggml/src/ggml-cuda/moe-expert-reduce.cu

Lines 29 to 35 in 4146d6a

	#pragma unroll
	for (int i = 0; i < n_expert_used_template; ++i) {
	ggml_cuda_mad(acc, experts[col], weights[i]);
	experts += n_cols;
	}
	dst[col] = acc;
	}

[JohannesGaessler]

I've received a Strix Halo system from AMD in the mail though I haven't set it up for development yet and I'm currently busy in general. @earthquake02 if you could do a git bisect to identify the exact commit (or commits) that are causing the issue, that would be very helpful.

[IMbackK]

I've received a Strix Halo system from AMD in the mail though I haven't set it up for development yet and I'm currently busy in general.

Oh nice, i never heard back from them.

@earthquake02 if you could do a git bisect to identify the exact commit (or commits) that are causing the issue, that would be very helpful.

he has done so already.

[JohannesGaessler]

he has done so already.

Ah sorry, I confused this issue with another one.

[am17an]

This should be fixed by the fact that this kernel was reverted in master