[Issue]: GPU hang with preemption failure on RX 7600 XT (gfx1102) via llama.cpp + TheRock - Vulkan works fine

[Ericson246]

Problem Description

llama-server (llama.cpp commit d2462f8f7) with ROCm/TheRock backend (tarball
therock-dist-linux-gfx110X-all-7.14.0a20260608) hangs on the first inference
request on AMD Radeon RX 7600 XT (gfx1102, Navi 33). GPU stays at 100%
utilization indefinitely with zero tokens produced. Kernel log shows a
consistent fatal signature: cp queue preemption failed, Failed to evict
process queues, MODE1 reset, followed by queue exhaustion
(-151 queues were already created).

The bug is 100% reproducible with bare minimum server flags (only
--n-gpu-layers 999 --cont-batching). The same llama.cpp build with Vulkan
backend (RADV) runs flawlessly at 6.6-7.6 t/s with zero kernel errors on
the same hardware, confirming the issue is in the ROCm/HIP/KFD stack.

Operating System

Ubuntu 26.04 LTS (Resolute Raccoon)

CPU

Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz

GPU

AMD Radeon RX 7600 XT 16GB (Navi 33, gfx1102, PCI ID 0x7480)

ROCm Version

TheRock 7.14.0a20260608 (nightly, therock-dist-linux-gfx110X-all-7.14.0a20260608.tar.gz)

ROCm Component

No response

Steps to Reproduce

Prerequisites

• Docker with /dev/kfd and /dev/dri device passthrough
• Container user in the render group (GID 991)
• Host ROCm not required for this reproduction — the Docker build uses a self-contained TheRock tarball inside the container

1. Build the Docker image

Using the upstream llama.cpp ROCm TheRock Dockerfile at commit d2462f8f7:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout d2462f8f7
docker buildx build \
  -t llama-cpp-rocm-therock:latest \
  -f .devops/rocm-therock.Dockerfile \
  --target server \
  --load .

The Dockerfile downloads TheRock from:
https://rocm.nightlies.amd.com/tarball-multi-arch/therock-dist-linux-gfx110X-all-7.14.0a20260608.tar.gz

It compiles for gfx1102 with: -DGGML_HIP=ON -DGGML_HIP_ROCWMMA_FATTN=ON -DAMDGPU_TARGETS='gfx1102'

2. Start the server

services:
  llama-api:
    image: llama-cpp-rocm-therock:latest
    container_name: llama-cpp-api
    devices:
      - "/dev/kfd:/dev/kfd"
      - "/dev/dri:/dev/dri"
    group_add:
      - "991"
    volumes:
      - /path/to/models:/models
    command: >
      -m /models/Qwen3.6-27B-Q3_K_S.gguf
      --host 0.0.0.0 --port 8080
      --n-gpu-layers 999 --cont-batching
    ulimits:
      memlock:
        soft: -1
        hard: -1

docker compose up -d

3. Send a test request

docker exec llama-cpp-api curl -s -X POST \
  http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello"}], "max_tokens":10}'

4. Observe

• Server logs stop at slot create_check: created context checkpoint 1 of 32 ...
• rocm-smi shows GPU 100%, VRAM ~91%, no tokens produced
• dmesg shows preemption failures, MODE1 reset, queue exhaustion

Expected (Vulkan control)

Build with -f .devops/vulkan.Dockerfile instead. Same compose, same model.
Result: 6.6-7.6 t/s, zero kernel errors.

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

rocminfo --support output

rocminfo --support (output from container)

ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.21
Runtime Ext Version:     1.24
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
XNACK enabled:           NO
DMAbuf Support:          YES
VMM Support:             YES

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
  Uuid:                    CPU-XX
  Marketing Name:          Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    23978388(0x16de194) KB
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    23978388(0x16de194) KB
    Pool 3
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    23978388(0x16de194) KB
    Pool 4
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    23978388(0x16de194) KB
  ISA Info:

*******
Agent 2
*******
  Name:                    gfx1102
  Uuid:                    GPU-XX
  Marketing Name:          AMD Radeon RX 7600 XT
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      2048(0x800) KB
  Chip ID:                 29824(0x7480)
  ASIC Revision:           0(0x0)
  Cacheline Size:          128(0x80)
  Max Clock Freq. (MHz):   2493
  BDFID:                   1024
  Internal Node ID:        1
  Compute Unit:            32
  SIMDs per CU:            2
  Shader Engines:          2
  Shader Arrs. per Eng.:   2
  Coherent Host Access:    FALSE
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          32(0x20)
  Workgroup Max Size:      1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    1024(0x500)
  Packet Processor uCode:: 632
  SDMA engine uCode::      24
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    16760832(0xffc000) KB
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    16760832(0xffc000) KB
    Pool 3
      Segment:                 GROUP
      Size:                    64(0x40) KB
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx1102
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      FBarrier Max Size:       32
    ISA 2
      Name:                    amdgcn-amd-amdhsa--gfx11-generic
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      FBarrier Max Size:       32
*** Done ***

</details>

### Additional Information

### Kernel fault signatures

Bare minimum case:
Jun 11 08:32:19 kernel: amdgpu: cp queue pipe 1 queue 3 preemption failed
Jun 11 08:32:19 kernel: amdgpu 0000:04:00.0: Failed to evict process queues
Jun 11 08:33:51 kernel: amdgpu 0000:04:00.0: GPU reset begin!. Source:  5
Jun 11 08:33:55 kernel: amdgpu: cp queue pipe 1 queue 3 preemption failed
Jun 11 08:33:55 kernel: amdgpu 0000:04:00.0: Failed to evict process queues
Jun 11 08:33:55 kernel: amdgpu: Failed to suspend process pid 198214
Jun 11 08:33:55 kernel: amdgpu 0000:04:00.0: MODE1 reset
Jun 11 08:33:56 kernel: WARNING: kfd_device_queue_manager.c:880 at destroy_queue_nocpsch_locked amdgpu
Jun 11 08:34:14 kernel: amdgpu: Can't create new usermode queue because -151 queues were already created
Jun 11 08:34:14 kernel: amdgpu: process pid 217024 DQM create queue type 0 failed. ret -1

sdma0 timeout case (non-MTP model during init):
amdgpu 0000:04:00.0: ring sdma0 timeout, signaled seq=107, emitted seq=109
amdgpu 0000:04:00.0: ERROR ring sdma0 test failed (-110)
amdgpu 0000:04:00.0: Ring sdma0 reset failed
amdgpu 0000:04:00.0: GPU reset begin!. Source:  1
amdgpu 0000:04:00.0: GPU reset succeeded, trying to resume
amdgpu 0000:04:00.0: drm device wedged, but recovered through reset

Devcoredump (sdma0 case):
**** AMDGPU Device Coredump ****
Ring timed out details
IP Type: 2 Ring Name: sdma0
gfxhub Page fault observed
Faulty page starting at address: 0x0000000000000000

### ROCm-smi during hang

VRAM% 91% | GPU% 100% | Temp 71C | Power 123W

### Container logs at hang point

slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot create_check: id  3 | task 0 | created context checkpoint 1 of 32
  (pos_min = 11, pos_max = 11, n_tokens = 12, size = 149.626 MiB)

(No further output - infinite hang)

### Post-hang state

After GPU reset, VRAM stays at ~70% with GPU at 1% activity.
Subsequent runs fail with "ROCm error: out of memory" until cold reboot.

### Environment notes

- The server runs in a Docker container with TheRock runtime installed at /opt/therock from a self-contained tarball (no dependency on host ROCm packages)
- HSA_OVERRIDE_GFX_VERSION is empty in the container; rocminfo reports gfx1102 correctly
- llama.cpp compiled for gfx1102 target with -DGGML_HIP=ON -DAMDGPU_TARGETS='gfx1102'
- Container base: ubuntu:24.04 (TheRock), ubuntu:26.04 (Vulkan)
- GPU is RX 7600 XT 16GB, detected as amdgpu driver, kernel 7.0.0-22-generic

### Related issues

- #737 - gfx1151, identical MES queue eviction failure pattern
- ROCm/ROCm#6313 - BC-250/gfx1013, preemption + evict failures
- ROCm/ROCm#5225 - gfx1030, sporadic page faults
- https://bugzilla.kernel.org/show_bug.cgi?id=221012 - kernel page fault on RX 7600 XT

[darren-amd]

Hi @Ericson246,

Thanks for the report! I just gave this exact workload a try on the same card and was unable to reproduce the issue. Could you please try upgrading amdgpu and providing me with the full dmesg logs if the issue persists? Thanks!

[Ericson246]

Hi @darren-amd,

First of all, thank you very much for your time and attention to this issue.
I really appreciate that you took the time to try to reproduce it.

I want to apologize for any inconvenience — I'm not an expert and this is my
first time reporting an issue to AMD. Thank you for your patience.

---

What I tested

After your suggestion, I upgraded to amdgpu-dkms 6.19.4 and ran a complete
test matrix with different TheRock versions, kernel parameters, and models.

Test 1: amdgpu 6.19.4 + TheRock 7.14 + sched_policy=2

Result: HANG — same as the original bug. MODE1 reset, preemption failures.
The amdgpu upgrade alone did NOT fix it.

Kernel log signature:

amdgpu: cp queue pipe 1 queue 3 preemption failed
amdgpu 0000:04:00.0: Failed to evict process queues
amdgpu 0000:04:00.0: MODE1 reset

Test 2: amdgpu 6.19.4 + TheRock 7.15 + sched_policy=2

Result: Partial improvement. First ~1900 tokens generated at 12-15 t/s,
but then a wavefront reset occurred on the ~3rd prompt. Preemption failures
were persistent in the kernel log but non-fatal at first.

Test 3: amdgpu 6.19.4 + TheRock 7.15 + sched_policy=1 (default HWS)

Result: WORKING. Zero kernel errors. I tested multiple models (27B standard,
35B MoE with MTP/draft, up to 110K context) and all work flawlessly.
~14 t/s without MTP, ~44-50 t/s with MTP draft.

Additional finding: amdgpu.runpm=0

My initial kernel parameters included amdgpu.runpm=0 for compatibility with
my older X99 motherboard. I discovered this parameter causes false GPU hang
readings: the MES firmware counter gets stuck at 100% after a crash, the GPU
stays at maximum clock even when idle, and monitoring
tools like nvtop trigger the GPU to 100% just by opening /dev/dri.

After removing amdgpu.runpm=0 from the kernel command line, the GPU idles
correctly, and everything is stable.

---

Current status

The issue is resolved on my system. The fix requires two kernel parameter
changes:

1. amdgpu.sched_policy=2 → 1 — eliminates preemption failures
2. Remove amdgpu.runpm=0 — prevents false hang readings and keeps the
   GPU cool in idle

Both TheRock 7.13 and 7.15 work correctly with sched_policy=1 and default
power management. I'm currently running TheRock 7.13 in production.

---

Summary table

Config	sched	runpm	Result	dmesg
7.14 (original)	2	0	HANG	preemption fail + MODE1 reset
amdgpu 6.19.4 + 7.14	2	0	HANG	preemption fail + MODE1 reset
amdgpu 6.19.4 + 7.15	2	0	Partial (~1900 tokens)	preemption + wavefront reset
amdgpu 6.19.4 + 7.15	1	0	WORKING	0 errors
amdgpu 6.19.4 + 7.15	1	default	WORKING + stable host	0 errors

---

If you need more tests

I'm happy to run any additional tests you might need:

• Test with sched_policy=2 on a specific kernel/driver version to help
  diagnose the Gang scheduling issue on gfx1102
• Test specific TheRock nightly builds
• Any other diagnostic you consider useful

Otherwise, please feel free to close the issue as resolved on my end.

Thank you again for your help and for developing the TheRock project.
I really appreciate the effort AMD is putting into Linux GPU computing.

Best regards,
Ericson

[darren-amd]

Hi @Ericson246,

Thanks a lot for the detailed follow up, glad to hear that it's working for you now! In general, I wouldn't recommend setting amdgpu.sched_policy=2, as that disables HWS and is usually only used for debugging/bringup purposes. You can read a bit more about this here. I'm going to close this off as resolved, but please feel free to let me know if you run into more issues, thanks!