Name and Version
ggml commit 58c3805 (via stable-diffusion.cpp, not llama-cli directly)
Reproduced using the shared ggml Vulkan backend (ggml-vulkan.cpp)
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
No response
Command line
# Reproduced via stable-diffusion.cpp (shares ggml Vulkan backend):
sd-cli --diffusion-model flux-2-klein-4b-Q4_0.gguf --vae ae.safetensors \
--llm Qwen3-4B-Q4_K_M.gguf --cfg-scale 1.0 --steps 4 --vae-tiling \
-H 1024 -W 1024 -p "a cat sitting in Taipei" -o output.png
Problem description & steps to reproduce
On AMD Renoir APU (gfx90c, RADV), Vulkan compute jobs crash with vk::Queue::submit: ErrorDeviceLost when running large compute graphs.
Root Cause
The default Linux amdgpu.lockup_timeout is 2000ms. ggml_backend_vk_graph_compute batches up to 100 nodes per vkQueueSubmit. On slow integrated GPUs/APUs, the accumulated GPU work in a single submission exceeds this timeout, causing the kernel to reset the compute ring.
The relevant code in ggml-vulkan.cpp:
int nodes_per_submit = 100;
At smaller workloads each batch completes in time. At larger workloads (e.g., 1024×1024 diffusion with seq_len ≈ 4608), the quadratic attention scaling pushes batches past the 2-second limit.
Steps to Reproduce
Encountered via stable-diffusion.cpp (which vendors ggml, commit 58c38058) running Flux 2 Klein 4B at 1024×1024 on AMD Renoir APU:
sd-cli --diffusion-model flux-2-klein-4b-Q4_0.gguf --vae ae.safetensors \
--llm Qwen3-4B-Q4_K_M.gguf --cfg-scale 1.0 --steps 4 --vae-tiling \
-H 1024 -W 1024 -p "a cat sitting in Taipei" -o output.png
The same issue would affect llama.cpp with sufficiently long contexts on APU hardware, since the Vulkan backend code is shared.
Validated Fix
Changing nodes_per_submit from 100 to 1 resolves the crash with no measurable performance regression (actually ~3% faster at 512×512):
| Resolution |
Before |
After (nodes=1) |
| 512×512 |
54.45s |
52.75s |
| 1024×1024 |
CRASH |
261.60s |
Suggested Improvement
Rather than hardcoding nodes_per_submit = 1 globally, a proper fix could:
- Auto-detect integrated/APU devices and use a smaller batch size
- Expose an environment variable override (e.g.,
GGML_VK_NODES_PER_SUBMIT)
- Account for heavy non-matmul ops (flash attention) in the byte-based submission heuristic
Note
We encountered this in stable-diffusion.cpp's vendored ggml (commit 58c38058). We have a working local fix, but are filing here since llama.cpp is where the Vulkan backend is actively maintained and changes sync downstream.
Kernel timeout can also be increased as a user-side workaround:
sudo sh -c 'echo 60000 > /sys/module/amdgpu/parameters/lockup_timeout'
First Bad Commit
No response
Relevant log output
From kernel log (journalctl -k):
amdgpu: ring comp_1.2.0 timeout, signaled seq=12481, emitted seq=12485
amdgpu: Process sd-cli pid 33328 thread sd-cli pid 33328
amdgpu: Starting comp_1.2.0 ring reset
amdgpu: Ring comp_1.2.0 reset succeeded
[drm] device wedged, but recovered through reset
From sd-cli stderr:
radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery.
terminate called after throwing an instance of 'vk::DeviceLostError'
what(): vk::Queue::submit: ErrorDeviceLost
Name and Version
ggml commit 58c3805 (via stable-diffusion.cpp, not llama-cli directly)
Reproduced using the shared ggml Vulkan backend (ggml-vulkan.cpp)
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
No response
Command line
Problem description & steps to reproduce
On AMD Renoir APU (gfx90c, RADV), Vulkan compute jobs crash with
vk::Queue::submit: ErrorDeviceLostwhen running large compute graphs.Root Cause
The default Linux
amdgpu.lockup_timeoutis 2000ms.ggml_backend_vk_graph_computebatches up to 100 nodes pervkQueueSubmit. On slow integrated GPUs/APUs, the accumulated GPU work in a single submission exceeds this timeout, causing the kernel to reset the compute ring.The relevant code in
ggml-vulkan.cpp:At smaller workloads each batch completes in time. At larger workloads (e.g., 1024×1024 diffusion with seq_len ≈ 4608), the quadratic attention scaling pushes batches past the 2-second limit.
Steps to Reproduce
Encountered via stable-diffusion.cpp (which vendors ggml, commit
58c38058) running Flux 2 Klein 4B at 1024×1024 on AMD Renoir APU:sd-cli --diffusion-model flux-2-klein-4b-Q4_0.gguf --vae ae.safetensors \ --llm Qwen3-4B-Q4_K_M.gguf --cfg-scale 1.0 --steps 4 --vae-tiling \ -H 1024 -W 1024 -p "a cat sitting in Taipei" -o output.pngThe same issue would affect llama.cpp with sufficiently long contexts on APU hardware, since the Vulkan backend code is shared.
Validated Fix
Changing
nodes_per_submitfrom100to1resolves the crash with no measurable performance regression (actually ~3% faster at 512×512):Suggested Improvement
Rather than hardcoding
nodes_per_submit = 1globally, a proper fix could:GGML_VK_NODES_PER_SUBMIT)Note
We encountered this in stable-diffusion.cpp's vendored ggml (commit
58c38058). We have a working local fix, but are filing here since llama.cpp is where the Vulkan backend is actively maintained and changes sync downstream.Kernel timeout can also be increased as a user-side workaround:
sudo sh -c 'echo 60000 > /sys/module/amdgpu/parameters/lockup_timeout'First Bad Commit
No response
Relevant log output
From kernel log (journalctl -k):
amdgpu: ring comp_1.2.0 timeout, signaled seq=12481, emitted seq=12485
amdgpu: Process sd-cli pid 33328 thread sd-cli pid 33328
amdgpu: Starting comp_1.2.0 ring reset
amdgpu: Ring comp_1.2.0 reset succeeded
[drm] device wedged, but recovered through reset
From sd-cli stderr:
radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery.
terminate called after throwing an instance of 'vk::DeviceLostError'
what(): vk::Queue::submit: ErrorDeviceLost