Name and Version
llama.cpp version 9514 (commit 21444c8), built with GCC 16.1.1, ROCm 7.2 / HIP 7.2.53211
Operating systems
Linux
GGML backends
RPC
Hardware
- AMD Radeon RX 7900 XTX (gfx1100, 24GB) - sharedMemPerBlock: 64KB
- AMD Radeon Pro W7800 48GB (gfx1100) - sharedMemPerBlock: 64KB
- AMD Radeon Graphics (gfx1151, 120GB via RPC) - sharedMemPerBlock: 64KB
Models
unsloth/Qwen3.6-27B-MTP-GGUF (Q8_K_XL quantization, ~34GB) - https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF
Step-3.7-Flash-UD-IQ4_XS-00001-of-00003.gguf --spec-draft-model Step-3.7-Flash-MTP-Q8_0.gguf
Problem description & steps to reproduce
When using MTP (--spec-type draft-mtp) with backend_sampling=1 (the default), the top_k(10) sampler is offloaded to the GPU. The top_k op internally uses argsort_f32_i32_cuda_bitonic which requires more shared memory per block than AMD GPUs provide (64KB). This causes a GGML_ASSERT failure that aborts the process.
When using RPC, the assert kills the RPC server process, which causes the client to see:
ggml-rpc.cpp:498: Remote RPC server crashed or returned malformed response
recv failed (bytes_recv=0, size_to_recv=8)
Steps to reproduce:
-
Start RPC server: ./rpc-server --host 0.0.0.0
-
Start llama-server with MTP:./llama-server --model Qwen3.6-27B-UD-Q8_K_XL.gguf --gpu-layers 99 --rpc localhost:50052,192.168.1.19:50052 --device RPC0,RPC1,RPC2 -ts 19,48,96 -c 200000 --no-warmup --spec-type draft-mtp --spec-draft-device RPC0
-
Send any completion request - crashes after the first generated word
Can also be reproduced fully locally (no RPC needed) - the GGML_ASSERT fires on any AMD GPU.
Workaround: use --no-spec-draft-backend-sampling parameter
First Bad Commit
Unknown - the argsort bitonic sort kernel and backend_sampling for MTP appear to have always had this limitation on AMD hardware.
Relevant log output
RPC server side:
/root/llama.cpp/ggml/src/ggml-cuda/argsort.cu:224: GGML_ASSERT(shared_mem <= ggml_cuda_info().devices[ggml_cuda_get_device()].smpb) failed
Stack trace:
#6 argsort_f32_i32_cuda_bitonic
#7 ggml_cuda_op_top_k
#8 ggml_cuda_graph_evaluate_and_capture
#9 ggml_backend_cuda_graph_compute
#10 ggml_backend_graph_compute
#11 rpc_server::graph_compute
Client side:
ggml-rpc.cpp:498: Remote RPC server crashed or returned malformed response
Workaround: --no-spec-draft-backend-sampling keeps the draft sampling on the CPU and avoids the crash.
Name and Version
llama.cpp version 9514 (commit 21444c8), built with GCC 16.1.1, ROCm 7.2 / HIP 7.2.53211
Operating systems
Linux
GGML backends
RPC
Hardware
Models
unsloth/Qwen3.6-27B-MTP-GGUF (Q8_K_XL quantization, ~34GB) - https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF
Step-3.7-Flash-UD-IQ4_XS-00001-of-00003.gguf --spec-draft-model Step-3.7-Flash-MTP-Q8_0.gguf
Problem description & steps to reproduce
When using MTP (--spec-type draft-mtp) with backend_sampling=1 (the default), the top_k(10) sampler is offloaded to the GPU. The top_k op internally uses argsort_f32_i32_cuda_bitonic which requires more shared memory per block than AMD GPUs provide (64KB). This causes a GGML_ASSERT failure that aborts the process.
When using RPC, the assert kills the RPC server process, which causes the client to see:
ggml-rpc.cpp:498: Remote RPC server crashed or returned malformed response
recv failed (bytes_recv=0, size_to_recv=8)
Steps to reproduce:
Start RPC server: ./rpc-server --host 0.0.0.0
Start llama-server with MTP:./llama-server --model Qwen3.6-27B-UD-Q8_K_XL.gguf --gpu-layers 99 --rpc localhost:50052,192.168.1.19:50052 --device RPC0,RPC1,RPC2 -ts 19,48,96 -c 200000 --no-warmup --spec-type draft-mtp --spec-draft-device RPC0
Send any completion request - crashes after the first generated word
Can also be reproduced fully locally (no RPC needed) - the GGML_ASSERT fires on any AMD GPU.
Workaround: use --no-spec-draft-backend-sampling parameter
First Bad Commit
Unknown - the argsort bitonic sort kernel and backend_sampling for MTP appear to have always had this limitation on AMD hardware.
Relevant log output
RPC server side:
/root/llama.cpp/ggml/src/ggml-cuda/argsort.cu:224: GGML_ASSERT(shared_mem <= ggml_cuda_info().devices[ggml_cuda_get_device()].smpb) failed
Stack trace:
#6 argsort_f32_i32_cuda_bitonic
#7 ggml_cuda_op_top_k
#8 ggml_cuda_graph_evaluate_and_capture
#9 ggml_backend_cuda_graph_compute
#10 ggml_backend_graph_compute
#11 rpc_server::graph_compute
Client side:
ggml-rpc.cpp:498: Remote RPC server crashed or returned malformed response
Workaround: --no-spec-draft-backend-sampling keeps the draft sampling on the CPU and avoids the crash.