[Issue]: RCCL deadlock during vLLM TP=2 inference on dual R9700 (gfx1201/RDNA4)

[kyuz0]

Problem Description

Summary

RCCL deadlocks during multi-GPU inference (tensor parallelism = 2) on two AMD Radeon R9700 (gfx1201 / RDNA4) GPUs connected via PCIe. Both GPUs spin at 100% utilization indefinitely. The deadlock occurs during the first multi-GPU operation (CUDA graph capture or first forward pass), not during RCCL initialization.

Single-GPU RCCL init works fine. The same hardware and TP=2 configuration worked correctly with RCCL 2.27.3 on a previous ROCm nightly build (December 2025).

Environment

Component	Version
ROCm	7.2.1
RCCL	2.27.7 (built from source: rocm-7.2.1 branch, --amdgpu_targets gfx1201)
PyTorch	2.11.0 (built from source for gfx1201)
vLLM	v0.19.1
OS	Fedora 43 host, Ubuntu 24.04 container
Kernel	6.x
GPUs	2x AMD Radeon R9700 (gfx1201), 16 GB VRAM each
CPU	AMD Ryzen 9 9900X3D
iGPU	gfx1036 (excluded via HIP_VISIBLE_DEVICES=0,1)

Containers

• Container built like this: https://github.com/kyuz0/amd-r9700-vllm-toolboxes/blob/main/Dockerfile.rocm7.2.1

• Based on this rocm build for gfx1201: https://github.com/kyuz0/gfx1201-rocm-torch-base-toolboxes/blob/main/rocm/toolbox.rocm.Dockerfile

GPU topology

Device  Node  IDs              GFX Version
0       1     0x7551, 32448    gfx1201     (R9700, SKU G287BP00)
1       2     0x7551, 14696    gfx1201     (R9700, SKU 1E4990U)
2       3     0x13c0, 22984    gfx1036     (iGPU, excluded)

Both R9700 cards are consumer PCIe GPUs. No xGMI / Infinity Fabric link between them.

RCCL init works (single-process test)

HIP_VISIBLE_DEVICES=0 python3 -c "
import torch, torch.distributed as dist, os
os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_PORT'] = '29500'
dist.init_process_group(backend='nccl', rank=0, world_size=1)
print('SUCCESS')
dist.destroy_process_group()
"
# Output: SUCCESS — completes instantly

Reproduction

Launch vLLM with TP=2:

HIP_VISIBLE_DEVICES=0,1 NCCL_P2P_DISABLE=1 \
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --tensor-parallel-size 2 --enforce-eager \
  --attention-backend TRITON_ATTN

Observed Behavior

1. Both workers initialize successfully:

(Worker pid=10859) INFO [parallel_state.py:1402] world_size=2 rank=0 local_rank=0 ... backend=nccl
(Worker pid=10860) INFO [parallel_state.py:1402] world_size=2 rank=1 local_rank=1 ... backend=nccl
(Worker pid=10859) INFO [pynccl.py:111] vLLM is using nccl==2.27.7

2. Model loads successfully across both GPUs:

Loading safetensors checkpoint shards: 100% | 4/4 [00:01, 2.69it/s]
(Worker_TP0) INFO Loading weights took 1.49 seconds
(Worker_TP0) INFO Model loading took 7.58 GiB memory
(Worker_TP0) INFO Available KV cache memory: 22.83 GiB

3. Deadlock on first multi-GPU operation (either CUDA graph capture or first inference request with --enforce-eager):

   • Both GPUs jump to 100% utilization and stay there permanently
   • No tokens are generated
   • Inference requests hang until timeout
   • Without --enforce-eager, hangs at: Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%
   • With --enforce-eager, server starts but first request triggers torch.compile on both ranks and deadlocks

4. rocm-smi during hang shows:

GPU[0] GPU%: 100%  (should be idle — no requests)
GPU[1] GPU%: 100%  (should be idle — no requests)

What was tried

Mitigation	Result
NCCL_P2P_DISABLE=1	No effect — still deadlocked
--enforce-eager	Server starts but inference deadlocks on first request
vLLM v0.19.1 stable	Same behavior
vLLM v0.19.2rc1 dev	Same behavior
Clearing all JIT caches	No effect

Working Reference (regression evidence)

The same physical hardware ran TP=2 flawlessly on a previous stack:

Component	Working Version	Broken Version
vLLM	0.14.0rc1.dev27 (Dec 2025)	0.19.1 (Apr 2026)
RCCL	2.27.3	2.27.7
ROCm	Nightly (late 2025)	7.2.1
PyTorch	2.x nightly	2.11.0

Working TP=2 output from the old build:

INFO [pynccl.py:111] vLLM is using nccl==2.27.3
Capturing CUDA graphs: 100% | 2/2 [00:00, 2.88it/s]
INFO Graph capturing finished in 2 secs
INFO Starting vLLM API server on http://0.0.0.0:8000

Suspected Root Cause

RCCL initialization completes successfully, but actual multi-GPU communication deadlocks when running across two gfx1201 consumer GPUs over PCIe. The specific RCCL op that deadlocks is not visible in logs — both GPUs simply spin at 100% with no progress. This may be related to:

1. PCIe transport path changes between RCCL 2.27.3 and 2.27.7 for gfx1201
2. Missing or broken fallback to host-staged copies when P2P is unavailable between consumer RDNA4 GPUs (even with NCCL_P2P_DISABLE=1)
3. Changes in how gfx1201 topology is detected / handled in the RCCL transport layer

Expected Behavior

Multi-GPU communication should complete without deadlocking, using host-staged memory copies as fallback when PCIe P2P is not available between consumer GPUs — as worked in RCCL 2.27.3.

Additional Context

• TP=1 inference on a single R9700 works perfectly on the same stack
• The same ROCm 7.2.1 + RCCL build works for TP=4 on MI25 (gfx900) GPUs in a separate project, confirming the RCCL build itself is functional — the issue is specific to gfx1201 multi-GPU

Related Issues

• vLLM side: [Bug]: TP=2 deadlock on dual AMD R9700 (gfx1201/RDNA4) — GPUs spin at 100%, inference blocked vllm-project/vllm#40980

Operating System

Fedora 43 host, Ubuntu 24.04 container

CPU

AMD Ryzen 9 9900X3D

GPU

2x AMD Radeon R9700 (gfx1201)

ROCm Version

7.2.1

ROCm Component

rccl

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[tcgu-amd]

Hey @kyuz0, can you run your workload with AMD_LOG_LEVEL=4 AMD_LOG_LEVEL_FILE=out and attach the output? Thanks!

Edit, also, running NCCL_DEBUG=INFO vllm serve ... 2>&1 | grep "Using protocol" should help as well.

[kyuz0]

The issue was also being tracked here: vllm-project/vllm#40980 (comment).

The issue is resolved by using this env NCCL_PROTO=Simple at the moment.

[tcgu-amd]

Thanks, that confirms our suspicion.

[cztomsik]

Thanks, that confirms our suspicion.

Could you elaborate? Is there anything we could try out to fix in user-space? Is there anything we could fork, quickfix, and build ourselves?

[tcgu-amd]

Hi @cztomsik, sorry for the late reply -- my attention has been pulled else where.

Commit 9545ae0 ("Navi4 LL enablement and tuning") added rcclGetProtoForGfx12() (src/rccl_wrap.cc:69), which lets the LL protocol be auto-selected for single-node AllReduce/AllGather/ReduceScatter on gfx12 when the message is under ~890KB/~96KB thresholds. That's called from rcclUpdateCollectiveProtocol() at src/rccl_wrap.cc:105-106 whenever NCCL_PROTO isn't set explicitly.

The problem is; src/device/prims_ll.h never received the gfx12 memory-ordering fixes that src/device/prims_simple.h got (stronger atomic acquire semantics needed on RDNA4). Notably, P2P sends/recvs already explicitly disable LL for gfx12 (src/enqueue.cc:1012) — the collective path at line 105-106 is the one gap that still lets LL through, and that's what deadlocks.

A patch could would be.
--- a/src/rccl_wrap.cc
+++ b/src/rccl_wrap.cc
@@ -103,7 +103,7 @@ void rcclUpdateCollectiveProtocol(struct ncclComm* comm, size_t const& nBytes,
} else if (!userProtocolInput && IsArchMatch(comm->topo->nodes[GPU].nodes[0].gpu.gcn, "gfx12") && comm->nNodes == 1){

• info->protocol = rcclGetProtoForGfx12( info->func,sizePerRank);

• info->protocol = NCCL_PROTO_SIMPLE; // LL not yet validated on gfx12, see [Issue]: RCCL deadlock during vLLM TP=2 inference on dual R9700 (gfx1201/RDNA4) #5480
  } else if(!userProtocolInput && comm->nNodes >= 2 && ...

But that's equivalent to setting NCCL_PROTO=Simple.

[tcgu-amd]

Created a patch ROCm/rccl#2187

[cztomsik]

@tcgu-amd no worries, thank you for the context