[Bug]: all2all communication hangs when using DeepEP and PPLX for v0.9.2

[JoeyYoung]

Your current environment

See as below.

🐛 Describe the bug

Dear all,

we are trying to run large-scale EP (across DP) with DeepEP, using the deepep dockerfile provided by SGLang.
testbed: 2 nodes, each with 8 GPUs (H20-141 GB). mlx5 IB card with IBGDA enabled.
model: deepseek_r1

We have nvidia_peermem and gdr driver installed in the host.
We can successfully run DeepEP internode benchmark with high_throughput and low_latency kernels (IBGDA enabled).

However, when we are running vLLM-v0.9.2 dp=2, tp=8, ep=16, with all2all backend set to deepep_low_latency or deepep_high_throughput. We found the process hangs when doing warm-up requests.
The script we use is offline_inference/data_parallel.py.

We trace the code and find that the first MoE layer can complete the dispatch -> expert compute -> combine process. But in the second MoE layer, workers hang when calling dispatch (here) and the whole task will timeout after 5 minutes. Surprisingly, when we torch up the code a bit to skip the combine process, the task can complete.

We also find using PPLX hangs in the same place. But using naive all2all impl (i.e., broadcast + allreduce) works fine.

More information: we integrate DeepEP by ourselves in v0.8.4, and can run normally using offline DP example in the same environment. When it comes to the community version after v0.9.1, this bug occurs. Is that due to the development of online DP？

We would greatly appreciate it if you have any hints on debugging this problem. It stucks us for weeks.

Please let me know if more information is needed. Many thanks!

Running command:

VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_ALL2ALL_BACKEND=deepep_high_throughput \
NVSHMEM_HCA_LIST=mlx5_2 \
NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME=eth0 \
NCCL_SOCKET_IFNAME=eth0 \
GLOO_SOCKET_IFNAME=eth0  \
NCCL_IB_HCA=mlx5_2 \
python examples/offline_inference/data_parallel.py \
            --model="/vllm-workspace/models/DeepSeek-V3-0324" \
            --dp-size=2 \
            --tp-size=8 \
            --node-size=2 \
            --node-rank=0 \
            --master-addr=xxx.xxx.xxx.xxx \
            --master-port=12768 


VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_ALL2ALL_BACKEND=deepep_high_throughput \
NVSHMEM_HCA_LIST=mlx5_2 \
NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME=eth0 \
NCCL_SOCKET_IFNAME=eth0 \
GLOO_SOCKET_IFNAME=eth0  \
NCCL_IB_HCA=mlx5_2 \
python examples/offline_inference/data_parallel.py \
            --model="/vllm-workspace/models/DeepSeek-V3-0324" \
            --dp-size=2 \
            --tp-size=8 \
            --node-size=2 \
            --node-rank=1 \
            --master-addr=xxx.xxx.xxx.xxx \
            --master-port=12768 

The output screenshot:

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[JoeyYoung]

@youkaichao @WoosukKwon @DarkLight1337 @jeejeelee
Anyone could kindly help with that？

[youkaichao]

cc @varun-sundar-rabindranath @tlrmchlsmth @bnellnm

[bnellnm]

@JoeyYoung , can you provide some logs with VLLM_LOGGING_LEVEL=DEBUG and NCCL_DEBUG=INFO. That might give some insight as to why you are seeing lockups. You could also try --enforce-eager to see if that helps.

[JoeyYoung]

@JoeyYoung , can you provide some logs with VLLM_LOGGING_LEVEL=DEBUG and NCCL_DEBUG=INFO. That might give some insight as to why you are seeing lockups. You could also try --enforce-eager to see if that helps.

Thanks for your attention, please check the log info on our two nodes as below, when running with deepep and pplx respectively:

deepep:

deepep-ht-node1.txt

deepep-ht-node2.txt

pplx:

pplx-node1.txt

pplx-node2.txt

[JoeyYoung]

@bnellnm Hi, we would appreciate it if you could kindly have a look

[varun-sundar-rabindranath]

@JoeyYoung can you try dp=16, tp=1, dp=8, tp=2 and dp=4, tp=4 variations please. It'll help debugging to understand if atlease one of the config passes. Thanks.

[tlrmchlsmth]

@JoeyYoung could you try updating vLLM to the latest 0.10.0? The EP+DP MoE support has been under heavy development so this may have been fixed already

[JoeyYoung]

@JoeyYoung can you try dp=16, tp=1, dp=8, tp=2 and dp=4, tp=4 variations please. It'll help debugging to understand if atlease one of the config passes. Thanks.

For DeepEP, it will crash because the inter-machine communication scenario requires one DP rank to have 8 GPUs, which is coded as a fixed number in their kernels (issue104).
We have tried to launch dp=8, tp=1, dp=4, tp=2 and dp=2, tp=4 within one machine, and it works normally.

[JoeyYoung]

@JoeyYoung could you try updating vLLM to the latest 0.10.0? The EP+DP MoE support has been under heavy development so this may have been fixed already

We will have a try

[JoeyYoung]

@JoeyYoung can you try dp=16, tp=1, dp=8, tp=2 and dp=4, tp=4 variations please. It'll help debugging to understand if atlease one of the config passes. Thanks.

We will try PPLX to see if any more information

[JoeyYoung]

@JoeyYoung could you try updating vLLM to the latest 0.10.0? The EP+DP MoE support has been under heavy development so this may have been fixed already

@tlrmchlsmth We have tested vLLM 0.10.0, but the program hangs in the same place.

[JoeyYoung]

@JoeyYoung can you try dp=16, tp=1, dp=8, tp=2 and dp=4, tp=4 variations please. It'll help debugging to understand if atlease one of the config passes. Thanks.

@varun-sundar-rabindranath We have tried pplx with different dp settings, but there is no further information. The program hangs in the same place. Do you have any suggestions?