Hi sglang Team,
First, thank you for creating such a powerful inference engine. I've been testing expert_parallel performance with the deepseek-v2 model on a 2-node cluster (each node with 8x H100 GPUs), and I've encountered some confusion regarding the parallelism settings. I'm opening this issue to share my findings and suggest a documentation update to help future users.
My initial approach was based on information from the community, which led to some counter-intuitive results.
Initial Understanding from Issue #3704 and PR #2203:
I started by reading Issue #3704, which states, "currently when enabling tp/dp/ep together, their size must be the same since they share the same communication group (tp group)." This understanding was reinforced by the implementation details in PR #2203.
First Attempt & OOM:
Following this rule, I set tp=16, dp=16, and ep=16 for my 16-GPU cluster. However, this configuration resulted in an Out-of-Memory (OOM) error with a batch size of 1024. This was confusing. While I understood that ep must equal tp due to the shared communication group, it wasn't clear why dp also had to be equal, especially since this configuration logically implies a need for 256 GPUs (tp * dp).
Intermediate Experiments:
I then tried reducing all three parameters simultaneously (e.g., tp=dp=ep=8). I observed that the number of GPUs actually utilized seemed to be dictated only by the tp value, and it was interpreted as a global value across all nodes (e.g., with tp=8 on 2 nodes, only 4 GPUs per node were active). The dp parameter seemed to have no effect in this setup.
The Solution and Final Configuration
My breakthrough came from studying the official DeepSeek-V3 benchmark guide and Issue #3812. These resources suggested a different approach.
By setting dp=1 (or omitting it) and tp=ep=16, I was able to successfully utilize all 16 GPUs across both nodes without any OOM issues. This appears to be the correct and intended configuration.
Here is my final, working launch command:
This command is run on each node with the appropriate RANK (0 or 1)
python3 -m sglang.launch_server \
--model-path /path/to/your/model \
--host 0.0.0.0 \
--port 30010 \
--tp-size 16 \
--dp-size 1 \
--nnodes 2 \
--node-rank ${RANK} \
--dist-init-addr ${MASTER_ADDR}:5000 \
--enable-ep-moe \
--enable-dp-attention \
--trust-remote-codeSuggestion for Documentation Improvement
I would like to suggest enhancing the documentation to clarify the following points, especially for multi-node scenarios:
How tp_size is interpreted: Please explicitly state that in a multi-node setup (--nnodes > 1), tp_size is treated as the total tensor parallel size across all nodes, and the launcher will distribute this number of processes evenly among the nodes (i.e., processes_per_node = tp_size / nnodes).
The interaction between TP, DP, and EP: A clear explanation of how the model is placed on the GPUs when tp, dp, and ep are enabled together would be invaluable. The rule from Issue #3704 seems to apply only under certain conditions and can be misleading.
A canonical multi-node example: Providing a clear, recommended configuration for a standard 2-node, 8-GPU-per-node setup (like the one above) would serve as an excellent reference for users.
Thank you for considering this feedback. I believe these clarifications would make sglang even more accessible and user-friendly for the community.
Hi sglang Team,
First, thank you for creating such a powerful inference engine. I've been testing expert_parallel performance with the deepseek-v2 model on a 2-node cluster (each node with 8x H100 GPUs), and I've encountered some confusion regarding the parallelism settings. I'm opening this issue to share my findings and suggest a documentation update to help future users.
My initial approach was based on information from the community, which led to some counter-intuitive results.
Initial Understanding from Issue #3704 and PR #2203:
I started by reading Issue #3704, which states, "currently when enabling tp/dp/ep together, their size must be the same since they share the same communication group (tp group)." This understanding was reinforced by the implementation details in PR #2203.
First Attempt & OOM:
Following this rule, I set tp=16, dp=16, and ep=16 for my 16-GPU cluster. However, this configuration resulted in an Out-of-Memory (OOM) error with a batch size of 1024. This was confusing. While I understood that ep must equal tp due to the shared communication group, it wasn't clear why dp also had to be equal, especially since this configuration logically implies a need for 256 GPUs (tp * dp).
Intermediate Experiments:
I then tried reducing all three parameters simultaneously (e.g., tp=dp=ep=8). I observed that the number of GPUs actually utilized seemed to be dictated only by the tp value, and it was interpreted as a global value across all nodes (e.g., with tp=8 on 2 nodes, only 4 GPUs per node were active). The dp parameter seemed to have no effect in this setup.
The Solution and Final Configuration
My breakthrough came from studying the official DeepSeek-V3 benchmark guide and Issue #3812. These resources suggested a different approach.
By setting dp=1 (or omitting it) and tp=ep=16, I was able to successfully utilize all 16 GPUs across both nodes without any OOM issues. This appears to be the correct and intended configuration.
Here is my final, working launch command:
This command is run on each node with the appropriate RANK (0 or 1)
Suggestion for Documentation Improvement
I would like to suggest enhancing the documentation to clarify the following points, especially for multi-node scenarios:
How tp_size is interpreted: Please explicitly state that in a multi-node setup (--nnodes > 1), tp_size is treated as the total tensor parallel size across all nodes, and the launcher will distribute this number of processes evenly among the nodes (i.e., processes_per_node = tp_size / nnodes).
The interaction between TP, DP, and EP: A clear explanation of how the model is placed on the GPUs when tp, dp, and ep are enabled together would be invaluable. The rule from Issue #3704 seems to apply only under certain conditions and can be misleading.
A canonical multi-node example: Providing a clear, recommended configuration for a standard 2-node, 8-GPU-per-node setup (like the one above) would serve as an excellent reference for users.
Thank you for considering this feedback. I believe these clarifications would make sglang even more accessible and user-friendly for the community.