[https://nvbugs/6027560][fix] fix hang issues on DGX_B200-8_GPUs-PyTo…#12656
[https://nvbugs/6027560][fix] fix hang issues on DGX_B200-8_GPUs-PyTo…#12656bo-nv merged 7 commits intoNVIDIA:mainfrom
Conversation
|
/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-1" |
📝 WalkthroughWalkthroughAdded UCX TLS configuration ( Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes 🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/integration/defs/accuracy/test_disaggregated_serving.py (1)
270-283:⚠️ Potential issue | 🟠 MajorPreserve the
^ibworkaround instead of overwriting it later.Lines 12/45 set
UCX_TLS="^ib"(B200 hang mitigation), but lines 24/56 unconditionally overwrite it toUCX_TLS="^cuda_ipc"whenhas_nvlink()returns false, entirely dropping the hang mitigation. Use valid UCX syntax^ib,cuda_ipcto exclude both transports in a single assignment.Proposed fix
for i, port in enumerate(ctx_ports): env = base_env.copy() env["TRTLLM_USE_UCX_KVCACHE"] = "1" - # Need to set UCX_TLS to ^ib to avoid hangs on CI B200 cluster. - env["UCX_TLS"] = "^ib" + # Exclude ib to avoid hangs on CI B200 cluster; also exclude cuda_ipc on non-NVLink systems. + ucx_excludes = ["ib"] + if not has_nvlink(): + ucx_excludes.append("cuda_ipc") + env["UCX_TLS"] = "^" + ",".join(ucx_excludes) if enable_perf: env["TRTLLM_KVCACHE_TIME_OUTPUT_PATH"] = kv_cache_perf_dir @@ -22,8 +22,6 @@ gpu_range = range(current_gpu_offset, current_gpu_offset + ctx_total_gpus) env["CUDA_VISIBLE_DEVICES"] = ",".join(map(str, gpu_range)) - if not has_nvlink(): - env["UCX_TLS"] = "^cuda_ipc" current_gpu_offset += ctx_total_gpus for i, port in enumerate(gen_ports): env = base_env.copy() env["TRTLLM_USE_UCX_KVCACHE"] = "1" - # Need to set UCX_TLS to ^ib to avoid hangs on CI B200 cluster. - env["UCX_TLS"] = "^ib" + # Exclude ib to avoid hangs on CI B200 cluster; also exclude cuda_ipc on non-NVLink systems. + ucx_excludes = ["ib"] + if not has_nvlink(): + ucx_excludes.append("cuda_ipc") + env["UCX_TLS"] = "^" + ",".join(ucx_excludes) if enable_perf: env["TRTLLM_KVCACHE_TIME_OUTPUT_PATH"] = kv_cache_perf_dir @@ -52,8 +52,6 @@ gpu_range = range(current_gpu_offset, current_gpu_offset + gen_total_gpus) env["CUDA_VISIBLE_DEVICES"] = ",".join(map(str, gpu_range)) - if not has_nvlink(): - env["UCX_TLS"] = "^cuda_ipc" current_gpu_offset += gen_total_gpus🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/integration/defs/accuracy/test_disaggregated_serving.py` around lines 270 - 283, The current logic sets env["UCX_TLS"] = "^ib" then later overwrites it to "^cuda_ipc" when has_nvlink() is false, losing the "^ib" workaround; change the update inside the has_nvlink() check to preserve the ib exclusion by assigning env["UCX_TLS"] = "^ib,cuda_ipc" (or append ",cuda_ipc" to the existing env["UCX_TLS"] if present) so both transports are excluded; modify the code around the existing env assignment and the has_nvlink() branch (referencing env and has_nvlink()) to perform this combined exclusion instead of an unconditional overwrite.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@tests/integration/defs/accuracy/test_disaggregated_serving.py`:
- Around line 270-283: The current logic sets env["UCX_TLS"] = "^ib" then later
overwrites it to "^cuda_ipc" when has_nvlink() is false, losing the "^ib"
workaround; change the update inside the has_nvlink() check to preserve the ib
exclusion by assigning env["UCX_TLS"] = "^ib,cuda_ipc" (or append ",cuda_ipc" to
the existing env["UCX_TLS"] if present) so both transports are excluded; modify
the code around the existing env assignment and the has_nvlink() branch
(referencing env and has_nvlink()) to perform this combined exclusion instead of
an unconditional overwrite.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: cfdbdea3-8a31-4614-a76c-151681570382
📒 Files selected for processing (2)
tests/integration/defs/accuracy/test_disaggregated_serving.pytests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
- tests/integration/test_lists/waives.txt
|
/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-1" |
|
PR_Github #41176 [ run ] triggered by Bot. Commit: |
|
PR_Github #41178 [ run ] triggered by Bot. Commit: |
|
PR_Github #41176 [ run ] completed with state |
|
/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-1" |
|
PR_Github #41180 [ run ] triggered by Bot. Commit: |
|
PR_Github #41178 [ run ] completed with state |
|
PR_Github #41180 [ run ] completed with state |
|
/bot run |
|
PR_Github #41223 [ run ] triggered by Bot. Commit: |
|
PR_Github #41223 [ run ] completed with state
|
|
/bot run |
|
PR_Github #41249 [ run ] triggered by Bot. Commit: |
|
PR_Github #41249 [ run ] completed with state
|
|
/bot run |
|
PR_Github #41273 [ run ] triggered by Bot. Commit: |
|
PR_Github #41273 [ run ] completed with state
|
…rch-1 Signed-off-by: Bo Deng <deemod@nvidia.com>
Signed-off-by: Bo Deng <deemod@nvidia.com>
|
PR_Github #41427 [ run ] completed with state
|
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #41538 [ run ] triggered by Bot. Commit: |
|
PR_Github #41538 [ run ] completed with state
|
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #41604 [ run ] triggered by Bot. Commit: |
|
PR_Github #41604 [ run ] completed with state
|
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #41632 [ run ] triggered by Bot. Commit: |
|
PR_Github #41632 [ run ] completed with state
|
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #41652 [ run ] triggered by Bot. Commit: |
|
/bot run --add-multi-gpu-test --disable-fail-fast |
1 similar comment
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #41698 [ run ] triggered by Bot. Commit: |
|
PR_Github #41698 [ run ] completed with state
|
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #41792 [ run ] triggered by Bot. Commit: |
|
PR_Github #41792 [ run ] completed with state
|
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #41831 [ run ] triggered by Bot. Commit: |
|
PR_Github #41831 [ run ] completed with state
|
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #41856 [ run ] triggered by Bot. Commit: |
|
PR_Github #41856 [ run ] completed with state |
NVIDIA#12656) Signed-off-by: Bo Deng <deemod@nvidia.com>
NVIDIA#12656) Signed-off-by: Bo Deng <deemod@nvidia.com>
5 participants
…rch-1
Summary by CodeRabbit
Release Notes
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.