[Bug]: Multi-node regression: segfault/hang in NCCL fabric detection on dual DGX Spark (1.3.0rc10 vs 1.2.0rc6)

[yam-face]

System Info

System Info:

2x DGX Spark (GB10, SM121, 128GB unified memory)
Connected via QSFP 200GbE direct cable (RoCE)
Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10
Working container: nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc6
CUDA 13.0, aarch64
Inter-node TP/EP over RoCE

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Happens on multiple models but

1. follows the trtllm inference guide for two sparks
2. Serve the Qwen3-235B-A22-FP4
3. Follow playbook exactly - only change trtllm version from 1.2.0rc6 to 1.3.0rc10

Expected behavior

Expected behavior:
When MNNVL fabric is unavailable (finalMNNVL=0), TRT-LLM should fall through to standard NCCL IB transport as it did in 1.2.0rc6, not hang or segfault.

actual behavior

Description:
Regression from 1.2.0rc6 to 1.3.0rc10 on dual DGX Spark multinode inference. Two distinct failure modes observed at the same point — NCCL fabric/allreduce initialization during autotuner warmup.

Failure 1 — Segfault (before NCCL env workarounds):
Qwen3-235B with --tp_size 2 segfaults in ncclGinGdakiQueryLastError → AllreduceOp::run during autotuner warmup. This model runs successfully on 1.2.0rc6 with the same configuration.
!!!!!!! Segfault encountered !!!!!!!
File "transport/net_ib/gdaki/gin_host_gdaki.cc", line 991, in ncclGinGdakiQueryLastError(void*, bool*)
File "gin/gin_host.cc", line 289, in ncclGinQueryLastError(ncclGinState*, bool*)
File "/dvs/p4/build/sw/gpgpu/nccl/gitfusion/stable/src/init.cc", line 3047, in ncclCommGetAsyncError
File "/dvs/p4/build/sw/gpgpu/nccl/gitfusion/stable/src/init.cc", line 402, in ncclCommEnsureReady(ncclComm*)
File "/dvs/p4/build/sw/gpgpu/nccl/gitfusion/stable/src/init.cc", line 3073, in ncclCommCount
File "", line 0, in tensorrt_llm::_v1::torch_ext::(anonymous namespace)::AllreduceOp::run(...)
File "", line 0, in tensorrt_llm::_v1::torch_ext::allreduce_raw(...)
Failure 2 — Indefinite hang (with NCCL env workarounds):
After adding NCCL_GIN_ENABLE=0 the segfault is eliminated, but both Qwen3-235B (--tp_size 2) and Nemotron-3-Super-120B-A12B-NVFP4 (--tp_size 2 --ep_size 2) hang indefinitely at:
[TensorRT-LLM][INFO] Detecting local TP group for rank 0
[TensorRT-LLM][INFO] Found inter-node TP group for rank 0, checking MNNVL support via fabric info
[TensorRT-LLM][INFO] Inter-node topology: localNVLink=1, localFabricValid=0, allRanksSameFabric=0, finalMNNVL=0

The segfualt occurs after MNNVL correctly resolves to finalMNNVL=0. It never proceeds to the standard NCCL IB fallback path.
Environment variables attempted — none resolve the hang:
NCCL_MNNVL_ENABLE=0
NCCL_GIN_ENABLE=0
NCCL_IB_MERGE_NICS=0
NCCL_SYMMETRIC_ENABLE=0
TRTLLM_ALLREDUCE_STRATEGY=NCCL

Qwen3-235B-A22B with --tp_size 2 — works on 1.2.0rc6, segfaults/hangs on 1.3.0rc10

Is reproducible across other models - not just Qwen3

additional notes

Probable regression source:
PR #11174 ("Fallback to NCCL instead of NCCL symmetric") was merged into 1.2.0rc6.post3 and fixed a similar NCCL fallback issue for non-MNNVL topologies. The 1.3.0 branch appears to have reintroduced or bypassed this fallback path with the new GIN/GDAKI transport stack and MNNVL fabric probing logic.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[karljang]

@Tabrizian, could you please review this issue and assign it? Thank you!

[justcaca2-alt]

I can reproduce the exact same issue with the following environment and model, please help to check:

1. Core Environment
   TensorRT-LLM Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10
   Model: NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
   Hardware: Dual DGX Spark (2x DGX Spark nodes)
   Deployment Guide: Followed official steps: https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/trt-llm#step-10-serve-the-model (Run on two Sparks)

[Tabrizian]

@nv-lschneider By any chance have you seen this issue before?

[nv-lschneider]

I have not seen this issue before, and don't have access to a dual DGX spark to investigate. But I am doing inquiries about dual DGX spark support.
As a work around, we might want to disable NCCL_SYMMETRIC entirely for dual DGX spark (not just the fallback).

[nv-lschneider]

I created a WAR PR to disable NCCL_SYMMETRIC auto selection on DGX spark platforms until the problem can be resolved from within NCCL.

I do not have access to a dual dgx spark, so I cannot test this.

[nv-lschneider]

I got access to a dual spark setup.
The issue appears to be TRT-LLM specific, not general.
I could not confirm the issue with nccl benchmarking tools on identical version.

Next step is to verify WAR #12902 on that setup (if you don't get a chance earlier.)
And investigate exact failure mode to confirm it will be resolved in future NCCL versions.

[Tabrizian]

@yam-face Is it possible to confirm if @nv-lschneider's PR fixes the issue you were observing?

[yam-face]

I applied the fixes to 1.3.0rc10 and while it fixed the seg fault, the process would still hang right after checking the fabric. However, I noticed that the "connect 2 sparks" playbook was updated to use release 1.3.0rc5 so I tested that one as well without the fixes. 1.3.0rc5 works just fine and correctly falls through to NCCL IB.

[nv-lschneider]

Thank you for trying.
It is surprising that the fixes did not work.
There seems to be another issue as well. Do you happen to have logs from the still hanging runs?

Thank you.

[nv-lschneider]

I found a fix, and verified that this is also fixed in NCCL 2.30.4.
Please take a look at the PR, I applied the changes and got a working repro on dual-spark.