NCCL v2.32.6-1 Release

Vera Rubin Performance

• Adds a pipeline-based cost model with Vera Rubin tuning data for Ring, Tree, NVLS, and NVLSTree.
• Adds Vera Rubin models for symmetric AllGather, ReduceScatter, and AllReduce kernels.
• Speeds up the topology graph search on large NVLink domains such as 32 to 40 GPU VR200 systems, reducing communicator initialization time.
• Tuner plugin interface v7 adds Rubin platform support.

Device API Enhancements

• Adds an experimental Flow device APIs for building pipelined collective kernels over LSA and GIN connections with the Simple, LL, and LL128 protocols. The Flow API signatures are not stable and may change.
• Extends ncclGetPeerPointer and the resource-buffer peer pointer to address every load/store-reachable local peer by world rank, and adds ncclDevCommCanGetPeerPointer to report whether a peer is reachable.
• Adds the NCCL_GIN_CONNECTION_CUSTOM_ARRAY GIN connection type so applications can request connectivity to an explicit list of peers.
• Adds ncclLsaCopyTma, a TMA-based variant of the ncclLsaCopy broadcast on Blackwell and later GPUs.
• Adds a signal-efficient two-signal GIN barrier that GIN plugins can request through the v15 barrierOptions property.
• CFT teams now require and use CUDA logical-endpoint clique information to determine unicast and multicast team sizes.
• Validates GPU and NIC capabilities when establishing GDAKI connections and falls back to the proxy backend when GDAKI is unsupported and NCCL_GIN_TYPE is unset.
• Updates the GPI device backend for the latest libgpi device ABI, including 31-bit signal and counter IDs and extended transfer-size encoding, and updates the bundled GPUNetIO code to v5.0.0-rc3.
• Moves the IR-only C device ABI (ncclGin_C, ncclCoopAny) out of the public device headers into bindings/ir.

Collectives and Runtime Enhancements

• Adds experimental collective kernels called General Kernels (or GENK), built on top of the new Flow APIs. These kernels are compiled out by default to minimize binary size; build with SYM_GENK=1 and set NCCL_SYM_GENK_ENABLE=1 to try them.
• Adds experimental AlltoAll_LL, a latency-oriented symmetric AlltoAll kernel for small and medium per-peer messages on LSA and mixed LSA/GIN communicators, selectable with NCCL_SYM_KERNEL=AlltoAll_LL.
• Supports host RMA on rail-only configurations such as NCCL_CROSS_NIC=0, where host RMA was previously disabled.
• Updates the GIN and RMA plugin interfaces to v15 and v17: connection setup receives an abort flag so ncclCommAbort can interrupt blocking GIN and RMA initialization, and the plugin rankStride is replaced by an explicit peerArray.
• Improves hierarchical CE collective bandwidth when inter-node traffic is distributed across multiple RMA contexts, including under CUDA graph capture.
• Reduces host munmap latency on ATS platforms such as GB200 by granting ncclMemAlloc memory access to the owning GPU only.

Diagnostics and Profiling

• Adds profiler plugin interface v8 with opt-in NCCL-sourced GPU and CPU timestamps on every event.
• Supports loading up to two profiler plugins at a time through a comma-separated NCCL_PROFILER_PLUGIN, each with its own event mask.
• Allows NetPlugin profiler events to be collected independently of proxy events; profiler plugins must accept ProxyStep or NULL parents.
• Updates Inspector to record per-call profiler tags (requiring profiler v7 or later), emit the selected algorithm and protocol per collective in its JSON output, and ship a communication-pattern extractor. NCCL_INSPECTOR_NCCL_SOURCED_TIMINGS defaults to on.
• Adds a RAS diagnostics check verifying that the network devices NCCL selected are consistent across the nodes of a communicator and within each node.
• Adds the DIAG debug subsystem (NCCL_DEBUG_SUBSYS=DIAG) for active and RAS diagnostics details.
• Attaches all GIN/GDAKI queue pairs to a dedicated hardware Q counter set so aggregate counters can be queried for GIN traffic; controlled by NCCL_GIN_GDAKI_Q_COUNTER (default on).

Other Improvements

• Enables building NCCL for Windows on Arm64.
• Reduces library size by compressing device code with --compress-mode=size when building with CUDA 13 or newer.
• Adds channel-major and step-major NVLS staging FIFO layouts that reduce NVLink uTLB pressure for large-message NVLS and NVLSTree on Rubin. NCCL_NVLS_FIFO_LAYOUT controls the selection.
• Builds libnccl_device.bc and libnccl_device.ltoir with Clang/LLVM 23, and unifies the IR build options as IR_ARCH and NCCL_EMIT_LLVM_IR.
• Relaxes ncclCommGrow so existing ranks may pass either the original ncclUniqueId or NULL.
• Adds a thread-exclusive Put path to the EFA-GDA GIN backend and prints endpoint counts when a flush times out. (Github PR #2478) (Github PR #2552)
• Invokes the build-time Python generators through python3, and resolves Git version metadata from the NCCL source tree for out-of-tree CMake builds. (Github PR #2452) (Github PR #2451)
• Adds a user-guide page on the determinism of NCCL reductions, and improve various other documentation.

Bug Fixes

• Fixes a PXN deadlock caused by an unnecessary blocking cudaHostRegister of legacy shared memory in the proxy process.
• Fixes single-node NVLS AllGather and ReduceScatter returning data in the wrong rank order when CUDA_VISIBLE_DEVICES permutes GPUs. (Github PR #2377) (Github Issue #1906)
• Fixes a silent hang with network-registered buffers where the receiving proxy erased a rendezvous flag already published by the GPU. (Github PR #2390) (Github Issue #2389)
• Fixes a NET/IB hang when the request counter wraps at UINT32_MAX and completions are matched to the wrong receive request.
• Fixes races in the NET/Socket transport, including a task-slot publication race that caused NULL data sends on aarch64, and a missing load ordering on the NET/IB CTS FIFO. (Github PR #2365) (Github PR #2393) (Github Issue #1983)
• Fixes IB async-event threads handling events against the wrong device when NCCL_IB_DEVICE_PCI_ORDER reorders devices.
• Fixes communicator initialization failures on hosts with multi-port NIC when NCCL loads multiple network plugins.
• Skips invalid measurements in the example tuner's configuration optimizer. (Github PR #2453)
• Fixes CX9 NIC detection that misclassified BlueField-4 NICs as CX9 and missed CX9 virtual functions, preventing automatic rail and plane assignment.
• Fixes communicator initialization failures when combined per-rank topology data exceeds the topology XML fusion capacity. (Github PR #2386) (Github Issue #2160)
• Fixes ncclAllReduce with ncclAvg on symmetric-memory buffers failing with an unhandled CUDA error for floating-point types. (Github PR #2448) (Github Issue #2447)
• Fixes a non-blocking communicator becoming permanently unusable after one call returned ncclInvalidArgument. (Github PR #2460) (Github Issue #2459)
• Fixes ncclCommShrink putting the parent communicator into an error state when the excluded rank list is invalid; it now returns ncclInvalidArgument. (Github Issue #2457)
• Fixes a crash in ncclCommInitRank when NCCL_ALGO, NCCL_PROTO, or NCCL_SYM_KERNEL contain empty entries. (Github PR #2456) (Github Issue #2455)
• Fixes per-call maxCTAs bounds not being honored by kernel-based AlltoAll, Gather, and Scatter, and P2P channel counts exceeding maxCTAs after rounding to a power of two. (Github PR #2435) (Github Issue #2421) (Github Issue #1598)
• Fixes ncclCommRegister returning success without a handle for CPU-backed buffers; it now returns a NULL handle and rejects a NULL handle pointer. (Github PR #2433)
• Fixes ranks on distinct MIG compute instances of the same GPU being rejected as duplicate GPU ranks. (Github Issue #1716)
• Fixes failures when cuMem host memory support is inconsistent across ranks. The enablement logic is no longer affected by CUDA forward compatibility.
• Fixes GDRCopy initialization failures on GPUs that support DMA-BUF export but not CPU mmap of DMA-BUF allocations.
• Fixes intermittent encrypted bootstrap failures caused by unread TLS session tickets.
• Fixes a hang or illegal memory access when a child communicator split from a splitShare=1 parent reuses NVLS buffers in a different rank order.
• Fixes GIN and GIN-based symmetric kernels sending puts and signals to the wrong peer on communicators created with splitShare=1.
• Fixes CFT counted writes on the multicast path silently skipping memory binding, which could lead to hangs.
• Fixes a compiler error when calling put, putValue, or signal on a single-backend ncclGin_BackendOne or non-default ncclGin_BackendMask object. (Github PR #2436)
• Fixes out-of-place AllGather performance on symmetric memory when the per-rank size is not a multiple of 256 bytes.
• Fixes a TRACE-only crash during tuning initialization for symmetric cost-model entries. (Github Issue #2380)
• Fixes profiler kernel-channel completion timestamps being recorded before all warp groups finished. (Github PR #2473)
• Fixes a use-after-free in RAS peer updates and a null dereference in RAS client collective handling. (Github PR #2406)
• Fixes a command-injection vulnerability in the ib_write_bw diagnostics check by sanitizing host and device names.
• Fixes nccl4py cute GIN kernel compilation failures on CUDA 12, and CUDA compilation detection in headers when compiling with Clang. (Github PR #2392) (Github Issue #1738)
• Fixes memory, file-descriptor, and resource leaks on various error paths. (Github PR #2267) (Github PR #2364) (Github PR #2373) (Github PR #2376) (Github PR #2403) (Github PR #2404) (Github PR #2432) (Github Issue #2189)

Contrib/ Updates

• Moves nccl_ep and nccl_m2n out of the NCCL repository; they are now developed at https://github.com/NVIDIA/nccl-extensions.
• Adds contrib/tide, a community contribution for layout-aware FSDP2 collectives, built on top of NCCL device and host APIs.
• Updates nccl4rust for the NCCL 2.32 API and adds a minimal AllReduce example.

Acknowledgements

We thank the following contributors for their work on this release:

@akkart-aws, @anranxia, @anshumang, @enpimashin, @EylonKrause, @jakezwang, @jsolman, @kodlan, @LiRunGuo, @madeleineth, @NorthSecond, @rootkiller6788 for your PRs.

We also thank the community for issue reports, testing, and feedback.

Known Issues

• Compiling NCCL from source targeting PTX can fail due to CUDA device runtime library conflicts during linking. Directly targeting SASS does not have the same issue. This will be fixed in the next release.
• On sm_120, FP8 sum AllReduce using the symmetric AGxLL_R kernel may fail with an illegal memory access when NCCL is built with affected CUDA toolkits (reported with CUDA 12.8–13.0). This is a CUDA compiler issue, fixed in CUDA 13.1 and later.