NCCL v2.31.2-1 Release

Compute Fabric Transport (CFT)

• Adds CFT host and Device APIs for registering window memory with CUDA logical endpoints and issuing device-side Put, Get, and NVLS operations.
• Supports CFT on Blackwell GPUs with CUDA Toolkit 13.3 or later.

Per-Collective Configuration and Tuning

• Adds the new ncclCollConfig_t type and nccl*Config APIs for all collectives, including a vendor-defined field that makes custom-fork integration easier.
• Supports per-collective algorithm selection, CTA/CGA size overrides, and CTA policy overrides.
• Uses collConfig to add userTag to the profiler API, motivated by GitHub Issue #1916.

GIN Enhancements

• Adds the EFA GDA backend for GIN Put, Signal, and Flush operations, contributed by the AWS EFA team (GitHub PR #2273).
• Adds per-DevComm GIN backend selection, allowing users to create multiple DevComms with different backends.
• Supports connecting GIN with custom strides.
• Adds device-side timeouts to blocking GIN APIs such as Flush, Wait, WaitSignal, and WaitCounter operations.
• Reduces QP usage with railed GIN.
• Reduces file-descriptor consumption in GDA-KI when using many QPs.
• Optimizes GDA-KI error reporting with event-based CQ error reporting.
• Adds out-of-order delivery (DDP) support in GDA-KI on SPCX, improving performance without affecting GIN correctness guarantees.
• Adds GRH support and automatic path-MTU discovery to the GIN GDA-KI backend.

Parallel Aggregated Tree (PAT) Enhancements

• Enhances the PAT algorithm for ReduceScatter and AllGather with hierarchical kernels that use NVLS within a node and PAT across nodes.
• Improves small- and medium-message performance.
• Keeps the feature disabled by default. Set NCCL_ALGO=PAT or use the collConfig API to enable it.

One-Sided RMA and Copy Engine Collectives

• Adds multiple contexts and signals to one-sided RMA operations, allowing traffic from a single rank to use multiple NICs.
• Uses multiple contexts and signals for hierarchical 0-SM AllGather and AllToAll.
• Optimizes small-message latency by using NVLink multicast for AllGather.

Tuning and Cost Model

• Introduces a unified cost-model interface for querying cost estimates for both legacy and Device API-based kernels.
• Enables TMA kernels by default on Blackwell and integrates them into the cost model for symmetric registered memory.

Diagnostics and Profiling

• Adds NCCL RAS Diagnostics checks through NCCL_RUN_RAS_DIAGNOSTICS=1 or the RAS client, including GPU inventory, CUDA driver versions, ECC errors, NVLink state, and NCCL_* environment consistency checks.
• Adds NCCL Diagnostics through NCCL_RUN_DIAGNOSTICS=1, including an active P2P connectivity check and actionable P2P remediation guidance.
• Moves kernel-channel profiling to a dedicated per-communicator thread, extending kernel timing to proxy-less NVLink/SHM and graph-captured collectives.
• Updates Profiler v7 to expose per-kernel initial_sync, compute, and final_sync phase events and symmetric-kernel variant metadata.
• Adds per-QP CPU WQE post-to-poll latency monitoring for the IB transport.

Other Improvements

• Adds backward compatibility for applications that JIT-compile NCCL Device API code.
• Adds LTO IR support for the NCCL Device API.
• Reduces communicator host-memory use and topology initialization time on large systems by allocating topology paths according to their actual lengths.
• Adds multiple GIN proxy progress threads with per-thread endpoint assignment through GIN_PROXY_NTHREADS (GitHub PR #2279).
• Improves GIN host-proxy throughput by processing multiple GIN operations per progress iteration (GitHub PR #2232) and adding RMA plugin hints for request aggregation (GitHub PR #2254).
• Adds CuTeDSL bindings for GIN Get, Flush, Signal, ReadSignal, and PutValue operations (GitHub PR #2266).
• Adds an internal CUDA 13.3 DMA-BUF mmap backend for GDRCopy.
• Adds event-based load balancing for the net_ib transport.
• Refreshes local GIDs after IBV_EVENT_GID_CHANGE events during IB port recovery.
• Updates the RMA plugin interface to v15 (GitHub PR #2254).
• Improves AMD EPYC topology modeling (GitHub PR #2036).
• Improves algorithm selection on newer Intel CPUs.
• Makes nccl_device.h compatible with C99 host translation units.

Bug Fixes

• Fixes a hang when PXN connection initialization races with communicator teardown.
• Fixes data corruption when multiple symmetric windows are carved from the same backing memory allocation (GitHub Issue #2198).
• Fixes a hang during the first Copy Engine collective on a two-rank communicator (GitHub Issue #2241).
• Fixes NIC/GPU assignment performance regressions by limiting consistent start-NIC selection to Blackwell systems with ConnectX-8.
• Fixes missing GIN signal and counter requirements when Device API communicators are created asynchronously (GitHub PR #2208).
• Fixes a memory leak when Device API communicator creation fails (GitHub PR #2225).
• Fixes an incorrect GIN context count reported to applications when the allocated context count is rounded up (GitHub PR #2301).
• Fixes a resource leak when GIN connection setup fails (GitHub PR #2206).
• Fixes the CUDA thread capture mode remaining changed when GIN setup fails (GitHub PR #2229).
• Fixes inconsistent NVLS enablement when multiple ranks share a GPU (GitHub PR #2257).
• Fixes valid communicator configurations being rejected when only one of minCTAs or maxCTAs is set (GitHub PR #2256).
• Fixes communicator initialization failures on multi-system and MNNVL topologies when NCCL_P2P_PXN_LEVEL=1 (GitHub PR #2258).
• Fixes a Device API loadConst performance regression caused by __ldg.
• Fixes CuTeDSL GIN barrier initialization failures caused by under-aligned by-value arguments (GitHub PR #2243, GitHub Issue #2242).

Contrib Updates

• Adds PACE, a parallelism-aware collective engine that fuses layout conversion, data-type conversion, and scatter/gather operations into collective kernels (GitHub PR #2319).
• Adds experimental nccl4rust host and Device API bindings using LTO IR.
• Adds NIIN, a header-only NVSHMEM-compatible interface implemented with NCCL Device API primitives.
• Adds community-contributed architecture learning guides for communicator initialization, topology, tuning, transports, and collective execution (GitHub PR #2081).
• Adds a community-contributed analysis of DeepEPv2.

Acknowledgements

We thank the following contributors for their work on this release:

@akkart-aws, @alpha-baby, @anshumang, @baymaxhuang, @bhasunit, @dboyan, @EylonKrause, @Gaojiaqi, @hershys-aws, @mozarhua, @rauteric, @tianhao909, @voipmonitor, @wanglei875, @Xuan-1998, @yongxiangren for your PRs.

We thank the AWS EFA team for its GIN CPU Proxy optimizations and for adding the EFA GDA backend to GIN.

We also thank the community for issue reports, testing, and feedback.

Known Issues

• PAT with NVLS on H100: PAT with NVLS has performance regressions on H100 platforms. This will be fixed in the next release.
• B40 symmetric TMA kernels: On B40 GPUs, TMA-based symmetric kernels can exceed the available shared memory and fail with an illegal memory access. Set NCCL_SYM_TMA_ENABLE=0 to disable these kernels.
• B100 PCIe MLoPart: Multi-rank-per-GPU workloads using MPS MLoPart on B100 PCIe systems can fail during memory allocation with CUDA error 101 (invalid device ordinal). Set NCCL_CUMEM_ENABLE=0 as a workaround.