Repository navigation
Releases
NCCL v2.32.6-1 Release
Vera Rubin Performance
- Adds a pipeline-based cost model with Vera Rubin tuning data for Ring, Tree, NVLS, and NVLSTree.
- Adds Vera Rubin models for symmetric AllGather, ReduceScatter, and AllReduce kernels.
- Speeds up the topology graph search on large NVLink domains such as 32 to 40 GPU VR200 systems, reducing communicator initialization time.
- Tuner plugin interface v7 adds Rubin platform support.
Device API Enhancements
- Adds an experimental Flow device APIs for building pipelined collective kernels over LSA and GIN connections with the Simple, LL, and LL128 protocols. The Flow API signatures are not stable and may change.
- Extends
ncclGetPeerPointerand the resource-buffer peer pointer to address every load/store-reachable local peer by world rank, and addsncclDevCommCanGetPeerPointerto report whether a peer is reachable. - Adds the
NCCL_GIN_CONNECTION_CUSTOM_ARRAYGIN connection type so applications can request connectivity to an explicit list of peers. - Adds
ncclLsaCopyTma, a TMA-based variant of thencclLsaCopybroadcast on Blackwell and later GPUs. - Adds a signal-efficient two-signal GIN barrier that GIN plugins can request through the v15
barrierOptionsproperty. - CFT teams now require and use CUDA logical-endpoint clique information to determine unicast and multicast team sizes.
- Validates GPU and NIC capabilities when establishing GDAKI connections and falls back to the proxy backend when GDAKI is unsupported and
NCCL_GIN_TYPEis unset. - Updates the GPI device backend for the latest libgpi device ABI, including 31-bit signal and counter IDs and extended transfer-size encoding, and updates the bundled GPUNetIO code to v5.0.0-rc3.
- Moves the IR-only C device ABI (
ncclGin_C,ncclCoopAny) out of the public device headers intobindings/ir.
Collectives and Runtime Enhancements
- Adds experimental collective kernels called General Kernels (or GENK), built on top of the new Flow APIs. These kernels are compiled out by default to minimize binary size; build with
SYM_GENK=1and setNCCL_SYM_GENK_ENABLE=1to try them. - Adds experimental
AlltoAll_LL, a latency-oriented symmetric AlltoAll kernel for small and medium per-peer messages on LSA and mixed LSA/GIN communicators, selectable withNCCL_SYM_KERNEL=AlltoAll_LL. - Supports host RMA on rail-only configurations such as
NCCL_CROSS_NIC=0, where host RMA was previously disabled. - Updates the GIN and RMA plugin interfaces to v15 and v17: connection setup receives an abort flag so
ncclCommAbortcan interrupt blocking GIN and RMA initialization, and the pluginrankStrideis replaced by an explicitpeerArray. - Improves hierarchical CE collective bandwidth when inter-node traffic is distributed across multiple RMA contexts, including under CUDA graph capture.
- Reduces host
munmaplatency on ATS platforms such as GB200 by grantingncclMemAllocmemory access to the owning GPU only.
Diagnostics and Profiling
- Adds profiler plugin interface v8 with opt-in NCCL-sourced GPU and CPU timestamps on every event.
- Supports loading up to two profiler plugins at a time through a comma-separated
NCCL_PROFILER_PLUGIN, each with its own event mask. - Allows NetPlugin profiler events to be collected independently of proxy events; profiler plugins must accept
ProxySteporNULLparents. - Updates Inspector to record per-call profiler tags (requiring profiler v7 or later), emit the selected algorithm and protocol per collective in its JSON output, and ship a communication-pattern extractor.
NCCL_INSPECTOR_NCCL_SOURCED_TIMINGSdefaults to on. - Adds a RAS diagnostics check verifying that the network devices NCCL selected are consistent across the nodes of a communicator and within each node.
- Adds the
DIAGdebug subsystem (NCCL_DEBUG_SUBSYS=DIAG) for active and RAS diagnostics details. - Attaches all GIN/GDAKI queue pairs to a dedicated hardware Q counter set so aggregate counters can be queried for GIN traffic; controlled by
NCCL_GIN_GDAKI_Q_COUNTER(default on).
Other Improvements
- Enables building NCCL for Windows on Arm64.
- Reduces library size by compressing device code with
--compress-mode=sizewhen building with CUDA 13 or newer. - Adds channel-major and step-major NVLS staging FIFO layouts that reduce NVLink uTLB pressure for large-message NVLS and NVLSTree on Rubin.
NCCL_NVLS_FIFO_LAYOUTcontrols the selection. - Builds
libnccl_device.bcandlibnccl_device.ltoirwith Clang/LLVM 23, and unifies the IR build options asIR_ARCHandNCCL_EMIT_LLVM_IR. - Relaxes
ncclCommGrowso existing ranks may pass either the originalncclUniqueIdorNULL. - Adds a thread-exclusive Put path to the EFA-GDA GIN backend and prints endpoint counts when a flush times out. (Github PR #2478) (Github PR #2552)
- Invokes the build-time Python generators through
python3, and resolves Git version metadata from the NCCL source tree for out-of-tree CMake builds. (Github PR #2452) (Github PR #2451) - Adds a user-guide page on the determinism of NCCL reductions, and improve various other documentation.
Bug Fixes
- Fixes a PXN deadlock caused by an unnecessary blocking
cudaHostRegisterof legacy shared memory in the proxy process. - Fixes single-node NVLS AllGather and ReduceScatter returning data in the wrong rank order when
CUDA_VISIBLE_DEVICESpermutes GPUs. (Github PR #2377) (Github Issue #1906) - Fixes a silent hang with network-registered buffers where the receiving proxy erased a rendezvous flag already published by the GPU. (Github PR #2390) (Github Issue #2389)
- Fixes a NET/IB hang when the request counter wraps at
UINT32_MAXand completions are matched to the wrong receive request. - Fixes races in the NET/Socket transport, including a task-slot publication race that caused NULL data sends on aarch64, and a missing load ordering on the NET/IB CTS FIFO. (Github PR #2365) (Github PR #2393) (Github Issue #1983)
- Fixes IB async-event threads handling events against the wrong device when
NCCL_IB_DEVICE_PCI_ORDERreorders devices. - Fixes communicator initialization failures on hosts with multi-port NIC when NCCL loads multiple network plugins.
- Skips invalid measurements in the example tuner's configuration optimizer. (Github PR #2453)
- Fixes CX9 NIC detection that misclassified BlueField-4 NICs as CX9 and missed CX9 virtual functions, preventing automatic rail and plane assignment.
- Fixes communicator initialization failures when combined per-rank topology data exceeds the topology XML fusion capacity. (Github PR #2386) (Github Issue #2160)
- Fixes
ncclAllReducewithncclAvgon symmetric-memory buffers failing with an unhandled CUDA error for floating-point types. (Github PR #2448) (Github Issue #2447) - Fixes a non-blocking communicator becoming permanently unusable after one call returned
ncclInvalidArgument. (Github PR #2460) (Github Issue #2459) - Fixes
ncclCommShrinkputting the parent communicator into an error state when the excluded rank list is invalid; it now returnsncclInvalidArgument. (Github Issue #2457) - Fixes a crash in
ncclCommInitRankwhenNCCL_ALGO,NCCL_PROTO, orNCCL_SYM_KERNELcontain empty entries. (Github PR #2456) (Github Issue #2455) - Fixes per-call
maxCTAsbounds not being honored by kernel-based AlltoAll, Gather, and Scatter, and P2P channel counts exceedingmaxCTAsafter rounding to a power of two. (Github PR #2435) (Github Issue #2421) (Github Issue #1598) - Fixes
ncclCommRegisterreturning success without a handle for CPU-backed buffers; it now returns aNULLhandle and rejects aNULLhandle pointer. (Github PR #2433) - Fixes ranks on distinct MIG compute instances of the same GPU being rejected as duplicate GPU ranks. (Github Issue #1716)
- Fixes failures when cuMem host memory support is inconsistent across ranks. The enablement logic is no longer affected by CUDA forward compatibility.
- Fixes GDRCopy initialization failures on GPUs that support DMA-BUF export but not CPU mmap of DMA-BUF allocations.
- Fixes intermittent encrypted bootstrap failures caused by unread TLS session tickets.
- Fixes a hang or illegal memory access when a child communicator split from a
splitShare=1parent reuses NVLS buffers in a different rank order. - Fixes GIN and GIN-based symmetric kernels sending puts and signals to the wrong peer on communicators created with
splitShare=1. - Fixes CFT counted writes on the multicast path silently skipping memory binding, which could lead to hangs.
- Fixes a compiler error when calling
put,putValue, orsignalon a single-backendncclGin_BackendOneor non-defaultncclGin_BackendMaskobject. (Github PR #2436) - F...
NCCL v2.32.3-1 Release
Vera Rubin Support
- Adds initial Rubin platform support including support for sm107, CX9 rail and plane detection, and MPS+MLoPart.
- 2.32.3 focuses on new functionalities and does not contain performance model tuning for Rubin. This will be part of the next release to improve out-of-box experience for Rubin users.
Device API Enhancements
- Adds Compute Fabric Transport (CFT) counted-write and wait support.
- Adds socket-based GIN support, enabling custom kernel development with TCP sockets.
- Adds support in GDAKI for LAG-aware QP assignment based on the context ID. (Github PR #2315)
- Adds NCCL_WIN_GIN_ONLY so users can register a window only for GIN usage.
- Optimize GIN performance by skipping mcst operation when possible.
Collectives and Runtime Enhancements
- Adds a ring-based hierarchical copy-engine AllGather implementation selectable with NCCL_HIER_CE_COLL_AG_RAIL_RING_ENABLE. (Github PR #2299)
- Improves Blackwell symmetric AllGather performance, resource overhead modeling and kernel selection with a new cost model.
- Adds optional TLS encryption for NCCL-owned socket traffic when NCCL is built with OpenSSL3, configured through the ncclSetEncryption API.
- Adds ncclCollConfig_t::launchCompletionEvent, allowing callers to observe kernel-launch completion.
- Adds ncclNvlsHostMode_t to ncclConfig_t so applications can disable host NVLS collectives per communicator.
- Optimizes NVLS slot consumption to avoid resource exhaustion issues when using multiple communicators with NVLS.
Diagnostics and Profiling
- Adds the ATTN log level for important non-fatal conditions, such as configuration fallbacks and plugin initialization failures.
- Expands RAS capability with GPU-resident progress counters and a watchdog DMA mirror for diagnosing stalled collective kernels.
- Adds additional RAS diagnostics functionalities for NVLink/NIC state and speed, PCI and GDR configuration, Xid/SXid events and others.
- Reports degraded NVLink fabric bandwidth with an ATTN message during communicator initialization.
- Reports mismatched NCCL Git revisions across communicator ranks during initialization.
Other Improvements
- Fixes PAT+NVLS performance drops on H100 platforms.
- Avoid address space exhaustion when repeatedly registering symmetric windows backed by a single physical memory allocation.
- Reduces communicator initialization overhead at large scale by avoiding scans of inactive proxy poll descriptors.
- Adds an experimental built-in NetworkDirect transport on Windows, with automatic socket fallback.
- EFA team improved EFA GDA support with gin.get API and others. (Github PR #2382)(Github PR #2398)(Github PR #2405)
- Reports Inspector ring-buffer drops and operation counts in JSON. (Github PR #2304)
- NCCL now supports communicators using multiple MIG instances.
- Generates llms.txt for agents to better read NCCL documentation.
Bug Fixes
- Fixes profiler overhead when no profiler plugin is loaded. (Github Issue #2355)
- Fixes P2P IPC registration reuse producing out-of-bounds remote addresses when a registered allocation spans multiple cuMem segments. (Github PR #2362)
- Fixes PAT connection-setup deadlocks when runtime connection is disabled and nodes have uneven local-rank counts. (Github Issue #2385)
- Fixes non-thread-safe token parsing that could corrupt concurrent configuration parsing. (Github Issue #2361)
- Fixes B40 TMA symmetric kernels crash due to insufficient shared memory.
- Fix GIN GDAKI bug related to hop limit when using DOCA SDK.
- Fixes GIN Proxy and GPI descriptor shared-memory sizing and alignment.
- Fixed CFT window registration performed before ncclDevCommCreate.
- Fixes profiler API events reporting rank 0 instead of the originating communicator rank. (Github Issue #2300)
- Fixes tuner plugins receiving uninitialized cost-model constants after the tuning rework.
Contrib/ Updates
- Adds more functionality in NiiN.
- Fixes contrib/nccl_checkpoint rejecting otherwise compatible NCCL versions. (Github Issue #2347)
Acknowledgements
We thank the following contributors for their work on this release:
@madeleineth, @alpha-baby, @akkart-aws, @anshumang, @cesar-stuardo-bd, @hexagonal-banana,
@rauteric, @shaq918, @yshalabi, @zrss for your contributions.
We also thank the community for issue reports, testing, and feedback.
Known Issues
- Rubin performance model has not been optimized with 2.32.3. Users can observe better performance on Rubin by increasing the number of CTAs NCCL uses. Automatic tuning will be improved in the next release.
- Socket GIN currently requires users to opt-in to enable GDRCopy.
- B100 PCIe MLoPart workloads can fail during memory allocation with CUDA error 101 (invalid device ordinal). Set NCCL_CUMEM_ENABLE=0 as a workaround.
NCCL4Py v0.6.0 Release
Highlights
- Added experimental CuTe DSL ReduceCopy APIs for LSA, multimem, and local memory operations.
- Added NCCL 2.32 host APIs for collective launch completion events, NVLS configuration, CFT capability inspection, and window registration.
- Added explicit CuTe DSL barrier-session teardown and corrected
ThreadScope.THREADto match libcu++.
New Features
Collective Launch Completion Events
-
Added per-collective CUDA launch completion events through
NCCLCollConfig.APIs:
NCCLCollConfig.launch_completion_eventNcclEventSpec
NCCL 2.32 Configuration and Capability Inspection
-
Added host-side NVLS configuration, CFT capability reporting, and specialized window registration flags.
APIs:
NCCLConfig.nvls_host_modeNcclNvlsHostModeNCCLCommProperties.cft_supportNCCLCommProperties.cft_multicast_supportNCCLCommProperties.cft_counted_supportWindowFlag.GIN_ONLYWindowFlag.CFT_COUNTED
CuTe DSL ReduceCopy
-
Added device APIs for reduction and copy operations across LSA windows, multimem pointers, and local tensors.
APIs:
lsa_reduce_sum(),multimem_reduce_sum()lsa_copy(),multimem_copy()lsa_reduce_sum_copy(),multimem_reduce_sum_copy()local_reduce_sum_copy()
CuTe DSL Barrier Lifecycle
-
Added explicit session destruction so barrier handles and indexes can be safely reused.
APIs:
LsaBarrierSession.destroy()GinBarrierSession.destroy()BarrierSession.destroy()
Examples and Documentation
- Added
08_reduce_copy.py, demonstrating an LSA reduction across registered windows.
Breaking Changes
ThreadScope.THREADnow has the libcu++ numeric value10instead of3. Code using the enum member requires no changes; code storing or passing its raw integer value must be updated.- CuTe DSL barrier sessions must call
destroy()exactly once after their final operation. Every thread in the session’s cooperative group must call it from uniform control flow.
Fixes and Enhancements
- The NCCL 2.31.2 device IR issue reported in nccl4py v0.5.0, which could prevent
Gin.put()from JIT-compiling with EFA GDA enabled, is resolved in the matching NCCL 2.32.3 IR. - Added a
py.typedmarker so compatible static type checkers can usenccl.coreannotations.
Compatibility Notes
- The nccl4py 0.6.0 host-side bindings and CuTe DSL API are generated from NCCL 2.32.3 headers.
- When using the CuTe DSL API, use matching NCCL 2.32.3
libnccl.soandlibnccl_device.bc; nccl4py does not verify this automatically. NCCLCollConfig.launch_completion_eventrequires a timing-disabled, non-interprocess CUDA event. Every rank must either provide an event or omit it. CUDA 12.3 or later is required for post-launch recording semantics.- The CuTe DSL device API remains experimental.
NCCL4Py v0.5.0 Release
Highlights
- Added host and CuTe DSL APIs for CFT setup and logical-endpoint addressing.
- Added per-collective configuration for algorithm selection, CTA tuning, profiler tags, and vendor options.
- Added communicator capability inspection and support for new NCCL 2.31.2 GIN features.
- Added compile-only CuTe DSL arguments for compiling kernels before creating NCCL resources.
New Features
Per-Collective Configuration
-
Added per-call configuration to all collective operations.
APIs:
NCCLCollConfigVendorOption- The
configargument onCommunicator.allreduce(),broadcast(),reduce(),allgather(),reduce_scatter(),alltoall(),gather(), andscatter()
Compute Fabric Transport Setup
-
Added APIs for configuring CFT resources, querying CFT teams, and resolving logical endpoint addresses. CFT communication and barrier operations are not yet exposed.
Host APIs:
NcclHostCftMode,NcclCftTeamMode,NcclCftCapNCCLConfig.host_cft_modeNCCLDevCommRequirements.cft_caps,NCCLDevCommRequirements.cft_barrier_countCommunicator.team_cft(),Communicator.team_cft_multimemCftLeInfoRegisteredWindowHandle.get_cft_le_info()RegisteredWindowHandle.get_peer_le_info()RegisteredWindowHandle.get_multimem_le_info()
CuTe DSL APIs:
CftTeamMode,CftLeInfoDevComm.team_cft(),DevComm.team_cft_multimemWindow.cft_le_info(),Window.peer_le_info(),Window.multimem_le_info()DevComm.resource_buffer_cft_le_info()DevComm.resource_buffer_peer_le_info()DevComm.resource_buffer_multimem_le_info()
Communicator and GIN Capabilities
-
Added consolidated communicator property inspection and support for new NCCL 2.31.2 configuration and GIN capabilities.
APIs:
NCCLCommProperties,Communicator.propertiesNCCLConfig.launch_order_implicitNCCLConfig.num_rma_sigNCCLConfig.rma_eager_initNCCLDevCommRequirements.gin_typeNCCLDevCommRequirements.gin_custom_strideNcclGinType.EFA_GDANcclGinConnectionType.CUSTOM_STRIDEGinBackendMask.EFA_GDA
Compile Before Creating NCCL Resources
-
Added type-only arguments for compiling CuTe DSL kernels before communicators, windows, or device resources are created.
APIs:
nccl.core.device.cute.runtime.make_fake_dev_comm()nccl.core.device.cute.runtime.make_fake_window()nccl.core.device.cute.runtime.make_fake_multimem_handle()nccl.core.device.cute.runtime.make_fake_lsa_barrier_handle()nccl.core.device.cute.runtime.make_fake_gin_barrier_handle()
Examples and Documentation
- Added a compile-before-resource-creation example and renamed the introductory CuTe DSL example to
00_basic.py. - See the complete CuTe DSL example suite.
Fixes and Enhancements
- Registered window handles created within
nccl.core.group()are now updated correctly after the group completes.
Compatibility Notes
- The nccl4py 0.5.0 host-side bindings and CuTe DSL API are generated from NCCL 2.31.2 headers.
- When using the CuTe DSL API, use the matching NCCL 2.31.2
libnccl.soandlibnccl_device.bc; nccl4py does not verify this automatically. - The CuTe DSL device API remains experimental.
Known Issues
- The NCCL 2.31.2 device IR may fail to JIT-compile
Gin.put()when EFA GDA is enabled. Apply this patch to NCCL 2.31.2 and rebuildlibnccl_device.bc.
NCCL v2.31.2-1 Release
Compute Fabric Transport (CFT)
- Adds CFT host and device APIs for registering window memory with CUDA logical endpoints and issuing device-side Put, Get, Red and NVLS operations.
- CFT is supported on Blackwell GPUs with CUDA toolkit 13.3 or later.
Per-Collective Configuration and Tuning
- Adds new
ncclCollConfig_tandnccl*ConfigAPIs for all collectives, with a vendor-defined field to make it easier for custom forks to integrate. - Supports per-collective algorithm selection, CTA/CGA size and CTA policy overrides.
- Leverages
collConfigto adduserTagin profiler API. Motivated by community RFE #1916.
GIN Enhancements
- AWS-EFA team contributed the EFA GDA backend for GIN Put, Signal, and Flush operations (Github PR #2273).
- Adds per-DevComm GIN backend selection, allowing users to create multiple DevComms with different backends.
- Supports connecting GIN with custom strides.
- Adds device-side timeouts to blocking GIN APIs such as Flush, Wait, WaitSignal, and WaitCounter operations.
- Reduces QP usage when using railed GIN.
- Reduces file-descriptor consumption in GDAKI when using many QPs.
- Optimizes error reporting in GDAKI via event-based CQ error reporting.
- Add support for out-of-order delivery (DDP) in GDAKI on SPCX, improving performance without impacting GIN correctness guarantees.
- Adds GRH support and automatic path-MTU discovery to the GIN GDAKI backend.
Parallel Aggregated Tree (PAT) Enhancements
- Enhances the PAT algorithm for ReduceScatter/AllGather by adding hierarchical kernels that use NVLS within the node and PAT across nodes.
- Enables better small/medium message performance.
- Disabled by default. Use
NCCL_ALGO=PAT, or thecollConfigAPI to enable.
One-Sided RMA and Copy Engine Collectives
- Adds support for multiple contexts and multiple signals to one-sided RMA operations, allowing one-sided RMA traffic from a single rank to use multiple NICs.
- Leverage multiple contexts and signals for hierarchical 0-SM AllGather and AllToAll.
- Optimizes small message latency by using NVLink multicast for AllGather.
Tuning and Cost Model
- Introduces a new unified cost model interface for querying the cost estimate for both legacy and device-API-based kernels.
- On Blackwell, TMA kernels are now enabled by default and integrated in the cost model for symmetric registered memory.
Diagnostics and Profiling
- Adds NCCL RAS Diagnostics checks through
NCCL_RUN_RAS_DIAGNOSTICS=1or RAS client, including GPU inventory, CUDA driver versions, ECC errors, NVLink state, andNCCL_*environment consistency checks. - Adds NCCL Diagnostics through
NCCL_RUN_DIAGNOSTICS=1, including active P2P connectivity check and actionable P2P remediation guidance. - Kerne-channel profiling moved to a dedicated per-communicator thread, extending kernel timing to proxy-less NVLink/SHM and graph-captured collectives.
- Profiler V7 exposes per-kernel
initial_sync,computeandfinal_syncphase events and symmetric-kernel variant metadata. - Adds per-QP CPU WQE post-to-poll latency monitoring for the IB transport.
Other Improvements
- Adds backward compatibility support for applications that JIT-compile NCCL Device API code.
- Adds support for LTO IR for NCCL device API.
- Reduces communicator host-memory use and topology initialization time on large systems by allocating topology paths according to their actual lengths.
- Adds multiple GIN proxy progress threads with per-thread endpoint assignment through
GIN_PROXY_NTHREADS(Github PR #2279). - Improves GIN host-proxy throughput by processing multiple GIN operations per progress iteration (Github PR #2232) and adding hints to the RMA plugin to aggregate requests (Github PR #2254).
- Adds CuTeDSL bindings for GIN Get, Flush, Signal, ReadSignal, and PutValue operations (Github PR #2266).
- Adds an internal CUDA 13.3 DMA-BUF mmap backend for GDRCopy.
- Adds event-based load balancing for the
net_ibtransport. - Refreshes local GIDs after
IBV_EVENT_GID_CHANGEevents during IB port recovery. - Updates the RMA plugin interface to v15 (Github PR #2254).
- Improves AMD EPYC topology modeling (Github PR #2036).
- Improves algorithm selection on newer Intel CPUs.
- Makes
nccl_device.hcompatible with C99 host translation units.
Bug Fixes
- Fixes a hang when PXN connection initialization races with communicator teardown.
- Fixes data corruption when multiple symmetric windows are carved from the same backing memory allocation (Github Issue #2198).
- Fixes a hang during the first Copy Engine collective on a two-rank communicator (Github Issue #2241).
- Fixes NIC/GPU assignment perf regressions by limiting consistent start-NIC selection to Blackwell systems with ConnectX-8.
- Fixes missing GIN signal and counter requirements when Device API communicators are created asynchronously (Github PR #2208).
- Fixes a memory leak when Device API communicator creation fails (Github PR #2225).
- Fixes an incorrect GIN context count reported to applications when the allocated context count is rounded up (Github PR #2301).
- Fixes a resource leak when GIN connection setup fails (Github PR #2206).
- Fixes the CUDA thread capture mode remaining changed when GIN setup fails (Github PR #2229).
- Fixes inconsistent NVLS enablement when multiple ranks share a GPU (Github PR #2257).
- Fixes valid communicator configurations being rejected when only one of
minCTAsormaxCTAsis set (Github PR #2256). - Fixes communicator initialization failures on multi-system and MNNVL topologies when
NCCL_P2P_PXN_LEVEL=1(Github PR #2258). - Fixes a Device API
loadConstperformance regression caused by__ldg. - Fixes CuTeDSL GIN barrier initialization failures caused by under-aligned by-value arguments (Github PR #2243, Issue #2242).
Contrib/ Updates
- Adds PACE, a parallelism-aware collective engine that fuses layout conversion, data-type conversion, and scatter/gather operations into collective kernels (Github PR #2319).
- Adds experimental
nccl4rusthost and Device API bindings using LTO IR. - Adds NIIN, a header-only NVSHMEM-compatible interface implemented with NCCL Device API primitives.
- Adds community-contributed architecture learning guides for communicator initialization, topology, tuning, transports, and collective execution (Github PR #2081).
- Adds a community-contributed analysis of DeepEPv2.
Acknowledgements
We thank the following contributors for their work on this release:
@akkart-aws, @alpha-baby, @anshumang, @baymaxhuang, @bhasunit, @dboyan, @EylonKrause, @Gaojiaqi, @hershys-aws, @mozarhua, @rauteric, @tianhao909, @voipmonitor, @wanglei875, @Xuan-1998, @yongxiangren for your PRs.
We thank the AWS EFA team for their optimizations to GIN CPU Proxy and for adding EFA GDA backend in GIN.
We also thank the community for issue reports, testing, and feedback.
Known Issues
- PAT + NVLS has performance regressions on H100 platforms. This will be fixed in the next release.
- B40 symmetric TMA kernels: On B40 GPUs, TMA-based symmetric kernels can exceed available shared memory and fail with an illegal memory access. Set
NCCL_SYM_TMA_ENABLE=0to disable these kernels. - B100 PCIe MLoPart: Multi-rank-per-GPU workloads using MPS MLoPart on B100 PCIe systems can fail during memory allocation with CUDA error 101 (
invalid device ordinal). SetNCCL_CUMEM_ENABLE=0as a workaround.
NCCL4Py v0.4.1 Release
Highlights
- Added host APIs for NCCL teams, rank translation, and device resource configuration.
- Expanded the experimental CuTe DSL device API with additional GIN operations, resource addressing, and direct support for host-created resources.
- Added live NCCL parameter access and improved version reporting.
New Features
Teams and Device Resources
-
Added APIs for inspecting communicator teams, translating ranks, and configuring resources for device-side communication.
APIs:
NCCLTeamCommunicator.team_world,Communicator.team_lsa,Communicator.team_railCommunicator.team_rank_to_world(),Communicator.team_rank_to_lsa()NCCLDevCommRequirements.teams,NCCLDevCommRequirements.resourcesNCCLDevCommRequirements.gin_traffic_classTeamRequirementLsaBarrierRequirement,GinBarrierRequirement,LLA2ARequirementMultimemHandle,LsaBarrierHandle,GinBarrierHandle,LLA2AHandleDevCommResource.multimem_handle(),DevCommResource.resource_handlesRegisteredWindowHandle.get_multimem_device_pointer()
CuTe DSL Device API
-
Added device-side rank translation, resource addressing, and additional GIN operations.
APIs:
DevComm.team_rank_to_world(),DevComm.team_rank_to_lsa()DevComm.resource_buffer_local_pointer()DevComm.resource_buffer_lsa_pointer()DevComm.resource_buffer_peer_pointer()DevComm.resource_buffer_multimem_pointer()DevComm.resource_buffer_lsa_multimem_pointer()Window.multimem_pointer(),Window.lsa_multimem_pointer()Gin.put_value(),Gin.get(),Gin.signal(),Gin.flush()Gin.read_signal(),Gin.reset_signal(),Gin.signal_shadow_pointer()Gin.read_counter(),Gin.wait_counter(),Gin.reset_counter()GIN_ALL_CONTEXTSThreadScope,GinResourceSharingModeDevComm.gin(..., resource_sharing_mode=...)opt_flagssupport forGin.put(),Gin.put_value(),Gin.get(), andGin.signal()
-
Host-created device communicators, registered windows, multimem handles, and LSA/GIN barrier handles can now be passed directly as arguments to
@cute.jitfunctions.
Runtime Inspection
-
Added live NCCL parameter access and consolidated version reporting under
nccl.core.APIs:
nccl.core.paramsnccl.core.dump_params()nccl.core.get_version()nccl.core.show_versions()
Examples and Documentation
- Added CuTe DSL examples covering teams, LSA all-reduce, multimem, GIN operations, barriers, and resource buffers.
Breaking Changes
nccl.epis no longer included in nccl4py. The NCCL EP Python bindings are now distributed separately from the NVIDIA/nccl-extensions repository.- Version helpers are no longer exported from the
ncclnamespace. Import them fromnccl.core. - Version reporting was consolidated:
nccl.core.get_version()now returnsVersionInfo.nccl.core.Version,get_lib_version(), andget_lib_path()were removed.VersionInfo.ncclwas renamed toVersionInfo.libnccl.VersionInfo.nccl_epwas removed.VersionInfo.nccl_bindingswas added.
Fixes and Enhancements
- Fixed the default release semantics of
Gin.put(). - Improved NCCL parameter error handling and string decoding.
Compatibility Notes
- The top-level
ncclPython package is now a PEP 420 namespace package. - The CuTe DSL device API remains experimental.
- The selected
libnccl_device.bcmust exactly match the NCCL version used by the bindings; nccl4py does not perform this check automatically. - When one thread manages multiple local ranks, call
Communicator.finalize()withinnccl.core.group().
Known Issues
-
The following CuTe DSL APIs require NCCL 2.31:
Gin.get()gin_session(),world_gin(),rail_gin(),hybrid_session(),world_hybrid()
-
CuTe DSL JIT compilation with CUDA 12 fails for kernels using the following APIs. CUDA 13 is not affected.
Affected APIs:
Gin.put(),Gin.put_value(),Gin.get(),Gin.signal(),Gin.flush()GinBarrierSession.sync(),BarrierSession.sync()
NCCL4py v0.3.1 Release
Highlights
- Added
nccl.ep, a Pythonic interface tolibnccl_ep.sofor expert
parallel dispatch/combine workflows. The package exposesGroup,Handle,
Tensor, typed config dataclasses,Algorithm,Layout,PassDir, and the
named input/output structs used by the NCCL EP API. - Added
nccl.core.device.cute, enabling CuTeDSL kernels to call NCCL device
APIs. - Added top-level stack diagnostics with
nccl.get_version()and
nccl.show_versions(), reportingnccl4py,libnccl.so, and
libnccl_ep.soversions, CUDA build variants, and loaded shared-library
paths. - Added free-threaded CPython support.
New Features
NCCL EP Python API
- New
nccl.eppackage provides Pythonic access to the NCCL EP extension
library. Group.create()creates EP groups from aCommunicatorandGroupConfig;
Group.create_handle()creates handles with an explicitLayout.Handlesupportsupdate(),dispatch(),combine(),complete(), and
destroy().DispatchInputs,DispatchOutputs,CombineInputs,CombineOutputs, and
LayoutInfoprovide named containers for the tensors and metadata used by
dispatch, combine, and handle setup.Tensorresolves Python buffers intoncclEpTensor_tdescriptors.GroupConfig,HandleConfig,DispatchConfig,CombineConfig, and
AllocConfigexpose typed configuration objects.AllocFnandFreeFnexpose caller-controlled EP allocation hooks.nccl.ep.interop.torch.get_nccl_comm_from_group()provides PyTorch interop
for creating an NCCL communicator from a PyTorch process group's rank and
world-size information.- Importing
nccl.epsets defaultNCCL_EP_HOMEwhen bundled EP JIT headers
are present, andNCCL_HOMEwhen NCCL public headers are available from the
installednvidia.ncclpackage. nccl.epchecks that the loadedlibnccl.soandlibnccl_ep.sowere built
with the same CUDA major version. CUDA minor differences are accepted.
Communicator Configuration
- Added
graph_stream_orderingtoNCCLConfig.
Device API and CuTe DSL
- New
nccl.core.device.cutemodule exposes the NCCL device API to CuTeDSL
kernels, including communicator/window access, GIN primitives, barrier
operations, and typed structs. - Added
bindings/nccl4py/examples/cute/main.py, a GIN put/wait example with
host-side validation. - Added
gin_strong_signals_requiredandgin_va_signals_requiredto
NCCLDevCommRequirementsfor configuring device communicator requirements. - Added
NcclGinType.GPIfor the GPU-Push Interface transport.
Version and Diagnostics API
- Top-level
nccl.get_version()returns aVersionInfodataclass containing
thenccl4pypackage version plusLibraryInfoentries for the loaded
libnccl.soand, when available,libnccl_ep.so. - Top-level
nccl.show_versions()prints the same stack information in a
human-readable version block. - Direct library probes are available for each native library:
nccl.core.get_lib_version()andnccl.core.get_lib_path()report the
loadedlibnccl.so;nccl.ep.get_lib_version()and
nccl.ep.get_lib_path()report the loadedlibnccl_ep.so. - Each
LibraryInfoincludes release version, CUDA build variant, and loaded
shared-library path.
Installation and Packaging
- CuTeDSL support can be installed through the CUDA-specific extras:
nccl4py[cu12]installsnvidia-cutlass-dsl>=4.5.2,<5.0, and
nccl4py[cu13]installsnvidia-cutlass-dsl[cu13]>=4.5.2,<5.0. - Wheels include package data for
nccl/ep/lib/libnccl_ep.soplus EP JIT
headers. The bundledlibnccl_ep.sois built with CUDA 13, regardless of
whether thecu12orcu13extra is installed. Users who want to use a
CUDA 12 build oflibnccl_ep.somust provide that library themselves, for
example throughLD_PRELOADorLD_LIBRARY_PATH. - Wheels are available for free-threaded CPython 3.14t.
Examples and Documentation
- Added Python examples for:
- multiple devices in one process:
docs/examples/01_communicators/01_multiple_devices_single_process/python/; - one device per MPI process:
docs/examples/01_communicators/03_one_device_per_process_mpi/python/; - point-to-point ring pattern:
docs/examples/02_point_to_point/01_ring_pattern/python/; - allreduce:
docs/examples/03_collectives/01_allreduce/python/; - user-buffer allreduce:
docs/examples/04_user_buffer_registration/01_allreduce/python/; - symmetric-memory allreduce:
docs/examples/05_symmetric_memory/01_allreduce/python/; - symmetric-memory allgather:
docs/examples/05_symmetric_memory/02_allgather/python/.
- multiple devices in one process:
- Added nccl4py documentation under
docs/userguide/source/nccl4py/, with the
main entry point atdocs/userguide/source/nccl4py.rst.
Breaking Changes
Removed APIs
-
nccl.core.group_simulate_end()has been removed. Use
nccl.core.group_end(simulate=True):from nccl.core import group_end, group_start group_start() # enqueue operations info = group_end(simulate=True)
-
NCCL_SPLIT_NOCOLORhas been removed from the public constants. Use
color=Nonewhen a rank should opt out ofCommunicator.split().
Deprecated APIs
nccl.core.get_version()remains available, but is deprecated. Use top-level
nccl.get_version()for structured version information, or
nccl.show_versions()for human-readable output.
Other Compatibility Notes
- Public NCCL enum wrappers are pure-Python
IntEnumorIntFlagclasses.
Integer compatibility is preserved, and dtype conversion remains supported.
Code that depends on binding-backed enum class identity from earlier releases
may need updates. - Enum members now follow the Python enum convention of
UPPER_SNAKE_CASE
names, such asCTAPolicy.DEFAULT,CommShrinkFlag.ABORT,
WindowFlag.COLL_SYMMETRIC, andNcclCommMemStat.GPU_MEM_TOTAL. The
previous PascalCase/camelCase aliases, such asCTAPolicy.Defaultand
NcclCommMemStat.GpuMemTotal, still work in 0.3.1 for compatibility, but
will be removed in a future release. New code should use the uppercase names.
Fixes and Enhancements
- Fixed pointer lifetime handling for non-blocking communicator and window
initialization. - Torch interop covers
torch.uint32andtorch.uint64when those dtypes are
available.
API Stability
nccl.epandnccl.core.device.cuteare initial API support. Their public
interfaces may change in future releases as the NCCL EP and CuTeDSL device
API integration matures.
NCCL EP v0.1.0 Release
NCCL EP is a high-performance NCCL API extension for efficient Mixture-of-Experts (MoE) communication. It provides optimized dispatch and combine primitives for Expert Parallelism (EP) across distributed GPU systems implemented on top of NCCL Device API: Load-Store Accessible (LSA) and GPU-Initiated Networking (GIN) operations.
API Improvements and Extensions
- Refactor the API signatures to improve user experience and support backward compatibility.
- Change the device memory ownership for EP Tensor data. The user is now responsible for device memory allocations for EP Tensors.
- Refactor the EP tensor data structure management for the host-side NCCL EP Tensor object. EP tensor now supports both dynamic allocation for long-term storage and static on-stack allocation for malloc-free usage on the data path.
- Add lightweight and CUDA Graph-compatible EP Handle management on the data path.
ncclEpCreateHandleis split intoncclEpInitHandle, which is a control-path operation that may allocate device memory and may be collective, andncclEpUpdateHandle, which updates the Handle's routing information before calling the Dispatch operation. - Allow users to set the number of SMs used by NCCL EP.
- Extend the API to associate an NCCL EP Tensor with an NCCL Window to enable zero-copy optimizations.
- Add flexible Dispatch output layout configurations:
- HT mode supports Flat and Expert-major layouts.
- Enable users to provide expert padding to align with GEMM requirements.
- LL mode supports Expert- and Rank-major layouts.
- Add active rank mask support to identify failed ranks and exclude them from future communication, allowing operation to continue instead of aborting the process.
- Introduce an explicit Forward/Backward pass selector in Dispatch and Combine operations.
- Drop top-K indices from the Dispatch operation signature and use the tensor provided to the Handle update.
Implementation Improvements
- Migrate to Just-In-Time (JIT) compilation for HT mode. This addresses performance issues and a number of limitations. LL migration to JIT is planned in the next release.
- Add full Multi-node-NVLINK (MNNVL) support.
- Remove limitations on the number of ranks in an LSA team. This has been tested on NVLink72.
- Fully migrate to NCCL infrastructure. All CUDA IPC references are removed and the code only depends on NCCL.
- Enable CUDA Graph support through EP handle management API changes and implementation changes.
- Support MoE and prefill workloads by enabling a variable number of tokens per sender on Dispatch.
Performance Optimizations
- Improve the performance of Dispatch for HT mode by leveraging NCCL Device API extensions available starting from NCCL v2.30.
- Improve the performance of the Combine operation in HT mode by leveraging JIT compilation.
- Enable zero-copy flows for HT mode.
- Optimize LL performance for NVLink-only configurations by avoiding the send-side staging buffer.
- Update
ep_benchto measure kernel-only performance. - Introduce
ncclTeamRailin HT mode instead of a split communicator. - Improve Dispatch/recv and Combine/send parallelization in LL mode.
Memory Footprint Optimizations
- Optimize the Dispatch staging buffer in LL mode. Use per-rank token deduplication and rank-major layout to reduce the staging buffer size by a factor of experts per rank.
- Expose rank-major layout at the API level in LL mode. Rank-major mode reduces the memory footprint by a factor of the number of experts per rank.
- Optimize HT mode handle memory usage by moving the global routing map buffer from the handle to the group scope. This allows different handles to share the buffer.
Python Bindings
- Expose NCCL EP through
nccl4py. - Make Python bindings more pythonic compared to the original 1-to-1 C-Python mapping.
Performance Benchmark (ep_bench)
- Report kernel-only performance metrics through CUPTI, if available.
- Extend the number of settings: number of SMs, number of experts, and layout selection.
- Add sophisticated validation for Dispatch and Combine phases to detect memory corruption and routing issues.
Bug Fixes
- Fix the bug in HT mode preventing launches on more than 8 nodes.
- Fix HT mode inter-node flags sizing that would cause overflow for 9 or more nodes.
- Clean the API and tools from quantization-related code. Quantization support is planned to be re-enabled in the following release.
- Fix memory ordering in Dispatch/Combine grid barriers.
- Fix a bug causing crashes in LL mode for batch sizes.
- Fix integer overflow in inter-node N2N warp at 8 or more nodes. Thanks to Mozar Huang.
Known Issues and Limitations
- The number of RDMA domains, or NCCL LSA Teams, in HT mode is limited to 32 due to algorithmic limitations.
nccl4py0.3 wheel is shipped withlibnccl_ep.sobuilt with CUDA 13. To use CUDA 12, users have to buildlibnccl_ep.sofrom source and specify the.sofile path usingLD_PRELOADorLD_LIBRARY_PATH. In addition,NCCL_EP_HOMEneeds to be set to point to the correspondingnccl_epinstallation directory.- NCCL EP v0.1 does not support quantization. While the API has appearances of quantization-related parameters, such as the scales tensor, the implementation was not tested and is not guaranteed to work. Elements of quantization support are expected to be introduced in the next release.
- The Dispatch operation has resource limitations associated with the amount of available shared on-chip memory. Consumption is impacted by two factors:
- The hidden dimension of the token.
- LSA team size, which is the size of the NVLink domain.
- If a job launch is aborted due to shared memory overflow, try to reduce the current stage or pipeline settings. In v0.1, this can only be done statically at build time: reduce the
HYBRIDEP_DISPATCH_NUM_OF_STAGESand/orHYBRIDEP_DISPATCH_NUM_OF_PIPELINES_PER_BLOCKmacro values inhybridep_configs.cuh, rebuild, and retry.
LL Mode Limitations
- Maximum top-K: 9.
- Hidden dimensions: 2048, 2560, 4096, 5120, 6144, 7168, and 8192.
Known Bugs
- In LL mode,
ep_benchreports Combine verification failure when a batch size of 1 token is used. - In LL mode,
ep_benchreports Combine verification failures on 16 nodes and 64 GPUs for the batch size of 8K tokens.
NCCL v2.30.7-1 Release
Zero-SM Collectives
- Adds hierarchical zero-SM collectives (AllGather and All2all) that use RMA CPU proxy for inter-node communication and Copy Engines for intra-node communication.
- Enables better overlap of compute and communication.
- Enable hierarchical zero-SM collectives with
NCCL_CTA_POLICY_ZEROflag.
GIN Enhancements
- Adds new experimental GPU Push Interface (GPI) backend for GIN.
- Adds explicit signal semantics with Strong and Weak signals.
- Adds proper
ncclGinFenceLevelsemantics for barriers. - Adds separate
NCCL_GIN_IB_TCtoggle to control traffic class used by GIN. - Adds
NCCL_GIN_RESOURCE_SHARING_THREADto enable more optimizations. - Optimizes QP overhead, including GDAKI mode when counters are not used.
- Ensures GIN is usable when NIC fusion is enabled.
- Adds GIN plugin example in
plugins/gin/example.
Symmetric Memory Improvements
- Restructures RMA plugin architecture.
- Adds support for asymmetric buffer sizes during window registration.
- Optimizes ReduceScatter symmetric kernel performance.
- Optimizes performance for RMA operations using CE.
- Adds batched CE operations to improve performance in the RMA CE put/wait path.
- Adds support for window registration during CUDA graph capture.
MPS with MLOPart Support (Experimental)
- NCCL now leverages CUDA feature Memory Locality Optimized Partition (MLOPart).
- Supports up to 2 ranks per physical GPU with MPS+mlopart.
Other Improvements
- Adds support for IB ports that require global route headers (GRH).
- Adds logic to
gin.flushto ensure all prior gets are visible. - Adds makefile support to compile python wheels from source.
- Adds
NCCL_RMA_DISABLEenv to enable/disable RMA (Github PR #2151). - Implements reset-without-zeroing for signals and counters in GIN (Github PR #2155).
- Pins GIN proxy thread to NUMA-local CPU set (Github PR #2182).
- Adds optimized weight transfer APIs in
contrib/nccl_xfer. - Adds custom kernels in
contrib/custom_algosfor alltoall and allreduce using NCCL Device API. - Adds examples of Root Mean Square Normalization (RMSNorm), demonstrating the fusion of computation and communication using the device API.
- Unifies coding style by using clang-format. Please see
docs/dev_guide/nccl_coding_style.mdfor more details. - Drops support for v11 and v12 GIN plugin APIs.
Bug Fixes
- Fixes a deadlock caused by cuda stream allocation under PXN when memseting a buffer at runtime.
- Reintroduce
cudaGridDependencySynchronizein built-in symmetric kernels, ensuring that newly launched kernels cannot access memory modified by prior kernels before it reaches point of coherency. - Ignores system headers in include/header processing, thereby avoiding excessive realpath calls in some builds (Github PR #1806).
- Improves QP load balancing on systems configured with RoCE LAG with the round-robin queue affinity policy (Github PR #2150).
- Fixes issue when receiving an external TCP request causes the proxy thread's
ncclProxyServiceto hang (Github PR #1834). - Fixes
rma_proxyMR registration type for host-NUMAcpuAccessSignals, which ensures that the net plugin does not reject the registration due to wrong memory type (Github PR #2187). - Fixes GIN init context leak (Github PR #2179).
- Fixes issue with one-sided host APIs when a custom GIN plugin is used.
- Fixes one-sided host API issue where requests are dropped at a high message rate (Github Issue #2119).
Acknowledgements
We thank the following contributors for their work on this release:
@andrewjcg, @baymaxhuang, @bhasunit, @fishautumn, @mozarhua, @ngoyal2707, @wanglei875 for your PRs.
We also thank the community for issue reports, testing, and feedback.
Known Issues
- NCCL one-sided host RMA APIs, e.g.,
ncclPutSignal, require every rank to call the API as a one-time initialization warm-up. This has been fixed on dev branch: e12963e - NCCL one-sided RMA operations have a possible corruption issue when multiple symmetric windows are carved from the same backing memory allocation. See #2198. This has been fixed on dev branch.
- Hang is possible when specific workload uses PXN (e.g. ncclSend/ncclRecv/All2all) and calls ncclCommDestroy() without waiting for all ranks to complete. This has been fixed on dev branch: 08b3662
- Hang is possible when workload uses 2-rank communicator and uses CE collectives. This has been fixed on dev branch: 430afbc
nccl4py v0.2.0 Release
Release Notes — nccl4py 0.2.0
This release adds Python bindings for the new NCCL 2.30 one-sided RMA, Device API (GIN), and elastic communicator features, along with substantially more control over communicator configuration.
Highlights
- One-sided RMA (point-to-point) — New
Communicator.put_signal(),Communicator.signal(), andCommunicator.wait_signal()methods, plus aWaitSignalDeschelper for describing signal values and match operations. - NCCL Device API host side setup — New
Communicator.create_dev_comm()that produces aDevCommResourcefor use with device-side NCCL kernels. Configure the device communicator through the newNCCLDevCommRequirementsclass, and introspect support viadevice_api_support,gin_type,railed_gin_type,host_rma_support, andn_lsa_teamsproperties. - Device pointer access for registered windows —
RegisteredWindowHandlenow exposesuser_ptr,get_lsa_device_pointer(),get_lsa_multimem_device_pointer(), andget_peer_device_pointer()for direct access to LSA, multimem, and peer mappings. - Elastic and fault-tolerant communicators — New
Communicator.grow(),revoke(),suspend(), andresume()methods to support elastic topology changes and error-handling flows.CommSuspendFlagadded alongside existingCommShrinkFlag. - More flexible construction — In addition to
init(), communicators can now be created with class methodinit_all()and instance methodinitialize().Communicator.get_mem_stat()reports per-communicator memory statistics.
Configuration
New tuning knobs on NCCLConfig:
graph_usage_mode,num_rma_ctx,max_p2p_peers.
NCCLDevCommRequirements — passed to Communicator.create_dev_comm() to describe the resources and capabilities a device communicator needs:
- LSA:
lsa_multimem,barrier_count,lsa_barrier_count,rail_gin_barrier_count,world_gin_barrier_count,lsa_ll_a2a_block_count,lsa_ll_a2a_slot_count. - GIN:
gin_force_enable,gin_context_count,gin_signal_count,gin_counter_count,gin_queue_depth,gin_connection_type,gin_exclusive_contexts.
Device / topology introspection
New Communicator properties: cuda_dev, nvml_dev, device_api_support, multimem_support, gin_type, railed_gin_type, n_lsa_teams, host_rma_support.
Other changes
CTAPolicyis now anIntFlag(wasIntEnum) so multiple policies can be combined.- Interop submodules
nccl.core.cupyandnccl.core.torchare now lazy-loaded via__getattr__and only imported on first attribute access, soimport nccl.coreno longer pulls in CuPy or PyTorch.