Skip to content

Releases

NCCL v2.32.6-1 Release

Choose a tag to compare

@justus-nv justus-nv released this 09 Nov 19:28
ad2d192

Vera Rubin Performance

  • Adds a pipeline-based cost model with Vera Rubin tuning data for Ring, Tree, NVLS, and NVLSTree.
  • Adds Vera Rubin models for symmetric AllGather, ReduceScatter, and AllReduce kernels.
  • Speeds up the topology graph search on large NVLink domains such as 32 to 40 GPU VR200 systems, reducing communicator initialization time.
  • Tuner plugin interface v7 adds Rubin platform support.

Device API Enhancements

  • Adds an experimental Flow device APIs for building pipelined collective kernels over LSA and GIN connections with the Simple, LL, and LL128 protocols. The Flow API signatures are not stable and may change.
  • Extends ncclGetPeerPointer and the resource-buffer peer pointer to address every load/store-reachable local peer by world rank, and adds ncclDevCommCanGetPeerPointer to report whether a peer is reachable.
  • Adds the NCCL_GIN_CONNECTION_CUSTOM_ARRAY GIN connection type so applications can request connectivity to an explicit list of peers.
  • Adds ncclLsaCopyTma, a TMA-based variant of the ncclLsaCopy broadcast on Blackwell and later GPUs.
  • Adds a signal-efficient two-signal GIN barrier that GIN plugins can request through the v15 barrierOptions property.
  • CFT teams now require and use CUDA logical-endpoint clique information to determine unicast and multicast team sizes.
  • Validates GPU and NIC capabilities when establishing GDAKI connections and falls back to the proxy backend when GDAKI is unsupported and NCCL_GIN_TYPE is unset.
  • Updates the GPI device backend for the latest libgpi device ABI, including 31-bit signal and counter IDs and extended transfer-size encoding, and updates the bundled GPUNetIO code to v5.0.0-rc3.
  • Moves the IR-only C device ABI (ncclGin_C, ncclCoopAny) out of the public device headers into bindings/ir.

Collectives and Runtime Enhancements

  • Adds experimental collective kernels called General Kernels (or GENK), built on top of the new Flow APIs. These kernels are compiled out by default to minimize binary size; build with SYM_GENK=1 and set NCCL_SYM_GENK_ENABLE=1 to try them.
  • Adds experimental AlltoAll_LL, a latency-oriented symmetric AlltoAll kernel for small and medium per-peer messages on LSA and mixed LSA/GIN communicators, selectable with NCCL_SYM_KERNEL=AlltoAll_LL.
  • Supports host RMA on rail-only configurations such as NCCL_CROSS_NIC=0, where host RMA was previously disabled.
  • Updates the GIN and RMA plugin interfaces to v15 and v17: connection setup receives an abort flag so ncclCommAbort can interrupt blocking GIN and RMA initialization, and the plugin rankStride is replaced by an explicit peerArray.
  • Improves hierarchical CE collective bandwidth when inter-node traffic is distributed across multiple RMA contexts, including under CUDA graph capture.
  • Reduces host munmap latency on ATS platforms such as GB200 by granting ncclMemAlloc memory access to the owning GPU only.

Diagnostics and Profiling

  • Adds profiler plugin interface v8 with opt-in NCCL-sourced GPU and CPU timestamps on every event.
  • Supports loading up to two profiler plugins at a time through a comma-separated NCCL_PROFILER_PLUGIN, each with its own event mask.
  • Allows NetPlugin profiler events to be collected independently of proxy events; profiler plugins must accept ProxyStep or NULL parents.
  • Updates Inspector to record per-call profiler tags (requiring profiler v7 or later), emit the selected algorithm and protocol per collective in its JSON output, and ship a communication-pattern extractor. NCCL_INSPECTOR_NCCL_SOURCED_TIMINGS defaults to on.
  • Adds a RAS diagnostics check verifying that the network devices NCCL selected are consistent across the nodes of a communicator and within each node.
  • Adds the DIAG debug subsystem (NCCL_DEBUG_SUBSYS=DIAG) for active and RAS diagnostics details.
  • Attaches all GIN/GDAKI queue pairs to a dedicated hardware Q counter set so aggregate counters can be queried for GIN traffic; controlled by NCCL_GIN_GDAKI_Q_COUNTER (default on).

Other Improvements

  • Enables building NCCL for Windows on Arm64.
  • Reduces library size by compressing device code with --compress-mode=size when building with CUDA 13 or newer.
  • Adds channel-major and step-major NVLS staging FIFO layouts that reduce NVLink uTLB pressure for large-message NVLS and NVLSTree on Rubin. NCCL_NVLS_FIFO_LAYOUT controls the selection.
  • Builds libnccl_device.bc and libnccl_device.ltoir with Clang/LLVM 23, and unifies the IR build options as IR_ARCH and NCCL_EMIT_LLVM_IR.
  • Relaxes ncclCommGrow so existing ranks may pass either the original ncclUniqueId or NULL.
  • Adds a thread-exclusive Put path to the EFA-GDA GIN backend and prints endpoint counts when a flush times out. (Github PR #2478) (Github PR #2552)
  • Invokes the build-time Python generators through python3, and resolves Git version metadata from the NCCL source tree for out-of-tree CMake builds. (Github PR #2452) (Github PR #2451)
  • Adds a user-guide page on the determinism of NCCL reductions, and improve various other documentation.

Bug Fixes

  • Fixes a PXN deadlock caused by an unnecessary blocking cudaHostRegister of legacy shared memory in the proxy process.
  • Fixes single-node NVLS AllGather and ReduceScatter returning data in the wrong rank order when CUDA_VISIBLE_DEVICES permutes GPUs. (Github PR #2377) (Github Issue #1906)
  • Fixes a silent hang with network-registered buffers where the receiving proxy erased a rendezvous flag already published by the GPU. (Github PR #2390) (Github Issue #2389)
  • Fixes a NET/IB hang when the request counter wraps at UINT32_MAX and completions are matched to the wrong receive request.
  • Fixes races in the NET/Socket transport, including a task-slot publication race that caused NULL data sends on aarch64, and a missing load ordering on the NET/IB CTS FIFO. (Github PR #2365) (Github PR #2393) (Github Issue #1983)
  • Fixes IB async-event threads handling events against the wrong device when NCCL_IB_DEVICE_PCI_ORDER reorders devices.
  • Fixes communicator initialization failures on hosts with multi-port NIC when NCCL loads multiple network plugins.
  • Skips invalid measurements in the example tuner's configuration optimizer. (Github PR #2453)
  • Fixes CX9 NIC detection that misclassified BlueField-4 NICs as CX9 and missed CX9 virtual functions, preventing automatic rail and plane assignment.
  • Fixes communicator initialization failures when combined per-rank topology data exceeds the topology XML fusion capacity. (Github PR #2386) (Github Issue #2160)
  • Fixes ncclAllReduce with ncclAvg on symmetric-memory buffers failing with an unhandled CUDA error for floating-point types. (Github PR #2448) (Github Issue #2447)
  • Fixes a non-blocking communicator becoming permanently unusable after one call returned ncclInvalidArgument. (Github PR #2460) (Github Issue #2459)
  • Fixes ncclCommShrink putting the parent communicator into an error state when the excluded rank list is invalid; it now returns ncclInvalidArgument. (Github Issue #2457)
  • Fixes a crash in ncclCommInitRank when NCCL_ALGO, NCCL_PROTO, or NCCL_SYM_KERNEL contain empty entries. (Github PR #2456) (Github Issue #2455)
  • Fixes per-call maxCTAs bounds not being honored by kernel-based AlltoAll, Gather, and Scatter, and P2P channel counts exceeding maxCTAs after rounding to a power of two. (Github PR #2435) (Github Issue #2421) (Github Issue #1598)
  • Fixes ncclCommRegister returning success without a handle for CPU-backed buffers; it now returns a NULL handle and rejects a NULL handle pointer. (Github PR #2433)
  • Fixes ranks on distinct MIG compute instances of the same GPU being rejected as duplicate GPU ranks. (Github Issue #1716)
  • Fixes failures when cuMem host memory support is inconsistent across ranks. The enablement logic is no longer affected by CUDA forward compatibility.
  • Fixes GDRCopy initialization failures on GPUs that support DMA-BUF export but not CPU mmap of DMA-BUF allocations.
  • Fixes intermittent encrypted bootstrap failures caused by unread TLS session tickets.
  • Fixes a hang or illegal memory access when a child communicator split from a splitShare=1 parent reuses NVLS buffers in a different rank order.
  • Fixes GIN and GIN-based symmetric kernels sending puts and signals to the wrong peer on communicators created with splitShare=1.
  • Fixes CFT counted writes on the multicast path silently skipping memory binding, which could lead to hangs.
  • Fixes a compiler error when calling put, putValue, or signal on a single-backend ncclGin_BackendOne or non-default ncclGin_BackendMask object. (Github PR #2436)
  • F...
Read more

NCCL v2.32.3-1 Release

Choose a tag to compare

@bhramesh-nvidia bhramesh-nvidia released this 17 Sep 21:09
12df1a1

Vera Rubin Support

  • Adds initial Rubin platform support including support for sm107, CX9 rail and plane detection, and MPS+MLoPart.
  • 2.32.3 focuses on new functionalities and does not contain performance model tuning for Rubin. This will be part of the next release to improve out-of-box experience for Rubin users.

Device API Enhancements

  • Adds Compute Fabric Transport (CFT) counted-write and wait support.
  • Adds socket-based GIN support, enabling custom kernel development with TCP sockets.
  • Adds support in GDAKI for LAG-aware QP assignment based on the context ID. (Github PR #2315)
  • Adds NCCL_WIN_GIN_ONLY so users can register a window only for GIN usage.
  • Optimize GIN performance by skipping mcst operation when possible.

Collectives and Runtime Enhancements

  • Adds a ring-based hierarchical copy-engine AllGather implementation selectable with NCCL_HIER_CE_COLL_AG_RAIL_RING_ENABLE. (Github PR #2299)
  • Improves Blackwell symmetric AllGather performance, resource overhead modeling and kernel selection with a new cost model.
  • Adds optional TLS encryption for NCCL-owned socket traffic when NCCL is built with OpenSSL3, configured through the ncclSetEncryption API.
  • Adds ncclCollConfig_t::launchCompletionEvent, allowing callers to observe kernel-launch completion.
  • Adds ncclNvlsHostMode_t to ncclConfig_t so applications can disable host NVLS collectives per communicator.
  • Optimizes NVLS slot consumption to avoid resource exhaustion issues when using multiple communicators with NVLS.

Diagnostics and Profiling

  • Adds the ATTN log level for important non-fatal conditions, such as configuration fallbacks and plugin initialization failures.
  • Expands RAS capability with GPU-resident progress counters and a watchdog DMA mirror for diagnosing stalled collective kernels.
  • Adds additional RAS diagnostics functionalities for NVLink/NIC state and speed, PCI and GDR configuration, Xid/SXid events and others.
  • Reports degraded NVLink fabric bandwidth with an ATTN message during communicator initialization.
  • Reports mismatched NCCL Git revisions across communicator ranks during initialization.

Other Improvements

  • Fixes PAT+NVLS performance drops on H100 platforms.
  • Avoid address space exhaustion when repeatedly registering symmetric windows backed by a single physical memory allocation.
  • Reduces communicator initialization overhead at large scale by avoiding scans of inactive proxy poll descriptors.
  • Adds an experimental built-in NetworkDirect transport on Windows, with automatic socket fallback.
  • EFA team improved EFA GDA support with gin.get API and others. (Github PR #2382)(Github PR #2398)(Github PR #2405)
  • Reports Inspector ring-buffer drops and operation counts in JSON. (Github PR #2304)
  • NCCL now supports communicators using multiple MIG instances.
  • Generates llms.txt for agents to better read NCCL documentation.

Bug Fixes

  • Fixes profiler overhead when no profiler plugin is loaded. (Github Issue #2355)
  • Fixes P2P IPC registration reuse producing out-of-bounds remote addresses when a registered allocation spans multiple cuMem segments. (Github PR #2362)
  • Fixes PAT connection-setup deadlocks when runtime connection is disabled and nodes have uneven local-rank counts. (Github Issue #2385)
  • Fixes non-thread-safe token parsing that could corrupt concurrent configuration parsing. (Github Issue #2361)
  • Fixes B40 TMA symmetric kernels crash due to insufficient shared memory.
  • Fix GIN GDAKI bug related to hop limit when using DOCA SDK.
  • Fixes GIN Proxy and GPI descriptor shared-memory sizing and alignment.
  • Fixed CFT window registration performed before ncclDevCommCreate.
  • Fixes profiler API events reporting rank 0 instead of the originating communicator rank. (Github Issue #2300)
  • Fixes tuner plugins receiving uninitialized cost-model constants after the tuning rework.

Contrib/ Updates

  • Adds more functionality in NiiN.
  • Fixes contrib/nccl_checkpoint rejecting otherwise compatible NCCL versions. (Github Issue #2347)

Acknowledgements

We thank the following contributors for their work on this release:

@madeleineth, @alpha-baby, @akkart-aws, @anshumang, @cesar-stuardo-bd, @hexagonal-banana,
@rauteric, @shaq918, @yshalabi, @zrss for your contributions.

We also thank the community for issue reports, testing, and feedback.

Known Issues

  • Rubin performance model has not been optimized with 2.32.3. Users can observe better performance on Rubin by increasing the number of CTAs NCCL uses. Automatic tuning will be improved in the next release.
  • Socket GIN currently requires users to opt-in to enable GDRCopy.
  • B100 PCIe MLoPart workloads can fail during memory allocation with CUDA error 101 (invalid device ordinal). Set NCCL_CUMEM_ENABLE=0 as a workaround.

NCCL4Py v0.6.0 Release

Choose a tag to compare

@xiakun-lu xiakun-lu released this 23 Sep 17:51

Highlights

  • Added experimental CuTe DSL ReduceCopy APIs for LSA, multimem, and local memory operations.
  • Added NCCL 2.32 host APIs for collective launch completion events, NVLS configuration, CFT capability inspection, and window registration.
  • Added explicit CuTe DSL barrier-session teardown and corrected ThreadScope.THREAD to match libcu++.

New Features

Collective Launch Completion Events

  • Added per-collective CUDA launch completion events through NCCLCollConfig.

    APIs:

    • NCCLCollConfig.launch_completion_event
    • NcclEventSpec

NCCL 2.32 Configuration and Capability Inspection

  • Added host-side NVLS configuration, CFT capability reporting, and specialized window registration flags.

    APIs:

    • NCCLConfig.nvls_host_mode
    • NcclNvlsHostMode
    • NCCLCommProperties.cft_support
    • NCCLCommProperties.cft_multicast_support
    • NCCLCommProperties.cft_counted_support
    • WindowFlag.GIN_ONLY
    • WindowFlag.CFT_COUNTED

CuTe DSL ReduceCopy

  • Added device APIs for reduction and copy operations across LSA windows, multimem pointers, and local tensors.

    APIs:

    • lsa_reduce_sum(), multimem_reduce_sum()
    • lsa_copy(), multimem_copy()
    • lsa_reduce_sum_copy(), multimem_reduce_sum_copy()
    • local_reduce_sum_copy()

CuTe DSL Barrier Lifecycle

  • Added explicit session destruction so barrier handles and indexes can be safely reused.

    APIs:

    • LsaBarrierSession.destroy()
    • GinBarrierSession.destroy()
    • BarrierSession.destroy()

Examples and Documentation

Breaking Changes

  • ThreadScope.THREAD now has the libcu++ numeric value 10 instead of 3. Code using the enum member requires no changes; code storing or passing its raw integer value must be updated.
  • CuTe DSL barrier sessions must call destroy() exactly once after their final operation. Every thread in the session’s cooperative group must call it from uniform control flow.

Fixes and Enhancements

  • The NCCL 2.31.2 device IR issue reported in nccl4py v0.5.0, which could prevent Gin.put() from JIT-compiling with EFA GDA enabled, is resolved in the matching NCCL 2.32.3 IR.
  • Added a py.typed marker so compatible static type checkers can use nccl.core annotations.

Compatibility Notes

  • The nccl4py 0.6.0 host-side bindings and CuTe DSL API are generated from NCCL 2.32.3 headers.
  • When using the CuTe DSL API, use matching NCCL 2.32.3 libnccl.so and libnccl_device.bc; nccl4py does not verify this automatically.
  • NCCLCollConfig.launch_completion_event requires a timing-disabled, non-interprocess CUDA event. Every rank must either provide an event or omit it. CUDA 12.3 or later is required for post-launch recording semantics.
  • The CuTe DSL device API remains experimental.

NCCL4Py v0.5.0 Release

Choose a tag to compare

@xiakun-lu xiakun-lu released this 01 Sep 19:41

Highlights

  • Added host and CuTe DSL APIs for CFT setup and logical-endpoint addressing.
  • Added per-collective configuration for algorithm selection, CTA tuning, profiler tags, and vendor options.
  • Added communicator capability inspection and support for new NCCL 2.31.2 GIN features.
  • Added compile-only CuTe DSL arguments for compiling kernels before creating NCCL resources.

New Features

Per-Collective Configuration

  • Added per-call configuration to all collective operations.

    APIs:

    • NCCLCollConfig
    • VendorOption
    • The config argument on Communicator.allreduce(), broadcast(), reduce(), allgather(), reduce_scatter(), alltoall(), gather(), and scatter()

Compute Fabric Transport Setup

  • Added APIs for configuring CFT resources, querying CFT teams, and resolving logical endpoint addresses. CFT communication and barrier operations are not yet exposed.

    Host APIs:

    • NcclHostCftMode, NcclCftTeamMode, NcclCftCap
    • NCCLConfig.host_cft_mode
    • NCCLDevCommRequirements.cft_caps, NCCLDevCommRequirements.cft_barrier_count
    • Communicator.team_cft(), Communicator.team_cft_multimem
    • CftLeInfo
    • RegisteredWindowHandle.get_cft_le_info()
    • RegisteredWindowHandle.get_peer_le_info()
    • RegisteredWindowHandle.get_multimem_le_info()

    CuTe DSL APIs:

    • CftTeamMode, CftLeInfo
    • DevComm.team_cft(), DevComm.team_cft_multimem
    • Window.cft_le_info(), Window.peer_le_info(), Window.multimem_le_info()
    • DevComm.resource_buffer_cft_le_info()
    • DevComm.resource_buffer_peer_le_info()
    • DevComm.resource_buffer_multimem_le_info()

Communicator and GIN Capabilities

  • Added consolidated communicator property inspection and support for new NCCL 2.31.2 configuration and GIN capabilities.

    APIs:

    • NCCLCommProperties, Communicator.properties
    • NCCLConfig.launch_order_implicit
    • NCCLConfig.num_rma_sig
    • NCCLConfig.rma_eager_init
    • NCCLDevCommRequirements.gin_type
    • NCCLDevCommRequirements.gin_custom_stride
    • NcclGinType.EFA_GDA
    • NcclGinConnectionType.CUSTOM_STRIDE
    • GinBackendMask.EFA_GDA

Compile Before Creating NCCL Resources

  • Added type-only arguments for compiling CuTe DSL kernels before communicators, windows, or device resources are created.

    APIs:

    • nccl.core.device.cute.runtime.make_fake_dev_comm()
    • nccl.core.device.cute.runtime.make_fake_window()
    • nccl.core.device.cute.runtime.make_fake_multimem_handle()
    • nccl.core.device.cute.runtime.make_fake_lsa_barrier_handle()
    • nccl.core.device.cute.runtime.make_fake_gin_barrier_handle()

Examples and Documentation

Fixes and Enhancements

  • Registered window handles created within nccl.core.group() are now updated correctly after the group completes.

Compatibility Notes

  • The nccl4py 0.5.0 host-side bindings and CuTe DSL API are generated from NCCL 2.31.2 headers.
  • When using the CuTe DSL API, use the matching NCCL 2.31.2 libnccl.so and libnccl_device.bc; nccl4py does not verify this automatically.
  • The CuTe DSL device API remains experimental.

Known Issues

  • The NCCL 2.31.2 device IR may fail to JIT-compile Gin.put() when EFA GDA is enabled. Apply this patch to NCCL 2.31.2 and rebuild libnccl_device.bc.

NCCL v2.31.2-1 Release

Choose a tag to compare

@bhramesh-nvidia bhramesh-nvidia released this 11 Aug 00:24
7b83616

Compute Fabric Transport (CFT)

  • Adds CFT host and device APIs for registering window memory with CUDA logical endpoints and issuing device-side Put, Get, Red and NVLS operations.
  • CFT is supported on Blackwell GPUs with CUDA toolkit 13.3 or later.

Per-Collective Configuration and Tuning

  • Adds new ncclCollConfig_t and nccl*Config APIs for all collectives, with a vendor-defined field to make it easier for custom forks to integrate.
  • Supports per-collective algorithm selection, CTA/CGA size and CTA policy overrides.
  • Leverages collConfig to add userTag in profiler API. Motivated by community RFE #1916.

GIN Enhancements

  • AWS-EFA team contributed the EFA GDA backend for GIN Put, Signal, and Flush operations (Github PR #2273).
  • Adds per-DevComm GIN backend selection, allowing users to create multiple DevComms with different backends.
  • Supports connecting GIN with custom strides.
  • Adds device-side timeouts to blocking GIN APIs such as Flush, Wait, WaitSignal, and WaitCounter operations.
  • Reduces QP usage when using railed GIN.
  • Reduces file-descriptor consumption in GDAKI when using many QPs.
  • Optimizes error reporting in GDAKI via event-based CQ error reporting.
  • Add support for out-of-order delivery (DDP) in GDAKI on SPCX, improving performance without impacting GIN correctness guarantees.
  • Adds GRH support and automatic path-MTU discovery to the GIN GDAKI backend.

Parallel Aggregated Tree (PAT) Enhancements

  • Enhances the PAT algorithm for ReduceScatter/AllGather by adding hierarchical kernels that use NVLS within the node and PAT across nodes.
  • Enables better small/medium message performance.
  • Disabled by default. Use NCCL_ALGO=PAT, or the collConfig API to enable.

One-Sided RMA and Copy Engine Collectives

  • Adds support for multiple contexts and multiple signals to one-sided RMA operations, allowing one-sided RMA traffic from a single rank to use multiple NICs.
  • Leverage multiple contexts and signals for hierarchical 0-SM AllGather and AllToAll.
  • Optimizes small message latency by using NVLink multicast for AllGather.

Tuning and Cost Model

  • Introduces a new unified cost model interface for querying the cost estimate for both legacy and device-API-based kernels.
  • On Blackwell, TMA kernels are now enabled by default and integrated in the cost model for symmetric registered memory.

Diagnostics and Profiling

  • Adds NCCL RAS Diagnostics checks through NCCL_RUN_RAS_DIAGNOSTICS=1 or RAS client, including GPU inventory, CUDA driver versions, ECC errors, NVLink state, and NCCL_* environment consistency checks.
  • Adds NCCL Diagnostics through NCCL_RUN_DIAGNOSTICS=1, including active P2P connectivity check and actionable P2P remediation guidance.
  • Kerne-channel profiling moved to a dedicated per-communicator thread, extending kernel timing to proxy-less NVLink/SHM and graph-captured collectives.
  • Profiler V7 exposes per-kernel initial_sync, compute and final_sync phase events and symmetric-kernel variant metadata.
  • Adds per-QP CPU WQE post-to-poll latency monitoring for the IB transport.

Other Improvements

  • Adds backward compatibility support for applications that JIT-compile NCCL Device API code.
  • Adds support for LTO IR for NCCL device API.
  • Reduces communicator host-memory use and topology initialization time on large systems by allocating topology paths according to their actual lengths.
  • Adds multiple GIN proxy progress threads with per-thread endpoint assignment through GIN_PROXY_NTHREADS (Github PR #2279).
  • Improves GIN host-proxy throughput by processing multiple GIN operations per progress iteration (Github PR #2232) and adding hints to the RMA plugin to aggregate requests (Github PR #2254).
  • Adds CuTeDSL bindings for GIN Get, Flush, Signal, ReadSignal, and PutValue operations (Github PR #2266).
  • Adds an internal CUDA 13.3 DMA-BUF mmap backend for GDRCopy.
  • Adds event-based load balancing for the net_ib transport.
  • Refreshes local GIDs after IBV_EVENT_GID_CHANGE events during IB port recovery.
  • Updates the RMA plugin interface to v15 (Github PR #2254).
  • Improves AMD EPYC topology modeling (Github PR #2036).
  • Improves algorithm selection on newer Intel CPUs.
  • Makes nccl_device.h compatible with C99 host translation units.

Bug Fixes

  • Fixes a hang when PXN connection initialization races with communicator teardown.
  • Fixes data corruption when multiple symmetric windows are carved from the same backing memory allocation (Github Issue #2198).
  • Fixes a hang during the first Copy Engine collective on a two-rank communicator (Github Issue #2241).
  • Fixes NIC/GPU assignment perf regressions by limiting consistent start-NIC selection to Blackwell systems with ConnectX-8.
  • Fixes missing GIN signal and counter requirements when Device API communicators are created asynchronously (Github PR #2208).
  • Fixes a memory leak when Device API communicator creation fails (Github PR #2225).
  • Fixes an incorrect GIN context count reported to applications when the allocated context count is rounded up (Github PR #2301).
  • Fixes a resource leak when GIN connection setup fails (Github PR #2206).
  • Fixes the CUDA thread capture mode remaining changed when GIN setup fails (Github PR #2229).
  • Fixes inconsistent NVLS enablement when multiple ranks share a GPU (Github PR #2257).
  • Fixes valid communicator configurations being rejected when only one of minCTAs or maxCTAs is set (Github PR #2256).
  • Fixes communicator initialization failures on multi-system and MNNVL topologies when NCCL_P2P_PXN_LEVEL=1 (Github PR #2258).
  • Fixes a Device API loadConst performance regression caused by __ldg.
  • Fixes CuTeDSL GIN barrier initialization failures caused by under-aligned by-value arguments (Github PR #2243, Issue #2242).

Contrib/ Updates

  • Adds PACE, a parallelism-aware collective engine that fuses layout conversion, data-type conversion, and scatter/gather operations into collective kernels (Github PR #2319).
  • Adds experimental nccl4rust host and Device API bindings using LTO IR.
  • Adds NIIN, a header-only NVSHMEM-compatible interface implemented with NCCL Device API primitives.
  • Adds community-contributed architecture learning guides for communicator initialization, topology, tuning, transports, and collective execution (Github PR #2081).
  • Adds a community-contributed analysis of DeepEPv2.

Acknowledgements

We thank the following contributors for their work on this release:

@akkart-aws, @alpha-baby, @anshumang, @baymaxhuang, @bhasunit, @dboyan, @EylonKrause, @Gaojiaqi, @hershys-aws, @mozarhua, @rauteric, @tianhao909, @voipmonitor, @wanglei875, @Xuan-1998, @yongxiangren for your PRs.

We thank the AWS EFA team for their optimizations to GIN CPU Proxy and for adding EFA GDA backend in GIN.

We also thank the community for issue reports, testing, and feedback.

Known Issues

  • PAT + NVLS has performance regressions on H100 platforms. This will be fixed in the next release.
  • B40 symmetric TMA kernels: On B40 GPUs, TMA-based symmetric kernels can exceed available shared memory and fail with an illegal memory access. Set NCCL_SYM_TMA_ENABLE=0 to disable these kernels.
  • B100 PCIe MLoPart: Multi-rank-per-GPU workloads using MPS MLoPart on B100 PCIe systems can fail during memory allocation with CUDA error 101 (invalid device ordinal). Set NCCL_CUMEM_ENABLE=0 as a workaround.

NCCL4Py v0.4.1 Release

Choose a tag to compare

@xiakun-lu xiakun-lu released this 11 Aug 23:00

Highlights

  • Added host APIs for NCCL teams, rank translation, and device resource configuration.
  • Expanded the experimental CuTe DSL device API with additional GIN operations, resource addressing, and direct support for host-created resources.
  • Added live NCCL parameter access and improved version reporting.

New Features

Teams and Device Resources

  • Added APIs for inspecting communicator teams, translating ranks, and configuring resources for device-side communication.

    APIs:

    • NCCLTeam
    • Communicator.team_world, Communicator.team_lsa, Communicator.team_rail
    • Communicator.team_rank_to_world(), Communicator.team_rank_to_lsa()
    • NCCLDevCommRequirements.teams, NCCLDevCommRequirements.resources
    • NCCLDevCommRequirements.gin_traffic_class
    • TeamRequirement
    • LsaBarrierRequirement, GinBarrierRequirement, LLA2ARequirement
    • MultimemHandle, LsaBarrierHandle, GinBarrierHandle, LLA2AHandle
    • DevCommResource.multimem_handle(), DevCommResource.resource_handles
    • RegisteredWindowHandle.get_multimem_device_pointer()

CuTe DSL Device API

  • Added device-side rank translation, resource addressing, and additional GIN operations.

    APIs:

    • DevComm.team_rank_to_world(), DevComm.team_rank_to_lsa()
    • DevComm.resource_buffer_local_pointer()
    • DevComm.resource_buffer_lsa_pointer()
    • DevComm.resource_buffer_peer_pointer()
    • DevComm.resource_buffer_multimem_pointer()
    • DevComm.resource_buffer_lsa_multimem_pointer()
    • Window.multimem_pointer(), Window.lsa_multimem_pointer()
    • Gin.put_value(), Gin.get(), Gin.signal(), Gin.flush()
    • Gin.read_signal(), Gin.reset_signal(), Gin.signal_shadow_pointer()
    • Gin.read_counter(), Gin.wait_counter(), Gin.reset_counter()
    • GIN_ALL_CONTEXTS
    • ThreadScope, GinResourceSharingMode
    • DevComm.gin(..., resource_sharing_mode=...)
    • opt_flags support for Gin.put(), Gin.put_value(), Gin.get(), and Gin.signal()
  • Host-created device communicators, registered windows, multimem handles, and LSA/GIN barrier handles can now be passed directly as arguments to @cute.jit functions.

Runtime Inspection

  • Added live NCCL parameter access and consolidated version reporting under nccl.core.

    APIs:

    • nccl.core.params
    • nccl.core.dump_params()
    • nccl.core.get_version()
    • nccl.core.show_versions()

Examples and Documentation

  • Added CuTe DSL examples covering teams, LSA all-reduce, multimem, GIN operations, barriers, and resource buffers.

Breaking Changes

  • nccl.ep is no longer included in nccl4py. The NCCL EP Python bindings are now distributed separately from the NVIDIA/nccl-extensions repository.
  • Version helpers are no longer exported from the nccl namespace. Import them from nccl.core.
  • Version reporting was consolidated:
    • nccl.core.get_version() now returns VersionInfo.
    • nccl.core.Version, get_lib_version(), and get_lib_path() were removed.
    • VersionInfo.nccl was renamed to VersionInfo.libnccl.
    • VersionInfo.nccl_ep was removed.
    • VersionInfo.nccl_bindings was added.

Fixes and Enhancements

  • Fixed the default release semantics of Gin.put().
  • Improved NCCL parameter error handling and string decoding.

Compatibility Notes

  • The top-level nccl Python package is now a PEP 420 namespace package.
  • The CuTe DSL device API remains experimental.
  • The selected libnccl_device.bc must exactly match the NCCL version used by the bindings; nccl4py does not perform this check automatically.
  • When one thread manages multiple local ranks, call Communicator.finalize() within nccl.core.group().

Known Issues

  • The following CuTe DSL APIs require NCCL 2.31:

    • Gin.get()
    • gin_session(), world_gin(), rail_gin(), hybrid_session(), world_hybrid()
  • CuTe DSL JIT compilation with CUDA 12 fails for kernels using the following APIs. CUDA 13 is not affected.

    Affected APIs:

    • Gin.put(), Gin.put_value(), Gin.get(), Gin.signal(), Gin.flush()
    • GinBarrierSession.sync(), BarrierSession.sync()

NCCL4py v0.3.1 Release

Choose a tag to compare

@bhramesh-nvidia bhramesh-nvidia released this 11 Jun 18:45
5067397

Highlights

  • Added nccl.ep, a Pythonic interface to libnccl_ep.so for expert
    parallel dispatch/combine workflows. The package exposes Group, Handle,
    Tensor, typed config dataclasses, Algorithm, Layout, PassDir, and the
    named input/output structs used by the NCCL EP API.
  • Added nccl.core.device.cute, enabling CuTeDSL kernels to call NCCL device
    APIs.
  • Added top-level stack diagnostics with nccl.get_version() and
    nccl.show_versions(), reporting nccl4py, libnccl.so, and
    libnccl_ep.so versions, CUDA build variants, and loaded shared-library
    paths.
  • Added free-threaded CPython support.

New Features

NCCL EP Python API

  • New nccl.ep package provides Pythonic access to the NCCL EP extension
    library.
  • Group.create() creates EP groups from a Communicator and GroupConfig;
    Group.create_handle() creates handles with an explicit Layout.
  • Handle supports update(), dispatch(), combine(), complete(), and
    destroy().
  • DispatchInputs, DispatchOutputs, CombineInputs, CombineOutputs, and
    LayoutInfo provide named containers for the tensors and metadata used by
    dispatch, combine, and handle setup.
  • Tensor resolves Python buffers into ncclEpTensor_t descriptors.
  • GroupConfig, HandleConfig, DispatchConfig, CombineConfig, and
    AllocConfig expose typed configuration objects.
  • AllocFn and FreeFn expose caller-controlled EP allocation hooks.
  • nccl.ep.interop.torch.get_nccl_comm_from_group() provides PyTorch interop
    for creating an NCCL communicator from a PyTorch process group's rank and
    world-size information.
  • Importing nccl.ep sets default NCCL_EP_HOME when bundled EP JIT headers
    are present, and NCCL_HOME when NCCL public headers are available from the
    installed nvidia.nccl package.
  • nccl.ep checks that the loaded libnccl.so and libnccl_ep.so were built
    with the same CUDA major version. CUDA minor differences are accepted.

Communicator Configuration

  • Added graph_stream_ordering to NCCLConfig.

Device API and CuTe DSL

  • New nccl.core.device.cute module exposes the NCCL device API to CuTeDSL
    kernels, including communicator/window access, GIN primitives, barrier
    operations, and typed structs.
  • Added bindings/nccl4py/examples/cute/main.py, a GIN put/wait example with
    host-side validation.
  • Added gin_strong_signals_required and gin_va_signals_required to
    NCCLDevCommRequirements for configuring device communicator requirements.
  • Added NcclGinType.GPI for the GPU-Push Interface transport.

Version and Diagnostics API

  • Top-level nccl.get_version() returns a VersionInfo dataclass containing
    the nccl4py package version plus LibraryInfo entries for the loaded
    libnccl.so and, when available, libnccl_ep.so.
  • Top-level nccl.show_versions() prints the same stack information in a
    human-readable version block.
  • Direct library probes are available for each native library:
    nccl.core.get_lib_version() and nccl.core.get_lib_path() report the
    loaded libnccl.so; nccl.ep.get_lib_version() and
    nccl.ep.get_lib_path() report the loaded libnccl_ep.so.
  • Each LibraryInfo includes release version, CUDA build variant, and loaded
    shared-library path.

Installation and Packaging

  • CuTeDSL support can be installed through the CUDA-specific extras:
    nccl4py[cu12] installs nvidia-cutlass-dsl>=4.5.2,<5.0, and
    nccl4py[cu13] installs nvidia-cutlass-dsl[cu13]>=4.5.2,<5.0.
  • Wheels include package data for nccl/ep/lib/libnccl_ep.so plus EP JIT
    headers. The bundled libnccl_ep.so is built with CUDA 13, regardless of
    whether the cu12 or cu13 extra is installed. Users who want to use a
    CUDA 12 build of libnccl_ep.so must provide that library themselves, for
    example through LD_PRELOAD or LD_LIBRARY_PATH.
  • Wheels are available for free-threaded CPython 3.14t.

Examples and Documentation

  • Added Python examples for:
    • multiple devices in one process:
      docs/examples/01_communicators/01_multiple_devices_single_process/python/;
    • one device per MPI process:
      docs/examples/01_communicators/03_one_device_per_process_mpi/python/;
    • point-to-point ring pattern:
      docs/examples/02_point_to_point/01_ring_pattern/python/;
    • allreduce: docs/examples/03_collectives/01_allreduce/python/;
    • user-buffer allreduce:
      docs/examples/04_user_buffer_registration/01_allreduce/python/;
    • symmetric-memory allreduce:
      docs/examples/05_symmetric_memory/01_allreduce/python/;
    • symmetric-memory allgather:
      docs/examples/05_symmetric_memory/02_allgather/python/.
  • Added nccl4py documentation under docs/userguide/source/nccl4py/, with the
    main entry point at docs/userguide/source/nccl4py.rst.

Breaking Changes

Removed APIs

  • nccl.core.group_simulate_end() has been removed. Use
    nccl.core.group_end(simulate=True):

    from nccl.core import group_end, group_start
    
    group_start()
    # enqueue operations
    info = group_end(simulate=True)
  • NCCL_SPLIT_NOCOLOR has been removed from the public constants. Use
    color=None when a rank should opt out of Communicator.split().

Deprecated APIs

  • nccl.core.get_version() remains available, but is deprecated. Use top-level
    nccl.get_version() for structured version information, or
    nccl.show_versions() for human-readable output.

Other Compatibility Notes

  • Public NCCL enum wrappers are pure-Python IntEnum or IntFlag classes.
    Integer compatibility is preserved, and dtype conversion remains supported.
    Code that depends on binding-backed enum class identity from earlier releases
    may need updates.
  • Enum members now follow the Python enum convention of UPPER_SNAKE_CASE
    names, such as CTAPolicy.DEFAULT, CommShrinkFlag.ABORT,
    WindowFlag.COLL_SYMMETRIC, and NcclCommMemStat.GPU_MEM_TOTAL. The
    previous PascalCase/camelCase aliases, such as CTAPolicy.Default and
    NcclCommMemStat.GpuMemTotal, still work in 0.3.1 for compatibility, but
    will be removed in a future release. New code should use the uppercase names.

Fixes and Enhancements

  • Fixed pointer lifetime handling for non-blocking communicator and window
    initialization.
  • Torch interop covers torch.uint32 and torch.uint64 when those dtypes are
    available.

API Stability

  • nccl.ep and nccl.core.device.cute are initial API support. Their public
    interfaces may change in future releases as the NCCL EP and CuTeDSL device
    API integration matures.

NCCL EP v0.1.0 Release

Choose a tag to compare

@bhramesh-nvidia bhramesh-nvidia released this 08 Jun 22:54
63cf786

NCCL EP is a high-performance NCCL API extension for efficient Mixture-of-Experts (MoE) communication. It provides optimized dispatch and combine primitives for Expert Parallelism (EP) across distributed GPU systems implemented on top of NCCL Device API: Load-Store Accessible (LSA) and GPU-Initiated Networking (GIN) operations.

API Improvements and Extensions

  • Refactor the API signatures to improve user experience and support backward compatibility.
  • Change the device memory ownership for EP Tensor data. The user is now responsible for device memory allocations for EP Tensors.
  • Refactor the EP tensor data structure management for the host-side NCCL EP Tensor object. EP tensor now supports both dynamic allocation for long-term storage and static on-stack allocation for malloc-free usage on the data path.
  • Add lightweight and CUDA Graph-compatible EP Handle management on the data path. ncclEpCreateHandle is split into ncclEpInitHandle, which is a control-path operation that may allocate device memory and may be collective, and ncclEpUpdateHandle, which updates the Handle's routing information before calling the Dispatch operation.
  • Allow users to set the number of SMs used by NCCL EP.
  • Extend the API to associate an NCCL EP Tensor with an NCCL Window to enable zero-copy optimizations.
  • Add flexible Dispatch output layout configurations:
    • HT mode supports Flat and Expert-major layouts.
    • Enable users to provide expert padding to align with GEMM requirements.
    • LL mode supports Expert- and Rank-major layouts.
  • Add active rank mask support to identify failed ranks and exclude them from future communication, allowing operation to continue instead of aborting the process.
  • Introduce an explicit Forward/Backward pass selector in Dispatch and Combine operations.
  • Drop top-K indices from the Dispatch operation signature and use the tensor provided to the Handle update.

Implementation Improvements

  • Migrate to Just-In-Time (JIT) compilation for HT mode. This addresses performance issues and a number of limitations. LL migration to JIT is planned in the next release.
  • Add full Multi-node-NVLINK (MNNVL) support.
  • Remove limitations on the number of ranks in an LSA team. This has been tested on NVLink72.
  • Fully migrate to NCCL infrastructure. All CUDA IPC references are removed and the code only depends on NCCL.
  • Enable CUDA Graph support through EP handle management API changes and implementation changes.
  • Support MoE and prefill workloads by enabling a variable number of tokens per sender on Dispatch.

Performance Optimizations

  • Improve the performance of Dispatch for HT mode by leveraging NCCL Device API extensions available starting from NCCL v2.30.
  • Improve the performance of the Combine operation in HT mode by leveraging JIT compilation.
  • Enable zero-copy flows for HT mode.
  • Optimize LL performance for NVLink-only configurations by avoiding the send-side staging buffer.
  • Update ep_bench to measure kernel-only performance.
  • Introduce ncclTeamRail in HT mode instead of a split communicator.
  • Improve Dispatch/recv and Combine/send parallelization in LL mode.

Memory Footprint Optimizations

  • Optimize the Dispatch staging buffer in LL mode. Use per-rank token deduplication and rank-major layout to reduce the staging buffer size by a factor of experts per rank.
  • Expose rank-major layout at the API level in LL mode. Rank-major mode reduces the memory footprint by a factor of the number of experts per rank.
  • Optimize HT mode handle memory usage by moving the global routing map buffer from the handle to the group scope. This allows different handles to share the buffer.

Python Bindings

  • Expose NCCL EP through nccl4py.
  • Make Python bindings more pythonic compared to the original 1-to-1 C-Python mapping.

Performance Benchmark (ep_bench)

  • Report kernel-only performance metrics through CUPTI, if available.
  • Extend the number of settings: number of SMs, number of experts, and layout selection.
  • Add sophisticated validation for Dispatch and Combine phases to detect memory corruption and routing issues.

Bug Fixes

  • Fix the bug in HT mode preventing launches on more than 8 nodes.
  • Fix HT mode inter-node flags sizing that would cause overflow for 9 or more nodes.
  • Clean the API and tools from quantization-related code. Quantization support is planned to be re-enabled in the following release.
  • Fix memory ordering in Dispatch/Combine grid barriers.
  • Fix a bug causing crashes in LL mode for batch sizes.
  • Fix integer overflow in inter-node N2N warp at 8 or more nodes. Thanks to Mozar Huang.

Known Issues and Limitations

  • The number of RDMA domains, or NCCL LSA Teams, in HT mode is limited to 32 due to algorithmic limitations.
  • nccl4py 0.3 wheel is shipped with libnccl_ep.so built with CUDA 13. To use CUDA 12, users have to build libnccl_ep.so from source and specify the .so file path using LD_PRELOAD or LD_LIBRARY_PATH. In addition, NCCL_EP_HOME needs to be set to point to the corresponding nccl_ep installation directory.
  • NCCL EP v0.1 does not support quantization. While the API has appearances of quantization-related parameters, such as the scales tensor, the implementation was not tested and is not guaranteed to work. Elements of quantization support are expected to be introduced in the next release.
  • The Dispatch operation has resource limitations associated with the amount of available shared on-chip memory. Consumption is impacted by two factors:
    • The hidden dimension of the token.
    • LSA team size, which is the size of the NVLink domain.
  • If a job launch is aborted due to shared memory overflow, try to reduce the current stage or pipeline settings. In v0.1, this can only be done statically at build time: reduce the HYBRIDEP_DISPATCH_NUM_OF_STAGES and/or HYBRIDEP_DISPATCH_NUM_OF_PIPELINES_PER_BLOCK macro values in hybridep_configs.cuh, rebuild, and retry.

LL Mode Limitations

  • Maximum top-K: 9.
  • Hidden dimensions: 2048, 2560, 4096, 5120, 6144, 7168, and 8192.

Known Bugs

  • In LL mode, ep_bench reports Combine verification failure when a batch size of 1 token is used.
  • In LL mode, ep_bench reports Combine verification failures on 16 nodes and 64 GPUs for the batch size of 8K tokens.

NCCL v2.30.7-1 Release

Choose a tag to compare

@bhramesh-nvidia bhramesh-nvidia released this 04 Jun 22:33
73cf112

Zero-SM Collectives

  • Adds hierarchical zero-SM collectives (AllGather and All2all) that use RMA CPU proxy for inter-node communication and Copy Engines for intra-node communication.
  • Enables better overlap of compute and communication.
  • Enable hierarchical zero-SM collectives with NCCL_CTA_POLICY_ZERO flag.

GIN Enhancements

  • Adds new experimental GPU Push Interface (GPI) backend for GIN.
  • Adds explicit signal semantics with Strong and Weak signals.
  • Adds proper ncclGinFenceLevel semantics for barriers.
  • Adds separate NCCL_GIN_IB_TC toggle to control traffic class used by GIN.
  • Adds NCCL_GIN_RESOURCE_SHARING_THREAD to enable more optimizations.
  • Optimizes QP overhead, including GDAKI mode when counters are not used.
  • Ensures GIN is usable when NIC fusion is enabled.
  • Adds GIN plugin example in plugins/gin/example.

Symmetric Memory Improvements

  • Restructures RMA plugin architecture.
  • Adds support for asymmetric buffer sizes during window registration.
  • Optimizes ReduceScatter symmetric kernel performance.
  • Optimizes performance for RMA operations using CE.
  • Adds batched CE operations to improve performance in the RMA CE put/wait path.
  • Adds support for window registration during CUDA graph capture.

MPS with MLOPart Support (Experimental)

  • NCCL now leverages CUDA feature Memory Locality Optimized Partition (MLOPart).
  • Supports up to 2 ranks per physical GPU with MPS+mlopart.

Other Improvements

  • Adds support for IB ports that require global route headers (GRH).
  • Adds logic to gin.flush to ensure all prior gets are visible.
  • Adds makefile support to compile python wheels from source.
  • Adds NCCL_RMA_DISABLE env to enable/disable RMA (Github PR #2151).
  • Implements reset-without-zeroing for signals and counters in GIN (Github PR #2155).
  • Pins GIN proxy thread to NUMA-local CPU set (Github PR #2182).
  • Adds optimized weight transfer APIs in contrib/nccl_xfer.
  • Adds custom kernels in contrib/custom_algos for alltoall and allreduce using NCCL Device API.
  • Adds examples of Root Mean Square Normalization (RMSNorm), demonstrating the fusion of computation and communication using the device API.
  • Unifies coding style by using clang-format. Please see docs/dev_guide/nccl_coding_style.md for more details.
  • Drops support for v11 and v12 GIN plugin APIs.

Bug Fixes

  • Fixes a deadlock caused by cuda stream allocation under PXN when memseting a buffer at runtime.
  • Reintroduce cudaGridDependencySynchronize in built-in symmetric kernels, ensuring that newly launched kernels cannot access memory modified by prior kernels before it reaches point of coherency.
  • Ignores system headers in include/header processing, thereby avoiding excessive realpath calls in some builds (Github PR #1806).
  • Improves QP load balancing on systems configured with RoCE LAG with the round-robin queue affinity policy (Github PR #2150).
  • Fixes issue when receiving an external TCP request causes the proxy thread's ncclProxyService to hang (Github PR #1834).
  • Fixes rma_proxy MR registration type for host-NUMA cpuAccessSignals, which ensures that the net plugin does not reject the registration due to wrong memory type (Github PR #2187).
  • Fixes GIN init context leak (Github PR #2179).
  • Fixes issue with one-sided host APIs when a custom GIN plugin is used.
  • Fixes one-sided host API issue where requests are dropped at a high message rate (Github Issue #2119).

Acknowledgements

We thank the following contributors for their work on this release:

@andrewjcg, @baymaxhuang, @bhasunit, @fishautumn, @mozarhua, @ngoyal2707, @wanglei875 for your PRs.

We also thank the community for issue reports, testing, and feedback.

Known Issues

  • NCCL one-sided host RMA APIs, e.g., ncclPutSignal, require every rank to call the API as a one-time initialization warm-up. This has been fixed on dev branch: e12963e
  • NCCL one-sided RMA operations have a possible corruption issue when multiple symmetric windows are carved from the same backing memory allocation. See #2198. This has been fixed on dev branch.
  • Hang is possible when specific workload uses PXN (e.g. ncclSend/ncclRecv/All2all) and calls ncclCommDestroy() without waiting for all ranks to complete. This has been fixed on dev branch: 08b3662
  • Hang is possible when workload uses 2-rank communicator and uses CE collectives. This has been fixed on dev branch: 430afbc

nccl4py v0.2.0 Release

Choose a tag to compare

@xiakun-lu xiakun-lu released this 24 Apr 21:33
nccl4py-v0.2.0

Release Notes — nccl4py 0.2.0

This release adds Python bindings for the new NCCL 2.30 one-sided RMA, Device API (GIN), and elastic communicator features, along with substantially more control over communicator configuration.

Highlights

  • One-sided RMA (point-to-point) — New Communicator.put_signal(), Communicator.signal(), and Communicator.wait_signal() methods, plus a WaitSignalDesc helper for describing signal values and match operations.
  • NCCL Device API host side setup — New Communicator.create_dev_comm() that produces a DevCommResource for use with device-side NCCL kernels. Configure the device communicator through the new NCCLDevCommRequirements class, and introspect support via device_api_support, gin_type, railed_gin_type, host_rma_support, and n_lsa_teams properties.
  • Device pointer access for registered windows — RegisteredWindowHandle now exposes user_ptr, get_lsa_device_pointer(), get_lsa_multimem_device_pointer(), and get_peer_device_pointer() for direct access to LSA, multimem, and peer mappings.
  • Elastic and fault-tolerant communicators — New Communicator.grow(), revoke(), suspend(), and resume() methods to support elastic topology changes and error-handling flows. CommSuspendFlag added alongside existing CommShrinkFlag.
  • More flexible construction — In addition to init(), communicators can now be created with class method init_all() and instance method initialize(). Communicator.get_mem_stat() reports per-communicator memory statistics.

Configuration

New tuning knobs on NCCLConfig:

  • graph_usage_mode, num_rma_ctx, max_p2p_peers.

NCCLDevCommRequirements — passed to Communicator.create_dev_comm() to describe the resources and capabilities a device communicator needs:

  • LSA: lsa_multimem, barrier_count, lsa_barrier_count, rail_gin_barrier_count, world_gin_barrier_count, lsa_ll_a2a_block_count, lsa_ll_a2a_slot_count.
  • GIN: gin_force_enable, gin_context_count, gin_signal_count, gin_counter_count, gin_queue_depth, gin_connection_type, gin_exclusive_contexts.

Device / topology introspection

New Communicator properties: cuda_dev, nvml_dev, device_api_support, multimem_support, gin_type, railed_gin_type, n_lsa_teams, host_rma_support.

Other changes

  • CTAPolicy is now an IntFlag (was IntEnum) so multiple policies can be combined.
  • Interop submodules nccl.core.cupy and nccl.core.torch are now lazy-loaded via __getattr__ and only imported on first attribute access, so import nccl.core no longer pulls in CuPy or PyTorch.