Skip to content

AWS OFI NCCL v1.20.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 25 Jun 04:54
v1.20.0
This tag was signed with the committer’s verified signature.
bibrakc Bibrak Qamar
GPG key ID: 4A0F230AEF407203
Verified
Learn about vigilant mode.
a2a6d08

v1.20.0 (2026-06)

The 1.20.0 release series has been tested with NCCL v2.28.9-1, NCCL v2.29.7-1, and NCCL v2.30.4-1 while maintaining backward compatibility with older NCCL versions (NCCL v2.17.1 and later).

This release has been tested with Libfabric v2.4.0amzn5.0. The plugin requires at least Libfabric v1.11.0, and compiling AWS-specific support requires at least Libfabric v1.22.0.

New Features:

  • Multi-receive support: Added support for NCCL multi-receive, which improves alltoall performance. This optimization is enabled automatically and requires no user configuration changes. In previous releases, this feature required setting NCCL_OFI_EAGER_MAX_SIZE=-1 (which disabled eager sends); this restriction has been removed.
  • NCCL v2.30 support: Added support for the NCCL net v12 plugin interface introduced in NCCL v2.30. Older NCCL versions (v2.17.1 and later) continue to work via backward-compatible plugin interfaces.
  • NGC container compatibility: Added a build option (--enable-ngc-v1-compat) to produce plugin libraries compatible with NVIDIA NGC containers that expect the v1 plugin naming convention.

Bug Fixes and Improvements:

  • Fixed a crash that could occur when establishing a connection to a peer that has lost network access (e.g., due to resource exhaustion)
  • Fixed tuner segfault when NCCL_ALGO or NCCL_PROTO is explicitly set
  • Removed tuner v1 plugin support
  • Fixed an issue where errors from firmware-cancelled operations (e.g., due to MR or address handle teardown) could be silently lost, potentially causing jobs to hang instead of cleanly reporting a failure
  • Fixed a memory leak in the SendRecv protocol where one memory registration handle was leaked on every receive communicator close, which could exhaust host memory in long-running jobs
  • Fixed DMA-BUF support detection and fd retrieval on ROCm 7.x (AMD MI300 and later), which previously caused domain initialization failures
  • Fixed a hang that could occur on providers using manual progress mode (e.g., OPX/Slingshot) when the send path did not drive CQ progress, preventing inbound control messages from landing

Other Improvements:

  • Improved code reliability through modernized memory management and resource lifetime handling

Environment Variable Changes:

Variable Change Description
NCCL_OFI_EAGER_MAX_SIZE Changed behavior Multi-receive no longer requires -1 to opt in. Eager sends are now auto-detected based on libfabric provider capability (mixed_hmem_iov). Default remains 8192 (eager buffer size in bytes).

Checksum (sha512) for the release tarball aws-ofi-nccl-1.20.0.tar.gz:

2f1ea41a263b9729e730273d0f49aa18b2d295120b67f2045e61380aa64d5c04f1b15c8d2d5630fe67f5f9ac7c2a7f256efa0e76097cca90e729e812ffabea67  aws-ofi-nccl-1.20.0.tar.gz
Assets 6
Loading