Skip to content

NVIDIA Megatron-Bridge 0.6.2

Latest
Latest

Choose a tag to compare

View all tags
@nemo-automation-bot nemo-automation-bot released this 18 Sep 23:54
v0.6.2
c0e164e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

  • The GCP NCCL plugin package is now installed in the container. On Google Cloud Platform hosts (A3 Ultra, A4, A4X and A4X Max machine types), the container automatically detects and activates the GCP NCCL network, tuner, and environment plugins for optimized multi-node communication.
  • Experimental NCCL EP backend support for Hopper and Blackwell, with GPT-OSS and Nemotron benchmark examples (#5887).

Performance

  • Nemotron 3.5 Lightning BF16, MXFP8, and NVFP4 performance recipes for B200/B300/GB300, including a GB300 MXFP8 FSDP variant (#5734, #5921).
  • Nemotron 3 Ultra NVFP4 performance recipes for 256-GPU GB200/GB300 systems with FSDP/HSDP and activation offloading (#6080).
  • DeepSeek-V4-Flash MXFP8 B200/B300 performance recipes and GB200/GB300 expert-parallelism and batch-size tuning (#5902, #5867). Note that the Megatron Core dev branch (SHA 2b31f24598a0ef9a9e871bd82ffcdd2f61ca38af) is required.
  • Nemotron 3 Nano and Super GB200/GB300 MXFP8 recipe improvements with CuTeDSL fused grouped MLP and FP8 attention (#5632).
  • DeepSeek-V3 GB300 NVFP4 recipe improvements with CuTeDSL fused grouped MLP, MLA fusion, FP8 attention, and full-iteration CUDA graphs (#5764).
  • Qwen3-235B NVFP4 B200/B300/GB200/GB300 recipe improvements with CuTeDSL fused grouped MLP, HybridEP communication overlap, and full-iteration CUDA graphs (#6039).

Known issues

  • SPCX grouped all-to-all payload corruption at high expert parallelism: in the affected NCCL 2.30.5/SPCX stack, stale or uninitialized ncclIbRequest.send.onlyWriteImm state can misclassify sends in grouped variable-split BF16 all-to-all operations, causing payload loss or corruption. MCore's MoE alltoall dispatcher can expose this issue at high EP configurations, but the fault originates in the SPCX transport rather than the model or router. As a workaround, disable the external NCCL network plugin and use NCCL's internal IB transport (NCCL_NET_PLUGIN=none NCCL_NET=IB), or use the HybridEP dispatcher (issue #5462).
  • Step-3.7-Flash checkpoint parity is not yet verified.
  • Large-model round-trip validation can exceed the NCCL process-group timeout: every rank participates in export, but rank 0 lazily loads and exhaustively compares the original Hugging Face tensors while peers may reach the synchronization collective first. This expected rank skew can exceed the default 600-second timeout even though rank 0 is progressing. Set --distributed-timeout-minutes above the worst-case verification time; for checkpoints too large for practical exhaustive validation, use separate import and export --distributed-save workflows instead of roundtrip (NVIDIA-NeMo/Megatron-Bridge#5491).
Changelog Details

Contributors

bhargaveede, balasaajay, and 7 other contributors
Assets 2
Loading