Skip to content

NVIDIA Megatron-Bridge 0.5.1

Choose a tag to compare

View all tags
@nemo-automation-bot nemo-automation-bot released this 21 Jul 11:05
v0.5.1
5cb3444
Changelog Details

Features

  • Nemotron 3 Ultra conversion and training recipes.

Performance

  • Performance tuned Deepseek-V4-Pro MXFP8 GB300 recipe with DSA kernels, CuTeDSL fused GGEMM, and full iteration CUDA graphs (#4808). Note that the Megatron Core dev branch is required.
  • Deepseek-V3 GB200/GB300 large-scale proxy improvement with CuTeDSL fused grouped GEMM, and full iteration CUDA graphs (#4700)
  • Deepseek-V3 B300 recipe improvement with CuTeDSL fused grouped GEMM, and full iteration CUDA graphs (#4498)
  • Qwen3 B300/B200 recipe improvement with CuTeDSL fused grouped GEMM, and full iteration CUDA graphs (#4606)

Known issues

  • Step-3.7-Flash forward-pass outputs have not been fully verified.
  • GPT-OSS GB200/GB300 MXFP8 regression due to regressed CuTeDSL fused kernels (cudnn-frontend/#256), expect to be fixed in NeMo 26.08 with cudnn-frontend v1.26
  • Some MoE training configurations that combine tensor parallelism and expert parallelism may run slower in 26.06 after upgrading from NCCL 2.29 to NCCL 2.30.
    • Root cause: NCCL 2.30 fixed a CPU-affinity leak and now correctly restores the launcher's original CPU affinity after communicator initialization. Earlier NCCL versions could inadvertently leave application threads bound to CPUs local to each GPU. Training launchers without explicit CPU and memory binding may therefore expose cross-NUMA scheduling overhead after the upgrade.
    • Workaround: As a workaround, bind each training rank and its host-memory allocations to the NUMA node local to its assigned GPU: numactl --cpunodebind=<NUMA_NODE> --membind=<NUMA_NODE> <training command>. The GPU-local NUMA node can be determined programmatically from the GPU's PCI bus ID. For Slurm or torchrun launchers, the training command can be wrapped as follows:
Code 
LID=${SLURM_LOCALID}
PCI_BUS=$(nvidia-smi -i $LID --query-gpu=pci.bus_id --format=csv,noheader 2>/dev/null | head -1 | tr '[:upper:]' '[:lower:]')
NUMA_NODE=$(cat /sys/bus/pci/devices/$PCI_BUS/numa_node 2>/dev/null || echo -1)
echo "[numactl_local] rank=$LID gpu_pci=$PCI_BUS numa=$NUMA_NODE"
exec numactl --cpunodebind=$NUMA_NODE --membind=$NUMA_NODE "$@"
Assets 2
Loading