Skip to content

NVIDIA Megatron-Bridge 0.7.1

Latest

Choose a tag to compare

@nemo-automation-bot nemo-automation-bot released this 20 Nov 00:46
e5e4889

Features

  • Dataloader prefetch_factor — new prefetch_factor option on the dataloader config lets workers buffer more than PyTorch's default of two batches. Full-iteration CUDA graphs need this so a whole step of microbatches is loaded before the graph replays (PR#6511).
  • Nemotron 3 Super GB200 fine-tuning recipes — dedicated GB200 SFT and PEFT recipes with GB200-tuned defaults. This resolves the 0.7.0 known issue where the published GB200 SFT/PEFT commands used H100 HybridEP topology settings (PR#6690).

Performance

  • Devices 26.10.01 NeMo container was tested on include B200, B300, GB200, GB300 and VRNVL72 across dense, MoE’s models and BF16, FP8, MXFP8 and NVFP4 precisions (depending on model).
  • VRNVL72 — preliminary support for all benchmarks and precisions as supported on GB300.
  • NCCL EP by default on GB200 and VR NVL72 — MoE performance recipes (DeepSeek V3, DeepSeek V4 Flash, Nemotron 3 Super, Nemotron 3.5 Lightning, Qwen3 MoE, GPT-OSS) now use the NCCL EP dispatcher with grouped tensors (PR#6373).
  • NCCL EP on B200 / B300 with grouped tensors:
    • Nemotron 3.5 Lightning BF16 / MXFP8 (PR#6567)
    • GPT-OSS 120B BF16 (PR#6542)
    • Nemotron 3 Super BF16 / MXFP8 / NVFP4, with retuned parallelism, recompute and CUDA graph scopes (PR#6718)
    • Qwen3 MoE (PR#6722)
  • DeepSeek V3
    • cuDNN normalization for GB200 / GB300 benchmarks (PR#6661)
    • Vera Rubin (VR NVL72) MXFP8 tuning — fused MLA KV-RoPE quantization, MXFP8 dispatch and combine, a full step of prefetched microbatches, and nondeterministic TE kernels allowed (PR#6723, PR#6985, PR#6994, PR#6766)
  • Qwen3-VL training speedups (PR#6496)
    • With vision CUDA graphs, the vision patch merger now runs on the graph-padded rows, so cuDNN LayerNorm plans are built once instead of once per new patch count (+3.3–3.5% steady-state throughput on Qwen3.5-VL 35B-A3B).
    • Opt-in padding mask keeps padding tokens out of MoE router statistics and HybridEP dispatch (+12% same-batch throughput with native MoE on Qwen3.5-VL 35B-A3B and 397B-A17B).
  • MLPerf recipes
    • Llama 3.1 8B pretraining on GB200 / GB300 — NVFP4 and FP8-CS, 8 to 512 GPUs (PR#6475)
    • Llama 2 70B LoRA on GB200 / GB300, with a Llama 2 70B PEFT recipe and FP8 delayed-scaling CLI support (PR#6476)
    • GPT-OSS 20B configs restored after the 26.08 perf recipe refactor (PR#6473)
    • DeepSeek V3 pretraining on VRNVL72 — MXFP8, throughput is 3301 TFLOPs/sec/GPU

Known Issues

  • Evaluator tutorial: WikiText failure with tensor parallelism. The WikiText evaluator tutorial may fail when using Ray in-framework deployment with tensor parallelism greater than 1 (TP > 1). Workaround: Set TP=1 or use the Triton backend.
Changelog Details
  • beep boop 🤖: Bumping Megatron-Bridge to v0.7.1 by @nemo-automation-bot[bot] :: PR: #6730
  • perf(recipes): enable NCCL EP for B200 and B300 Lightning BF16 and MXFP8 (6567) by @svcnvidia-nemo-ci :: PR: #6706
  • fix(training): fix local CUDA graph deletion by explicitly clearing local CG runners (6681) by @svcnvidia-nemo-ci :: PR: #6701
  • perf(recipe): use NCCL EP for GPT-OSS 120B BF16 on B200 and B300 (6542) by @svcnvidia-nemo-ci :: PR: #6545
  • Add MLPerf Llama 3.1 8B Recipes (6475) by @svcnvidia-nemo-ci :: PR: #6727
  • 26.10 perf summary page (6734) by @svcnvidia-nemo-ci :: PR: #6738
  • docs(perf): refresh Qwen GB300 MXFP8 benchmark results (6740) by @svcnvidia-nemo-ci :: PR: #6741
  • perf(recipe): default GB200 and VRNVL72 MoE perf recipes to NCCL EP and grouped tensors (6373) by @malay-nagda :: PR: #6747
  • Restore MLPerf GPTOSS 20B configs after 26.08 perf recipe refactor (6473) by @svcnvidia-nemo-ci :: PR: #6708
  • feat(model)!: pass a padding mask to Qwen3-VL MoE layers and run the vision merger on graph-padded rows (6496) by @malay-nagda :: PR: #6758
  • perf(recipe): enable cuDNN normalization for DeepSeek-V3 E2E benchmarks (6661) by @svcnvidia-nemo-ci :: PR: #6765
  • fix(ckpt): support MFSDP v2 MoE checkpoint resume (6469) by @svcnvidia-nemo-ci :: PR: #6602
  • perf(recipes): tune Nemotron 3 Super B200 and B300 NCCL-EP configs (6718) by @svcnvidia-nemo-ci :: PR: #6807
  • perf(recipe): backport Qwen3 B200/B300 NCCL EP to r0.7.0 by @rhmukundan :: PR: #6815
  • cp: backport GB200 finetuning defaults to r0.7.0 by @svcnemo-autobot :: PR: #6783
  • fix(container): require patched framework dependencies on r0.7.0 by @balasaajay :: PR: #6805
  • fix(docker): patch Go HTTP and fsspec vulnerabilities on r0.7.0 by @balasaajay :: PR: #6860
  • cp: Add MLPerf Llama 2 70B LoRA recipes to r0.7.0 by @svcnemo-autobot :: PR: #6901
  • fix(docker): backport TransformerEngine#3748 onto TE 2.21.0 on r0.7.0 by @malay-nagda :: PR: #6907
  • perf(recipe): port Rubin DeepSeek V3 MXFP8 tuning to r0.7.0 by @malay-nagda :: PR: #6906
  • fix(config): reject checkpoint downloader targets (6929) by @svcnvidia-nemo-ci :: PR: #6935
  • fix(container): pin grpcio-reflection for Ray Serve on r0.7.0 by @balasaajay :: PR: #6992
  • fix(docker): remove unused Ray Java archive on r0.7.0 by @balasaajay :: PR: #6980
  • fix(docker): install libdw headers for vLLM wheel build by @balasaajay :: PR: #6997
  • feat(data): add prefetch_factor to the dataloader config (6511) by @dingqingy-nv :: PR: #6976
  • fix(perf): keep the recipe's create_attention_mask when --data mock rebuilds the dataset (6802) by @dingqingy-nv :: PR: #6977
  • fix(perf): allow nondeterministic TE kernels for DeepSeek-V3 VRNVL72 … by @rsalagame-nvidia :: PR: #6994
  • perf(recipe): prefetch a full step of DeepSeek V3 VRNVL72 microbatches (6985) by @dingqingy-nv :: PR: #6986
  • fix(perf): adjust Rubin recipes and allow nondeterministic dprob kernels (6766) by @malay-nagda :: PR: #6999
  • fix(docker): pin SentencePiece for protobuf 6 compatibility by @balasaajay :: PR: #7012
  • fix(docker): require patched aiohttp on r0.7.0 by @balasaajay :: PR: #7015
  • docs: align r0.7.0 docs and latest pointer with 0.7.1 by @balasaajay :: PR: #7044
  • beep boop 🤖: Bumping Megatron-Bridge to v0.7.1 [skip ci] by @github-actions[bot]
  • perf(recipes): enable NCCL EP for B200 and B300 Lightning BF16 and MXFP8 (6567) (#6706) by @svcnvidia-nemo-ci
  • fix(training): fix local CUDA graph deletion by explicitly clearing local CG runners (6681) (#6701) by @svcnvidia-nemo-ci
  • perf(recipe): use NCCL EP for GPT-OSS 120B BF16 on B200 and B300 (6542) (#6545) by @svcnvidia-nemo-ci
  • Add MLPerf Llama 3.1 8B Recipes (6475) (#6727) by @svcnvidia-nemo-ci
  • 26.10 perf summary page (6734) (#6738) by @svcnvidia-nemo-ci
  • docs(perf): refresh Qwen GB300 MXFP8 benchmark results (6740) (#6741) by @svcnvidia-nemo-ci
  • perf(recipe): default GB200 and VRNVL72 MoE perf recipes to NCCL EP and grouped tensors (6373) (#6747) by @malay-nagda
  • Restore MLPerf GPTOSS 20B configs after 26.08 perf recipe refactor (6473) (#6708) by @svcnvidia-nemo-ci
  • feat(model)!: pass a padding mask to Qwen3-VL MoE layers and run the vision merger on graph-padded rows (6496) (#6758) by @malay-nagda
  • perf(recipe): enable cuDNN normalization for DeepSeek-V3 E2E benchmarks (6661) (#6765) by @svcnvidia-nemo-ci
  • fix(ckpt): support MFSDP v2 MoE checkpoint resume (6469) (#6602) by @svcnvidia-nemo-ci
  • perf(recipes): tune Nemotron 3 Super B200 and B300 NCCL-EP configs (6718) (#6807) by @svcnvidia-nemo-ci
  • perf(recipe): backport Qwen3 B200/B300 NCCL EP to r0.7.0 (#6815) by @rhmukundan
  • cp: backport GB200 finetuning defaults to r0.7.0 (#6783) by @svcnemo-autobot
  • fix(container): require patched framework dependencies on r0.7.0 (#6805) by @balasaajay
  • fix(docker): patch Go HTTP and fsspec vulnerabilities on r0.7.0 (#6860) by @balasaajay
  • cp: Add MLPerf Llama 2 70B LoRA recipes to r0.7.0 (#6901) by @svcnemo-autobot
  • fix(docker): backport TransformerEngine#3748 onto TE 2.21.0 on r0.7.0 (#6907) by @malay-nagda
  • perf(recipe): port Rubin DeepSeek V3 MXFP8 tuning to r0.7.0 (#6906) by @malay-nagda
  • fix(config): reject checkpoint downloader targets (6929) (#6935) by @svcnvidia-nemo-ci
  • fix(container): pin grpcio-reflection for Ray Serve on r0.7.0 (#6992) by @balasaajay
  • fix(docker): remove unused Ray Java archive on r0.7.0 (#6980) by @balasaajay
  • fix(docker): install libdw headers for vLLM wheel build (#6997) by @balasaajay
  • feat(data): add prefetch_factor to the dataloader config (6511) (#6976) by @dingqingy-nv
  • fix(perf): keep the recipe's create_attention_mask when --data mock rebuilds the dataset (6802) (#6977) by @dingqingy-nv
  • fix(perf): allow nondeterministic TE kernels for DeepSeek-V3 VRNVL72 … (#6994) by @rsalagame-nvidia
  • perf(recipe): prefetch a full step of DeepSeek V3 VRNVL72 microbatches (6985) (#6986) by @dingqingy-nv
  • fix(perf): adjust Rubin recipes and allow nondeterministic dprob kernels (6766) (#6999) by @malay-nagda
  • fix(docker): pin SentencePiece for protobuf 6 compatibility (#7012) by @balasaajay
  • fix(docker): require patched aiohttp on r0.7.0 (#7015) by @balasaajay
  • docs: align r0.7.0 docs and latest pointer with 0.7.1 (#7044) by @balasaajay