You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Features
Dataloaderprefetch_factor — new prefetch_factor option on the dataloader config lets workers buffer more than PyTorch's default of two batches. Full-iteration CUDA graphs need this so a whole step of microbatches is loaded before the graph replays (PR#6511).
Nemotron 3 Super GB200 fine-tuning recipes — dedicated GB200 SFT and PEFT recipes with GB200-tuned defaults. This resolves the 0.7.0 known issue where the published GB200 SFT/PEFT commands used H100 HybridEP topology settings (PR#6690).
Performance
Devices 26.10.01 NeMo container was tested on include B200, B300, GB200, GB300 and VRNVL72 across dense, MoE’s models and BF16, FP8, MXFP8 and NVFP4 precisions (depending on model).
VRNVL72 — preliminary support for all benchmarks and precisions as supported on GB300.
NCCL EP by default on GB200 and VR NVL72 — MoE performance recipes (DeepSeek V3, DeepSeek V4 Flash, Nemotron 3 Super, Nemotron 3.5 Lightning, Qwen3 MoE, GPT-OSS) now use the NCCL EP dispatcher with grouped tensors (PR#6373).
cuDNN normalization for GB200 / GB300 benchmarks (PR#6661)
Vera Rubin (VR NVL72) MXFP8 tuning — fused MLA KV-RoPE quantization, MXFP8 dispatch and combine, a full step of prefetched microbatches, and nondeterministic TE kernels allowed (PR#6723, PR#6985, PR#6994, PR#6766)
With vision CUDA graphs, the vision patch merger now runs on the graph-padded rows, so cuDNN LayerNorm plans are built once instead of once per new patch count (+3.3–3.5% steady-state throughput on Qwen3.5-VL 35B-A3B).
Opt-in padding mask keeps padding tokens out of MoE router statistics and HybridEP dispatch (+12% same-batch throughput with native MoE on Qwen3.5-VL 35B-A3B and 397B-A17B).
MLPerf recipes
Llama 3.1 8B pretraining on GB200 / GB300 — NVFP4 and FP8-CS, 8 to 512 GPUs (PR#6475)
Llama 2 70B LoRA on GB200 / GB300, with a Llama 2 70B PEFT recipe and FP8 delayed-scaling CLI support (PR#6476)
GPT-OSS 20B configs restored after the 26.08 perf recipe refactor (PR#6473)
DeepSeek V3 pretraining on VRNVL72 — MXFP8, throughput is 3301 TFLOPs/sec/GPU
Known Issues
Evaluator tutorial: WikiText failure with tensor parallelism. The WikiText evaluator tutorial may fail when using Ray in-framework deployment with tensor parallelism greater than 1 (TP > 1). Workaround: Set TP=1 or use the Triton backend.
Changelog Details
beep boop 🤖: Bumping Megatron-Bridge to v0.7.1 by @nemo-automation-bot[bot] :: PR: #6730
perf(recipes): enable NCCL EP for B200 and B300 Lightning BF16 and MXFP8 (6567) by @svcnvidia-nemo-ci :: PR: #6706
fix(training): fix local CUDA graph deletion by explicitly clearing local CG runners (6681) by @svcnvidia-nemo-ci :: PR: #6701
perf(recipe): use NCCL EP for GPT-OSS 120B BF16 on B200 and B300 (6542) by @svcnvidia-nemo-ci :: PR: #6545