Skip to content

NVIDIA Megatron-Bridge 0.7.0

Latest
Latest

Choose a tag to compare

View all tags
@nemo-automation-bot nemo-automation-bot released this 30 Oct 03:39
v0.7.0
04ce2b3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

Model Collection Support

LLM

  • DeepSeek V4 — enhanced support with MCore HybridModel on main branch; library configs modernized; multi-node inference launcher (PR#5860, PR#5554, PR#6163)
  • GLM-5.2 — enhanced support with MCore HybridModel, with shared GLM-5.2/5.3 bridge compatibility (PR#6158, PR#6117)
  • Kimi K3 — language backbone (PR#5130)
  • Muse Glimmer 30B (PR#5479, PR#6255)
  • K-EXAONE 2.0 (PR#5191, community @eagle705)
  • Muse Spark Open 816B-A42B and Beam 1.0 501B-A23B are on main, not in 0.7.0. Verified with Megatron Bridge main in the NeMo 26.10 container plus Transformers 5.20.1; see their verification cards.

VLM / Multimodal

  • Nemotron 3.5 Super VL — support, text-only conversion, MoE fine-tuning state and verification refresh (PR#6100, PR#6130, PR#6220)
  • BAGEL-7B-MoT — T2I / editing / VLM WebDataset conversion, BAGEL MIMO construction, flow-matching schedule, H100 pretrain and finetune recipes (PR#5605, community @ympcMark)

Diffusion

Training & Functionality

  • Multi-LoRA and MoE PEFT
  • SFT data: in-batch packing and multi-source blending — GPT SFT in-batch packing (PR#5493), native Energon packing for Qwen3-VL (PR#5285), direct-HF and JSONL blending (PR#5438, PR#5951), DeepSeek chat collation (PR#5519, PR#5535, PR#5541), CoderForge preset (PR#5610), DataComp Energon prep (PR#5087), custom Energon task encoders (PR#6141)
  • Deterministic training — --deterministic now reuses Megatron-Core determinism setup (PR#6439). A new apply_determinism_overrides function applies the model overrides and runtime environment defaults needed for bit-exact reproducibility to any recipe, and is importable from megatron.bridge.recipes.utils (PR#6440). Expansion to more recipes is coming in the next release.
  • MegatronMIMO — in-batch sequence packing (PR#5131), scalable canonical-grid reads (PR#5525), intra-microbatch reordering (PR#6306, community @going-song)

Checkpoint Conversion

  • Low-memory, distributed, and quantized conversion/export
    • Lower GPU memory for import/export (PR#5154); distributed CPU export (PR#5830)
    • Local refit views (PR#5776), local BF16 export iterator (PR#6005), source Megatron param names on exports (PR#6074)
    • Native MXFP8 export tasks and param export (PR#5917, PR#5868)
    • Pipeline-stage-local HF export (PR#6291, community @Bsgg1)

Performance

  • Vera Rubin (VR200 NVL72) performance recipes — the NeMo 26.10 container is an early-access container for Vera Rubin support. Preliminary performance recipes for Vera Rubin NVL72 live under the vrnvl72 namespace and are launched through the flat perf-recipe names, e.g. train.sh --recipe <model>_pretrain_<N>gpu_vrnvl72_<precision>_config. Pretrain coverage:
    • LLM: Llama 3 / 3.1, DeepSeek V3, DeepSeek V4 Flash, GPT-OSS, Qwen3 MoE, Nemotron 3 Super / Ultra, Nemotron 3.5 Lightning
    • VLM: Qwen3.5-VL
    • Precisions: BF16, FP8, MXFP8, NVFP4, depending on model
  • Nemotron 3 Ultra — performance updates (PR#6437); GTP, partial CUDA graphs and NCCL EP for NVFP4 on GB200 / GB300 / VRNVL72 (PR#6478); deterministic GB200 MXFP8 perf recipe (PR#6635); H100 SFT now runs on the documented 48-node layout (PR#6423)
  • Qwen3.5-VL — cuDNN GDN for GB300 GDN layers (PR#6330); 8-GPU GB300 MXFP8 pretrain recipe for 35B-A3B (PR#6433); updated VRNVL72 35B/397B MXFP8 recipes (PR#6455) and 397B GB300 MXFP8 recipe (PR#6633)
  • Qwen3.5 long-context SFT — text MXFP8 recipe (PR#6300); VL GB200 recipes (PR#6324)
  • NCCL EP — GB300 MoE perf recipes default to NCCL EP (PR#6171), GPT-OSS GB200 (PR#6261). HybridEP deprecation and full migration are deferred.
  • GB300 MXFP8 proxy recipes (PR#6604)

Examples & Documentation

Breaking Changes & Removals

  • Legacy model families removed — the families deprecated in 26.08 are removed in 0.7.0 (PR#6370). Loading them now raises an error pointing to Megatron Bridge 0.6.x; pin megatron-bridge<0.7.0 if you still need them.
    • DeepSeek V2 and V2 Lite
    • Gemma 1 and Gemma 2
    • Llama 2
    • Mistral 7B and Mistral Small 3 24B
    • Llama Nemotron (70B, Nano 8B/4B, Super, Ultra)
    • Nemotron H v1 and Nemotron Nano v2
    • Nemotron Nano v2 VL
    • Legacy Nemotron (including Nemotron-4 340B)
  • Nemotron example directories flattened — script paths changed (PR#6371).

Community Contributions

  • @linmuchuiyang — DeepSeek V4 multi-node inference launcher
  • @bbuschkaemper — BERT bridge
  • @ympcMark — BAGEL-7B-MoT
  • @eagle705 — K-EXAONE 2.0
  • @BenDGeist — DiffusionGemma
  • @Bsgg1 — pipeline-stage-local HF export
  • @going-song — MegatronMIMO intra-microbatch reordering
  • @j316chuck — fused expert LoRA output merge
  • @mathewjhan — multi-LoRA for Megatron models
  • @yushengsu-thu — multi-LoRA on grouped MoE expert linears, seeded slot re-initialization
  • @HollowMan6 — LoRALinear wrap, merge-before-quantize, expand_shared_outer export, LoRA+ LR split, 3D-MoE LoRA export

A big thank you to our community contributors for their valuable support!

Known Issues

  • Nemotron Super text recipes on GB200: Published SFT and PEFT commands may fail because of incorrect H100 HybridEP topology settings. A fix is in PR #6690.
  • LoRA+ warmup: Unequal A/B learning rates require lr_warmup_init=0. Configurations combining a nonzero warmup initial learning rate with lora_plus_ratio != 1 are unsupported and rejected.
  • BAGEL: Support remains experimental. Training recipes may fail with incompatible Megatron-Bridge and Megatron-Core versions.
  • All other known issues from 26.08 are resolved in the 26.10 release.
  • There is a known issue with Export-Deploy in the 26.10 release: deploy_ray_inframework.py fails during initialization when tensor parallelism (TP) is greater than 1. The rank-1 ModelWorker exits due to an assertion failure in generate_other_ranks() → _assert_primary. The fix in PR #782 is approved but not yet merged and is targeted for the upcoming patch release. In the meantime, users can apply the PR, which has been validated with the WikiText tutorial.
Changelog Details
  • fix(model): keep pretrained GLM-5 router bias fixed by @hershg :: PR: #6036
  • [build] chore: bump MCore main to d476c21 by @cuichenx :: PR: #6160
  • fix(training): restore gradient synchronization for copied model configs by @aroshanghias-nvd :: PR: #6067
  • reuse save_tokenizer_assets from MLM by @dimapihtar :: PR: #5532
  • fix(ci): repair wheel packaging and release documentation by @cuichenx :: PR: #6167
  • feat(model): add Nemotron 3.5 Super VL support by @cuichenx :: PR: #6100
  • build: bump MCore main to bde6af2 by @yaoyu-33 :: PR: #6169
  • fix(moe): require explicit dispatcher backend selection by @cuichenx :: PR: #6150
  • feat(ckpt): add nemotron super vl text-only conversion by @cuichenx :: PR: #6130
  • build: bump MCore main to c711dc0 by @cuichenx :: PR: #6173
  • fix(data): chunked materialization with fork-based parallelism by @kenkenpa2126 :: PR: #5869
  • build: pin flashinfer-python in final image by @balasaajay :: PR: #6178
  • ci: skip PyNvVideoCodec in driverless import check by @balasaajay :: PR: #6181
  • fix(peft): expose gather_output on output-layer substitutes by @lauradang :: PR: #6180
  • [build] chore: bump MCore main to c343bb6d3 by @cuichenx :: PR: #6179
  • fix(nemotron-omni): forward padding mask to language model by @rohitrango :: PR: #6113
  • fix(training): allow context parallelism with MFSDP v2 by @wujingyue :: PR: #6176
  • ci: bump preflight workflows to v1.11.0 by @ko3n1g :: PR: #6193
  • 1f1b cuTE DSL fusion full-iter CG | rubin- NT3 ultra, DSv4 flash by @malay-nagda :: PR: #6046
  • build: bump MCore main to 6a366090 by @cuichenx :: PR: #6191
  • fix(training): propagate loss mask to MTP forward for gpt_step by @zhongbozhu :: PR: #6175
  • Fix(qwen3vl): finalize TE precision after decoder replacement by @vasunvidia :: PR: #6140
  • fix(model): preserve Gemma 4 image attention across pipeline stages by @cuichenx :: PR: #6187
  • fix(data): preserve nested payloads when splitting HF datasets by @cuichenx :: PR: #6182
  • feat(recipe): Enable Vision Encoder CUDA Graph for QWEN3.5 35B and 397B pretrain recipe by @tomlifu :: PR: #6183
  • perf(recipe): default GB300 MoE perf recipes to NCCL EP and grouped tensors by @YangFei1990 :: PR: #6171
  • build(deps): pin mamba-ssm to the version the patched build requires by @ko3n1g :: PR: #6204
  • [build] chore: bump MCore main to 8dded20c by @cuichenx :: PR: #6202
  • docs(training): sync packed sequence guidance for Fern by @cuichenx :: PR: #6207
  • fix(qwen3-asr): support legacy configs with native Transformers by @cuichenx :: PR: #6194
  • fix(training): account packed GPT FLOPs across context parallel ranks by @zhongbozhu :: PR: #6189
  • perf(docker): parallelize TE, DeepEP, and Mamba compiles in the CI container (opt-in) by @ko3n1g :: PR: #6214
  • Add Gemma2 TransformerEngine flash-attention path with softcap (opt-in) by @nvegesna-netizen :: PR: #5628
  • build: bump MCore main to 8c487209 by @cuichenx :: PR: #6206
  • build: bump MCore main to d113016b by @cuichenx :: PR: #6217
  • [peft] fix: scale grouped-expert LoRA at the bottleneck, not the output by @cuichenx :: PR: #5913
  • feat(conversion): extract Qwen3.5 MoE text checkpoints with MTP by @cuichenx :: PR: #6185
  • feat(qwen): configure vision recomputation independently of decoder by @cuichenx :: PR: #6186
  • fix(ckpt): sync training resume state and compatibility fixes by @cuichenx :: PR: #6209
  • fix(training): correct diagnostic rank selection and repeated logs by @cuichenx :: PR: #6211
  • feat(data): support custom Energon task encoder configs by @cuichenx :: PR: #6141
  • fix(docker): preserve validated base runtime on opt-in builds by @malay-nagda :: PR: #6190
  • ci: enable Claude review for trusted fork PRs by @balasaajay :: PR: #5844
  • fix(diffusion): align WAN checkpoint prefixes by @cuichenx :: PR: #6094
  • fix(training): correct loss averages and hybrid FLOPs reporting by @cuichenx :: PR: #6210
  • fix(training): sync optimizer buffer and model setup correctness guards by @cuichenx :: PR: #6208
  • fix(data): propagate unpacked SFT padding masks to MoE routers by @cuichenx :: PR: #6188
  • fix(ckpt): preserve MoE fine-tuning state and refresh Super VL verification by @cuichenx :: PR: #6220
  • [build] chore: bump MCore main to c035a426 by @cuichenx :: PR: #6224
  • fix(ckpt): derive checkpoint replica_id from the gtp_remat-inclusive DP group by @wdykas :: PR: #6102
  • [training] fix: define optimizer chain in loss test fixture by @cuichenx :: PR: #6238
  • fix(training): give Qwen3-VL a distillation forward step for context parallelism by @kevalmorabia97 :: PR: #6230
  • ci(review): AUT-2439 migrate legacy Claude commands to /review by @svcnemo-autobot :: PR: #6231
  • [build] chore: bump MCore main to 07147d69 by @cuichenx :: PR: #6240
  • [model, recipe] feat: migrate DeepSeek V4 to HybridModel on MCore main by @guihong-nv :: PR: #5860
  • docs(model): refresh Qwen3.8-27B GB200 verification and add H100 pretrain performance by @kamran-nvidia :: PR: #6234
  • fix(diffusion): scatter WAN text embeddings for sequence parallelism by @cuichenx :: PR: #6232
  • fix(training): keep mrope position ids full-length under context parallelism by @kevalmorabia97 :: PR: #6243
  • fix(ckpt): pass GTP-inclusive group to model-only load metadata by @liding-nv :: PR: #6235
  • [build] chore: bump MCore main to b4d72b79 by @cuichenx :: PR: #6245
  • feat(recipe): add Nemotron 3 Ultra 256-GPU VR200 NVFP4 perf recipe by @malay-nagda :: PR: #6205
  • ci(review): replace automatic Claude reviews with command guidance by @ko3n1g :: PR: #6264
  • perf(recipe): use NCCL EP for GPT-OSS GB200 and remove temporary examples by @malay-nagda :: PR: #6261
  • build: bump MCore main and preserve optimizer checkpoint compatibility by @cuichenx :: PR: #6260
  • docs: clarify model support and parity investigation skills by @cuichenx :: PR: #6259
  • perf(recipe): raise Muse Glimmer 30B H100 pretrain to 205.6 TFLOP/s/GPU by @kamran-nvidia :: PR: #6255
  • fix(docs): AUT-2456 preview approved PR mirror branches by @svcnemo-autobot :: PR: #6253
  • [build] chore: bump MCore main to bab851b by @cuichenx :: PR: #6265
  • fix(recipe): point GLM-5.2 SFT card commands at library aliases by @kamran-nvidia :: PR: #6267
  • docs(model): record GLM-5.2 GB200 warm-start pretrain verification by @kamran-nvidia :: PR: #6257
  • fix(docs): remove excluded review rubric from skills index by @cuichenx :: PR: #6280
  • feat(ckpt): support MXFP8 parameter export by @yuukidach :: PR: #5868
  • fix(security): reject unsafe checkpoint config targets before import by @chtruong814 :: PR: #6246
  • refactor(recipe): rename Nemotron multimodal recipe package by @cuichenx :: PR: #6283
  • fix(peft): pack fused-expert LoRA in PEFT's ParamWrapper layout by @jerryhong21 :: PR: #6233
  • build: update Nsight Systems to 2026.5.1 and wandb to 0.29.0 by @balasaajay :: PR: #6285
  • fix(perf): fix VR200 recipes by @malay-nagda :: PR: #6212
  • build: bump MCore main to e998be072 and align TE dependencies by @cuichenx :: PR: #6277
  • fix(model): generate DSA-aligned pipeline layouts for GLM-5.2 conversion by @kamran-nvidia :: PR: #6288
  • docs: correct Nemotron verification artifact paths by @cuichenx :: PR: #6286
  • fix(misc): map --max-position-embeddings onto model.seq_length by @pruprakash :: PR: #5598
  • fix(model): default multi_latent_attention to True on the MLA config by @pruprakash :: PR: #5597
  • fix(ckpt): handle missing newer MLM checkpoint args by @caohch-1 :: PR: #5715
  • feat(training): expose GTP rematerialization runtime controls by @liding-nv :: PR: #6226
  • build: update image cuBLAS to 13.8.1 by @balasaajay :: PR: #6252
  • docs(misc): add catalog regeneration step to model verification card skill by @kamran-nvidia :: PR: #6297
  • fix(ckpt): disable shared-expert overlap in single-rank flex fallback by @kamran-nvidia :: PR: #6287
  • fix(data): reduce peak memory when building memmap indices by @kenkenpa2126 :: PR: #5850
  • docs(model): verify GLM-5.2 CPU export as a distributed export by @kamran-nvidia :: PR: #6292
  • fix(peft): preserve GPT-OSS gate/up order in adapter export by @cuichenx :: PR: #6244
  • docs(model): record GLM-5.2 H100 HybridEP warm-start pretrain and resume verification by @kamran-nvidia :: PR: #6295
  • [ci] fix: check GitHub file links through raw content by @cuichenx :: PR: #6302
  • perf(recipe): enable Nemotron 3.5 MXFP8 full graphs on GB300 and VR200 by @malay-nagda :: PR: #6289
  • feat(model): add bert support. by @bbuschkaemper :: PR: #4737
  • fix(training): count attention output gates in hybrid FLOPs by @cuichenx :: PR: #6301
  • [build] chore: bump MCore main to feba4e1e by @cuichenx :: PR: #6299
  • fix(ckpt): support GTP checkpoint groups and HF conversion by @yaoyu-33 :: PR: #6116
  • fix(models): forward layer-wise optimizer flag to DDP wrapping by @liding-nv :: PR: #6308
  • no non det algo by @malay-nagda :: PR: #6282
  • fix(perf): allow --nccl_ub false to disable NCCL UB by @jayzuccarelli :: PR: #6241
  • Support For NVCRE Executor by @sjayaram-nv :: PR: #5254
  • [build] chore: bump MCore main to ef8cd10b8 by @cuichenx :: PR: #6309
  • feat(ckpt): add Nemotron-H router replay diagnostic by @cuichenx :: PR: #6322
  • fix(recipe): select alltoall for Qwen3 235B pretrain recipe by @kamran-nvidia :: PR: #6317
  • docs(misc): list EXAONE, Kimi K3, MiniMax-M3 and Nemotron-3 Ultra as supported by @pruprakash :: PR: #5743
  • fix(build): remove stale cuDNN Frontend override from FW image by @balasaajay :: PR: #6323
  • feat(training): global-batch online packing with Megatron-Core's sequence-packing scheduler by @ilml :: PR: #6127
  • docs: list BAGEL and Muse Glimmer in supported models by @cuichenx :: PR: #6328
  • test: add copyright header to router replay test by @cuichenx :: PR: #6334
  • fix(misc): fail sync inference when a prompt returns no result by @pruprakash :: PR: #5765
  • fix(recipe): pin expert tensor parallelism on Qwen3.5-VL MoE pretrain recipes by @pruprakash :: PR: #5754
  • fix(data): apply the null-content contract to legacy chat turns by @pruprakash :: PR: #5742
  • fix(data): scope GLM assistant boundary scan to the current turn by @pruprakash :: PR: #5753
  • docs(models): correct Super and GPT-OSS verification provenance by @cuichenx :: PR: #6293
  • fix: pin FW cuDNN Frontend and complete supported model table by @balasaajay :: PR: #6329
  • build: install Mixture of Kittens in Docker on amd64 and arm64 by @balasaajay :: PR: #6290
  • [build] chore: bump MCore main to fbbe6eb5 by @cuichenx :: PR: #6333
  • docs(recipe): keep recipe functions self-contained by @kamran-nvidia :: PR: #6337
  • fix(training): count full runtime sequences in GPT FLOPs by @cuichenx :: PR: #6227
  • feat(mimo): add Nemotron Omni Megatron MIMO conversion support by @liding-nv :: PR: #6313
  • docs(model): save a BERT tokenizer in the megatron-bert README checkpoint step by @pruprakash :: PR: #6331
  • feat(recipe): add Qwen3.5 text MXFP8 long-context SFT by @cuichenx :: PR: #6300
  • build: bump MCore main to 5052ce8 by @cuichenx :: PR: #6339
  • fix(model): keep GLM-5 export max_position_embeddings from the reference config by @linmuchuiyang :: PR: #5996
  • fix(misc): migrate BERT conversion example to scripts/conversion/convert.sh by @pruprakash :: PR: #6332
  • perf(ckpt): build QKV/KV split indices on the input device by @ting-hong-shieh :: PR: #6236
  • test(model): guard verification card launcher commands by @pruprakash :: PR: #5835
  • build(docker): remove Nsight Systems EFA metrics plugin by @balasaajay :: PR: #6341
  • fix(recipe): set TORCH_CPP_LOG_LEVEL=ERROR in DeepSeek V3 and Qwen3 235B bf16 recipes by @malay-nagda :: PR: #6336
  • build(deps): support Transformers 5.15-5.17 and PEFT 0.20-0.21 by @cuichenx :: PR: #6223
  • fix(model): pass pp_rank to callable transformer layer specs by @pruprakash :: PR: #5752
  • [build] chore: Bump MCore main to 159b443c by @cuichenx :: PR: #6344
  • fix(ckpt): preserve NemotronH checkpoint embedding keys by @cuichenx :: PR: #6343
  • feat(recipe): add Qwen3.5 VL long-context GB200 SFT recipes by @cuichenx :: PR: #6324
  • fix(data): reject structured conversations in selected chat-named columns by @pruprakash :: PR: #6358
  • fix(recipe): give Moonlight-16B 8-chunk pipeline layouts 27 decoder layers by @pruprakash :: PR: #6359
  • fix(ckpt): reject unowned export metadata at every PP size by @pruprakash :: PR: #5585
  • fix(model): accept is_mtp_layer in Kimi K3 attention by @pruprakash :: PR: #6362
  • docs(misc): point GB200 fine-tunes at the card's own import by @pruprakash :: PR: #6366
  • perf(recipe): Enable cuDNN GDN for GB300 Qwen3.5-VL 35B/397B GDN layers by @tomlifu :: PR: #6330
  • Migrate GLM from GPTModel to HybridModel by @xuantengh :: PR: #6158
  • [build] chore: Bump MCore main to 52fbcbc1 by @cuichenx :: PR: #6355
  • LoRA: Stop pre-quantizing input to LoRA branch by @timmoon10 :: PR: #6342
  • AUT-2508 [build] chore: Bump MCore main to 441a9874 by @svcnemo-autobot :: PR: #6369
  • fix(perf): size Ultra HSDP instances from launch GPU count by @rhmukundan :: PR: #6318
  • feat: add DiffusionGemma support by @BenDGeist :: PR: #6284
  • fix(training): guard eval loss reduction and surface zero-token steps by @cuichenx :: PR: #6239
  • docs(model): unpublish Nemotron 3.5 Super VL guides by @cuichenx :: PR: #6380
  • refactor(recipe)!: flatten Nemotron example directories by @cuichenx :: PR: #6371
  • refactor(model): add Qwen3 builder-backed model config with YaRN support by @kamran-nvidia :: PR: #6365
  • feat(peft): fuse expert LoRA output merge by @j316chuck :: PR: #6041
  • fix(qwen3-asr): restore vendored generation on transformers 5 by @cuichenx :: PR: #6368
  • AUT-2511 [build] chore: Bump MCore main to 8949def8 by @svcnemo-autobot :: PR: #6374
  • fix(conversion): sanitize nested PEFT hooks before pipeline export by @huvunvidia :: PR: #6340
  • docs(nemotron): port IITC LoRA tutorial to main by @cuichenx :: PR: #6403
  • fix(model): build Sarvam MLA without a query norm when there is no query LoRA by @pushkar-sarvam :: PR: #6349
  • feat(training): dynamic context parallel on global-batch packing with a Qwen3.5-35B-A3B recipe by @ilml :: PR: #6275
  • feat(data): intra-microbatch reordering for MegatronMIMO scalable reads by @going-song :: PR: #6306
  • docs(model): verify Gemma 3 4B card SFT on 8 H100 GPUs by @pruprakash :: PR: #6400
  • fix(data): preserve native HF conversation fields for validation by @cuichenx :: PR: #6378
  • feat: support pipeline-stage-local HF export by @Bsgg1 :: PR: #6291
  • docs(models): re-verify Nemotron Super H100 long-context SFT at a recipe-containing revision by @pruprakash :: PR: #6363
  • build: update PyTorch base image to 26.09 by @balasaajay :: PR: #6258
  • chore: AUT-2517 bump MCore main to fde43b6d by @svcnemo-autobot :: PR: #6405
  • AUT-2521 [build] chore: Bump MCore main to a8a5977f by @svcnemo-autobot :: PR: #6411
  • build: bump MCore main to e0c5e186 by @cuichenx :: PR: #6412
  • refactor(model)!: remove retired model support by @cuichenx :: PR: #6370
  • Non-uniform HSDP Update by @gautham-kollu :: PR: #6382
  • fix(models): default the HF config file lock to HF_HOME instead of ~/.cache/huggingface by @linmuchuiyang :: PR: #6162
  • feat(deepseek-v4): let inference.sh launch across nodes by @linmuchuiyang :: PR: #6163
  • docs: pin 0.4.2 links to models removed from main to v0.4.2 by @ZhiyuLi-Nvidia :: PR: #6416
  • ci: use writable config locks for offline test containers by @cuichenx :: PR: #6420
  • fix(recipe)!: run Nemotron 3 Ultra H100 SFT on the documented 48-node layout (6376) by @svcnvidia-nemo-ci :: PR: #6423
  • refactor(recipe)!: backport vrnvl72 rename to r0.7.0 (#6419) by @malay-nagda :: PR: #6436
  • perf(recipe): add an 8-GPU GB300 MXFP8 pretrain recipe for Qwen3.5-VL 35B-A3B (6413) by @svcnvidia-nemo-ci :: PR: #6433
  • feat(recipe): backport NT3 Ultra performance updates to r0.7.0 (#6228) by @malay-nagda :: PR: #6437
  • build: install cudnn-frontend from feat/1.29.0-megatron in fw_final image by @ZhiyuLi-Nvidia :: PR: #6427
  • fix(build): install pinned cuDNN Frontend Git ref in fw-final by @balasaajay :: PR: #6447
  • fix(docker): remove EFA metrics plugin for all Linux targets (r0.7.0) by @balasaajay :: PR: #6449
  • perf(recipe): update VRNVL72 Qwen3.5-VL 35B/397B MXFP8 recipes (6450) by @svcnvidia-nemo-ci :: PR: #6455
  • feat(nemotron): Add GTP, Partial CG and NCCL-EP support for NT3 Ultra NVFP4 on GB200/GB300/VRNVL72 (6468) by @svcnvidia-nemo-ci :: PR: #6478
  • fix(training): reuse Megatron-Core determinism setup under --deterministic (6271) by @svcnvidia-nemo-ci :: PR: #6439
  • feat(recipe): add deterministic Nemotron 3 Ultra, Qwen3.5-9B and Moonlight-16B pretrain recipes (6440) by @ZhiyuLi-Nvidia :: PR: #6497
  • docs(model): re-verify Nemotron 3.5 Lightning training at a recipe-containing revision (6425) by @svcnvidia-nemo-ci :: PR: #6442
  • test(recipe): run DeepSeek V4 SFT smoke test from toy init (6515) by @svcnvidia-nemo-ci :: PR: #6525
  • fix(model): backport verification card and DiffusionGemma parity fixes to r0.7.0 by @cuichenx :: PR: #6512
  • build: update r0.7.0 MCore release and validation by @balasaajay :: PR: #6540
  • fix(inference): resolve coordinator addresses and release unused CUDA cache (6535) by @svcnvidia-nemo-ci :: PR: #6550
  • build(security): update Framework Python dependencies for r0.7.0 by @balasaajay :: PR: #6558
  • fix(nemotron-omni): backport video embedder export recovery to r0.7.0 by @huvunvidia :: PR: #6561
  • docs(recipe): document TORCHINDUCTOR_COMPILE_THREADS=1 for Nemotron 3 Ultra H100 SFT (6556) by @svcnvidia-nemo-ci :: PR: #6562
  • docs(model-card): note Omni H100 SFT runs with the default CUDA allocator (6560) by @svcnvidia-nemo-ci :: PR: #6563
  • docs(model-card): verify K-EXAONE 2 distributed inference (6294) by @svcnvidia-nemo-ci :: PR: #6557
  • fix(build): pin leptonai for NeMo-Run API compatibility (r0.7.0) by @balasaajay :: PR: #6577
  • fix(docker): restore NeMo-FW directory traversal permissions (r0.7.0) by @balasaajay :: PR: #6571
  • fix(recipe): use HybridEP and TileLang DSA in GLM-5.2 H100 recipes (6548) by @svcnvidia-nemo-ci :: PR: #6575
  • docs(resiliency): replace removed upstream restart guide on r0.7.0 by @cuichenx :: PR: #6584
  • build(security): update Pillow to 13.0.0 on r0.7.0 by @balasaajay :: PR: #6580
  • cp: build(security): backport #6605 to r0.7.0 by @svcnemo-autobot :: PR: #6611
  • feat(recipe): backport GB300 MXFP8 proxy to r0.7.0 by @malay-nagda :: PR: #6604
  • cp: backport FFmpeg security fixes to r0.7.0 by @svcnemo-autobot :: PR: #6626
  • fix(nemotronh): enable moe_preprocess and moe_router in CUDA graph modules for NT3 Ultra GB200/GB300/VRNVL72 configurations (6607) by @svcnvidia-nemo-ci :: PR: #6632
  • perf(recipe): update Qwen3.5-VL 397B GB300 MXFP8 recipe (6627) by @svcnvidia-nemo-ci :: PR: #6633
  • feat(recipe): add deterministic Nemotron 3 Ultra GB200 MXFP8 perf recipe (6617) by @svcnvidia-nemo-ci :: PR: #6635
  • fix(recipe): use NCCL EP in DeepSeek V3 GB300 deterministic recipe (6622) by @svcnvidia-nemo-ci :: PR: #6636
  • fix(peft): reject nonzero warmup initialization for LoRA+ (6464) by @svcnvidia-nemo-ci :: PR: #6582
  • fix(build): preserve patched AnyIO in Framework images (r0.7.0) by @balasaajay :: PR: #6641
  • build(security): require GitPython 3.2.1 on r0.7.0 by @balasaajay :: PR: #6637
  • build: use PyPI nvidia-cudnn-frontend 1.29.1 in fw_final image (6524) by @svcnvidia-nemo-ci :: PR: #6589
  • fix(misc): run Pyxis conversion and inference tasks from the repo root (6625) by @svcnvidia-nemo-ci :: PR: #6644
  • fix(ckpt): copy source artifacts from the local HF cache when the Hub is offline (6643) by @svcnvidia-nemo-ci :: PR: #6646
  • fix(build): cap PyNvVideoCodec RTSP redirects on r0.7.0 by @balasaajay :: PR: #6655
  • build: fix MLflow and filelock scan findings on r0.7.0 by @balasaajay :: PR: #6659
  • docs(model): start the BAGEL FSDP card leaf from scratch (6586) by @kamran-nvidia :: PR: #6665
  • docs(model): publish a two-node H100 geometry for Super VL conversion (6606) by @svcnvidia-nemo-ci :: PR: #6669
  • test: isolate config loader retry clock mocks (6505) by @svcnvidia-nemo-ci :: PR: #6673
  • build: remove PyNvVideoCodec from release container by @balasaajay :: PR: #6667
  • fix(model): pass --bagel-repo through the conversion launcher for BAGEL import (6623) by @svcnvidia-nemo-ci :: PR: #6676
  • docs(model): re-verify BAGEL card on NeMo 26.10 at fd8ab82 (6613) by @kamran-nvidia :: PR: #6678
  • docs(model): backport Super VL card fixes to r0.7.0 (6599, 6600, 6601, 6616) by @kamran-nvidia :: PR: #6675
  • docs(model): re-pin Super VL GPU conversion entries to include the launcher fix (6674) by @svcnvidia-nemo-ci :: PR: #6685
  • docs: mark Megatron Bridge 0.7.0 as latest by @balasaajay :: PR: #6694
  • fix(model): keep pretrained GLM-5 router bias fixed (#6036) by @hershg
  • [build] chore: bump MCore main to d476c21 (#6160) by @cuichenx
  • fix(training): restore gradient synchronization for copied model configs (#6067) by @aroshanghias-nvd
  • reuse save_tokenizer_assets from MLM (#5532) by @dimapihtar
  • fix(ci): repair wheel packaging and release documentation (#6167) by @cuichenx
  • feat(model): add Nemotron 3.5 Super VL support (#6100) by @cuichenx
  • build: bump MCore main to bde6af2 (#6169) by @yaoyu-33
  • fix(moe): require explicit dispatcher backend selection (#6150) by @cuichenx
  • feat(ckpt): add nemotron super vl text-only conversion (#6130) by @cuichenx
  • build: bump MCore main to c711dc0 (#6173) by @cuichenx
  • fix(data): chunked materialization with fork-based parallelism (#5869) by @kenkenpa2126
  • build: pin flashinfer-python in final image (#6178) by @balasaajay
  • ci: skip PyNvVideoCodec in driverless import check (#6181) by @balasaajay
  • fix(peft): expose gather_output on output-layer substitutes (#6180) by @lauradang
  • [build] chore: bump MCore main to c343bb6d3 (#6179) by @cuichenx
  • fix(nemotron-omni): forward padding mask to language model (#6113) by @rohitrango
  • fix(training): allow context parallelism with MFSDP v2 (#6176) by @wujingyue
  • ci: bump preflight workflows to v1.11.0 (#6193) by @ko3n1g
  • 1f1b cuTE DSL fusion full-iter CG | rubin- NT3 ultra, DSv4 flash (#6046) by @malay-nagda
  • build: bump MCore main to 6a366090 (#6191) by @cuichenx
  • fix(training): propagate loss mask to MTP forward for gpt_step (#6175) by @zhongbozhu
  • Fix(qwen3vl): finalize TE precision after decoder replacement (#6140) by @vasunvidia
  • fix(model): preserve Gemma 4 image attention across pipeline stages (#6187) by @cuichenx
  • fix(data): preserve nested payloads when splitting HF datasets (#6182) by @cuichenx
  • feat(recipe): Enable Vision Encoder CUDA Graph for QWEN3.5 35B and 397B pretrain recipe (#6183) by @tomlifu
  • perf(recipe): default GB300 MoE perf recipes to NCCL EP and grouped tensors (#6171) by @YangFei1990
  • build(deps): pin mamba-ssm to the version the patched build requires (#6204) by @ko3n1g
  • [build] chore: bump MCore main to 8dded20c (#6202) by @cuichenx
  • docs(training): sync packed sequence guidance for Fern (#6207) by @cuichenx
  • fix(qwen3-asr): support legacy configs with native Transformers (#6194) by @cuichenx
  • fix(training): account packed GPT FLOPs across context parallel ranks (#6189) by @zhongbozhu
  • perf(docker): parallelize TE, DeepEP, and Mamba compiles in the CI container (opt-in) (#6214) by @ko3n1g
  • Add Gemma2 TransformerEngine flash-attention path with softcap (opt-in) (#5628) by @nvegesna-netizen
  • build: bump MCore main to 8c487209 (#6206) by @cuichenx
  • build: bump MCore main to d113016b (#6217) by @cuichenx
  • [peft] fix: scale grouped-expert LoRA at the bottleneck, not the output (#5913) by @cuichenx
  • feat(conversion): extract Qwen3.5 MoE text checkpoints with MTP (#6185) by @cuichenx
  • feat(qwen): configure vision recomputation independently of decoder (#6186) by @cuichenx
  • fix(ckpt): sync training resume state and compatibility fixes (#6209) by @cuichenx
  • fix(training): correct diagnostic rank selection and repeated logs (#6211) by @cuichenx
  • feat(data): support custom Energon task encoder configs (#6141) by @cuichenx
  • fix(docker): preserve validated base runtime on opt-in builds (#6190) by @malay-nagda
  • ci: enable Claude review for trusted fork PRs (#5844) by @balasaajay
  • fix(diffusion): align WAN checkpoint prefixes (#6094) by @cuichenx
  • fix(training): correct loss averages and hybrid FLOPs reporting (#6210) by @cuichenx
  • fix(training): sync optimizer buffer and model setup correctness guards (#6208) by @cuichenx
  • fix(data): propagate unpacked SFT padding masks to MoE routers (#6188) by @cuichenx
  • fix(ckpt): preserve MoE fine-tuning state and refresh Super VL verification (#6220) by @cuichenx
  • [build] chore: bump MCore main to c035a426 (#6224) by @cuichenx
  • fix(ckpt): derive checkpoint replica_id from the gtp_remat-inclusive DP group (#6102) by @wdykas
  • [training] fix: define optimizer chain in loss test fixture (#6238) by @cuichenx
  • fix(training): give Qwen3-VL a distillation forward step for context parallelism (#6230) by @kevalmorabia97
  • ci(review): AUT-2439 migrate legacy Claude commands to /review (#6231) by @svcnemo-autobot
  • [build] chore: bump MCore main to 07147d69 (#6240) by @cuichenx
  • [model, recipe] feat: migrate DeepSeek V4 to HybridModel on MCore main (#5860) by @guihong-nv
  • docs(model): refresh Qwen3.8-27B GB200 verification and add H100 pretrain performance (#6234) by @kamran-nvidia
  • fix(diffusion): scatter WAN text embeddings for sequence parallelism (#6232) by @cuichenx
  • fix(training): keep mrope position ids full-length under context parallelism (#6243) by @kevalmorabia97
  • fix(ckpt): pass GTP-inclusive group to model-only load metadata (#6235) by @liding-nv
  • [build] chore: bump MCore main to b4d72b79 (#6245) by @cuichenx
  • feat(recipe): add Nemotron 3 Ultra 256-GPU VR200 NVFP4 perf recipe (#6205) by @malay-nagda
  • ci(review): replace automatic Claude reviews with command guidance (#6264) by @ko3n1g
  • perf(recipe): use NCCL EP for GPT-OSS GB200 and remove temporary examples (#6261) by @malay-nagda
  • build: bump MCore main and preserve optimizer checkpoint compatibility (#6260) by @cuichenx
  • docs: clarify model support and parity investigation skills (#6259) by @cuichenx
  • perf(recipe): raise Muse Glimmer 30B H100 pretrain to 205.6 TFLOP/s/GPU (#6255) by @kamran-nvidia
  • fix(docs): AUT-2456 preview approved PR mirror branches (#6253) by @svcnemo-autobot
  • [build] chore: bump MCore main to bab851b (#6265) by @cuichenx
  • fix(recipe): point GLM-5.2 SFT card commands at library aliases (#6267) by @kamran-nvidia
  • docs(model): record GLM-5.2 GB200 warm-start pretrain verification (#6257) by @kamran-nvidia
  • fix(docs): remove excluded review rubric from skills index (#6280) by @cuichenx
  • feat(ckpt): support MXFP8 parameter export (#5868) by @yuukidach
  • fix(security): reject unsafe checkpoint config targets before import (#6246) by @chtruong814
  • refactor(recipe): rename Nemotron multimodal recipe package (#6283) by @cuichenx
  • fix(peft): pack fused-expert LoRA in PEFT's ParamWrapper layout (#6233) by @jerryhong21
  • build: update Nsight Systems to 2026.5.1 and wandb to 0.29.0 (#6285) by @balasaajay
  • fix(perf): fix VR200 recipes (#6212) by @malay-nagda
  • build: bump MCore main to e998be072 and align TE dependencies (#6277) by @cuichenx
  • fix(model): generate DSA-aligned pipeline layouts for GLM-5.2 conversion (#6288) by @kamran-nvidia
  • docs: correct Nemotron verification artifact paths (#6286) by @cuichenx
  • fix(misc): map --max-position-embeddings onto model.seq_length (#5598) by @pruprakash
  • fix(model): default multi_latent_attention to True on the MLA config (#5597) by @pruprakash
  • fix(ckpt): handle missing newer MLM checkpoint args (#5715) by @caohch-1
  • feat(training): expose GTP rematerialization runtime controls (#6226) by @liding-nv
  • build: update image cuBLAS to 13.8.1 (#6252) by @balasaajay
  • docs(misc): add catalog regeneration step to model verification card skill (#6297) by @kamran-nvidia
  • fix(ckpt): disable shared-expert overlap in single-rank flex fallback (#6287) by @kamran-nvidia
  • fix(data): reduce peak memory when building memmap indices (#5850) by @kenkenpa2126
  • docs(model): verify GLM-5.2 CPU export as a distributed export (#6292) by @kamran-nvidia
  • fix(peft): preserve GPT-OSS gate/up order in adapter export (#6244) by @cuichenx
  • docs(model): record GLM-5.2 H100 HybridEP warm-start pretrain and resume verification (#6295) by @kamran-nvidia
  • [ci] fix: check GitHub file links through raw content (#6302) by @cuichenx
  • perf(recipe): enable Nemotron 3.5 MXFP8 full graphs on GB300 and VR200 (#6289) by @malay-nagda
  • feat(model): add bert support. (#4737) by @bbuschkaemper
  • fix(training): count attention output gates in hybrid FLOPs (#6301) by @cuichenx
  • [build] chore: bump MCore main to feba4e1e (#6299) by @cuichenx
  • fix(ckpt): support GTP checkpoint groups and HF conversion (#6116) by @yaoyu-33
  • fix(models): forward layer-wise optimizer flag to DDP wrapping (#6308) by @liding-nv
  • no non det algo (#6282) by @malay-nagda
  • fix(perf): allow --nccl_ub false to disable NCCL UB (#6241) by @jayzuccarelli
  • Support For NVCRE Executor (#5254) by @sjayaram-nv
  • [build] chore: bump MCore main to ef8cd10b8 (#6309) by @cuichenx
  • feat(ckpt): add Nemotron-H router replay diagnostic (#6322) by @cuichenx
  • fix(recipe): select alltoall for Qwen3 235B pretrain recipe (#6317) by @kamran-nvidia
  • docs(misc): list EXAONE, Kimi K3, MiniMax-M3 and Nemotron-3 Ultra as supported (#5743) by @pruprakash
  • fix(build): remove stale cuDNN Frontend override from FW image (#6323) by @balasaajay
  • feat(training): global-batch online packing with Megatron-Core's sequence-packing scheduler (#6127) by @ilml
  • docs: list BAGEL and Muse Glimmer in supported models (#6328) by @cuichenx
  • test: add copyright header to router replay test (#6334) by @cuichenx
  • fix(misc): fail sync inference when a prompt returns no result (#5765) by @pruprakash
  • fix(recipe): pin expert tensor parallelism on Qwen3.5-VL MoE pretrain recipes (#5754) by @pruprakash
  • fix(data): apply the null-content contract to legacy chat turns (#5742) by @pruprakash
  • fix(data): scope GLM assistant boundary scan to the current turn (#5753) by @pruprakash
  • docs(models): correct Super and GPT-OSS verification provenance (#6293) by @cuichenx
  • fix: pin FW cuDNN Frontend and complete supported model table (#6329) by @balasaajay
  • build: install Mixture of Kittens in Docker on amd64 and arm64 (#6290) by @balasaajay
  • [build] chore: bump MCore main to fbbe6eb5 (#6333) by @cuichenx
  • docs(recipe): keep recipe functions self-contained (#6337) by @kamran-nvidia
  • fix(training): count full runtime sequences in GPT FLOPs (#6227) by @cuichenx
  • feat(mimo): add Nemotron Omni Megatron MIMO conversion support (#6313) by @liding-nv
  • docs(model): save a BERT tokenizer in the megatron-bert README checkpoint step (#6331) by @pruprakash
  • feat(recipe): add Qwen3.5 text MXFP8 long-context SFT (#6300) by @cuichenx
  • build: bump MCore main to 5052ce8 (#6339) by @cuichenx
  • fix(model): keep GLM-5 export max_position_embeddings from the reference config (#5996) by @linmuchuiyang
  • fix(misc): migrate BERT conversion example to scripts/conversion/convert.sh (#6332) by @pruprakash
  • perf(ckpt): build QKV/KV split indices on the input device (#6236) by @ting-hong-shieh
  • test(model): guard verification card launcher commands (#5835) by @pruprakash
  • build(docker): remove Nsight Systems EFA metrics plugin (#6341) by @balasaajay
  • fix(recipe): set TORCH_CPP_LOG_LEVEL=ERROR in DeepSeek V3 and Qwen3 235B bf16 recipes (#6336) by @malay-nagda
  • build(deps): support Transformers 5.15-5.17 and PEFT 0.20-0.21 (#6223) by @cuichenx
  • fix(model): pass pp_rank to callable transformer layer specs (#5752) by @pruprakash
  • [build] chore: Bump MCore main to 159b443c (#6344) by @cuichenx
  • fix(ckpt): preserve NemotronH checkpoint embedding keys (#6343) by @cuichenx
  • feat(recipe): add Qwen3.5 VL long-context GB200 SFT recipes (#6324) by @cuichenx
  • fix(data): reject structured conversations in selected chat-named columns (#6358) by @pruprakash
  • fix(recipe): give Moonlight-16B 8-chunk pipeline layouts 27 decoder layers (#6359) by @pruprakash
  • fix(ckpt): reject unowned export metadata at every PP size (#5585) by @pruprakash
  • fix(model): accept is_mtp_layer in Kimi K3 attention (#6362) by @pruprakash
  • docs(misc): point GB200 fine-tunes at the card's own import (#6366) by @pruprakash
  • perf(recipe): Enable cuDNN GDN for GB300 Qwen3.5-VL 35B/397B GDN layers (#6330) by @tomlifu
  • Migrate GLM from GPTModel to HybridModel (#6158) by @xuantengh
  • [build] chore: Bump MCore main to 52fbcbc1 (#6355) by @cuichenx
  • LoRA: Stop pre-quantizing input to LoRA branch (#6342) by @timmoon10
  • AUT-2508 [build] chore: Bump MCore main to 441a9874 (#6369) by @svcnemo-autobot
  • fix(perf): size Ultra HSDP instances from launch GPU count (#6318) by @rhmukundan
  • feat: add DiffusionGemma support (#6284) by @BenDGeist
  • fix(training): guard eval loss reduction and surface zero-token steps (#6239) by @cuichenx
  • docs(model): unpublish Nemotron 3.5 Super VL guides (#6380) by @cuichenx
  • refactor(recipe)!: flatten Nemotron example directories (#6371) by @cuichenx
  • refactor(model): add Qwen3 builder-backed model config with YaRN support (#6365) by @kamran-nvidia
  • feat(peft): fuse expert LoRA output merge (#6041) by @j316chuck
  • fix(qwen3-asr): restore vendored generation on transformers 5 (#6368) by @cuichenx
  • AUT-2511 [build] chore: Bump MCore main to 8949def8 (#6374) by @svcnemo-autobot
  • fix(conversion): sanitize nested PEFT hooks before pipeline export (#6340) by @huvunvidia
  • docs(nemotron): port IITC LoRA tutorial to main (#6403) by @cuichenx
  • fix(model): build Sarvam MLA without a query norm when there is no query LoRA (#6349) by @pushkar-sarvam
  • feat(training): dynamic context parallel on global-batch packing with a Qwen3.5-35B-A3B recipe (#6275) by @ilml
  • feat(data): intra-microbatch reordering for MegatronMIMO scalable reads (#6306) by @going-song
  • docs(model): verify Gemma 3 4B card SFT on 8 H100 GPUs (#6400) by @pruprakash
  • fix(data): preserve native HF conversation fields for validation (#6378) by @cuichenx
  • feat: support pipeline-stage-local HF export (#6291) by @Bsgg1
  • docs(models): re-verify Nemotron Super H100 long-context SFT at a recipe-containing revision (#6363) by @pruprakash
  • build: update PyTorch base image to 26.09 (#6258) by @balasaajay
  • chore: AUT-2517 bump MCore main to fde43b6d (#6405) by @svcnemo-autobot
  • AUT-2521 [build] chore: Bump MCore main to a8a5977f (#6411) by @svcnemo-autobot
  • build: bump MCore main to e0c5e186 (#6412) by @cuichenx
  • refactor(model)!: remove retired model support (#6370) by @cuichenx
  • Non-uniform HSDP Update (#6382) by @gautham-kollu
  • fix(models): default the HF config file lock to HF_HOME instead of ~/.cache/huggingface (#6162) by @linmuchuiyang
  • feat(deepseek-v4): let inference.sh launch across nodes (#6163) by @linmuchuiyang
  • docs: pin 0.4.2 links to models removed from main to v0.4.2 (#6416) by @ZhiyuLi-Nvidia
  • ci: use writable config locks for offline test containers (#6420) by @cuichenx
  • [🤖]: Update docs-versions after code-freeze for r0.7.0 by @github-actions[bot]
  • Bump docs version to r0.7.0 by @github-actions[bot]
  • fix(recipe)!: run Nemotron 3 Ultra H100 SFT on the documented 48-node layout (6376) (#6423) by @svcnvidia-nemo-ci
  • refactor(recipe)!: backport vrnvl72 rename to r0.7.0 (#6419) (#6436) by @malay-nagda
  • perf(recipe): add an 8-GPU GB300 MXFP8 pretrain recipe for Qwen3.5-VL 35B-A3B (6413) (#6433) by @svcnvidia-nemo-ci
  • feat(recipe): backport NT3 Ultra performance updates to r0.7.0 (#6228) (#6437) by @malay-nagda
  • build: install cudnn-frontend from feat/1.29.0-megatron in fw_final image (#6427) by @ZhiyuLi-Nvidia
  • fix(build): install pinned cuDNN Frontend Git ref in fw-final (#6447) by @balasaajay
  • fix(docker): remove EFA metrics plugin for all Linux targets (r0.7.0) (#6449) by @balasaajay
  • perf(recipe): update VRNVL72 Qwen3.5-VL 35B/397B MXFP8 recipes (6450) (#6455) by @svcnvidia-nemo-ci
  • feat(nemotron): Add GTP, Partial CG and NCCL-EP support for NT3 Ultra NVFP4 on GB200/GB300/VRNVL72 (6468) (#6478) by @svcnvidia-nemo-ci
  • fix(training): reuse Megatron-Core determinism setup under --deterministic (6271) (#6439) by @svcnvidia-nemo-ci
  • feat(recipe): add deterministic Nemotron 3 Ultra, Qwen3.5-9B and Moonlight-16B pretrain recipes (6440) (#6497) by @ZhiyuLi-Nvidia
  • docs(model): re-verify Nemotron 3.5 Lightning training at a recipe-containing revision (6425) (#6442) by @svcnvidia-nemo-ci
  • test(recipe): run DeepSeek V4 SFT smoke test from toy init (6515) (#6525) by @svcnvidia-nemo-ci
  • fix(model): backport verification card and DiffusionGemma parity fixes to r0.7.0 (#6512) by @cuichenx
  • build: update r0.7.0 MCore release and validation (#6540) by @balasaajay
  • fix(inference): resolve coordinator addresses and release unused CUDA cache (6535) (#6550) by @svcnvidia-nemo-ci
  • build(security): update Framework Python dependencies for r0.7.0 (#6558) by @balasaajay
  • fix(nemotron-omni): backport video embedder export recovery to r0.7.0 (#6561) by @huvunvidia
  • docs(recipe): document TORCHINDUCTOR_COMPILE_THREADS=1 for Nemotron 3 Ultra H100 SFT (6556) (#6562) by @svcnvidia-nemo-ci
  • docs(model-card): note Omni H100 SFT runs with the default CUDA allocator (6560) (#6563) by @svcnvidia-nemo-ci
  • docs(model-card): verify K-EXAONE 2 distributed inference (6294) (#6557) by @svcnvidia-nemo-ci
  • fix(build): pin leptonai for NeMo-Run API compatibility (r0.7.0) (#6577) by @balasaajay
  • fix(docker): restore NeMo-FW directory traversal permissions (r0.7.0) (#6571) by @balasaajay
  • fix(recipe): use HybridEP and TileLang DSA in GLM-5.2 H100 recipes (6548) (#6575) by @svcnvidia-nemo-ci
  • docs(resiliency): replace removed upstream restart guide on r0.7.0 (#6584) by @cuichenx
  • build(security): update Pillow to 13.0.0 on r0.7.0 (#6580) by @balasaajay
  • cp: build(security): backport #6605 to r0.7.0 (#6611) by @svcnemo-autobot
  • feat(recipe): backport GB300 MXFP8 proxy to r0.7.0 (#6604) by @malay-nagda
  • cp: backport FFmpeg security fixes to r0.7.0 (#6626) by @svcnemo-autobot
  • fix(nemotronh): enable moe_preprocess and moe_router in CUDA graph modules for NT3 Ultra GB200/GB300/VRNVL72 configurations (6607) (#6632) by @svcnvidia-nemo-ci
  • perf(recipe): update Qwen3.5-VL 397B GB300 MXFP8 recipe (6627) (#6633) by @svcnvidia-nemo-ci
  • feat(recipe): add deterministic Nemotron 3 Ultra GB200 MXFP8 perf recipe (6617) (#6635) by @svcnvidia-nemo-ci
  • fix(recipe): use NCCL EP in DeepSeek V3 GB300 deterministic recipe (6622) (#6636) by @svcnvidia-nemo-ci
  • fix(peft): reject nonzero warmup initialization for LoRA+ (6464) (#6582) by @svcnvidia-nemo-ci
  • fix(build): preserve patched AnyIO in Framework images (r0.7.0) (#6641) by @balasaajay
  • build(security): require GitPython 3.2.1 on r0.7.0 (#6637) by @balasaajay
  • build: use PyPI nvidia-cudnn-frontend 1.29.1 in fw_final image (6524) (#6589) by @svcnvidia-nemo-ci
  • fix(misc): run Pyxis conversion and inference tasks from the repo root (6625) (#6644) by @svcnvidia-nemo-ci
  • fix(ckpt): copy source artifacts from the local HF cache when the Hub is offline (6643) (#6646) by @svcnvidia-nemo-ci
  • fix(build): cap PyNvVideoCodec RTSP redirects on r0.7.0 (#6655) by @balasaajay
  • build: fix MLflow and filelock scan findings on r0.7.0 (#6659) by @balasaajay
  • docs(model): start the BAGEL FSDP card leaf from scratch (6586) (#6665) by @kamran-nvidia
  • docs(model): publish a two-node H100 geometry for Super VL conversion (6606) (#6669) by @svcnvidia-nemo-ci
  • test: isolate config loader retry clock mocks (6505) (#6673) by @svcnvidia-nemo-ci
  • build: remove PyNvVideoCodec from release container (#6667) by @balasaajay
  • fix(model): pass --bagel-repo through the conversion launcher for BAGEL import (6623) (#6676) by @svcnvidia-nemo-ci
  • docs(model): re-verify BAGEL card on NeMo 26.10 at fd8ab82 (6613) (#6678) by @kamran-nvidia
  • docs(model): backport Super VL card fixes to r0.7.0 (6599, 6600, 6601, 6616) (#6675) by @kamran-nvidia
  • docs(model): re-pin Super VL GPU conversion entries to include the launcher fix (6674) (#6685) by @svcnvidia-nemo-ci
  • docs: mark Megatron Bridge 0.7.0 as latest (#6694) by @balasaajay

Contributors

wujingyue, timmoon10, and 49 other contributors
Assets 2
Loading