Release NVIDIA NeMo-Automodel 0.5.0 · NVIDIA-NeMo/Automodel

Release Notes

Highlights
- Blackwell and FP8 support. Run DeepSeek-V4 with TileLang kernels, train MoE models with MXFP8 on GB200, and use context parallelism for Qwen3.5/3.6 MoE VLM. More details in our performance summary
- Agent SFT. Train Qwen2.5-3B on multi-turn, function-calling ChatML or ShareGPT data with full-SFT and LoRA recipes. The data adapter preserves tool calls, tool responses, and optional reasoning_content.
- Speculative decoding. Train EAGLE 1/2/3/3.1, P-EAGLE, and DFlash drafts for Llama, Phi, Qwen3, Qwen3-MoE, GPT-OSS, and Nemotron Nano targets.
- dLLM. Train DFlash drafts and fine-tune LLaDA2 or Nemotron-Labs-Diffusion with the new discrete-diffusion workflows.
- Diffusion. Fine-tune and generate with FLUX.2-dev, Qwen-Image, and Wan 2.2 T2V-A14B; update FLUX, Hunyuan, and Wan 2.1 recipes for LoRA and throughput.
New Hardware and Precision Support
- GB200 MXFP8 MoE. Adds Transformer Engine MXFP8 grouped-expert training for Qwen3-MoE-30B, GPT-OSS-120B, and Qwen3-MoE-235B, and standardizes the bf16 expert backend on Transformer Engine grouped linear layers.
New and Expanded Model Support
LLM and MoE
- DeepSeek-V4 Flash & Pro. Adds a native model, checkpoint adapter, HellaSwag full-SFT and LoRA recipes, a packed-sequence recipe, and large-scale TileLang PP/EP recipes. The implementation covers hash routing, hyper-connections, sparse/compressed attention, FP4/FP8 checkpoint loading, and multi-token prediction.
- ERNIE 4.5. Adds HellaSwag recipes for the 0.3B dense and 21B-A3B MoE checkpoints.
- MiMo-V2-Flash. Adds a HellaSwag fine-tuning recipe and a native model implementation with checkpoint conversion.
- Ling 2.0. Adds Mini, Flash, and 1T model paths with SFT, LoRA, HellaSwag, SQuAD, and pipeline-parallel recipes.
- Hy3-preview. Adds a 295B MoE DeepEP fine-tuning and LoRA recipe, with model-specific expert-bias handling.
- Hy-MT2-30B-A3B. Adds a supervised fine-tuning recipe and native model support.
- MiniMax-M2.7. Adds pipeline-parallel full-SFT and LoRA recipes.
- Falcon-H1. Adds 0.5B, 1.5B, and 7B full-SFT recipes, plus 7B and 34B PEFT recipes.
- MiniCPM5-1B. Adds full-SFT and PEFT recipes.
- Nemotron-3-Ultra-550B-A55B. Adds a 128-GPU throughput benchmark and a 128-GPU full-SFT recipe using a real router, THD sequence packing, and a repeated MTP head.
- Qwen3.5/3.6 dense and MoE. Adds 27B dense and 35B-A3B VLM SFT/LoRA recipes, a 35B EP8/CP2 long-context recipe, MTP support, and a native dense Qwen3.5 backbone that keeps its SSM-gate parameters in fp32.
VLM
- Qwen3.6 VLM. Adds 27B dense and 35B-A3B SFT/LoRA recipes, including a 4K MedPix EP8/CP2 recipe for the MoE model.
- Mistral Medium 3.5. Adds a 128B MedPix full-finetuning recipe and a matching LoRA recipe.
- LLaVA-OneVision 1.5. Adds a 4B full-finetuning recipe and an 8B LoRA recipe.
- VLM QLoRA. Enables the existing quantization: config block for VLM finetuning, including BitsAndBytes 4-bit NF4 base weights with LoRA adapters.
- VLM knowledge distillation. Adds a Qwen3.5-VL 9B-to-4B KD recipe with a chunked KD loss, frozen vision/audio towers, and separate CE/KD metrics.
- Step-3.7-Flash. Adds support for Stepfun AI’s MoE VLM model Step-3.7-Flash with SFT and LoRa recipes using EP + PP across 16 nodes.
Omni and Multimodal
- Bagel. Adds the Bagel model, dataset, pretraining recipe, finetuning recipe, EMA support, checkpoint adapter, and distributed initialization. The one-node examples use token budgets validated for parity runs.
- Qwen2.5-Omni ASR. Adds AMI and multi-language SFT recipes for the 3B and 7B checkpoints.
- Qwen3-Omni ASR. Adds AMI and multi-language audio SFT recipes.
- Nemotron-Omni vision. Adds a dynamic-resolution image path aligned with vLLM's image preprocessing, preventing training/rollout vision-embedding divergence.
- Nemotron-Omni parallelism. Adds activation checkpointing for the nested language model and an EP8/CP2 VLM recipe that prepares and shards multimodal embeddings across the context mesh.
Diffusion
- FLUX.2-dev. Adds a model adapter, dataset processor, full-finetuning recipe, LoRA recipe, and generation configuration for its 4D positional-ID and Mistral3 text-embedding architecture.
- Qwen-Image. Adds model and processor support, full-finetuning, pretraining, generation, and LoRA recipes. The LoRA recipe targets attention and image/text MLP layers, and the data collator pads variable-length cached prompt embeddings.
- Wan 2.2 T2V-A14B. Adds preprocessing, high-noise and low-noise stage finetuning, and generation that loads independently trained checkpoints into the matching denoisers.
- FLUX, Hunyuan, and Wan 2.1. Adds LoRA recipes for each workflow, a multinode Wan 2.1 recipe, and tuned full-finetune/LoRA settings including compile and FSDP reduction-dtype controls.
Gemma 4 Optimizations
- TP/PP VLM support. Adds TP4/PP2 and TP4/PP4 recipes for Gemma 4 31B, including pipeline-stage handling for image position IDs.
- LoRA coverage. Adds PEFT recipes for the 2B, 4B, 31B, and 26B-A4B MoE Gemma 4 VLM models.
- Joint drafter. Adds a Gemma 4 joint-drafter model, MedPix recipe, and MTP inference benchmark.
Retrieval and Embeddings
- Late-interaction retrieval. Adds multi-vector/MaxSim training for ColBERT-style encoders, including distributed in-batch negatives.
- Bi-encoder positives. Adds cycling over every positive passage instead of choosing only one positive document per query.
- Embedding and reranker recipes. Adds a Ministral 3 3B bi-encoder recipe and model-coverage pages for bidirectional Ministral and Llama models.
Agent SFT and Speculative Decoding
- Agent-chat data adapter. Accepts ChatML messages and ShareGPT conversations, merges consecutive tool calls, pairs tool responses to call IDs, and supervises assistant content and tool-call arguments.
- Final-turn SFT. Adds train_on_last_turn_only to mask every earlier assistant response and train only on the final assistant turn.
- Reasoning traces. Preserves reasoning_content during data conversion and adds mask_reasoning_content to include the trace in the prompt while excluding it from loss.
- Tool-call evaluation. Adds generation-based validation metrics for call presence, tool-name accuracy, JSON validity, argument precision/recall, and exact argument match.
- EAGLE recipes. Adds EAGLE 1/2 recipes for Llama, Phi, Qwen3, and Qwen3-MoE; EAGLE 3 recipes for Llama, Phi, Qwen3, Qwen3-MoE, and GPT-OSS; and EAGLE 3.1 recipes for Llama.
- Remote EAGLE target. Adds a remote target server/client path and an example that trains a draft model while target supervision runs on a separate GPU.
- P-EAGLE sequence partitioning. Adds a sequence_partitions knob that splits a long-context draft step into loss-equivalent segments to reduce activation memory.
- EAGLE packing and reasoning controls. Adds packed variable-length training with block-causal attention and a switch to keep, save, or mask target-model reasoning traces.
- EAGLE performance tools. Adds an SGLang acceptance/speedup benchmark, offline target-output caching, and a fused Triton soft-target cross-entropy kernel that avoids allocating a full fp32 log-probability tensor.
dLLM
- DFlash. Adds a draft-training strategy that distills frozen target hidden states with a position-decay loss, with Nemotron Nano 30B and Qwen3-4B example recipes.
- LLaDA2. Adds a discrete-diffusion SFT recipe alongside the existing LLaDA workflow.
- Nemotron-Labs-Diffusion. Adds a hybrid diffusion-LLM SFT recipe.
Training, Parallelism, and Performance
- Selective FSDP2 activation checkpointing. Adds distributed.activation_checkpointing: selective, which saves attention and communication operations while alternating save/recompute for matrix multiplies and grouped MoE GEMMs. It supports dense models, MoE with expert parallelism, single-GPU runs, and torch.compile.
- Memory-efficient LoRA. Adds use_memory_efficient_lora for LoRA training configurations.
- Qwen3.5/3.6 VLM context parallelism. Prepares image/video embeddings and mRoPE positions before sequence sharding, passes the dense sequence index to linear attention, and uses a CP-aware validation denominator.
- Nemotron v3 parallelism. Adds context and pipeline parallelism support for Nemotron v3 models and MTP-related train-loop support.
- Cross-entropy kernels. Adds fused linear cross-entropy for custom models and a CUDA Triton implementation of EAGLE soft-target cross-entropy.
- DeepSeek-V4 kernels. Adds TileLang implementations for sparse MLA, the lightning indexer, and MHC Sinkhorn; the CUDA environment now installs TileLang and TileKernels for the DeepSeek-V4 recipes.
- Large-checkpoint loading. Uses memory-mapped local HF DCP reads so each rank maps only its requested tensor slice instead of copying full tensors into host RAM.
Data, Recipes, and Operations
- Object storage. Adds S3 and multi-storage-client sources to the Megatron pretraining dataset path.
- Dataset blends. Adds data_dir_list entries in [num_samples, path] form for deterministic per-source sampling before concatenation, while retaining plain-path entries.
- Blend files. Adds support for flat-list blend JSON files.
- Lazy preprocessing. Adds lazy dataset preprocessing to defer work until it is needed by the data pipeline.
- Retrieval runtime tuning. Exposes DDP options for bucket size, static graph, buffer broadcast, unused parameters, and gradient bucket views; exposes FSDP2 reshard_after_forward; and adds averaged retrieval-loss logging.
- Recipe coverage. Adds runnable examples for agent SFT, EAGLE/DFlash, dLLM, diffusion, ASR, retrieval, DeepSeek-V4, Gemma 4, Qwen3.6, Nemotron Ultra, and Bagel.
- Packaging. Marks the package as Production/Stable, pins Transformers 5.8.1, requires Transformer Engine 2.14.1 or newer, and resolves Linux PyTorch packages from the CUDA 13.0 index.
Media dependencies are now opt-in
For VLM and Diffusion recipes, video/media packages are no longer installed by default through the base, VLM, or diffusion extras. Install the media extra when a workflow needs OpenCV, decord, Qwen media utilities, or FFmpeg: pip install 'nemo-automodel[media]' Use nemo-automodel[vlm-media] or nemo-automodel[diffusion-media] when only one of those media stacks is required.
Known Issues
- Rope fusion was disabled for numerical reasons, will be re-enabled in 26.08; expect up to 10% perf regression, depending on model size and specific config.
- Adam/AdamW with foreach enabled have known perf regression [PyT#174879] [PyT#177187[
- DeepSeek-V4 Flash fine-tune produces NaN grad norm and eventually NaN loss
- Inter-node DeepEP is incompatible with the nvshmem version installed in the container. This will be fixed in 26.08, please use HybridEP for inter-node EP in 26.06

Changelog Details

fix(vlm): align n_images_per_sample with batch_size in kimi_k25 collate by @khazic :: PR: #2175
fix(deepseek-v4): keep MoE routing scores and attention softmax in fp32 by @khazic :: PR: #2201
fix(vlm): forward get_rope_index to neat packing for mRoPE models by @khazic :: PR: #2172
fix(vlm): fail loudly in PP chunker when pixel_values cannot be aligned by @khazic :: PR: #2181
ci: Major refactor of release-workflows by @ko3n1g :: PR: #2127
fix(nemotron-v3): support THD with input_embeds instead of input_ids by @pzelasko :: PR: #2185
fix(tests): clean up sys.modules pollution in training fixtures by @rob-luke :: PR: #2168
fix(infra): keep model.to(device) on unsharded post-shard load by @HuiyingLi :: PR: #2146
fix(recipes): correct validation loss averaging in LLM KD recipe by @khazic :: PR: #2204
feat(recipes): add VLM knowledge distillation recipe with chunked KD loss by @khazic :: PR: #2205
fix(pp): preserve VLM forward when class opts in via _pp_keep_self_forward by @khazic :: PR: #2192
fix(vlm): chunk video inputs for pipeline parallelism by @khazic :: PR: #2177
fix(vlm): ceil-divide PP chunker so trailing samples are not dropped by @khazic :: PR: #2180
fix(gemma4 recipes): Change attn_impl to eager for gemma4 TP, PP configs by @athitten :: PR: #2199
fix(deepseek_v4): support DeepSeek-V4-Flash-Base by @zpqiu :: PR: #2064
docs: release notes by @akoumpa :: PR: #2141
ci: emit MODEL_FAMILY variable in generated CI YAML by @kajalj22 :: PR: #2203
feat(vlm): support teacher offload for VLM KD by @khazic :: PR: #2211
test(vlm): cover KD TP and CP correctness by @khazic :: PR: #2215
feat(config): add example YAML linter by @zeel2104 :: PR: #2119
refactor: move VLM PP media chunking into pipelining by @HuiyingLi :: PR: #2210
fix(tests): stub _offload_teacher_model on VLM KD recipe fixture by @khazic :: PR: #2225
fix: Remove Qwen3.5 packing known issue marker by @HuiyingLi :: PR: #2222
feat(nemotron-v3): add Multi-Token Prediction (MTP) support by @adil-a :: PR: #2161
chore: missing docstrings, update pyproject by @akoumpa :: PR: #2219
fix(dsv4): preserve reference fp32 parameters by @khazic :: PR: #2216
feat(eagle3): add Llama EAGLE-3 draft training recipe by @khazic :: PR: #2224
fix(vlm): align KD distributed train step by @khazic :: PR: #2212
refactor(eagle3): align draft model module names with sglang by @khazic :: PR: #2235
fix(megatron): make helpers_cpp compile portable across env activations by @hawkoli1987 :: PR: #2232
fix: switch default reduce_dtype to float32 by @akoumpa :: PR: #2237
feat(megatron): accept flat-list blend JSON files by @hawkoli1987 :: PR: #2233
perf(diffusion): improve flow matching training throughput by @pthombre :: PR: #2238
fix: broken links in docs by @akoumpa :: PR: #2245
fix: Fix fake balanced gate bias update by @HuiyingLi :: PR: #2253
fix(distributed): error on default TP plan fallthrough at tp_size>1 by @bzantium :: PR: #2244
feat(eagle3): add FlashAttention-2 backend for draft attention by @khazic :: PR: #2254
feat: add MiMo V2 Flash by @HuiyingLi :: PR: #2250
feat: add ernie4.5 by @HuiyingLi :: PR: #2246
fix(qwen3_5): preserve packed-sample boundaries in GatedDeltaNet by @HuiyingLi :: PR: #2147
docs: Update README with new features and remove old entries by @snowmanwwg :: PR: #2262
fix(test): initialize weights in mimo_v2_flash round-trip fixture by @khazic :: PR: #2261
fix(eagle3): avoid UnboundLocalError on empty train dataloader by @khazic :: PR: #2258
fix(eagle3): validate ttt_steps >= 1 instead of returning NaN loss by @khazic :: PR: #2259
feat(loggers): MLflow run resumption, accurate run status, and VLM/MoE coverage by @rob-luke :: PR: #2231
feat: Support for TokenClassification in Automodel by @hXl3s :: PR: #1634
fix: Add overview section to README.md by @snowmanwwg :: PR: #2270
fix: Fix Qwen3-VL-MoE lm_head checkpoint loading by @HuiyingLi :: PR: #2268
fix(eagle3): drop dead cur_loss_mask, raise on too-shallow aux recipe by @khazic :: PR: #2256
fix(eagle3): reject non-positive draft_vocab_size / target_vocab_size by @khazic :: PR: #2260
fix(eagle3): flush trailing partial grad-accum window each epoch by @khazic :: PR: #2257
feat(deepseek-v4): add Multi-Token Prediction (MTP) training support by @khazic :: PR: #2191
feat(datasets): add S3/MSC object-storage support for MegatronPretrai… by @hawkoli1987 :: PR: #2234
feat(model): add Ling 2.0 / BailingMoeV2 (mini, flash, 1T) (#2242) by @Hayden727 :: PR: #2255
feat: add support for data_dir_list in dict form by @rnyak :: PR: #2229
docs(fern): scaffold Fern docs site mirroring published v0.4.0 sidebar by @lbliii :: PR: #2196
fix(streaming): raise TypeError from iterable dataset len/getitem by @qiaochuz-nv :: PR: #2272
fix: adjust param_dtype during fully_shard_by_dtype traversal by @akoumpa :: PR: #2271
feat(speculative): add llama eagle1 training recipe by @khazic :: PR: #2263
feat(speculative): add llama eagle2 recipe alias by @khazic :: PR: #2264
feat: qwen-omni3 audio ASR SFT recipe by @yuekaizhang :: PR: #2280
fix(deepseek-v4): align HCA backward graph under FSDP2 by @HaloWorld :: PR: #2277
fix: update VLM first-microbatch state during gradient accumulation by @HuiyingLi :: PR: #2282
ci: add retrieval bi-encoder and cross-encoder nightly tests by @oliverholworthy :: PR: #2042
docs: add embedding + reranker model coverage by @akoumpa :: PR: #1843
refactor(ci): unify job construction in generate_ci_tests + add PR check by @thomasdhc :: PR: #2288
feat: Support sliding-window masks for SDPA attention by @HuiyingLi :: PR: #2281
feat(speculative): add checkpoint resume for EAGLE recipes by @khazic :: PR: #2285
refactor(ci): extract finetune launcher config into ci_config + resolver by @thomasdhc :: PR: #2290
docs: tutorial: add llama PEFT tutorial by @krishnakalyan3 :: PR: #2293
docs: add ERNIE, MiMo, and Ling announcements by @HuiyingLi :: PR: #2292
feat: add optimized DeepSeek V4 kernels by @hemildesai :: PR: #2076
feat: Support VLM context parallel inputs in PP by @HuiyingLi :: PR: #2286
ci: skip uv lock generation on forks by @chtruong814 :: PR: #2252
feat: Add Nemotron-Labs-Diffusion (hybrid) SFT support by @zyzhou5 :: PR: #2273
feat(speculative): add SGLang serve helper for trained EAGLE drafters by @khazic :: PR: #2294
feat(speculative): add offline dataset regeneration with EAGLE target model by @khazic :: PR: #2307
docs(fern): add EAGLE speculative decoding e2e tutorial by @khazic :: PR: #2308
fix(speculative): make EAGLE recipes complete a checkpoint save end-to-end by @khazic :: PR: #2310
feat(speculative): add Phi-3 support for EAGLE-1/2/3 + correctness fixes by @khazic :: PR: #2312
feat(speculative): add Qwen3 dense target support for EAGLE-1/2/3 by @khazic :: PR: #2313
feat(examples): add two-node QLoRA recipe by @HuiyingLi :: PR: #2309
fix(datasets): align reasoning + assistant loss masks for left padding by @khazic :: PR: #2314
feat(vlm): add Nemotron-Omni RADIO post-load patches by @yuekaizhang :: PR: #2311
fix(speculative): EAGLE-3 vocab-shrunk checkpoint resume shape mismatch by @qiaochuz-nv :: PR: #2319
feat(speculative): add Qwen3-MoE target support to EAGLE-1/2/3 by @khazic :: PR: #2317
ci: validate release branch-rules by @ko3n1g :: PR: #2295
fix(speculative): EAGLE recipes re-track target_model on .to(device) by @qiaochuz-nv :: PR: #2326
fix(datasets): serialize dict tool arguments to JSON string in xlam by @khazic :: PR: #2323
ci: Add NVSkills request workflow by @akoumpa :: PR: #2331
chore: Move contributor skills out of catalog path by @akoumpa :: PR: #2330
fix(datasets): parse JSON-encoded tools field in ChatDataset by @khazic :: PR: #2324
ci: remove build-docs workflow by @ko3n1g :: PR: #2206
feat(datasets): add multi-turn agent SFT dataset adapter by @khazic :: PR: #2321
chore: add skill evaluation datasets by @akoumpa :: PR: #2332
feat(speculative): add EAGLE-3.1 drafter toggles (fc_norm, norm_output) by @khazic :: PR: #2322
feat(models): add Hy-MT2-30B-A3B SFT support by @khazic :: PR: #2320
fix(datasets): merge tool_calls into prior assistant turn in agent_chat by @khazic :: PR: #2325
fix(examples): switch agent SFT loss to FusedLinearCrossEntropy to avoid OOM by @khazic :: PR: #2336
fix(training): clarify mixed-precision optimizer-state setup by @yuhezhang-ai :: PR: #2248
chore: prefix AutoModel public skill names by @akoumpa :: PR: #2339
ci: Update transformers to latest version 5.8.1 by @svcnvidia-nemo-ci :: PR: #2223
feat: Add StepFun 3.7 support by @athitten :: PR: #2344
feat: support Qwen2.5-Omni model by @yuekaizhang :: PR: #2345
feat(datasets): preserve and optionally mask reasoning_content in agent SFT by @khazic :: PR: #2348
ci: Clear up disk space for lint jobs by @chtruong814 :: PR: #2346
ci: add cluster_tag for reserved-cluster opt-in by @thomasdhc :: PR: #2353
feat(datasets): add train_on_last_turn_only to agent chat SFT dataset by @khazic :: PR: #2347
ci: override ep size for benchmark gptoss 120b by @thomasdhc :: PR: #2352
feat: Add gemma4 drafter model support by @athitten :: PR: #2240
perf(diffusion): improve Flux training throughput by @pthombre :: PR: #2251
feat(datasets): drop history reasoning_content from agent SFT prompt by @khazic :: PR: #2349
feat(eval): add tool-call accuracy evaluator for agent SFT validation by @khazic :: PR: #2338
feat(vlm): wire BitsAndBytes QLoRA quantization into VLM finetune recipe by @HuiyingLi :: PR: #2358
fix(datasets): tag agent SFT row errors with the example id by @khazic :: PR: #2361
perf(datasets): skip full-conversation re-tokenization in chat loss mask by @khazic :: PR: #2363
fix(datasets): warn when agent SFT seq_length has no effect by @khazic :: PR: #2360
test(datasets): cover agent SFT recipe data path end-to-end by @khazic :: PR: #2367
ci: Remove sync-skills workflow and revert recent skills symlink push by @chtruong814 :: PR: #2378
feat(examples): add offline tool-call eval for agent SFT checkpoints by @khazic :: PR: #2368
ci(linting): free disk space before installing toolchain by @ko3n1g :: PR: #2380
fix(datasets): warn on agent SFT rows with no supervised tokens by @khazic :: PR: #2362
ci: align CUDA 13.2 / cu130 toolchain for TE 2.14.1 bump by @thomasdhc :: PR: #2121
feat: Add late interaction model training support for retrieval by @rnyak :: PR: #2283
docs(fern): add multi-turn agent (tool-calling) SFT tutorial by @khazic :: PR: #2364
fix(speculative): serve EAGLE-3 drafts by resolving eagle_meta.pt by @khazic :: PR: #2374
feat(datasets): auto-convert ShareGPT conversations to OpenAI messages by @khazic :: PR: #2375
fix(models): make LlamaRotaryEmbedding honor position_ids values by @khazic :: PR: #2377
perf(speculative): cache the EAGLE-3 draft-vocab token map across runs by @khazic :: PR: #2371
feat(examples): add MiniCPM5-1B fine-tuning recipes by @Ckcinnabar :: PR: #2341
fix(checkpoint): harden consolidated safetensors export by @yuhezhang-ai :: PR: #2289
perf(diffusion): improve Hunyuan training throughput by @pthombre :: PR: #2383
fix(speculative): declare flash_attention_2 support on EAGLE-3 draft by @khazic :: PR: #2387
fix(speculative): rescale EAGLE-1/2 trailing grad-accum window by @khazic :: PR: #2369
feat(datasets): add turn-aware history truncation to agent SFT dataset by @khazic :: PR: #2366
fix(speculative): make EAGLE dataloader CUDA-fork-safe by @khazic :: PR: #2386
fix(gpt_oss): free quantized expert tensors per-layer to reduce peak memory by @stanley1208 :: PR: #2149
feat(speculative): add EAGLE-3 offline target-output cache (legacy path) by @khazic :: PR: #2373
feat(speculative): add SGLang acceptance/speedup benchmark for EAGLE drafters by @khazic :: PR: #2372
chore: bump _code_freeze workflow to v1.4.2 by @ko3n1g :: PR: #2396
feat: Add Gemma 4 12B config by @HuiyingLi :: PR: #2402
ci: bump _release_library.yml to v1.4.3 by @ko3n1g :: PR: #2401
feat(dllm): add DFlash and LLaDA2 SFT recipes by @kashif :: PR: #2315
fix(agent-sft): harden tool-call eval and fix last-turn loss masking by @khazic :: PR: #2365
perf(speculative): skip redundant DDP all-reduce in EAGLE grad accumulation by @khazic :: PR: #2370
feat: adding PP and CP for nemotron v3 models by @adil-a :: PR: #2316
feat(speculative): add gpt-oss EAGLE-3 draft model by @khazic :: PR: #2399
feat(speculative): add remote target serving for EAGLE-3 training by @khazic :: PR: #2398
feat: add nemotron-3-ultra configs and docs by @adil-a :: PR: #2411
feat(speculative): add step-based and final checkpointing to EAGLE recipes by @khazic :: PR: #2405
feat: Enable cycling through all positive documents in biencoder training #907 by @yuhezhang-ai :: PR: #933
docs(fern): add Nemotron-3-Ultra-550B fine-tuning guide by @adil-a :: PR: #2420
feat(speculative): add P-EAGLE parallel-drafting training for EAGLE-3 drafts by @khazic :: PR: #2376
feat(speculative): log the draft model summary in EAGLE training by @khazic :: PR: #2407
feat(speculative): add DFlash draft-model training recipe by @khazic :: PR: #2406
feat(speculative): add P-EAGLE sequence partitioning for long-context training by @khazic :: PR: #2409
feat(speculative): log EAGLE training metrics to Weights & Biases by @khazic :: PR: #2408
perf(speculative): add fused Triton soft cross-entropy kernel for EAGLE-3 by @khazic :: PR: #2428
refactor(speculative): decouple P-EAGLE from the EAGLE-3 code path by @khazic :: PR: #2429
perf(diffusion): optimize Wan2.1 finetuning recipes by @pthombre :: PR: #2403
feat(dllm): add Qwen3-4B dflash recipe, surface FSDP2 prefetch knobs by @kashif :: PR: #2412
build: add managed = true to [tool.uv] by @kajalj22 :: PR: #2434
refactor: expose shared recipe builders from components by @HuiyingLi :: PR: #2190
feat: mixture of mutiple ASR datasets training recipe by @yuekaizhang :: PR: #2414
fix(speculative): guard PEAGLE flex attention compile by @akoumpa :: PR: #2443
feat(models): support fused linear cross-entropy across custom models by @akoumpa :: PR: #2397
chore(skills): refresh distributed training signature by @akoumpa :: PR: #2438
refactor(datasets): unify reasoning_content coercion in agent chat by @khazic :: PR: #2440
fix(tokenizer): make NeMoAutoTokenizerWithBosEosEnforced picklable by @khazic :: PR: #2439
feat(speculative): add target_attn_implementation knob for EAGLE-3 target by @kashif :: PR: #2415
feat(examples): add Falcon H1 fine-tuning recipes by @Ckcinnabar :: PR: #2334
chore(ci): update codeowners to use NVIDIA-NeMo/core-am by @akoumpa :: PR: #2453
fix: fp32 master weights for custom MoE models under FSDP2 by @zpqiu :: PR: #1896
feat(bagel): add multimodal Bagel training support by @zyzhou5 :: PR: #2275
feat(models): add Qwen3.5 MTP support by @HuiyingLi :: PR: #2417
feat: add use_memory_efficient_lora knob by @akoumpa :: PR: #2239
fix(deepseek_v3): initialize weights in fp32 and default router to fp32 by @yuhezhang-ai :: PR: #2450
feat(speculative): add EAGLE-3 sequence packing and reasoning-mode control by @khazic :: PR: #2444
feat(distributed): add selective activation checkpointing for FSDP2 by @yuhezhang-ai :: PR: #2389
feat(diffusion): improve qwen image finetuning configs by @pthombre :: PR: #2442
ci: add nemo-run, split qwen-vl-utils from decord for arm by @thomasdhc :: PR: #2456
fix(precision): dtype contract bug fixes for FSDP2 mixed-dtype loads by @yuhezhang-ai :: PR: #2419
docs(speculative): add subsystem README, fold in regeneration guide by @khazic :: PR: #2448
feat(diffusion): add Wan2.2 T2V-A14B two-stage finetuning support by @linnanwang :: PR: #2284
fix(checkpoint): exclude TE _extra_state keys from load-time mismatch warning by @adil-a :: PR: #2247
feat(moe): enable MXFP8 MoE training on GB200 (TransformerEngine + torchao) by @hemildesai :: PR: #2394
fix(speculative): embed d2t/t2d vocab remap in EAGLE-3 draft checkpoint by @khazic :: PR: #2447
feat: Add query functionality of Model Capability Registry by @athitten :: PR: #2423
refactor(speculative): reuse shared dflash mask and loss in the trainer by @kashif :: PR: #2433
fix(moe): include MTP modules in FSDP sync traversal by @HuiyingLi :: PR: #2441
ci: update package version to 0.5.0 (2472) by @svcnvidia-nemo-ci :: PR: #2473
feat: make mesh accept meshcontext (2266) by @svcnvidia-nemo-ci :: PR: #2474
feat(vlm): enable Qwen3.5 MoE VLM CP (2432) by @svcnvidia-nemo-ci :: PR: #2483
feat(model): flux2 (2145) by @svcnvidia-nemo-ci :: PR: #2489
fix(config): glm4.7 yaml (2527) by @svcnvidia-nemo-ci :: PR: #2528
fix(gemma4): cast dense params without casting buffers (2359) by @svcnvidia-nemo-ci :: PR: #2525
fix: unwrap ModelOutput to extract logits (2523) by @svcnvidia-nemo-ci :: PR: #2530
fix(qwen3_5): make dense VLM pipeline-parallel safe (2524) by @svcnvidia-nemo-ci :: PR: #2554
feat(examples): add Nemotron-3-Ultra-550B benchmark and full-SFT recipes (2539) by @svcnvidia-nemo-ci :: PR: #2550
ci: schedule ep-parallel finetune recipes at documented node counts (2546) by @svcnvidia-nemo-ci :: PR: #2558
fix(diffusion): resolve flux nightly CI failures (2529) by @svcnvidia-nemo-ci :: PR: #2567
fix(ci): bump ling_1t_lora_pp local_batch_size to satisfy PP assert (2575) by @svcnvidia-nemo-ci :: PR: #2578
test: fix all 5 vllm_deploy tests (token drift, nemotron OOM + mamba merge) (2559) by @svcnvidia-nemo-ci :: PR: #2576
fix(checkpoint): preserve tied lm_head on resume (2511) by @svcnvidia-nemo-ci :: PR: #2573
fix(ci): set node counts for multi-node VLM finetune recipes (2574) by @svcnvidia-nemo-ci :: PR: #2579
fix(recipe): reshard MoE experts after forward in nemotron_nano_v3_cp_test (2577) by @svcnvidia-nemo-ci :: PR: #2580
ci: use digits for spark recipes (2581) by @svcnvidia-nemo-ci :: PR: #2583
ci: Enable activation checkpointing for gemma_2_9b_it_squad (AM-464) (2585) by @svcnvidia-nemo-ci :: PR: #2586
fix(test): load checkpoint-robustness HF reference via device_map (2582) by @svcnvidia-nemo-ci :: PR: #2588
fix(peft): LoRA MLP QLoRA/PP/gemma3n fixes (AM-435, AM-447, AM-453) (2584) by @svcnvidia-nemo-ci :: PR: #2597
fix(vlm): enable activation checkpointing for 35B Qwen3.5/3.6 VLM recipes (2600) by @svcnvidia-nemo-ci :: PR: #2602
fix(gemma4): FSDP2-safe kv-sharing + skip frozen audio tower on grad-accum (2566) by @svcnvidia-nemo-ci :: PR: #2599
fix(vlm): use FusedLinearCrossEntropy for qwen3_5_9b to avoid logits OOM (2603) by @svcnvidia-nemo-ci :: PR: #2604
fix(distributed): register Falcon-H1 TP plan to fix 34B PEFT OOM (2589) by @akoumpa :: PR: #2605
fix(models): keep RoPE frequency buffers fp32 under bf16 model cast (2549) by @akoumpa :: PR: #2606
fix: use TE attention for gpt_oss packed-sequence recipe (AM-438) (2587) by @svcnvidia-nemo-ci :: PR: #2611
fix(oom): use FusedLinearCrossEntropy in qwen3 tulu3 configs to avoid OOM (2609) by @svcnvidia-nemo-ci :: PR: #2612
perf(distributed): add retrieval tuning knobs (2452) by @svcnvidia-nemo-ci :: PR: #2607
fix(moe): weight GroupedExpertsTE down-projection bias by routing probability (2591) by @svcnvidia-nemo-ci :: PR: #2610
fix(qwen35moe):convert MTP experts as grouped(AM-442)(2595) by @svcnvidia-nemo-ci :: PR: #2618
fix(bagel): distributed setup init (2608) by @svcnvidia-nemo-ci :: PR: #2613
fix(transformers): keep gemma3n KV sharing working under FSDP2 (AM-454) (2594) by @svcnvidia-nemo-ci :: PR: #2619
fix(docker): build DeepEP against the NVSHMEM wheel matching the apt runtime (2614) by @svcnvidia-nemo-ci :: PR: #2629
fix(qwen3_moe): keep native forward under PP so CP+THD works (2625) by @svcnvidia-nemo-ci :: PR: #2628
fix(config): validate pp_size against distributed.pipeline (2616) by @svcnvidia-nemo-ci :: PR: #2631
fix(moe): default ignore_router_for_ac=True for activation checkpointing (2635) by @svcnvidia-nemo-ci :: PR: #2636
ci: raise ci.time for slow finetune recipes hitting 10-min default (2637) by @svcnvidia-nemo-ci :: PR: #2640
ci: cap MAX_STEPS to 10 for slow vlm_finetune recipes (2639) by @svcnvidia-nemo-ci :: PR: #2641
fix(parallelizer): resolve NemotronH decoder blocks for Nemotron-V3 (2638) by @svcnvidia-nemo-ci :: PR: #2642
refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep (2630) by @akoumpa :: PR: #2644
fix(datasets): decode MedPix images on demand instead of up front (2645) by @svcnvidia-nemo-ci :: PR: #2650
fix(devstral2,ministral3): load FP8 checkpoints via custom mistral3_vlm path (drop HF FineGrainedFP8) (2654) by @svcnvidia-nemo-ci :: PR: #2666
feat(qwen3_5): port dense Qwen3.5 to a custom-model(2557) by @akoumpa :: PR: #2663
fix(examples): enable ac for phi_4_squad (2634) by @akoumpa :: PR: #2661
feat(config): disable W&B in example configs (2643) by @akoumpa :: PR: #2662
feat(moe): MTP FLOPs accounting fix (2486) by @akoumpa :: PR: #2660
cp: fix(merge_lora) (#2653) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2667
cp: fix(datasets) (#2649) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2665
cp: fix(distributed) (#2655) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2668
cp: fix(models) (#2652) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2669
fix(moe): preserve fp32 A_log in Qwen3.5-{MoE,Next GatedDeltaNet} (2484) by @akoumpa :: PR: #2664
cp: fix(model) (#2657) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2671
fix(recipe): disable fused RoPE for MLA packed-sequence MoE recipes (2675) by @svcnvidia-nemo-ci :: PR: #2677
fix(docker): bump DeepEP to 42144303 to pad HybridEP token capacity (2678) by @svcnvidia-nemo-ci :: PR: #2680
cp: fix(training) (#2672) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2679
cp: fix(checkpoint) (#2682) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2685
cp: fix(qwen3_moe) (#2687) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2688
fix(vlm): bump mistral3p5_128b_medpix max_length 1024->2048 (2689) by @svcnvidia-nemo-ci :: PR: #2693
cp: fix(llama3_3) (#2673) to r0.5.0 by @svcnvidia-nemo-ci :: PR: #2684
test(models): speed up qwen3.5 moe/vl-moe from_pretrained unit tests (2698) by @svcnvidia-nemo-ci :: PR: #2699
perf(checkpoint): mmap HF DCP read_data to avoid host-RAM OOM on large loads (2690) by @svcnvidia-nemo-ci :: PR: #2701
fix(mistral3): remap FP8 VLM checkpoint prefixes (2692) by @svcnvidia-nemo-ci :: PR: #2702
ci: add cluster_tag to gb200 benchmarks (2714) by @svcnvidia-nemo-ci :: PR: #2715
fix(loss): reuse LM head gather for MTP loss (2694) by @akoumpa :: PR: #2708
fix(gemma4_moe): re-tie lm_head to active embed_tokens on MoE path (2601) by @svcnvidia-nemo-ci :: PR: #2709
ci: fix 26.06 release cves (2705) by @svcnvidia-nemo-ci :: PR: #2711
fix(moe): handle non-EP expert weight DTensors (2697) by @svcnvidia-nemo-ci :: PR: #2712
build: install tilelang + tile_kernels for DeepSeek-V4 recipes (2683) by @akoumpa :: PR: #2717
fix(checkpoint): super-49B consolidated reload and vllm_deploy (2626) by @svcnvidia-nemo-ci :: PR: #2718
fix(models): use bool sparse masks for sdpa (2624) by @svcnvidia-nemo-ci :: PR: #2721
ci: add time budgets for 12 new timeout failures (2707) by @svcnvidia-nemo-ci :: PR: #2724
cp: DeciLM Nemotron TP plan (#2703) to r0.5.0 by @akoumpa :: PR: #2726
fix(gemma4): avoid DynamicCache OOM on dense E2B/E4B via kv-share holder by @athitten :: PR: #2725
fix(qwen3_5): handle packed MTP attention (2727) by @svcnvidia-nemo-ci :: PR: #2729
ci: add gb200 cluster specification for nemotron_ultra recipe (2733) by @svcnvidia-nemo-ci :: PR: #2734
fix(ci): pin qwen3_moe_30b mxfp8 finetune to gb200 (2735) by @svcnvidia-nemo-ci :: PR: #2737
ci: address diffusers cve (2706) by @svcnvidia-nemo-ci :: PR: #2738
ci: address thrift cve bump to 0.23.0 (2736) by @svcnvidia-nemo-ci :: PR: #2742
fix(deepseek-v4): restore batch axis for packed-sequence (THD) forward (2651) by @svcnvidia-nemo-ci :: PR: #2745
fix(deepseek-v4): avoid bf16 -inf overflow in additive attention mask (2658) by @svcnvidia-nemo-ci :: PR: #2746
fix(ci): reduce mixtral release smoke batch (2728) by @akoumpa :: PR: #2751
build: install TileKernels for DeepSeek V4 (2740) by @svcnvidia-nemo-ci :: PR: #2750
fix(qwen3_moe): step-0 NaN in MXFP8 packed finetune — expert unload + fused RoPE (2722) by @svcnvidia-nemo-ci :: PR: #2753
fix(fsdp2): guard uninitialized accumulated grads (2744) by @svcnvidia-nemo-ci :: PR: #2752
ci: bump benchmark glm_4.7_flash_te_deepep time (2757) by @svcnvidia-nemo-ci :: PR: #2759
fix(diffusion): raise qwen-image dist timeout for checkpoint consolidation (2748) by @svcnvidia-nemo-ci :: PR: #2756
fix(diffusion): reuse warm HF cache instead of re-downloading models (2747) by @svcnvidia-nemo-ci :: PR: #2754
fix(mistral3): preserve medium VLM checkpoint layout (2758) by @svcnvidia-nemo-ci :: PR: #2762
fix(sdpa): apply resolved backend constraints to custom models (2761) by @svcnvidia-nemo-ci :: PR: #2765
ci: run dsv32_lora, kimi_k2 and qwen3_moe_235b deepep benchmarks online (2773) by @svcnvidia-nemo-ci :: PR: #2774
fix(distributed): use flattened CP FSDP mesh (2768) by @svcnvidia-nemo-ci :: PR: #2769
fix: Remove dali from container (2770) by @svcnvidia-nemo-ci :: PR: #2771
fix(benchmark): skip unsupported MTP flops (2767) by @svcnvidia-nemo-ci :: PR: #2784
fix: Qwen3.5 MedPix EP32 NCCL timeout (2777) by @svcnvidia-nemo-ci :: PR: #2785
fix: skip fused LoRA MLP install for meta weights (2775) by @svcnvidia-nemo-ci :: PR: #2781
fix(ci): address go-git/go-billy and rustls-webpki CVEs (2780) by @svcnvidia-nemo-ci :: PR: #2787
fix(ci): stabilize diffusion finetune smoke tests (2788) by @svcnvidia-nemo-ci :: PR: #2791
build(deps): move ffmpeg/opencv deps to opt-in media extra (2743) by @svcnvidia-nemo-ci :: PR: #2792
fix(ci): HybridEP bench + LoRA OOM fixes (2789) by @svcnvidia-nemo-ci :: PR: #2794
fix(vlm): keep Qwen3.5 media tokens aligned (2772) by @akoumpa :: PR: #2793
fix: qwen3.5 and 3.6 mtp expert checkpoint layout (2778) by @svcnvidia-nemo-ci :: PR: #2795
fix(ci): drop base-image uv/wandb copies flagged for CVEs (2800) by @svcnvidia-nemo-ci :: PR: #2803
fix(optim): align Dion mesh with FSDP sharding (2808) by @svcnvidia-nemo-ci :: PR: #2812
fix(ci): stabilize failed benchmark recipes (2817) by @svcnvidia-nemo-ci :: PR: #2818
feat(diffusion): support Hugging Face datasets (2816) by @svcnvidia-nemo-ci :: PR: #2831
cp: docs: document opt-in media extras (vlm-media/diffusion-media) (#2799) by @chtruong814 :: PR: #2848
feat: integrate NeMo-Run launcher for managed job submission (#1668) by @hemildesai
feat: MoE model benchmarks, LoRA configs, and flops calculators (#1676) by @hemildesai
feat: adding lora to diffusion (#1653) by @linnanwang
fix: resolve PT 2.11 DeviceMesh deprecation warnings and unify EP mesh (#1684) by @hemildesai
fix: handle dict-typed chat_template in format_chat_template (#1696) by @adil-a
docs: update model coverage/docs for glm5.1 (#1720) by @HuiyingLi
feat: add dynamic sequence length support for pipeline parallelism (#1689) by @hemildesai
fix: mute warning spam (#1721) by @akoumpa
feat: add save_checkpoint_every_epoch flag and max_steps support for … (#1723) by @pthombre
feat: nemotron Parse fine-tuning notebook and assets (#1655) by @krishnakalyan3
fix: fixing the pooling error for non-llama models for biencoder training (#1645) by @rnyak
ci: Remove relative path in codcov with explicit path (#1695) by @thomasdhc
ci: Update install test scope (#1697) by @thomasdhc
fix: propagate brev link to docs (#1735) by @akoumpa
fix(docker): replace deprecated pynvml with nvidia-ml-py (#1725) by @ooooo-create
fix: Qwen3.5 dense CP support and FSDP mixed-dtype fix (#1710) by @HuiyingLi
ci: Address timeout is ci tests (#1733) by @thomasdhc
test: Checkpoint robustness skips atexit-registered destroy_process_group() (#1730) by @thomasdhc
feat: Add lora recipes for gemma4 (#1731) by @athitten
test: add vLLM deployment tests for checkpoint robustness (#1656) by @adil-a
fix: Update lora configs for gemma4 (#1748) by @athitten
docs: minor changes to tutorials (#1747) by @krishnakalyan3
docs: update brev links (#1751) by @HuiyingLi
fix: Baichuan2 checkpoint robustness test CI failures (#1727) by @adil-a
build: drop rc0 pre-release tag and add dynamic git versioning (#1729) by @ko3n1g
ci: Address container and source code cve (#1753) by @thomasdhc
ci: Update test timeout and add ci_tests readme (#1752) by @thomasdhc
fix: Allow use_cache when activation_checkpointing is True (#1726) by @athitten
fix: MoE gate bias defaults and configurable gate_bias_update_factor (#1768) by @hemildesai
fix: skip embedding[padding_idx] = 0 with TP (#1675) by @akoumpa
fix: launcher option from being consumed as a config override. (#1766) by @akoumpa
feat: FSDP2 w weight prefetching and async TP optimization (#1711) by @ZhiyuLi-Nvidia
ci: add missing recipe owners (#1775) by @akoumpa
ci: Resolve cve and remove uv cache (#1774) by @thomasdhc
fix: add THD logit unsqueeze for GPT-OSS model (#1757) by @hemildesai
fix: update yamls for vllm_deploy (#1780) by @akoumpa
feat: minimax m27 (#1785) by @HuiyingLi
docs: Add nightly CI test summary for LLM and VLM finetune configs (#1791) by @thomasdhc
docs: update index (#1788) by @akoumpa
feat: Enable benchmark CI testing with llm_benchmark and vlm_benchmark (#1793) by @thomasdhc
fix: Add per-tensor conversion in gemma4 state_dict_adapter.py (#1764) by @athitten
fix: NotImplementedError: aten::equal on meta tensors during multi-GPU init (#1769) by @harshareddy832
fix: Coerce plain-dict backend to BackendConfig in model init (#1784) by @adil-a
fix: Restrict auto-discovery scopes in generate_ci_tests.py (#1805) by @thomasdhc
chore: Update GPT-OSS and Qwen3 recipe configs (#1811) by @hemildesai
ci: RC6 timeout fixes for release test recipes (#1801) by @thomasdhc
ci: Increase benchmark timeout for GLM and Qwen3.5 MoE LoRA recipes (#1818) by @thomasdhc
fix: enable dequantization for ministral3 and dataset limit (#1807) by @akoumpa
fix: meta init with force_hf=True (#1810) by @akoumpa
fix: tie weights outside _init_model (#1817) by @akoumpa
fix: in-place state dict conversion to reduce peak VRAM by ~50% (#1742) by @stanley1208
fix: Align benchmark TEST_LEVEL check with generate_ci_tests scope (#1831) by @thomasdhc
docs: fix GLM-5.1 attention mechanism name in README (#1833) by @Taishi-N324
fix: stop resolve_yaml_env_vars from scanning runtime data in instantiate() (#1827) by @khazic
fix: gpt_oss_20b_single_gpu_peft CI crash with nproc_per_node override (#1835) by @adil-a
fix: Re-apply PyTorch dependency overrides after full COPY in Dockerfile (#1847) by @thomasdhc
fix: install ffmpeg and rebuild torchcodec for phi4mm audio decoding (#1826) by @HuiyingLi
fix: rotary embeddings for v4 (#1821) by @akoumpa
fix: mute unsupported field attribute warning on startup (#1773) by @akoumpa
ci: Update to transformers v5.5 (#1734) by @athitten
fix: relax checkpoint robustness HF KL threshold for nemotron_nano_8b_v1 (#1839) by @adil-a
fix: pre-cache HF dynamic modules to prevent filesystem race in robustness test (#1840) by @adil-a
fix: trust_remote_code guard in robustness test (#1845) by @adil-a
ci: add NMP customizer contract test configs (#1712) by @adil-a
style: run prek run --all-files to format all files (#1065) by @ooooo-create
fix: PyTorch 2.9.x compatibility for DeviceMesh private APIs (#1825) by @khazic
chore: bump FW-CI-templates to v0.80.2 (#1585) by @ko3n1g
ci: Reduce default finetune step count from 100 to 50 (#1874) by @thomasdhc
fix: gpt oss ci (#1877) by @akoumpa
fix: gradient checkpointing broken for MoE models on single GPU (ep_size=1) (#1873) by @VM-IPA
fix: Setup vllm testing with uv --no-config (#1875) by @thomasdhc
fix: Skip snapshot_download when HF_HUB_OFFLINE=1 (#1834) by @thomasdhc
feat: add Qwen3.6-35B-A3B VLM finetune recipe (#1882) by @HuiyingLi
feat: add mock VLM dataset and Gemma4 pretokenize support (#1682) by @HuiyingLi
fix(ci): retry apt-get and Azure CLI installs to handle mirror sync failures (#1872) by @ko3n1g
feat: LLaVA-OneVision-1.5 integration #1783 (#1790) by @vgauraha62
fix: lint in llava_onevision (#1884) by @akoumpa
cp: 1813 fix: FSDP2 meta-device crash for Qwen3.5 GatedDeltaNet fp32 params (#1869) by @HuiyingLi
fix(collate): auto-derive assistant turn markers for non-Qwen models (#1862) by @khazic
fix(metric_logger): handle non-scalar tensor metrics without crashing (#1871) by @khazic
fix: Create diffusion_kernels group to fix HF_HUB_OFFLINE compatibility (#1842) by @thomasdhc
fix: handle transformers.FineGrainedFP8Config quantization config (#1864) by @akoumpa
fix: resolve VLM CI failures for PP recipes and collate_fn (#1799) by @HuiyingLi
docs: Add dLLM SFT fine-tuning and generation guide (#1806) by @zyzhou5
fix: preserve root mesh for multi-node Gemma4 TP4 FSDP2 runs (#1868) by @khazic
ci(action): surface launch info and pass/fail banner; fix exit_code capture (#1887) by @ko3n1g
chore: move recipes to have perf CI/CD coverage (#1885) by @ZhiyuLi-Nvidia
fix: relax KL thresholds and remove invalid kwargs in Qwen3Next linear attn (#1867) by @hemildesai
fix: Fix bug in diffusion generation (#1850) by @pthombre
ci: Add Dockerfile.deploy for deploy test environment (#1804) by @thomasdhc
fix: test in mesh_utils (#1898) by @akoumpa
fix: pass unnormalized residual to MoE gate in Gemma4 decoder layer (#1895) by @sharonyu-115
fix(gemma4_moe): vision-aware mask when use_bidirectional_attention==vision (#1905) by @jQizhang
fix: restore Qwen3.5 + Phi-4-MM nightly CI after transformers v5.5 update (#1906) by @HuiyingLi
feat: TP+PP support for Gemma4 VLM (with tied lm_head fix and 31B recipes) (#1904) by @khazic
fix: baichuan dynamic cache (#1865) by @zyzhou5
docs: add SkyPilot Kubernetes tutorial (#1667) by @zeel2104
chore: update mask helpers for Transformers inputs_embeds rename (#1782) by @ooooo-create
fix: add embed_vision to MULTIMODAL_SUFFIXES and set lbs=2 for Gemma4 PP2 recipe (#1911) by @khazic
build: move flash-linear-attention back to optional-dependencies (#1894) by @zpqiu
fix: support Qwen-Image finetune (T2I) (#1704) by @harshareddy832
fix: qlora ckpt loading (#1549) by @akoumpa
fix: Update recipe_owner for gemma4 (#1925) by @athitten
fix: update defer_fsdp_grad_sync in recipes (#1919) by @akoumpa
fix: chat dataset (#1921) by @akoumpa
fix: make _get_logits pp aware in ckpt robustness (#1923) by @akoumpa
fix: disable packed sequences for nemotron_nano_4b_squad (#1929) by @adil-a
fix: Step-3.5-Flash layer_types mismatch and related recipe fixes (#1916) by @hemildesai
fix(moe): align EP expert weight dtype with activation dtype (#1913) by @jQizhang
feat: Qwen3.5 VLM TP+PP support with per-microbatch grad reduce-scatter knob (#1859) by @akoumpa
ci: Add test_recipes for custom test scope (#1915) by @thomasdhc
fix: Update recipe test time based on release test run (#1955) by @thomasdhc
ci(feat): use AWS ephemeral runners for external contributors (#1892) by @ko3n1g
fix: AC silently skipped on all registered VLMs — flatten ModuleList (#1941) by @khazic
fix: Patch wandb-core Go CVEs: bump otel SDK, add go-jose (#1957) by @thomasdhc
fix: update docs (#1961) by @akoumpa
docs: Add container version to docs version picker (#1965) by @chtruong814
fix: Update gemm4 26b ci timeout (#1962) by @thomasdhc
chore: add @zyzhou5 and @athitten to codeowners (#1968) by @akoumpa
chore: add tests to 1941 (#1959) by @HuiyingLi
feat: add LoRA recipes for GLM-5.1, MiniMax-M2.7, and Qwen3.6-35B-A3B (#1970) by @HuiyingLi
fix: ministral tp plan (#1963) by @akoumpa
fix(vlm): qwen3_5_4b_neat_packing OOM - reduce seqlen to 4096 (#1975) by @HuiyingLi
fix: nemotron flash (#1973) by @akoumpa
fix: batch ckpt-robustness fixes for pipeline 48953745 (supersedes 9 PRs) (#1971) by @adil-a
fix(devstral): point 24B Squad recipes at official FP8 model (#1980) by @HuiyingLi
feat: add tqdm progress bar to all training recipe loops (#1983) by @khazic
fix: vllm deploy test should fail if vllm is not present (#1987) by @thomasdhc
fix: Move benchmark recipe out of llm_finetune nightly (#1989) by @thomasdhc
feat: add qwen3.6 27B config (#1992) by @HuiyingLi
fix: Address ci timeout test from rc8 (#1991) by @thomasdhc
docs: add SECURITY.md (#1996) by @chtruong814
fix: Address pillow CVE (#1994) by @thomasdhc
ci: Support per-recipe env_vars in CI config (#1999) by @thomasdhc
ci: Update test recipe list (#2001) by @thomasdhc
fix: Add changes for QwenImage Training (#1976) by @pthombre
fix: guard against zero label tokens causing NaN loss in VLM training (#1985) by @khazic
fix: batch Flash 1B + Super-49B PEFT + qwen2.5-7B ckpt-robustness (#1984) by @adil-a
test: add test to 1985 (#2006) by @HuiyingLi
fix: transformers v5.5.0 validation (#2010) by @akoumpa
fix: change drop_long_samples to True by default (#2009) by @akoumpa
ci: add base_sha to codecov/codecov-action upload step (#2016) by @ko3n1g
chore: bump pyt (#2003) by @akoumpa
ci: add --tb=short to pytest invocations in CI test scripts (#2018) by @thomasdhc
fix: switch from match_all_linear to target_modules (#2022) by @akoumpa
fix: add discover pp seq len (#2024) by @akoumpa
feat: Add diffusion finetuning CI pipeline for nightly runs (#1728) by @pthombre
fix: regression in tokenizer+auto_map with transformers 5.5.0 (#2025) by @akoumpa
fix: Propagate torch_dtype to sub-configs correctly (#2027) by @athitten
feat: inject TransformerEngine DotProductAttention into HF models (#2011) by @khazic
ci: add known_issue_id / allow_failure keys + triage (#2028) by @thomasdhc
fix: gradient clip with torch_mm + EP (gpt-oss 120b recipe) (#2012) by @akoumpa
fix: add required YAML frontmatter to skills (#2032) by @ooooo-create
test: add dataloader checkpoint integration test for retrieval recipes (#1800) by @oliverholworthy
ci: triage pipeline benchmark failures (#2040) by @thomasdhc
fix: lora checkpointing (#2037) by @linnanwang
fix: Unsafe deserialization via torch.load on dataset cache files (#2045) by @tomaioo
ci: triage rc9 finetune failures (#2043) by @thomasdhc
ci: triage vllm_deploy rc9 failures (#2047) by @thomasdhc
ci: add LoRA nightly tests for Wan, Hunyuan, Flux diffusion recipes (#2048) by @pthombre
docs(llm): add DeepSeek V4 Flash fine-tuning guide (#2053) by @khazic
docs(llm): drop validate-yaml reference from DeepSeek V4 Flash guide (#2054) by @khazic
docs: Update README with new finetuning support details (#2055) by @HuiyingLi
feat: DeepSeek V4 Flash support (#2039) by @khazic
ci: Update base container pillow version for cve (#2065) by @thomasdhc
ci: drop scheduled nightly-docs trigger, add manual dispatch (#2067) by @thomasdhc
fix(ci): cache heavy CUDA wheels in install-test (#1796) (#1798) by @KevinSailema
fix: llava onevision recipes (#1922) by @akoumpa
fix: Use uv Python for MCore dataset compilation (#438) (#807) by @YuHe Zhang
fix(vlm): mask fake vision tokens in kimi_k25_vl_collate_fn (#2062) by @khazic
fix(vlm): use build_labels_from_template in phi4_mm_collate_fn (#2061) by @khazic
feat: Add Nemotron 3 nano omni (#2063) by @HuiyingLi
fix(vlm): use build_labels_from_template in kimi_vl_collate_fn (#2060) by @khazic
fix(vlm): support multi-image token expansion in _expand_image_tokens (#2059) by @khazic
feat: lazy dataset preprocessing (#2007) by @edjson
fix(vlm): broaden fake-image vision-token detection beyond Qwen (#2058) by @khazic
fix(examples): use HF model ids for Nemotron-Omni and MiniMax-M2.7 yamls (#2069) by @HuiyingLi
docs: Bump docs version (#2073) by @thomasdhc
fix: fallback to safetensors if using peft (#1924) by @akoumpa
feat: Add Ministral3-3B bidirectional encoder training scripts (#1809) by @rnyak
ci: bump release workflow to 0.94.1 (#2078) by @thomasdhc
ci: Add codeowner for tests/ci_tests (#2080) by @thomasdhc
feat(llm): add Hy3-preview (HYV3) SFT support (#2072) by @khazic
docs(llm): add Hy3-preview fine-tuning guide (#2084) by @khazic
ci: onboard GB200 testing (#1893) by @ko3n1g
docs: add Mistral Medium 3.5 VLM coverage and fine-tuning guide (#2091) by @HuiyingLi
feat: add mistral medium 3.5 (#2090) by @HuiyingLi
fix(examples): switch Nemotron-Omni & DeepSeek-V4-Flash recipes to deepep (#2092) by @HuiyingLi
fix: update deployment requirements (#2093) by @thomasdhc
docs(nemotron-omni): document LoRA inference with FQN translation (#2094) by @HuiyingLi
feat: Enable AC and dynamic-resolution vision path for Nemotron-Omni (#2085) by @yuekaizhang
fix: omni lint (#2098) by @akoumpa
fix: update stale _peft import in vlm_generate example (#2097) by @akoumpa
fix(ci): add retry with backoff to approve-test-queue bot (#2099) by @ko3n1g
ci: harden generate_ci_tests against missing recipes + add PR check (#2101) by @thomasdhc
fix: use pre-shard HF state-dict keys when source index is missing (#2096) by @adil-a
chore(ty): surface unresolved imports + fix stale paths in vlm_generate (#2100) by @akoumpa
feat(dsv4): apply YaRN frequency interpolation to compress-rope (#2103) by @HuiyingLi
docs: use absolute raw.githubusercontent.com URLs for embedded images (#2104) by @HuiyingLi
fix: use config.expert_dim for MoE expert LoRA init (#2102) by @adil-a
fix: bump qwen3_moe_30b_lora ci time (#2107) by @thomasdhc
ci: add sync-skills workflow (#1841) by @ko3n1g
feat(devstral): restore 24B SQuAD recipes via force_hf (#2111) by @HuiyingLi
ci: Bump base pytorch image to 26.04 (#2108) by @thomasdhc
ci: replace wandb manual reinstall with version pin (#2112) by @thomasdhc
ci: restore repo vars for runner prefix, registry, and test data path (#1960) by @ko3n1g
feat: add inbatch neg sampling for training (#2077) by @rnyak
fix: save QLoRA PEFT checkpoints with HF adapter prefix (#2120) by @HuiyingLi
ci: update codecov thresholds (#2122) by @akoumpa
chore: refactor skill system (#2113) by @ko3n1g
fix: add diffusion_finetune to release test suite (#2124) by @thomasdhc
feat(qwen3_5): hide _fp32_params wrapping in HF-format saves (#2117) by @HuiyingLi
docs(nemotron-omni): use device_map fast path for SFT inference (#2126) by @HuiyingLi
ci: Update to production/stable and add maintainers (#2139) by @thomasdhc
fix: update readme (#2138) by @akoumpa
fix: MoE aux-loss dtype mismatch under activation checkpointing (#2083) by @pzelasko
ci: use HF cache for diffusion dataset download (#2151) by @thomasdhc
fix(deepseek-v4): transpose hyperconnection comb (#2159) by @jQizhang
fix: issue fixed (#2157) by @akoumpa
feat(nemotron-omni): enable context parallelism for VLM path (#2125) by @HuiyingLi
fix(deepseek_v4): use scatter_add_ for indexer_topk mask (#2152) by @jQizhang
fix(gemma4): Gemma4 packing, attention mask, and fixesMoE routing (#2116) by @shruthan
fix(dsv4/moe): clamp shared-expert SwiGLU in fp32 (#2173) by @sharonyu-115
ci: Separate internal vs external CI queue (#2187) by @chtruong814
feat: Add April LoRA finetuning recipes (#2186) by @HuiyingLi
ci: Remove unnecessary taint-node job (#2188) by @chtruong814
docs: Add build timestamp meta tag to docs (#2194) by @chtruong814
fix(peft): gate FP8+PEFT config kwarg injection on is_hf_model (#2169) by @HuiyingLi
fix: coverage for external members (#2197) by @thomasdhc
feat: add extract_submodel parameter to build_encoder_backbone (#1838) by @oliverholworthy
feat(recipes): auto-detect world_size in setup_distributed (#2183) by @adil-a
fix: fix Qwen3.5 PP ModuleDict layer extraction (#2200) by @HuiyingLi
fix(vlm): align n_images_per_sample with batch_size in kimi_k25 collate (#2175) by @khazic
fix(deepseek-v4): keep MoE routing scores and attention softmax in fp32 (#2201) by @khazic
fix(vlm): forward get_rope_index to neat packing for mRoPE models (#2172) by @khazic
fix(vlm): fail loudly in PP chunker when pixel_values cannot be aligned (#2181) by @khazic
ci: Major refactor of release-workflows (#2127) by @ko3n1g
fix(nemotron-v3): support THD with input_embeds instead of input_ids (#2185) by @pzelasko
fix(tests): clean up sys.modules pollution in training fixtures (#2168) by @rob-luke
fix(infra): keep model.to(device) on unsharded post-shard load (#2146) by @HuiyingLi
fix(recipes): correct validation loss averaging in LLM KD recipe (#2204) by @khazic
feat(recipes): add VLM knowledge distillation recipe with chunked KD loss (#2205) by @khazic
fix(pp): preserve VLM forward when class opts in via _pp_keep_self_forward (#2192) by @khazic
fix(vlm): chunk video inputs for pipeline parallelism (#2177) by @khazic
fix(vlm): ceil-divide PP chunker so trailing samples are not dropped (#2180) by @khazic
fix(gemma4 recipes): Change attn_impl to eager for gemma4 TP, PP configs (#2199) by @athitten
fix(deepseek_v4): support DeepSeek-V4-Flash-Base (#2064) by @zpqiu
docs: release notes (#2141) by @akoumpa
ci: emit MODEL_FAMILY variable in generated CI YAML (#2203) by @kajalj22
feat(vlm): support teacher offload for VLM KD (#2211) by @khazic
test(vlm): cover KD TP and CP correctness (#2215) by @khazic
feat(config): add example YAML linter (#2119) by @zeel2104
refactor: move VLM PP media chunking into pipelining (#2210) by @HuiyingLi
fix(tests): stub _offload_teacher_model on VLM KD recipe fixture (#2225) by @khazic
fix: Remove Qwen3.5 packing known issue marker (#2222) by @HuiyingLi
feat(nemotron-v3): add Multi-Token Prediction (MTP) support (#2161) by @adil-a
chore: missing docstrings, update pyproject (#2219) by @akoumpa
fix(dsv4): preserve reference fp32 parameters (#2216) by @khazic
feat(eagle3): add Llama EAGLE-3 draft training recipe (#2224) by @khazic
fix(vlm): align KD distributed train step (#2212) by @khazic
refactor(eagle3): align draft model module names with sglang (#2235) by @khazic
fix(megatron): make helpers_cpp compile portable across env activations (#2232) by @hawkoli1987
fix: switch default reduce_dtype to float32 (#2237) by @akoumpa
feat(megatron): accept flat-list blend JSON files (#2233) by @hawkoli1987
perf(diffusion): improve flow matching training throughput (#2238) by @pthombre
fix: broken links in docs (#2245) by @akoumpa
fix: Fix fake balanced gate bias update (#2253) by @HuiyingLi
fix(distributed): error on default TP plan fallthrough at tp_size>1 (#2244) by @bzantium
feat(eagle3): add FlashAttention-2 backend for draft attention (#2254) by @khazic
feat: add MiMo V2 Flash (#2250) by @HuiyingLi
feat: add ernie4.5 (#2246) by @HuiyingLi
fix(qwen3_5): preserve packed-sample boundaries in GatedDeltaNet (#2147) by @HuiyingLi
docs: Update README with new features and remove old entries (#2262) by @snowmanwwg
fix(test): initialize weights in mimo_v2_flash round-trip fixture (#2261) by @khazic
fix(eagle3): avoid UnboundLocalError on empty train dataloader (#2258) by @khazic
fix(eagle3): validate ttt_steps >= 1 instead of returning NaN loss (#2259) by @khazic
feat(loggers): MLflow run resumption, accurate run status, and VLM/MoE coverage (#2231) by @rob-luke
feat: Support for TokenClassification in Automodel (#1634) by @hXl3s
fix: Add overview section to README.md (#2270) by @snowmanwwg
fix: Fix Qwen3-VL-MoE lm_head checkpoint loading (#2268) by @HuiyingLi
fix(eagle3): drop dead cur_loss_mask, raise on too-shallow aux recipe (#2256) by @khazic
fix(eagle3): reject non-positive draft_vocab_size / target_vocab_size (#2260) by @khazic
fix(eagle3): flush trailing partial grad-accum window each epoch (#2257) by @khazic
feat(deepseek-v4): add Multi-Token Prediction (MTP) training support (#2191) by @khazic
feat(datasets): add S3/MSC object-storage support for MegatronPretrai… (#2234) by @hawkoli1987
feat(model): add Ling 2.0 / BailingMoeV2 (mini, flash, 1T) (#2242) (#2255) by @Hayden727
feat: add support for data_dir_list in [num_samples, path] form (#2229) by @rnyak
docs(fern): scaffold Fern docs site mirroring published v0.4.0 sidebar (#2196) by @lbliii
fix(streaming): raise TypeError from iterable dataset len/getitem (#2272) by @qiaochuz-nv
fix: adjust param_dtype during fully_shard_by_dtype traversal (#2271) by @akoumpa
feat(speculative): add llama eagle1 training recipe (#2263) by @khazic
feat(speculative): add llama eagle2 recipe alias (#2264) by @khazic
feat: qwen-omni3 audio ASR SFT recipe (#2280) by @yuekaizhang
fix(deepseek-v4): align HCA backward graph under FSDP2 (#2277) by @HaloWorld
fix: update VLM first-microbatch state during gradient accumulation (#2282) by @HuiyingLi
ci: add retrieval bi-encoder and cross-encoder nightly tests (#2042) by @oliverholworthy
docs: add embedding + reranker model coverage (#1843) by @akoumpa
refactor(ci): unify job construction in generate_ci_tests + add PR check (#2288) by @thomasdhc
feat: Support sliding-window masks for SDPA attention (#2281) by @HuiyingLi
feat(speculative): add checkpoint resume for EAGLE recipes (#2285) by @khazic
refactor(ci): extract finetune launcher config into ci_config + resolver (#2290) by @thomasdhc
docs: tutorial: add llama PEFT tutorial (#2293) by @krishnakalyan3
docs: add ERNIE, MiMo, and Ling announcements (#2292) by @HuiyingLi
feat: add optimized DeepSeek V4 kernels (#2076) by @hemildesai
feat: Support VLM context parallel inputs in PP (#2286) by @HuiyingLi
ci: skip uv lock generation on forks (#2252) by @chtruong814
feat: Add Nemotron-Labs-Diffusion (hybrid) SFT support (#2273) by @zyzhou5
feat(speculative): add SGLang serve helper for trained EAGLE drafters (#2294) by @khazic
feat(speculative): add offline dataset regeneration with EAGLE target model (#2307) by @khazic
docs(fern): add EAGLE speculative decoding e2e tutorial (#2308) by @khazic
fix(speculative): make EAGLE recipes complete a checkpoint save end-to-end (#2310) by @khazic
feat(speculative): add Phi-3 support for EAGLE-1/2/3 + correctness fixes (#2312) by @khazic
feat(speculative): add Qwen3 dense target support for EAGLE-1/2/3 (#2313) by @khazic
feat(examples): add two-node QLoRA recipe (#2309) by @HuiyingLi
fix(datasets): align reasoning + assistant loss masks for left padding (#2314) by @khazic
feat(vlm): add Nemotron-Omni RADIO post-load patches (#2311) by @yuekaizhang
fix(speculative): EAGLE-3 vocab-shrunk checkpoint resume shape mismatch (#2319) by @qiaochuz-nv
feat(speculative): add Qwen3-MoE target support to EAGLE-1/2/3 (#2317) by @khazic
ci: validate release branch-rules (#2295) by @ko3n1g
fix(speculative): EAGLE recipes re-track target_model on .to(device) (#2326) by @qiaochuz-nv
fix(datasets): serialize dict tool arguments to JSON string in xlam (#2323) by @khazic
ci: Add NVSkills request workflow (#2331) by @akoumpa
chore: Move contributor skills out of catalog path (#2330) by @akoumpa
chore(beep boop 🤖): symlink skills/ → .claude/skills, .agents/skills and AGENTS.md → CLAUDE.md by @github-actions[bot]
fix(datasets): parse JSON-encoded tools field in ChatDataset (#2324) by @khazic
ci: remove build-docs workflow (#2206) by @ko3n1g
feat(datasets): add multi-turn agent SFT dataset adapter (#2321) by @khazic
chore: add skill evaluation datasets (#2332) by @akoumpa
feat(speculative): add EAGLE-3.1 drafter toggles (fc_norm, norm_output) (#2322) by @khazic
feat(models): add Hy-MT2-30B-A3B SFT support (#2320) by @khazic
fix(datasets): merge tool_calls into prior assistant turn in agent_chat (#2325) by @khazic
fix(examples): switch agent SFT loss to FusedLinearCrossEntropy to avoid OOM (#2336) by @khazic
fix(training): clarify mixed-precision optimizer-state setup (#2248) by @yuhezhang-ai
chore: prefix AutoModel public skill names (#2339) by @akoumpa
ci: Update transformers to latest version 5.8.1 (#2223) by @svcnvidia-nemo-ci
docs: add Step-3.7-Flash docs by @athitten
feat: Add StepFun 3.7 support (#2344) by @athitten
feat: support Qwen2.5-Omni model (#2345) by @yuekaizhang
feat(datasets): preserve and optionally mask reasoning_content in agent SFT (#2348) by @khazic
ci: Clear up disk space for lint jobs (#2346) by @chtruong814
ci: add cluster_tag for reserved-cluster opt-in (#2353) by @thomasdhc
feat(datasets): add train_on_last_turn_only to agent chat SFT dataset (#2347) by @khazic
ci: override ep size for benchmark gptoss 120b (#2352) by @thomasdhc
feat: Add gemma4 drafter model support (#2240) by @athitten
perf(diffusion): improve Flux training throughput (#2251) by @pthombre
feat(datasets): drop history reasoning_content from agent SFT prompt (#2349) by @khazic
feat(eval): add tool-call accuracy evaluator for agent SFT validation (#2338) by @khazic
feat(vlm): wire BitsAndBytes QLoRA quantization into VLM finetune recipe (#2358) by @HuiyingLi
fix(datasets): tag agent SFT row errors with the example id (#2361) by @khazic
perf(datasets): skip full-conversation re-tokenization in chat loss mask (#2363) by @khazic
fix(datasets): warn when agent SFT seq_length has no effect (#2360) by @khazic
test(datasets): cover agent SFT recipe data path end-to-end (#2367) by @khazic
ci: Remove sync-skills workflow and revert recent skills symlink push (#2378) by @chtruong814
feat(examples): add offline tool-call eval for agent SFT checkpoints (#2368) by @khazic
ci(linting): free disk space before installing toolchain (#2380) by @ko3n1g
fix(datasets): warn on agent SFT rows with no supervised tokens (#2362) by @khazic
ci: align CUDA 13.2 / cu130 toolchain for TE 2.14.1 bump (#2121) by @thomasdhc
feat: Add late interaction model training support for retrieval (#2283) by @rnyak
docs(fern): add multi-turn agent (tool-calling) SFT tutorial (#2364) by @khazic
fix(speculative): serve EAGLE-3 drafts by resolving eagle_meta.pt (#2374) by @khazic
feat(datasets): auto-convert ShareGPT conversations to OpenAI messages (#2375) by @khazic
fix(models): make LlamaRotaryEmbedding honor position_ids values (#2377) by @khazic
perf(speculative): cache the EAGLE-3 draft-vocab token map across runs (#2371) by @khazic
feat(examples): add MiniCPM5-1B fine-tuning recipes (#2341) by @Ckcinnabar
fix(checkpoint): harden consolidated safetensors export (#2289) by @yuhezhang-ai
perf(diffusion): improve Hunyuan training throughput (#2383) by @pthombre
fix(speculative): declare flash_attention_2 support on EAGLE-3 draft (#2387) by @khazic
fix(speculative): rescale EAGLE-1/2 trailing grad-accum window (#2369) by @khazic
feat(datasets): add turn-aware history truncation to agent SFT dataset (#2366) by @khazic
fix(speculative): make EAGLE dataloader CUDA-fork-safe (#2386) by @khazic
fix(gpt_oss): free quantized expert tensors per-layer to reduce peak memory (#2149) by @stanley1208
feat(speculative): add EAGLE-3 offline target-output cache (legacy path) (#2373) by @khazic
feat(speculative): add SGLang acceptance/speedup benchmark for EAGLE drafters (#2372) by @khazic
chore: bump _code_freeze workflow to v1.4.2 (#2396) by @ko3n1g
feat: Add Gemma 4 12B config (#2402) by @HuiyingLi
ci: bump _release_library.yml to v1.4.3 (#2401) by @ko3n1g
feat(dllm): add DFlash and LLaDA2 SFT recipes (#2315) by @kashif
fix(agent-sft): harden tool-call eval and fix last-turn loss masking (#2365) by @khazic
perf(speculative): skip redundant DDP all-reduce in EAGLE grad accumulation (#2370) by @khazic
feat: adding PP and CP for nemotron v3 models (#2316) by @adil-a
feat(speculative): add gpt-oss EAGLE-3 draft model (#2399) by @khazic
feat(speculative): add remote target serving for EAGLE-3 training (#2398) by @khazic
feat: staging (#2411) by @adil-a
feat(speculative): add step-based and final checkpointing to EAGLE recipes (#2405) by @khazic
feat: Enable cycling through all positive documents in biencoder training #907 (#933) by @yuhezhang-ai
docs(fern): add Nemotron-3-Ultra-550B fine-tuning guide (#2420) by @adil-a
feat(speculative): add P-EAGLE parallel-drafting training for EAGLE-3 drafts (#2376) by @khazic
feat(speculative): log the draft model summary in EAGLE training (#2407) by @khazic
feat(speculative): add DFlash draft-model training recipe (#2406) by @khazic
feat(speculative): add P-EAGLE sequence partitioning for long-context training (#2409) by @khazic
feat(speculative): log EAGLE training metrics to Weights & Biases (#2408) by @khazic
perf(speculative): add fused Triton soft cross-entropy kernel for EAGLE-3 (#2428) by @khazic
refactor(speculative): decouple P-EAGLE from the EAGLE-3 code path (#2429) by @khazic
perf(diffusion): optimize Wan2.1 finetuning recipes (#2403) by @pthombre
feat(dllm): add Qwen3-4B dflash recipe, surface FSDP2 prefetch knobs (#2412) by @kashif
build: add managed = true to [tool.uv] (#2434) by @kajalj22
refactor: expose shared recipe builders from components (#2190) by @HuiyingLi
feat: mixture of mutiple ASR datasets training recipe (#2414) by @yuekaizhang
fix(speculative): guard PEAGLE flex attention compile (#2443) by @akoumpa
feat(models): support fused linear cross-entropy across custom models (#2397) by @akoumpa
chore(skills): refresh distributed training signature (#2438) by @akoumpa
refactor(datasets): unify reasoning_content coercion in agent chat (#2440) by @khazic
fix(tokenizer): make NeMoAutoTokenizerWithBosEosEnforced picklable (#2439) by @khazic
feat(speculative): add target_attn_implementation knob for EAGLE-3 target (#2415) by @kashif
feat(examples): add Falcon H1 fine-tuning recipes (#2334) by @Ckcinnabar
chore(ci): update codeowners to use NVIDIA-NeMo/core-am (#2453) by @akoumpa
fix: fp32 master weights for custom MoE models under FSDP2 (#1896) by @zpqiu
feat(bagel): add multimodal Bagel training support (#2275) by @zyzhou5
feat(models): add Qwen3.5 MTP support (#2417) by @HuiyingLi
feat: add use_memory_efficient_lora knob (#2239) by @akoumpa
fix(deepseek_v3): initialize weights in fp32 and default router to fp32 (#2450) by @yuhezhang-ai
feat(speculative): add EAGLE-3 sequence packing and reasoning-mode control (#2444) by @khazic
feat(distributed): add selective activation checkpointing for FSDP2 (#2389) by @yuhezhang-ai
feat(diffusion): improve qwen image finetuning configs (#2442) by @pthombre
ci: add nemo-run, split qwen-vl-utils from decord for arm (#2456) by @thomasdhc
fix(precision): dtype contract bug fixes for FSDP2 mixed-dtype loads (#2419) by @yuhezhang-ai
docs(speculative): add subsystem README, fold in regeneration guide (#2448) by @khazic
feat(diffusion): add Wan2.2 T2V-A14B two-stage finetuning support (#2284) by @linnanwang
fix(checkpoint): exclude TE _extra_state keys from load-time mismatch warning (#2247) by @adil-a
feat(moe): enable MXFP8 MoE training on GB200 (TransformerEngine + torchao) (#2394) by @hemildesai
fix(speculative): embed d2t/t2d vocab remap in EAGLE-3 draft checkpoint (#2447) by @khazic
feat: Add query functionality of Model Capability Registry (#2423) by @athitten
refactor(speculative): reuse shared dflash mask and loss in the trainer (#2433) by @kashif
fix(moe): include MTP modules in FSDP sync traversal (#2441) by @HuiyingLi
ci: update package version to 0.5.0 (2472) (#2473) by @svcnvidia-nemo-ci
feat: make mesh accept meshcontext (2266) (#2474) by @svcnvidia-nemo-ci
feat(vlm): enable Qwen3.5 MoE VLM CP (2432) (#2483) by @svcnvidia-nemo-ci
feat(model): flux2 (2145) (#2489) by @svcnvidia-nemo-ci
fix(config): glm4.7 yaml (2527) (#2528) by @svcnvidia-nemo-ci
fix(gemma4): cast dense params without casting buffers (2359) (#2525) by @svcnvidia-nemo-ci
fix: unwrap ModelOutput to extract logits (2523) (#2530) by @svcnvidia-nemo-ci
fix(qwen3_5): make dense VLM pipeline-parallel safe (2524) (#2554) by @svcnvidia-nemo-ci
feat(examples): add Nemotron-3-Ultra-550B benchmark and full-SFT recipes (2539) (#2550) by @svcnvidia-nemo-ci
ci: schedule ep-parallel finetune recipes at documented node counts (2546) (#2558) by @svcnvidia-nemo-ci
fix(diffusion): resolve flux nightly CI failures (2529) (#2567) by @svcnvidia-nemo-ci
fix(ci): bump ling_1t_lora_pp local_batch_size to satisfy PP assert (2575) (#2578) by @svcnvidia-nemo-ci
test: fix all 5 vllm_deploy tests (token drift, nemotron OOM + mamba merge) (2559) (#2576) by @svcnvidia-nemo-ci
fix(checkpoint): preserve tied lm_head on resume (2511) (#2573) by @svcnvidia-nemo-ci
fix(ci): set node counts for multi-node VLM finetune recipes (2574) (#2579) by @svcnvidia-nemo-ci
fix(recipe): reshard MoE experts after forward in nemotron_nano_v3_cp_test (2577) (#2580) by @svcnvidia-nemo-ci
ci: use digits for spark recipes (2581) (#2583) by @svcnvidia-nemo-ci
ci: Enable activation checkpointing for gemma_2_9b_it_squad (AM-464) (2585) (#2586) by @svcnvidia-nemo-ci
fix(test): load checkpoint-robustness HF reference via device_map (2582) (#2588) by @svcnvidia-nemo-ci
fix(peft): LoRA MLP QLoRA/PP/gemma3n fixes (AM-435, AM-447, AM-453) (2584) (#2597) by @svcnvidia-nemo-ci
fix(vlm): enable activation checkpointing for 35B Qwen3.5/3.6 VLM recipes (2600) (#2602) by @svcnvidia-nemo-ci
fix(gemma4): FSDP2-safe kv-sharing + skip frozen audio tower on grad-accum (2566) (#2599) by @svcnvidia-nemo-ci
fix(vlm): use FusedLinearCrossEntropy for qwen3_5_9b to avoid logits OOM (2603) (#2604) by @svcnvidia-nemo-ci
fix(distributed): register Falcon-H1 TP plan to fix 34B PEFT OOM (2589) (#2605) by @akoumpa
fix(models): keep RoPE frequency buffers fp32 under bf16 model cast (2549) (#2606) by @akoumpa
fix: use TE attention for gpt_oss packed-sequence recipe (AM-438) (2587) (#2611) by @svcnvidia-nemo-ci
fix(oom): use FusedLinearCrossEntropy in qwen3 tulu3 configs to avoid OOM (2609) (#2612) by @svcnvidia-nemo-ci
perf(distributed): add retrieval tuning knobs (2452) (#2607) by @svcnvidia-nemo-ci
fix(moe): weight GroupedExpertsTE down-projection bias by routing probability (2591) (#2610) by @svcnvidia-nemo-ci
fix(qwen35moe):convert MTP experts as grouped(AM-442)(2595) (#2618) by @svcnvidia-nemo-ci
fix(bagel): distributed setup init (2608) (#2613) by @svcnvidia-nemo-ci
fix(transformers): keep gemma3n KV sharing working under FSDP2 (AM-454) (2594) (#2619) by @svcnvidia-nemo-ci
fix(docker): build DeepEP against the NVSHMEM wheel matching the apt runtime (2614) (#2629) by @svcnvidia-nemo-ci
fix(qwen3_moe): keep native forward under PP so CP+THD works (2625) (#2628) by @svcnvidia-nemo-ci
fix(config): validate pp_size against distributed.pipeline (2616) (#2631) by @svcnvidia-nemo-ci
fix(moe): default ignore_router_for_ac=True for activation checkpointing (2635) (#2636) by @svcnvidia-nemo-ci
ci: raise ci.time for slow finetune recipes hitting 10-min default (2637) (#2640) by @svcnvidia-nemo-ci
ci: cap MAX_STEPS to 10 for slow vlm_finetune recipes (2639) (#2641) by @svcnvidia-nemo-ci
fix(parallelizer): resolve NemotronH decoder blocks for Nemotron-V3 (2638) (#2642) by @svcnvidia-nemo-ci
refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep (2630) (#2644) by @akoumpa
fix(datasets): decode MedPix images on demand instead of up front (2645) (#2650) by @svcnvidia-nemo-ci
fix(devstral2,ministral3): load FP8 checkpoints via custom mistral3_vlm path (drop HF FineGrainedFP8) (2654) (#2666) by @svcnvidia-nemo-ci
feat(qwen3_5): port dense Qwen3.5 to a custom-model(2557) (#2663) by @akoumpa
fix(examples): enable ac for phi_4_squad (2634) (#2661) by @akoumpa
feat(config): disable W&B in example configs (2643) (#2662) by @akoumpa
feat(moe): MTP FLOPs accounting fix (2486) (#2660) by @akoumpa
cp: fix(merge_lora) (#2653) to r0.5.0 (#2667) by @svcnvidia-nemo-ci
cp: fix(datasets) (#2649) to r0.5.0 (#2665) by @svcnvidia-nemo-ci
cp: fix(distributed) (#2655) to r0.5.0 (#2668) by @svcnvidia-nemo-ci
cp: fix(models) (#2652) to r0.5.0 (#2669) by @svcnvidia-nemo-ci
fix(moe): preserve fp32 A_log in Qwen3.5-{MoE,Next GatedDeltaNet} (2484) (#2664) by @akoumpa
cp: fix(model) (#2657) to r0.5.0 (#2671) by @svcnvidia-nemo-ci
fix(recipe): disable fused RoPE for MLA packed-sequence MoE recipes (2675) (#2677) by @svcnvidia-nemo-ci
fix(docker): bump DeepEP to 42144303 to pad HybridEP token capacity (2678) (#2680) by @svcnvidia-nemo-ci
cp: fix(training) (#2672) to r0.5.0 (#2679) by @svcnvidia-nemo-ci
cp: fix(checkpoint) (#2682) to r0.5.0 (#2685) by @svcnvidia-nemo-ci
cp: fix(qwen3_moe) (#2687) to r0.5.0 (#2688) by @svcnvidia-nemo-ci
fix(vlm): bump mistral3p5_128b_medpix max_length 1024->2048 (2689) (#2693) by @svcnvidia-nemo-ci
cp: fix(llama3_3) (#2673) to r0.5.0 (#2684) by @svcnvidia-nemo-ci
test(models): speed up qwen3.5 moe/vl-moe from_pretrained unit tests (2698) (#2699) by @svcnvidia-nemo-ci
perf(checkpoint): mmap HF DCP read_data to avoid host-RAM OOM on large loads (2690) (#2701) by @svcnvidia-nemo-ci
fix(mistral3): remap FP8 VLM checkpoint prefixes (2692) (#2702) by @svcnvidia-nemo-ci
ci: add cluster_tag to gb200 benchmarks (2714) (#2715) by @svcnvidia-nemo-ci
fix(loss): reuse LM head gather for MTP loss (2694) (#2708) by @akoumpa
fix(gemma4_moe): re-tie lm_head to active embed_tokens on MoE path (2601) (#2709) by @svcnvidia-nemo-ci
ci: fix 26.06 release cves (2705) (#2711) by @svcnvidia-nemo-ci
fix(moe): handle non-EP expert weight DTensors (2697) (#2712) by @svcnvidia-nemo-ci
build: install tilelang + tile_kernels for DeepSeek-V4 recipes (2683) (#2717) by @akoumpa
fix(checkpoint): super-49B consolidated reload and vllm_deploy (2626) (#2718) by @svcnvidia-nemo-ci
fix(models): use bool sparse masks for sdpa (2624) (#2721) by @svcnvidia-nemo-ci
ci: add time budgets for 12 new timeout failures (2707) (#2724) by @svcnvidia-nemo-ci
cp: DeciLM Nemotron TP plan (#2703) to r0.5.0 (#2726) by @akoumpa
fix(gemma4): avoid DynamicCache OOM on dense E2B/E4B via kv-share holder (#2725) by @athitten
fix(qwen3_5): handle packed MTP attention (2727) (#2729) by @svcnvidia-nemo-ci
ci: add gb200 cluster specification for nemotron_ultra recipe (2733) (#2734) by @svcnvidia-nemo-ci
fix(ci): pin qwen3_moe_30b mxfp8 finetune to gb200 (2735) (#2737) by @svcnvidia-nemo-ci
ci: address diffusers cve (2706) (#2738) by @svcnvidia-nemo-ci
ci: address thrift cve bump to 0.23.0 (2736) (#2742) by @svcnvidia-nemo-ci
fix(deepseek-v4): restore batch axis for packed-sequence (THD) forward (2651) (#2745) by @svcnvidia-nemo-ci
fix(deepseek-v4): avoid bf16 -inf overflow in additive attention mask (2658) (#2746) by @svcnvidia-nemo-ci
fix(ci): reduce mixtral release smoke batch (2728) (#2751) by @akoumpa
build: install TileKernels for DeepSeek V4 (2740) (#2750) by @svcnvidia-nemo-ci
fix(qwen3_moe): step-0 NaN in MXFP8 packed finetune — expert unload + fused RoPE (2722) (#2753) by @svcnvidia-nemo-ci
fix(fsdp2): guard uninitialized accumulated grads (2744) (#2752) by @svcnvidia-nemo-ci
ci: bump benchmark glm_4.7_flash_te_deepep time (2757) (#2759) by @svcnvidia-nemo-ci
fix(diffusion): raise qwen-image dist timeout for checkpoint consolidation (2748) (#2756) by @svcnvidia-nemo-ci
fix(diffusion): reuse warm HF cache instead of re-downloading models (2747) (#2754) by @svcnvidia-nemo-ci
fix(mistral3): preserve medium VLM checkpoint layout (2758) (#2762) by @svcnvidia-nemo-ci
fix(sdpa): apply resolved backend constraints to custom models (2761) (#2765) by @svcnvidia-nemo-ci
ci: run dsv32_lora, kimi_k2 and qwen3_moe_235b deepep benchmarks online (2773) (#2774) by @svcnvidia-nemo-ci
fix(distributed): use flattened CP FSDP mesh (2768) (#2769) by @svcnvidia-nemo-ci
fix: Remove dali from container (2770) (#2771) by @svcnvidia-nemo-ci
fix(benchmark): skip unsupported MTP flops (2767) (#2784) by @svcnvidia-nemo-ci
fix: Qwen3.5 MedPix EP32 NCCL timeout (2777) (#2785) by @svcnvidia-nemo-ci
fix: skip fused LoRA MLP install for meta weights (2775) (#2781) by @svcnvidia-nemo-ci
fix(ci): address go-git/go-billy and rustls-webpki CVEs (2780) (#2787) by @svcnvidia-nemo-ci
fix(ci): stabilize diffusion finetune smoke tests (2788) (#2791) by @svcnvidia-nemo-ci
build(deps): move ffmpeg/opencv deps to opt-in media extra (2743) (#2792) by @svcnvidia-nemo-ci
fix(ci): HybridEP bench + LoRA OOM fixes (2789) (#2794) by @svcnvidia-nemo-ci
fix(vlm): keep Qwen3.5 media tokens aligned (2772) (#2793) by @akoumpa
fix: qwen3.5 and 3.6 mtp expert checkpoint layout (2778) (#2795) by @svcnvidia-nemo-ci
fix(ci): drop base-image uv/wandb copies flagged for CVEs (2800) (#2803) by @svcnvidia-nemo-ci
fix(optim): align Dion mesh with FSDP sharding (2808) (#2812) by @svcnvidia-nemo-ci
fix(ci): stabilize failed benchmark recipes (2817) (#2818) by @svcnvidia-nemo-ci
feat(diffusion): support Hugging Face datasets (2816) (#2831) by @svcnvidia-nemo-ci
cp: docs: document opt-in media extras (vlm-media/diffusion-media) (#2799) (#2848) by @chtruong814

Contributors

Thank you to all our contributors for helping make NeMo AutoModel v0.5.0 possible!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

NVIDIA NeMo-Automodel 0.5.0

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

Contributors

Contributors

Uh oh!