NeMo AutoModel 26.10 Release Notes

Highlights

Expanded model coverage. New training paths for DeepSeek-V4.1-Flash,
GLM-5.3-Flash, Qwen3.8-Flash-Next, MiMo-V2.5-Pro and MiMo-V2.6, and additional
vision-language models.
More MoE training options. Mixture-of-Kittens (MoK) BF16 and MXFP8 experts,
compact HybridEP routing, and MXFP4-resident DeepSeek-V4-Flash experts for LoRA.
Speculative and discrete-diffusion training. DFlash 2, additional Kimi K3
and DeepSeek-V4.1 draft-training paths, streaming EAGLE-3 feature components,
I-DLM training, and SCDD training and sampling.
Text-to-image fine-tuning. HunyuanImage-3.0 with expert parallelism and
Qwen-Image-2.1 with full-parameter and LoRA recipes.
Training and integration improvements. Native FlashAttention-4, explicit
packing contracts, weighted domain mixtures, lower-memory checkpoint loading,
and public registration of custom model architectures.

New and Expanded Model Support

LLM and MoE

DeepSeek-V4.1-Flash: Native model and checkpoint conversion, attention and
expert LoRA, packed SFT and context parallelism, H100-80GB recipes, and MedPix
vision SFT. The release includes the CP8 gradient-synchronization fix.
(model guide, #3855, #3872, #3879, #3962, #4221)
GLM-5.3-Flash: Native hybrid KDA/DSA model, vision processing, distributed
checkpoint loading, and packed context-parallel training. The MedPix recipe
uses EP72/CP2 on 72 H100 GPUs; sparse attention combines FlashMLA forward with
cuDNN backward. TP, PP, video training, and full-model single-GPU loading are
outside this model's supported path. GLM-5.2 and GLM-5.3 also gain cuDNN DSA
training backends.
(model guide, #3699, #3127, #3747)
Qwen3.8: Adds 27B fine-tuning recipes and native Flash-Next language-only
training with a sharded Engram embedding table, sparse QSA attention, packed
sequences, and contiguous context parallelism. Flash-Next's QSA indexers remain
frozen; its checkpoint MTP head is not loaded. TP and PP are disabled in the
documented Flash-Next configurations.
(model guide, #3560, #3690)
MiMo: Adds MiMo-V2.5-Pro, MiMo-V2.6, and MiMo-V2.6-Pro support. The
MiMo-V2.6-Pro-RL recipes cover packed 64K Tulu3 text, MedPix full fine-tuning,
and MedPix LoRA on GB200. Keep expert-parallel groups within an NVLink domain.
(recipe guide, #2514, #3992, #4054)
Additional architectures: Dense Nemotron-H Nano 4B, a Laguna XS 2.1
training recipe, and encoder-decoder fine-tuning through
NeMoAutoModelForSeq2SeqLM, including a T5-small SQuAD example. The seq2seq
path does not wire up CP, TP, PP, or sequence packing.
(#2670, #3639, #2860)

Vision-Language Models

DeepSeek-V4-Flash Vision and Muse Glimmer: Adds the DeepSeek-V4 vision
path and native Muse Glimmer training, checkpoint conversion, and distributed
model support.
(#3772, #3476)
Cohere North Micro Vision: Adds Shopify product-catalogue and RDR LoRA
recipes, including checkpoint resume. This preview model uses remote-code
dependencies and is excluded from automatic release-scope CI discovery.
(#3518, #3400)
Existing VLM improvements: Gemma 4 gains D512 FFPA attention on SM100 and
sliding-window communication improvements. MiniMax-M3 gains SM100 sparse
attention with a custom backward pass. Qwen3.5 gains packed vision training
with Transformer Engine and MTP with VLM context parallelism.
(#3742, #2984, #3809, #4303, #3990)

Speculative Decoding and Discrete-Diffusion LLMs

DFlash 2: Draft model, trainer, Qwen3 and Qwen3.8-27B recipes, dynamic
convolution, and candidate-path selection. Sampling supports top-p and top-k.
(#3605, #3627)
Additional draft targets and parallelism: Kimi K3 DSpark, EAGLE-3, and
DFlash training; DeepSeek-V4.1 DSpark training; and Ulysses context parallelism
for the EAGLE-3 draft.
(#3263, #3286, #3287, #3875, #3356)
Streaming supervision components: EAGLE-3 feature production and loading,
an asynchronous producer, and a safetensors-backed shared-directory feature
store with backpressure.
(#3085, #3094)
I-DLM and SCDD: I-DLM all-masked training uses dllm.mode: idlm; SCDD
adds mixed mask/uniform corruption, training, and sampling with
dllm.mode: scdd. Neither new mode supports context parallelism. SCDD does
not currently maintain the reference method's EMA weights.
(#2437, #3354)

Diffusion

HunyuanImage-3.0: Native MoE text-to-image implementation with FSDP2,
expert parallelism, LoRA, full fine-tuning, preprocessing, and generation.
Recipes cover single-node LoRA and four-node full fine-tuning. TP, PP, CP,
image-to-image conditioning, and text-generation training are unsupported.
(model and recipe guide, #4113, #4129)
Qwen-Image-2.1: Full-parameter and LoRA text-to-image fine-tuning, a
dedicated latent/prompt preprocessing path, and generation recipes. Requires
a Diffusers build providing QwenImage21Pipeline.
(model and recipe guide, #4027)
Validation: Periodic diffusion validation loss, retention of validation
tail batches, and sample-weighted validation loss.
(#3579, #4291)

Retrieval and Embeddings

Context-aware reranking: Train rerankers with query-context pairs and
configurable prompt augmentation.
(#3786)
Mistral3 vision-language retrieval: Embedding training, reranking, and
multimodal hard-negative mining for text, images, and image-plus-text
documents. Exports preserve retrieval prompts, pooling, normalization,
attention policy, and scoring temperature; release backports fix portable
reranker exports and Pixtral image-attention boundaries.
(#3915, #4197, #4267)

Training, Parallelism, and Performance

MoK experts: Optional dispatcher: mok integration for fused dispatch
and expert computation, plus MXFP8 experts. Supported shapes, precision,
topology, and adapter combinations are checked explicitly.
(#3422, #3606)
Lower-memory DeepSeek-V4-Flash LoRA: MXFP4-resident frozen experts avoid
keeping a full BF16 expert copy. This path requires expert parallelism and
torch_mm experts with the torch dispatcher; DeepEP experts are unsupported.
(#2548)
MoE and Kimi K3 efficiency: Compact HybridEP routing, reuse of DeepEP
dispatch layouts during activation-checkpoint recomputation, fused gradient
norm reduction, pipeline receive-buffer reuse, and opt-in Kimi K3 kernels.
(#3404, #3684, #3685, #3780, #3954)
Kimi Linear: Adds packed sequences and context parallelism.
(#3243)
Weighted domain mixtures: Configure sampling frequency separately from
each domain's objective weight for Megatron continual pretraining, with
per-domain token metrics and weighted validation. This initial path rejects
packing, PP, CP, JSON blends, and other unvalidated combinations.
(#3303)
Correctness fixes: Expert-parallel gradient clipping, TP replica
synchronization, TP knowledge-distillation gradient reduction, per-token
parallel cross-entropy gradients, and preservation of model input/output
dtypes through FSDP.
(#3817, #3822, #3777, #3853, #4150)

Data, Checkpointing, and Extensibility

Data paths: SynthTraces agent SFT, Shopify product-catalogue VLM data,
typed Datum records and collate-time THD packing, and mask_generation_prompt
to exclude generation-prompt tokens from loss.
(#3371, #3400, #3514, #3739)
Checkpoint loading: DCP loading of standard Hugging Face safetensors,
broader low-memory MoE loading, and bounded quantized/GPT-OSS MXFP4 loading.
(#3616, #3610, #3619, #3623)
PEFT export and resume: Fixes for grouped-expert merged exports,
DeepSeek-V4 LoRA reload/resume, Nemotron adapter namespaces and generation
configuration, and processor artifacts in merged VLM adapters.
(#3834, #4072, #3866, #3891, #3270)
Custom architectures: Register an architecture through the public
nemo_automodel.register_architecture() API or the
nemo_automodel.architectures entry-point group.
(#3645)
Freezing: freeze_config.freeze_modules and unfreeze_modules accept
module selectors and glob patterns.
(#3681)

Compatibility and Upgrade Notes

Dependency baseline: The release branch pins transformers==5.18.0
and requires torchao>=0.16.0. Its Dockerfile uses the NVIDIA PyTorch
26.08-py3 base. Use the release branch's dependency files when building an
environment; individual model guides may still mention older intermediate
Transformers pins.
(package dependencies,
container definition)
Optional experiment tracking: MLflow and Weights & Biases are optional
extras. Install nemo-automodel[mlflow] for the tracking client,
nemo-automodel[mlflow-full] for the full MLflow stack, or
nemo-automodel[wandb] for W&B.
(#3785,
extra definitions)
Native FA4: Select model.backend.attn: fa4 in an INSTALL_FA4=true
image. The native backend supports BSHD inputs and explicitly selected NEAT
packing at cp_size: 1; legacy THD packing and dense attention masks are
unsupported. Packed Qwen3Next recurrent attention and DeepSeek V3.2 sparse
attention are also unsupported. On SM100/SM110, use SDPA for local attention
with 256-wide query/key and value heads. HF-dispatched models retain their
separate HF flash_attention_4 implementation.
(native FA4 notes)
Packing API migration: Packing follows the built model's attention
dispatch; packed_sequence.attn_implementation is ignored with a warning.
Replace get_attn_implementation(cfg_model, model) with
get_model_attn_implementation(model); configure multiple pipeline stages
with configure_packing_for_models(model_parts). Pass the returned contract
as packing= to the collater or packing_contract= to the dataloader's
build(). Custom integrations must forward model-specific packing metadata.
HF flash packing now uses public varlen kwargs without modifying Transformers
globals. The old VLM dataloader backend resolver was removed.
(migration details)
Qwen3.5 packed CP: Load-balanced packed context parallelism now raises
instead of silently losing document isolation. Use cp_size: 1, or the
supported SDPA block-diagonal CP path for Qwen3.5-MoE.
(packing support boundaries)
Custom retrieval loaders: Pass backbone options such as
attn_implementation="sdpa" by keyword; positional arguments after the
checkpoint path are forwarded to the backbone constructor.
(#3915)

External Contributors

Thank you to the following community contributors for the code, documentation,
tests, and fixes included in 26.10, and to everyone who reported issues,
reviewed changes, and helped validate the release.

@akx — expert-parallel collective deadlock fixes, Kimi-K2.5 checkpoint and quantization fixes, lint updates, and distributed-filesystem checkpoint handling. (#3631, #3632, #3633, #3686)
@Anai-Guo — removing a shadowed Kimi-K2.5-VL forward-test class. (#3873)
@andrewwhitecdw — removing an unused CI summary environment variable. (#3536)
@beccohov — typed packing metadata in place of the global Hugging Face monkeypatch, and periodic diffusion validation loss. (#3577, #3579)
@Butterfingrz — Gemma 4 D512 FFPA attention on SM100 and MiniMax-M3 SM100 sparse attention with a custom backward pass. (#3742, #3809)
@coderaBruce — preserving fused losses through MiniMax-M3-VL's logits_to_keep contract. (#3511)
@coding-famer — saving processor artifacts when merging VLM LoRA adapters. (#3270)
@dundysm — accepting the singular embedding-key alias in Nemotron V3 Hugging Face checkpoints. (#3999)
@edjson — preserving FP32 routing weights in DeepSeek and Ling models. (#3746, #3861)
Daniel Shen (@excepshenal, @dshen-crusoe) — MXFP4-resident MoE experts for DeepSeek-V4-Flash LoRA. (#2548)
@Faisal-Alsrheed — correcting the training dataloader annotation and modernizing optional types. (#3534)
@haowen-han — preserving FP32 rotary frequencies in the Qwen3.5 vision tower. (#3856)
@HuayiJin — keeping Qwen3.8 Flash Next flex-QSA membership lookup in int64. (#3802)
@kabirvashisht4-glitch — documentation corrections and fixes to dataset logging and collation, checkpoint sizing, sequence-classification loss, InfoNCE validation, scheduler resume, launcher overrides, and the torchao dependency floor. (#3762, #3830, #3883, #4131)
@kashif — I-DLM and SCDD training, streaming EAGLE-3 features, DFlash 2 training and sampling, DeepSeek-V4.1 DSpark support, and DiffusionGemma documentation. (#2437, #3094, #3354, #3605, #3875)
@khazic — speculative-decoding training, correctness fixes and documentation; Kimi Linear packing and context parallelism; SynthTraces data support; DeepSeek-V4.1-Flash training; and HunyuanImage-3.0 fine-tuning with expert parallelism. (#3243, #3286, #3356, #3855, #4129)
@linmuchuiyang — selective activation-checkpoint replay fixes, DeepSeek-V4.1 H100 recipes and data preparation, and the CP8 gradient-synchronization fix backported into this release. (#3961, #3962, #4203, #4221)
@mikemikimike — preventing tensor-parallel knowledge-distillation gradient over-reduction. (#3777)
@MohammedAlkindi — documentation and coverage-link repairs, UTF-8 configuration loading, multi-node SkyPilot rendezvous, SQuAD handling, PEFT compatibility and wildcard fixes, validation-loss reduction, and optimizer-group scheduling. (#3769, #4037, #4043, #4045, #4050)
@MrCapricornLiu — Mistral tokenizers in the distillation collator, InfoNCE masking order, and per-token gradients in parallel cross-entropy. (#3849, #3850, #3853)
@nathan-az — Gemma 4 sliding-window attention communication and FlashAttention improvements. (#2984)
@piyushumate — making MLflow and Weights & Biases optional dependencies. (#3785)
@ralovets — base-weight loading with DDP and Megatron FSDP, RTX PRO 6000 MFU recognition, and multi-source dataset blending fixes. (#3597, #3598, #3919)
@rouchenzi — keeping Qwen3.8 Flash Next PLE hashing and convolution within packed document boundaries. (#4132)
@sahel-sh — configurable metric buffering, retaining retrieval metrics on preemption or error, and context-aware reranker training. (#3541, #3583, #3786)
@Simar-malhotra09 and credited co-author Matilda-Coder — MiMo-V2.5-Pro model support. (#2514)
@stanley1208 — FP8 loading checks, dense Nemotron-H and encoder-decoder support, PEFT checkpoint naming and export fixes, expert-parallel gradient clipping, generation-config export, and shared PEFT reload tests. (#2350, #2670, #2860, #3817, #3945)
@taran-dev4u — VLM processor serialization tests and Nemotron Omni PEFT save/reload key conversion. (#3379, #3986)
@waynel96 — GLM-5.3-Flash installation and parity documentation, KDA execution fixes, and the GLM5 Next training chat template. (#3939, #4077, #4110, #4165)
@yh0903 — avoiding retained routed inputs in chunked MoE checkpoints. (#4103)
@yuvalnissan — honoring the checkpoint's GLM indexer rotary layout. (#4142)
@ZenAlexa — advancing the DeepSeek-V4 pipeline media cursor for text-only microbatches. (#3821)
@zeryx — the Shopify product-catalogue VLM dataset and Cohere Micro Vision model support. (#3400, #3518)

Changelog Details

fix(checkpoint): validate PEFT adapter-only state by @akoumpa :: PR: #3501
ci(convergence): fix gemma4 eval setup and re-baseline Qwen3-MoE by @athitten :: PR: #3493
fix(deepseek_v4): avoid TileLang boolx8 backward codegen by @HuiyingLi :: PR: #3467
fix(Qwen3.5): Support split FP8 expert checkpoints for Qwen3.5-MoE by @HuiyingLi :: PR: #3495
fix(checkpoint): preserve FSDP2 mixed precision during recompute by @jQizhang :: PR: #3513
docs(models): add Qwen3.8-Max coverage by @HuiyingLi :: PR: #3517
feat(models): add cohere micro vision by @zeryx :: PR: #3518
fix(test): use the recipe's attention backend for the HF reload parity check by @athitten :: PR: #3500
feat(examples): add Qwen3-32B Tulu-3 finetune recipes by @athitten :: PR: #3104
feat(moe): integrate Mixture-of-Kittens backend by @HuiyingLi :: PR: #3422
fix(deps): resolve Starlette and GitPython CVEs by @thomasdhc :: PR: #3523
docs(cohere): add absolute recipe links and remove dead blog reference by @HuiyingLi :: PR: #3526
ci: AUT-1501 bump SSO preflight to v1.8.10 by @svcnemo-autobot :: PR: #3537
feat(loss): return per-depth MTP losses by @Slyne :: PR: #3525
docs(models): rename Qwen3.8-Max to Qwen3.8-2.4T-A95B by @HuiyingLi :: PR: #3549
fix(retrieval): support canonical Sentence Transformers metadata by @qiaochuz-nv :: PR: #3546
feat(gemma4): add E-series tensor parallelism by @HuiyingLi :: PR: #3512
feat(datasets): Datum record and collate-time THD sequence packing by @HuiyingLi :: PR: #3514
fix(minimax_m3_vl): declare and honour logits_to_keep so fused losses are kept by @coderaBruce :: PR: #3511
ci: route DGX Spark recipes to GB10 by @akoumpa :: PR: #3538
fix(docker): build bitsandbytes for SM121 by @akoumpa :: PR: #3553
feat(qwen): add Qwen3.8-27B fine-tuning recipes by @akoumpa :: PR: #3560
fix(fsdp): uniform reduce dtype and EP-local expert gradients by @akoumpa :: PR: #3540
fix(nemotron_omni): support mixed-resolution image batches in tile path by @HuiyingLi :: PR: #3559
fix(vlm): use FP32 master weights for Nemotron Omni by @HuiyingLi :: PR: #3444
fix(checkpoint): restore Gemma4 Unified HF export keys by @khazic :: PR: #3260
fix(metric_logger): expose the logger buffer size in the factory builder by @sahel-sh :: PR: #3541
fix(training): correct dataloader annotation and modernize Optional syntax by @Faisal-Alsrheed :: PR: #3534
fix(docker): harmonize non-root runtime contract by @ko3n1g :: PR: #3484
fix(deps): resolve msgpack, wandb, and mistune container CVEs by @thomasdhc :: PR: #3563
fix(mtp): support context-parallel caller inputs by @Slyne :: PR: #3543
fix(moe): stabilize async DeepEP checkpointing by @pzelasko :: PR: #3522
fix(recipes): support MTP with context parallelism by @Slyne :: PR: #3544
test(checkpoint): improve resume correctness diagnostics by @yuhezhang-ai :: PR: #3562
fix(model_init): stream Devstral FP8 checkpoints by @yuhezhang-ai :: PR: #3413
fix(checkpoint): restore single-GPU custom-model DCP loading by @HuiyingLi :: PR: #3533
feat(nemotron): allow skipping logits materialization by @pzelasko :: PR: #3555
fix(kimi_linear): make long-context recipe CP8 by @HuiyingLi :: PR: #3571
perf(glm): skip top-k pipeline carry without IndexShare by @akoumpa :: PR: #3573
docs(dllm): align DiffusionGemma guides with the SFT recipe by @kashif :: PR: #3550
fix(minimax): declare packed-sequence and CP attention ownership by @athitten :: PR: #3547
test: add functional coverage for under-tested recipes by @akoumpa :: PR: #3584
fix(perf): honor configured peak in MFU logging by @yuhezhang-ai :: PR: #3569
feat(datasets): support SynthTraces agent SFT by @khazic :: PR: #3371
fix(kimi_k3): apply the HF renames to expert LoRA keys on save by @stanley1208 :: PR: #3435
refactor(checkpoint): gate direct loads by adapter capability by @yuhezhang-ai :: PR: #3574
fix(pp): keep accumulated grads across stage re-initialization by @athitten :: PR: #3530
fix(vlm): truncate only the token axis in gemma4 collate by @athitten :: PR: #3531
perf(Gemma4): reduce comms in sliding window, FA implementation, by @nathan-az :: PR: #2984
feat(vlm): add CP vision shape floor by @yuhezhang-ai :: PR: #3581
test(pp): add gemma4 parallelism parity tests [1/4] by @athitten :: PR: #3529
fix(diffusion): apply compute_dtype via autocast when FSDP2 skips by @zyzhou5 :: PR: #3589
feat(examples): Qwen3-32B CoderForge long-context CP data + SFT recipe by @athitten :: PR: #3594
feat(glm): add cuDNN DSA backend for GLM-5.2 by @HuiyingLi :: PR: #3127
chore(lint): enforce PEP 604 optional annotations by @akoumpa :: PR: #3535
fix(lint): update Qwen3 prefilter optional annotations by @akoumpa :: PR: #3602
fix(deps): resolve 26.08 rc9 container CVEs by @thomasdhc :: PR: #3607
fix(deps): align pyproject DeepEP pin with the container build by @akoumpa :: PR: #3608
feat(dflash): add DFlash 2 draft model, trainer, and recipe by @kashif :: PR: #3605
docs: announce DFlash 2 support in the README by @khazic :: PR: #3624
fix(perf): recognize RTX PRO 6000 for MFU by @ralovets :: PR: #3598
fix(ci): AUT-1630 route autobot through internal queue by @svcnemo-autobot :: PR: #3629
fix: remove unused RUN_ID env var from CICD summary job by @andrewwhitecdw :: PR: #3536
fix(checkpoint): load base weights under DDP and MegatronFSDP by @ralovets :: PR: #3597
feat(speculative): add Kimi K3 DFlash draft training by @khazic :: PR: #3287
feat(streaming): async producer + SharedDirFeatureStore by @kashif :: PR: #3094
feat(speculative): add Ulysses context parallelism for the EAGLE-3 draft by @khazic :: PR: #3356
fix: EP collective deadlock with variable-length token counts (LoRA flavor) by @akx :: PR: #3631
feat(laguna): add Laguna XS 2.1 training recipe by @akoumpa :: PR: #3639
fix(kimi_k25): map lm_head to model.language_model.lm_head by @akx :: PR: #3632
refactor(checkpoint): StateDictAdapter method target-module rename by @stanley1208 :: PR: #3591
test(pp): add deepseek_v4 PP and EP parity tests [2/4] by @athitten :: PR: #3613
feat(dllm): add SCDD training and sampling by @kashif :: PR: #3354
fix(deps): resolve 26.08 rc10 container CVEs by @thomasdhc :: PR: #3647
docs(examples): add Gemma4-31B SWE-bench eval steps + base-vs-SFT agentic result by @athitten :: PR: #3585
perf(checkpoint): reduce allocating grouped MoE load overhead by @yuhezhang-ai :: PR: #3580
test(checkpoint): expand parity metrics and phase coverage by @yuhezhang-ai :: PR: #3567
perf(checkpoint): avoid full GC scans during MoE export by @yuhezhang-ai :: PR: #3621
chore: bump lint tools + fix lint issues by @akx :: PR: #3634
test(ci): report Jensen-Shannon checkpoint parity metrics by @yuhezhang-ai :: PR: #3620
fix(test): unblock the Kimi-Linear vanilla-HF parity reference by @yuhezhang-ai :: PR: #3659
docs(examples): add Qwen3-32B SWE-bench Verified eval (Phase 3) by @athitten :: PR: #3671
fix(moe): equalize and align per-rank token counts for HybridEP dispatch by @HuiyingLi :: PR: #3641
fix(peft): support mixed-dtype memory-efficient LoRA backward by @akoumpa :: PR: #3675
docs(models): add Qwen3.8-Flash-Next coverage by @HuiyingLi :: PR: #3691
fix(retrieval): register Llama classifier auto class by @oliverholworthy :: PR: #2905
ci(convergence): fix gemma4 gate tolerance by @athitten :: PR: #3696
ci: merge L2 matrix entries so the e2e stage fits max-parallel: 10 by @akoumpa :: PR: #3695
fix(fsdp): preserve internal module output dtype by @HuiyingLi :: PR: #3657
feat(qwen3-8-flash-next): add Engram training by @HuiyingLi :: PR: #3690
fix(minimax): repair HF parity reference and gate at the measured envelope by @yuhezhang-ai :: PR: #3674
test(pp): fail parity tests when neither run trained by @athitten :: PR: #3701
ci: AUT-2165 trigger approval queue when CICD completes by @svcnemo-autobot :: PR: #3710
fix(ci): AUT-2165 serialize approval queue runs by @svcnemo-autobot :: PR: #3712
ci: AUT-2161 route external jobs through ephemeral v2 by @svcnemo-autobot :: PR: #3697
fix: catch FileExistsError when checkpointing to distributed fs by @akx :: PR: #3686
feat(glm5-next): add GLM-5.3-Flash training support by @HuiyingLi :: PR: #3699
fix(mtp): enable activation checkpointing for repeated dense MTP blocks by @pzelasko :: PR: #3660
test(pp): add qwen3_5_moe PP and EP parity tests [3/4] by @athitten :: PR: #3649
fix(glm): align and diagnose cross-framework router parity by @yuhezhang-ai :: PR: #3635
perf(qwen3.5): reuse packed GDN metadata across layers by @yuhezhang-ai :: PR: #3705
docs(models): expand GLM-5.3-Flash coverage by @HuiyingLi :: PR: #3744
fix(moe): Kimi-K3 correctness fixes (routing, NCCL warmup, PP, per-GPU tps) by @yisongbetter :: PR: #3731
feat(glm): add GLM-5.3 cuDNN training support by @HuiyingLi :: PR: #3747
docs(glm): add GLM-5.3 training coverage by @HuiyingLi :: PR: #3752
ci(build): bump FA4, quack and ffpa-attn by @akoumpa :: PR: #3733
perf(magi): avoid TE padding for magi by @HuiyingLi :: PR: #3757
feat: support generic freeze-config selectors by @pstjohn :: PR: #3681
feat(registry): add public architecture registration and entry-point discovery by @pstjohn :: PR: #3645
fix(retrieval): support Sentence Transformers v6 metadata by @oliverholworthy :: PR: #3600
fix(release): set version to 0.7.0 by @thomasdhc :: PR: #3770
docs: repair links left dangling by the Fern docs migration by @MohammedAlkindi :: PR: #3769
perf(moe): Kimi-K3 memory and throughput optimizations by @yisongbetter :: PR: #3732
fix(retrieval): do not lose metric records on preemption or error by @sahel-sh :: PR: #3583
fix(config): read YAML config files as UTF-8 by @MohammedAlkindi :: PR: #3782
build: bump ffpa-attn to 0.2.4 by @akoumpa :: PR: #3788
fix(checkpoint): support 64 GB RTX Spark training by @akoumpa :: PR: #3766
docs: correct stale CLI signature and PR title rules in agent docs by @kabirvashisht4-glitch :: PR: #3762
fix(kimi_k25): honour the quantization flag by @akx :: PR: #3633
feat(diffusion): periodic validation loss during training by @beccohov :: PR: #3579
fix(kimi-k3): correct MFU accounting (FLOPs formula, config defaults) by @yisongbetter :: PR: #3792
fix(loss): prevent TP KD gradient over-reduction by @mikemikimike :: PR: #3777
fix: preserve selected routing weights in FP32 in deepseek by @edjson :: PR: #3746
perf(kimi-k3): RMSNorm compile island and benchmark static-routing sync pins by @yisongbetter :: PR: #3779
build(container): bump PyTorch base to 26.08 by @thomasdhc :: PR: #3771
fix(qwen3_8_flash_next): keep flex QSA membership lookup in int64 by @HuayiJin :: PR: #3802
feat: support DeepSeek V4 Flash Vision by @HuiyingLi :: PR: #3772
fix(fsdp): restore full-layer activation checkpointing for KV-shared models by @athitten :: PR: #3677
feat(gemma4): support D512 FFPA attn on SM100 by @Butterfingrz :: PR: #3742
build(deps): Transformers upgrade to 5.15.1 by @athitten :: PR: #3781
test(deepseek): update router parity for transformers 5.15 by @akoumpa :: PR: #3811
feat(moe): add MoK MXFP8 expert support by @HuiyingLi :: PR: #3606
fix(deepseek-v4): advance PP media cursor for text-only microbatches by @ZenAlexa :: PR: #3821
perf(distributed): ring-pooled PP recv buffers and Kimi-K3 2k benchmark shape by @yisongbetter :: PR: #3780
feat(dataset): add mask_generation_prompt to keep prompt tokens out of loss by @khazic :: PR: #3739
feat(recipe): use HybridEP for Qwen3.5 35B by @HuiyingLi :: PR: #3827
fix(checkpoint): synchronize async daemon initialization by @HuiyingLi :: PR: #3824
fix(moe): export fused expert lora in the corrected peft >= 0.19.1 layout by @stanley1208 :: PR: #3626
fix(ci): calibrate checkpoint robustness recipes by @yuhezhang-ai :: PR: #3832
fix(ci): honor checkpoint tokenizer in vLLM deploy by @yuhezhang-ai :: PR: #3805
fix(fsdp): shard untied embeddings as separate units by @yuhezhang-ai :: PR: #3708
fix(checkpoint): skip size estimation on non-rank-0 processes by @kabirvashisht4-glitch :: PR: #3826
fix: AUT-2228 route partial reruns to the current attempt by @svcnemo-autobot :: PR: #3833
fix(ci): AUT-2247 use NCCL socket transport on GCP runners by @svcnemo-autobot :: PR: #3843
feat(vlm): add Nemotron 3.5 Super VL by @HuiyingLi :: PR: #3801
perf(checkpoint): expand low-memory DCP support across MoE adapters by @yuhezhang-ai :: PR: #3610
build(deps): make mlflow and wandb optional extras by @piyushumate :: PR: #3785
docs(vlm): add Nemotron 3.5 Super VL CORD-v2 fine-tuning and evaluation guide by @HuiyingLi :: PR: #3806
fix(vlm): drop removed freeze_embeddings from Nemotron 3.5 Super VL recipes by @HuiyingLi :: PR: #3851
fix(checkpoint): shard extracted exports by size by @oliverholworthy :: PR: #3423
feat(tokenizer): add explicit backend selection by @oliverholworthy :: PR: #3102
ci: drop the redundant -rA PASSES report from test runs by @akoumpa :: PR: #3858
perf(checkpoint): load standard HF safetensors with DCP by @yuhezhang-ai :: PR: #3616
fix(qwen3_omni_moe): put the thinker prefix inside the peft prefix on lora saves by @stanley1208 :: PR: #3630
fix(peft): export Nemotron adapters in the Transformers v5 namespace by @yuhezhang-ai :: PR: #3866
feat(vlm): Nemotron 3.5 Super VL text-to-SQL LoRA recipe and guide by @HuiyingLi :: PR: #3852
feat: add DeepSeek-V4.1-Flash model training support by @khazic :: PR: #3855
fix(ci): stabilize FlexAttention checkpoint parity by @yuhezhang-ai :: PR: #3868
test: enforce a 5s per-test timeout by @akoumpa :: PR: #3859
fix(transformers): honor dtype and retain torch_dtype compatibility by @yuhezhang-ai :: PR: #2379
fix(datasets): log the VLM pre-filter threshold that is actually applied by @kabirvashisht4-glitch :: PR: #3799
docs: add DeepSeek-V4.1-Flash to README news by @HuiyingLi :: PR: #3876
feat(deepseek-v41): add attention and expert LoRA recipe by @HuiyingLi :: PR: #3872
feat(minimax-m3): add SM100 MSA sparse attention with custom backward by @Butterfingrz :: PR: #3809
test(pp): add mistral3p5 PP and TP parity tests [4/4] by @athitten :: PR: #3709
docs(examples): fix gemma4 SWE-bench tool-call wire format by @athitten :: PR: #3885
chore: stop eagerly importing the transformers model zoo by @akoumpa :: PR: #3880
fix(mistral4): align scaling, routing, and reference precision by @yuhezhang-ai :: PR: #3865
fix(ling): fp32 default router projection and selected weights by @edjson :: PR: #3861
fix(tests): drop the shadowed TestKimiK25VLForConditionalGenerationForward by @Anai-Guo :: PR: #3873
fix(ci): restore Mistral3 routing and bound import probe by @akoumpa :: PR: #3888
fix(distributed): synchronize tensor-parallel replicas by @yuhezhang-ai :: PR: #3822
perf(kimi-k3): manual GC every 10 steps in the GB200 2k benchmark yaml by @yisongbetter :: PR: #3896
perf(checkpoint): bound quantized DCP load memory by @yuhezhang-ai :: PR: #3619
fix(optim): apply checkpoint weight decay and step count on scheduler resume by @kabirvashisht4-glitch :: PR: #3883
perf(moe): static-routing pad-size pin and Kimi-K3 shared-expert overlap by @yisongbetter :: PR: #3895
fix(qwen3_5): keep vision tower rotary inv_freq in fp32 across the dt… by @haowen-han :: PR: #3856
fix(loss): reject InfoNCE directions without candidates to score by @kabirvashisht4-glitch :: PR: #3870
fix(benchmark): avoid full GPT-OSS logits allocation by @janbernloehr :: PR: #3909
ci: separate unit-test budgets from hard timeouts by @akoumpa :: PR: #3900
feat(training): add weighted domain mixtures by @akoumpa :: PR: #3303
fix: deterministic implementation of _float32_rms_norm_fwd by @ashors1 :: PR: #2971
perf(training): fused multi-tensor gradient-norm reduction by @akoumpa :: PR: #3685
fix(recipes): normalize seq-cls loss and gradients by a global label count by @kabirvashisht4-glitch :: PR: #3830
fix(recipe): update Step-3.5-Flash LoRA topology by @janbernloehr :: PR: #3910
feat(recipe): add Super 3.5 VL continued pretraining on FineWeb by @HuiyingLi :: PR: #3889
fix(checkpoint): shard CP-only models before loading weights by @HuiyingLi :: PR: #3908
fix(init): initialize DTensor-sharded models whose embedding has a padding_idx by @yisongbetter :: PR: #3808
fix(datasets): drop NeMo-only branch that crashes multi-source blend_per_split by @ralovets :: PR: #3919
perf: add Super 3.5 VL GB200 benchmark config by @HuiyingLi :: PR: #3905
fix(loss): align MTP ignore index with configured loss by @akoumpa :: PR: #3897
fix(nemotron-omni): keep native RADIO names in convert_single_tensor_to_hf by @yuekaizhang :: PR: #3924
feat(deepseek-v41): add context parallelism and packed SFT by @HuiyingLi :: PR: #3879
fix(nemotron_v3): keep the generation config on consolidated exports by @stanley1208 :: PR: #3891
perf(checkpoint): bound GPT-OSS MXFP4 loading by @yuhezhang-ai :: PR: #3623
feat(dspark): add DeepSeek V4.1 training support by @kashif :: PR: #3875
fix(models): initialize attention projection biases by @janbernloehr :: PR: #3926
perf(moe): reuse the DeepEP dispatch layout on activation-checkpoint recompute by @akoumpa :: PR: #3684
fix(peft): export merged grouped-expert adapters by @pstjohn :: PR: #3834
feat(deepseek-v41): enable MedPix vision SFT by @HuiyingLi :: PR: #3916
fix(qwen2_5_omni): put the thinker prefix inside the peft prefix on lora saves by @stanley1208 :: PR: #3672
perf(qwen3.5): coalesce packed metadata transfers by @akoumpa :: PR: #3750
fix(ci): allow measured GPT-OSS PEFT HF numerical drift by @yuhezhang-ai :: PR: #3937
refactor(distributed): replace globals() monkey-patch, add no-globals lint gate by @akoumpa :: PR: #3938
perf(training,moe): pipeline-stage compile prewarm, HybridEP shared JIT cache by @yisongbetter :: PR: #3893
perf(kimi_k3): opt-in KDA kernel knobs (conv backend, recompute, state layout) by @yisongbetter :: PR: #3932
fix(deepseek-v41): register a FLOPs formula so BenchmarkingRecipe can run by @yisongbetter :: PR: #3955
fix(ci): capture GLM routing with the current HF gate API by @yuhezhang-ai :: PR: #3951
fix(ci): stabilize GPT-OSS and Mamba checkpoint parity by @yuhezhang-ai :: PR: #3953
fix(ci): serialize convergence model config preparation by @athitten :: PR: #3941
fix(mistral): freeze text-only visual branches and enable resume by @yuhezhang-ai :: PR: #3964
test(peft): add a shared HF PEFT reload suite by @stanley1208 :: PR: #3945
fix(fsdp): preserve already-sharded embedding units by @yuhezhang-ai :: PR: #3936
fix(examples): rename README_kimi_k3.md and repoint stale links by @MohammedAlkindi :: PR: #3972
feat(flops): register Qwen3.8-Flash-Next formula by @Agoniii :: PR: #3963
fix(data): support Mistral tokenizers in the distillation collator by @MrCapricornLiu :: PR: #3849
fix(datasets): stop default_collater reshaping the caller's example tensors by @kabirvashisht4-glitch :: PR: #3934
fix(deps): update DeepEP for Rubin C++20 builds by @akoumpa :: PR: #3973
fix(inkling): preserve raw checkpoint expert widths by @yuhezhang-ai :: PR: #3977
fix(ci): match Nemotron remote reloads with legacy PEFT exports by @yuhezhang-ai :: PR: #3952
fix(metrics): require explicit text FLOPs accounting by @HuiyingLi :: PR: #3901
fix(ci): align Kimi GB200 expert groups with NVLink blocks by @yuhezhang-ai :: PR: #3914
docs(model-coverage): fix GLM-5.3-Flash install command and transformers pin by @waynel96 :: PR: #3939
perf(kimi-k3): Triton SiTU + attn-residual kernels, fp32 glue compile islands by @yisongbetter :: PR: #3894
docs(fern): use gray code backgrounds by @akoumpa :: PR: #3981
fix(recipes): use HybridEP for Qwen3 MoE benchmark [AMINT-370] by @HuiyingLi :: PR: #3988
fix(recipes): use HybridEP for DeepSeek V3.2 LoRA [AMINT-369] by @HuiyingLi :: PR: #3987
fix(dflash): apply activation checkpointing to the draft by @kashif :: PR: #3946
perf(kimi_k3): fast-math Triton SiTU kernels (situ_backend: triton_fast_math) by @yisongbetter :: PR: #3985
fix(launcher): reject nemo_run overrides the executor has no attribute for by @kabirvashisht4-glitch :: PR: #3949
ci: bump preflight workflows to v1.11.0 by @ko3n1g :: PR: #3997
fix(ci): use compatible Transformers for Gemma 4 eval by @athitten :: PR: #3969
refactor(moe): use native grouped mm experts by @akoumpa :: PR: #3991
build(container): include FlashMLA for GLM-5.3 CI [AMINT-347] by @akoumpa :: PR: #4000
fix: update Nemotron 3.5 VL by @HuiyingLi :: PR: #4001
fix(deepseek-v41): honor BackendConfig.fake_balanced_gate by @yisongbetter :: PR: #3996
ci: align CUDA base image with 26.08 by @akoumpa :: PR: #4004
fix(ci): normalize inactive top-p before saving merged adapters by @yuhezhang-ai :: PR: #3998
perf(kimi_k3): opt-in fused KDA kernels (kda_chunk_impl=fused) by @yisongbetter :: PR: #3954
feat: add MiMo V2.6 support by @HuiyingLi :: PR: #3992
fix(distributed): ignore non-zero-dim FSDP copy-out ops in selective AC replay by @linmuchuiyang :: PR: #3961
fix(ci): restore MoE benchmark activation checkpointing [AMINT-372] by @akoumpa :: PR: #4011
feat(deepseek-v41): add H100-80GB recipes, Tulu3 data prep, and coverage notes by @linmuchuiyang :: PR: #3962
fix(ci): raise Qwen3 MoE and Step 3.5 Flash benchmark timeouts by @janbernloehr :: PR: #3966
feat(dflash): support top-p and top-k in speculative decoding by @kashif :: PR: #3627
fix(loss): apply per-token gradients in parallel cross entropy by @MrCapricornLiu :: PR: #3853
perf(qwen3-8-flash-next): vectorize QSA block selection, size chunks by memory by @Agoniii :: PR: #4006
docs: update Super 3.5 VL checkpoint names for GA by @HuiyingLi :: PR: #4013
perf(qwen3.8): add FA4 QSA backend for H100 by @HuiyingLi :: PR: #3925
perf(docker): add build parallelism opt-in build args by @ko3n1g :: PR: #4028
fix(docs): use incremental Fern publishing by @akoumpa :: PR: #4024
feat(mimo_v25): support MiMo-V2.5-Pro by @Simar-malhotra09 :: PR: #2514
fix(transformers): pin config and weights to the resolved Hub snapshot by @akoumpa :: PR: #4019
fix(deps): make MoE source builds work with CUDA 13 by @akoumpa :: PR: #4029
fix(nemotron_v3): accept singular embedding HF key alias by @dundysm :: PR: #3999
fix(distributed): retain TE context-parallel token indices [AMINT-353] by @HuiyingLi :: PR: #3984
docs(fern): organize model coverage by provider and model ID by @akoumpa :: PR: #3974
fix(nemotron-flash): support Transformers 5.15 layer types [AMINT-360] by @akoumpa :: PR: #4002
fix(distributed): prepare runtimes before pipeline traffic [AMINT-362] by @akoumpa :: PR: #4007
fix(ci): normalize recipe paths and git decoding in coverage table sync by @MohammedAlkindi :: PR: #3993
fix(benchmark): repair nightly benchmark configs by @akoumpa :: PR: #4033
fix(test): restore Step-3.7 HF PEFT checkpoint reload by @yuhezhang-ai :: PR: #4030
fix(checkpoint): preserve nested HF key renames by @yuhezhang-ai :: PR: #3982
fix(training): reduce grad-clip norms across the expert-parallel axis by @stanley1208 :: PR: #3817
perf(benchmark): disable MTP for Qwen3.5 MoE LoRA by @akoumpa :: PR: #4034
fix(ci): accept identical short greedy outputs [AMINT-15] by @akoumpa :: PR: #4003
fix(benchmark): use HybridEP for Kimi K2 [AMINT-385] by @akoumpa :: PR: #4040
fix(optim): exclude empty local shards from TE FusedAdam updates by @pzelasko :: PR: #4035
fix(init): support HF initialization of DTensor-sharded models by @yuhezhang-ai :: PR: #3983
test(peft): close the remaining reload-suite coverage gaps by @stanley1208 :: PR: #4026
ci(review): AUT-2439 migrate legacy Claude commands to /review by @svcnemo-autobot :: PR: #4044
fix(nemotron_omni): handle PEFT key conversion for adapter save and reload by @taran-dev4u :: PR: #3986
fix(dataset): reject pickle in Megatron index caches by @chtruong814 :: PR: #4047
fix(datasets): handle unanswerable SQuAD rows in the chat-template path by @MohammedAlkindi :: PR: #4038
fix: support native RADIO video projection in Nemotron Omni by @HuiyingLi :: PR: #4014
fix(launcher): make SkyPilot multi-node torchrun rendezvous at the head by @MohammedAlkindi :: PR: #4037
fix: skip None modules during parameter freezing by @athitten :: PR: #3970
fix(qwen3.5): avoid MedPix allocator fragmentation [AMINT-377] by @akoumpa :: PR: #4057
fix(loss): scale InfoNCE logits before masking padded negatives by @MrCapricornLiu :: PR: #3850
feat(diffusion): add Qwen-Image-2.1 text-to-image finetuning by @HuiyingLi :: PR: #4027
fix(ci): route GB200 DeepSeek-V4.1 recipes to GB200 runners [AMINT-358] by @HuiyingLi :: PR: #4066
fix(recipes): load DeepSeek-V4.1 Tulu3 inputs without env vars [AMINT-365] by @HuiyingLi :: PR: #4067
fix(data): load C++ helpers through PyTorch JIT by @akoumpa :: PR: #4062
fix(ci): use NCCL internal IB path for Qwen3.8 Engram recipes [AMINT-344] by @HuiyingLi :: PR: #4069
fix(moe): reduce peak expert MLP memory [AMINT-351] by @akoumpa :: PR: #4010
refactor(retrieval): configure text attention mode by @oliverholworthy :: PR: #1837
refactor(distributed): move parallelization into model sidecars by @akoumpa :: PR: #4032
fix(loss): chunk output projection in cross entropy by @akoumpa :: PR: #4046
test(minimax_m3_vl): raise lazy-import subprocess budget to 30s by @HuiyingLi :: PR: #4074
docs(glm-5.3-flash): state the Transformers reference and proxy for logit parity by @waynel96 :: PR: #4077
fix(docs): AUT-2456 preview approved PR mirror branches by @svcnemo-autobot :: PR: #4063
fix(optim): keep per-group lr and weight decay under the LR scheduler by @MohammedAlkindi :: PR: #4050
fix(qwen3.5): support MTP with VLM context parallelism [AMINT-352] by @HuiyingLi :: PR: #3990
fix(loss): fp32 hidden-grad accumulation and torch_compile impl for fused CE by @HuiyingLi :: PR: #4048
perf(qwen3-8-flash-next): replay QSA routes on activation-checkpoint recompute by @Agoniii :: PR: #4005
feat(moe): mxfp4-resident MoE experts for DeepSeek-V4-Flash LoRA by @excepshenal :: PR: #2548
fix(qwen2_5_omni): accept v4_compatible in map_peft_target_module_to_hf by @MohammedAlkindi :: PR: #4042
fix(peft): stop layer wildcards like .layers.1. matching layers 10-19 by @MohammedAlkindi :: PR: #4045
refactor(packing): replace global HF monkeypatch with typed metadata by @beccohov :: PR: #3577
fix(ci): relax GPT-OSS PEFT checkpoint resume tolerance by @yuhezhang-ai :: PR: #4071
fix(deps): AUT-2465 raise security floors and pin patched Nsight Systems by @svcnemo-autobot :: PR: #4084
fix(docker): backport DeepEP NVSHMEM queue-pair compatibility by @janbernloehr :: PR: #3968
ci: install VLM media dependencies during the image build by @akoumpa :: PR: #4119
fix(recipes): reduce seq-cls validation loss over all samples and DP ranks by @MohammedAlkindi :: PR: #4043
fix(deps): install MoK through pyproject and uv sync by @akoumpa :: PR: #4121
feat(mimo): add MiMo-V2.6-Pro support by @HuiyingLi :: PR: #4054
ci: raise test timeouts from 60 to 70 seconds by @akoumpa :: PR: #4125
fix(glm5_next, kimi_linear): always use chunk_kda by @waynel96 :: PR: #4110
fix(moe): avoid retaining routed inputs in chunked checkpoints by @yh0903 :: PR: #4103
fix(minimax-m3): plan MSA block selection once, outside checkpointed blocks by @akoumpa :: PR: #4123
feat(diffusion): add custom MoE transformers with expert parallelism by @HuiyingLi :: PR: #4113
perf(moe): add HybridEP compact routing by @akoumpa :: PR: #3404
feat(attention): add native FA4 and model-aware packed batches by @akoumpa :: PR: #3734
feat(retrieval): add Mistral3 VL embedding and reranking by @oliverholworthy :: PR: #3915
build: require torchao>=0.16.0 so peft can inject LoRA adapters by @kabirvashisht4-glitch :: PR: #4131
feat(diffusion): HunyuanImage-3.0 text-to-image fine-tuning with EP by @khazic :: PR: #4129
fix(qwen3-8-flash-next): keep PLE hash and conv inside packed documents by @rouchenzi :: PR: #4132
docs: add HunyuanImage-3.0 to README news by @HuiyingLi :: PR: #4139
fix(hunyuan-image3): import prompt batch keys from the model package by @HuiyingLi :: PR: #4144
fix(glm): honor checkpoint indexer rotary layout by @yuvalnissan :: PR: #4142
fix(gemma4): align unpacked vision masks with Transformers by @akoumpa :: PR: #4124
feat(retrieval): context aware reranker training by @sahel-sh :: PR: #3786
fix(ci): use accessible Super benchmark config by @athitten :: PR: #3980
docs: link Nemotron end-to-end embedding recipe by @akoumpa :: PR: #4159
fix(retrieval): align Mistral prompts and vLLM template detection by @oliverholworthy :: PR: #4153
fix(deps): include SQLite support in the MLflow extra by @akoumpa :: PR: #4161
feat(glm5_next): training chat template that supervises the end-of-turn token by @waynel96 :: PR: #4165
fix(peft): groundwork for boundary-owned PEFT prefix handling by @stanley1208 :: PR: #4115
perf(moe): fuse bounded weighted expert bias addition by @akoumpa :: PR: #4166
fix(deepseek-v32): align indexer RoPE layout and precision by @HuiyingLi :: PR: #4146
fix(diffusion): restore inference device placement [AMINT-388] by @HuiyingLi :: PR: #4136
fix(loss): inspect DDP model metadata for chunked cross entropy by @akoumpa :: PR: #4167
fix(deps): AUT-2490 enforce CVE floors and remove EFA plugins by @svcnemo-autobot :: PR: #4172
fix(fsdp): preserve model input and output dtypes by @HuiyingLi :: PR: #4150
fix(checkpoint): preserve HF root renames through FSDP wrapping by @yuhezhang-ai :: PR: #4163
fix(peft): freeze router biases in LoRA recipes by @yuhezhang-ai :: PR: #4076
fix(deepseek-v4): make LoRA checkpoints reload and resume by @yuhezhang-ai :: PR: #4072
fix(moe): initialize MOK expert weights on FSDP shards by @yuhezhang-ai :: PR: #4174
docs: refresh release log and prepare 26.10 container references (4162) by @svcnvidia-nemo-ci :: PR: #4184
test: fix reranker persistent worker timeout and cleanup (4183) by @svcnvidia-nemo-ci :: PR: #4193
fix(distributed): use CP-local pipeline token capacity [AMINT-399] (4137) by @svcnvidia-nemo-ci :: PR: #4192
fix(distributed): restore specialized Hugging Face parallelizers (4180) by @svcnvidia-nemo-ci :: PR: #4200
fix(retrieval): export portable Mistral3 reranker checkpoints (4188) by @svcnvidia-nemo-ci :: PR: #4197
fix(moe): restore bias-free throughput and GPT-OSS weight loading (4155) by @svcnvidia-nemo-ci :: PR: #4199
fix(ci): relax intermittent checkpoint resume loss tolerances (4212) by @svcnvidia-nemo-ci :: PR: #4222
cp: backport DeepSeek-V4.1 CP8 gradient sync fix to r0.7.0 (#4203) by @akoumpa :: PR: #4221
fix(llm): release unused model outputs before backward (4201) by @svcnvidia-nemo-ci :: PR: #4220
ci: Update transformers to latest version 5.18.0 (4116) by @svcnvidia-nemo-ci :: PR: #4225
fix(training): validate domain mixture corpus bindings (4195) by @svcnvidia-nemo-ci :: PR: #4226
perf(moe): raise biased expert MLP chunk threshold to 512 MiB (4234) by @svcnvidia-nemo-ci :: PR: #4235
fix(container): AUT-2510 use patched aiohttp in Ray runtime-env agent (4223) by @svcnvidia-nemo-ci :: PR: #4237
fix(ci): keep pyarrow compatible with CUDA build numpy (4244) by @svcnvidia-nemo-ci :: PR: #4247
fix(ci): include recipes in import contract checks (4224) by @svcnvidia-nemo-ci :: PR: #4248
docs(fern): unpublish Nemotron 3.5 Super VL guides (4253) by @svcnvidia-nemo-ci :: PR: #4254
fix(ci): relax DeepSeek V4 LoRA resume loss tolerance (4240) by @svcnvidia-nemo-ci :: PR: #4251
perf(minimax): optimize M3 attention and checkpoint recomputation (4176) by @svcnvidia-nemo-ci :: PR: #4249
fix(retrieval): backport Pixtral FlashAttention image boundaries (4204) by @svcnvidia-nemo-ci :: PR: #4267
fix(deps): raise security floors for audited Python packages (4246) by @svcnvidia-nemo-ci :: PR: #4272
test: keep L0 Hub access opt-in and fix HellaSwag mocks (4268) by @svcnvidia-nemo-ci :: PR: #4273
build(docker): AUT-2515 skip redundant rdma-core installation (4255) by @svcnvidia-nemo-ci :: PR: #4274
fix(ci): reuse verified wheelhouses and container dependencies (4269) by @svcnvidia-nemo-ci :: PR: #4280
fix(ci): relax customizer GPT-OSS resume loss tolerance (4278) by @svcnvidia-nemo-ci :: PR: #4287
fix(qwen3.8): materialize hybrid layer types before model construction (4206) by @svcnvidia-nemo-ci :: PR: #4239
fix(packing): validate TE masks and fix Qwen3-VL packed inputs (4241) by @svcnvidia-nemo-ci :: PR: #4259
fix(diffusion): retain validation tails and weight loss by samples (4227) by @svcnvidia-nemo-ci :: PR: #4291
ci: restore wheelhouse caches directly in pip and UV jobs (4289) by @svcnvidia-nemo-ci :: PR: #4301
fix(ci): stabilize DeepSeek-V4 MXFP4 validation (4245) by @svcnvidia-nemo-ci :: PR: #4262
fix(distributed): reject unsupported MTP CP PP from mesh topology (4252) by @svcnvidia-nemo-ci :: PR: #4296
fix(ci): restore device hooks after HF adapter reload (4277) by @svcnvidia-nemo-ci :: PR: #4297
fix(qwen3.5): support packed vision training with TE attention (4242) by @svcnvidia-nemo-ci :: PR: #4303
fix(vlm): pad variable-resolution images in Ministral recipes (4304) by @svcnvidia-nemo-ci :: PR: #4309
fix(qwen3.5-moe): isolate packed text metadata from vision (4306) by @svcnvidia-nemo-ci :: PR: #4316
fix(vlm): fit Muse Glimmer FP32 optimizer state on two nodes (4307) by @svcnvidia-nemo-ci :: PR: #4321
fix(checkpoint): apply Qwen MTP FP8 scales (AM-864) (4250) by @svcnvidia-nemo-ci :: PR: #4318
fix(deps): AUT-2528 raise Apache Thrift security floor to 0.25.0 (4315) by @svcnvidia-nemo-ci :: PR: #4319
fix(mimo): include the query token in sliding attention windows (4343) by @svcnvidia-nemo-ci :: PR: #4344
fix(benchmark): correct MoK CI topology and DeepSeek V4 FLOPs (4302) by @svcnvidia-nemo-ci :: PR: #4345
fix(ci): use compatible Nemotron Nano 9B deployment settings (4339) by @svcnvidia-nemo-ci :: PR: #4350
docs(fern): add recipe-linked model performance tables (4256) by @svcnvidia-nemo-ci :: PR: #4361
cp: backport Triton RoPE #4214 and rotary fusion #4346 to r0.7.0 by @akoumpa :: PR: #4360
docs: apply jgerh copyedits from PR 4159 (4191) by @svcnvidia-nemo-ci :: PR: #4383
cp: reduce routing and GLM overhead for r0.7.0 by @akoumpa :: PR: #4400
fix(peft): eliminate FP32 storage for frozen expert weights (4388) by @svcnvidia-nemo-ci :: PR: #4392
fix(qwen): use FA4 to avoid cuDNN SDPA backward NaNs (4423) by @svcnvidia-nemo-ci :: PR: #4426
docs: point release documentation links to r0.7.0 by @akoumpa :: PR: #4431
cp: backport homepage and training guides (#4425) to r0.7.0 by @akoumpa :: PR: #4432
cp: backport recipe deprecation metadata to r0.7.0 (#4436) by @akoumpa :: PR: #4440
fix(mimo): pin HybridEP domains for GB200 SFT recipes (4445) by @svcnvidia-nemo-ci :: PR: #4447
fix(deps): AUT-2606 backport Pillow 13 to r0.7.0 by @svcnemo-autobot :: PR: #4458
docs: add 0.6.0 and 0.7.0 release notes (4471) by @svcnvidia-nemo-ci :: PR: #4472
fix(packing): resolve packed mask layout from the built model (#3285) by @khazic
docs(speculative): document Gemma4, MSD, CP, and during-training tools (#3239) by @khazic
fix(docs): hide Kimi tokenizer regex from autodoc (#3288) by @akoumpa
fix(pp): use static metadata with PyTorch 2.13 (#3290) by @yuhezhang-ai
docs(distributed): review frozen multimodal FSDP guidance (#3272) by @yuhezhang-ai
fix(vlm): fused linear CE in gemma4 31B FFPA 8k recipe (AMINT-203) (#3273) by @athitten
feat: support AutoModelForSeq2SeqLM (encoder-decoder fine-tuning) (#2860) by @stanley1208
fix(distributed): preserve ERNIE router and dense Qwen3.5 SSM precision (#3255) by @yuhezhang-ai
feat(nemotron_v3): support dense Nemotron-H (Nano 4B) (#2670) by @stanley1208
ci: add Tulu-3 convergence + eval flow (#3171) by @athitten
feat(dllm): add I-DLM all-masked trainer (dllm.mode idlm) (#2437) by @kashif
perf(moe): add scoped partial CUDA graphs (#2917) by @akoumpa
fix(speculative): reject tensor and pipeline parallelism in DSpark (#3266) by @khazic
docs(models): add Inkling-Small recipe and coverage (#3315) by @HuiyingLi
fix(docs): avoid literal ampersand in Kimi autodoc (#3317) by @HuiyingLi
fix(config): repair GLM-5.2 LoRA recipe (#3313) by @yuhezhang-ai
fix(kd): preserve tensor-valued hidden states (#3324) by @akoumpa
fix(kd): use mesh-safe gradient clipping (#3302) by @akoumpa
fix(training): prewarm Mamba SSD autotune kernels (#3296) by @yuhezhang-ai
fix(ci): grant NVSkills status read permission (#3336) by @akoumpa
fix(nemotron-parse): sync RADIO preprocessing (#3331) by @akoumpa
refactor(docker): build torchao & FlashAttention as isolated wheel stages (#3329) by @thomasdhc
fix(speculative): correct DFlash anchor error and generated-span slice (#3268) by @khazic
fix(distributed): reuse default group for world-sized meshes (#3319) by @yuhezhang-ai
fix(distributed): restore Nemotron Flash TP2 training (#3345) by @HuiyingLi
refactor(dspark): pack the log window in a helper (#3049) by @kashif
chore(skills): refresh catalog signatures (#3333) by @akoumpa
fix(docs): make Fern autodoc metadata MDX-safe (#3355) by @akoumpa
fix(pp): preserve VLM media cursor with static metadata (#3344) by @yuhezhang-ai
fix(kimi_k25_vl): don't int4 quantize LoRA adapter keys on save (#3295) by @stanley1208
fix(deps): resolve 26.08 rc2 container CVEs (#3346) by @thomasdhc
fix(pp): route Qwen3.5 MoE pre-embedded inputs (#3294) by @HuiyingLi
fix(diffusion): use spawn start method for GPU preprocessing pools (#3339) by @pthombre
chore(ci): AUT-1231 bump preflight workflow template (#3365) by @svcnemo-autobot
fix(tokenizer): make special token insertion opt-in (#3337) by @akoumpa
fix(distributed): stabilize SAC replay with TE and FSDP (#3330) by @akoumpa
fix(tools): save processor artifacts in LoRA merge for #3256 (#3270) by @coding-famer
fix(ci): use a single container cache donor (#3377) by @thomasdhc
feat(models): make Inkling implementation standalone (#3358) by @HuiyingLi
fix(fsdp): resolve fp32 master-weight compute dtype per parameter (#3328) by @NancyFyong
perf(checkpoint): reduce distributed save overhead (#3369) by @akoumpa
fix(ci): enable LTX-2.3 diffusion finetuning (#3372) by @akoumpa
fix(model): correct Mistral4 attention and distributed MoE routing (#3348) by @yuhezhang-ai
fix(checkpoint): survive an interrupted save instead of hanging all ranks (#3261) by @yuhezhang-ai
ci: tune Nemotron single-GPU model load threads (#3370) by @yuhezhang-ai
chore(ci): AUT-1298 bump community workflow to v1.8.7 (#3378) by @svcnemo-autobot
ci: add gradient-determinism rule to PR review guidelines (#3392) by @akoumpa
docs(training): mark Mamba prewarm sections for review (#3340) by @yuhezhang-ai
perf: remove Python overhead from model hot paths (#3374) by @akoumpa
fix(retrieval): guard optional W&B imports (#3380) by @yuhezhang-ai
fix(checkpoint): harden PEFT PP checkpointing and Step-3.7 coverage (#3316) by @yuhezhang-ai
docs(models): automate latest model coverage (#3351) by @akoumpa
fix(moe): support EP-free DTensor state-dict conversion (#3397) by @akoumpa
fix(gemma4): enable expandable_segments on the CP tulu3 recipes (#3408) by @athitten
fix(deps): resolve OSS CVE findings (#3398) by @thomasdhc
test(checkpoint): harden MoE reload parity checks (#3335) by @yuhezhang-ai
test(tools): add focused mock coverage for VLM processor serialization (#3379) by @taran-dev4u
feat: Add {% generation %} chat template for DiffusionGemma SFT/LoRA examples (#3353) by @Lintianqianjin
fix(docs): AUT-1324 qualify LTX model coverage slug (#3418) by @svcnemo-autobot
fix(ci): run Kimi K3 HellaSwag on GB200 (#3420) by @akoumpa
docs(fern): add observed 404 redirects (#3425) by @lbliii
fix(transformers): default missing THD capability (#3406) by @akoumpa
perf(moe): vectorize deterministic expert bias gradients (#3387) by @akoumpa
ci(minimax): extend M2.7 LoRA timeout (#3428) by @yuhezhang-ai
feat(model_init): add FP8 pre-flight check for the force_hf path (#2350) by @stanley1208
ci(convergence): re-baseline Qwen3-MoE IFEval to 0.5989 (#3416) by @athitten
fix(retrieval): use stock Ministral embedding backbone (#3103) by @oliverholworthy
fix(distributed): avoid duplicate FSDP2 prefetch all-gathers (#3411) by @yuhezhang-ai
fix(#1755): Fix PEFT adapter outputs for v5 support (#3284) by @Mr-Neutr0n
fix(kimi_k25_vl): keep PEFT expert LoRA keys under the base_model.model. prefix (#3431) by @stanley1208
feat(kimi_linear): context parallelism and packed-sequence support (#3243) by @khazic
fix: Gemma4-31B CoderForge CP8/64K data processing and recipe (#3446) by @athitten
ci(vlm): raise Slurm wall time for MiniMax-M3 and Gemma4 recipes (#3448) by @athitten
fix(retrieval): repair Ministral3 training recipe (#3441) by @yuhezhang-ai
test(checkpoint): redesign resume robustness around a shared trajectory (#3427) by @yuhezhang-ai
fix: normalize MoE auxiliary loss during gradient accumulation (#3359) by @HuiyingLi
test(checkpoint): stabilize MoE checkpoint parity gates (#3440) by @yuhezhang-ai
fix(retrieval): export Sentence Transformers metadata (#3021) by @oliverholworthy
feat: nemotron 3.5 cfg (#3442) by @adil-a
fix(gemma4): exact router scalar and fp32 reference routing for MoE parity (#3456) by @athitten
feat(streaming): add EAGLE-3 producer and loader over the local store (#3085) by @kashif
fix(peft): enable v5 expert adapters for Nemotron and MiniMax (#3439) by @yuhezhang-ai
fix(bagel): type SFT backend before AutoModel init (#3449) by @zyzhou5
fix(vlm): correct CP gradients and vision sharding (#3414) by @yuhezhang-ai
docs: improve accessibility for agents (#3424) by @akoumpa
fix(checkpoint): save ranked RNG state per global rank (#3437) by @yuhezhang-ai
fix(docs): generate model coverage tables at build time (#3472) by @akoumpa
docs(models): add MuseGlimmer recipes and coverage (#3475) by @HuiyingLi
docs: announce MuseGlimmer support (#3478) by @akoumpa
fix(peft): align frozen LoRA tensors with compute dtype (#3470) by @akoumpa
fix(training): all-reduce grad-norm scalars on the mesh device (#3461) by @zyzhou5
fix(speculative): align the GLM-5.2 DSpark draft RoPE with its target (#3267) by @khazic
feat(speculative): add Kimi K3 DSpark training (#3263) by @khazic
feat(datasets): add Shopify product-catalogue VLM dataset (#3400) by @zeryx
feat(vlm): support packed THD vlm context parallelism (#3322) by @HuiyingLi
feat(models): add MuseGlimmer training support (#3476) by @HuiyingLi
fix(datasets): reopen IndexedDataset readers in spawned workers (#3382) by @HuiyingLi
feat(nemotron_v3): run MTP heads under eval for validation acceptance metrics (#2802) by @Slyne
feat(speculative): add Kimi K3 EAGLE-3 training (#3286) by @khazic
fix(speculative): record dataset identity in the EAGLE-3 precompute manifest (#3396) by @khazic
fix(ci): register flux2/wan2.2/qwen-image-edit diffusion recipes in CI (#3485) by @pthombre
chore: Bump gitpython to >= 3.1.59 (#3482) by @chtruong814
fix(recipe): shard GPT-OSS 120B across 64 experts (#3483) by @HuiyingLi
perf(benchmarks): recompute deterministic MoE routers under AC (#3474) by @akoumpa
ci: AUT-1408 replace cicd PAT with GitHub App tokens (#3463) by @svcnemo-autobot
perf(recipes): run GPT-OSS 120B with EP64 and no activation checkpointing (#3497) by @akoumpa
docs: add Nemotron 3.5 Lightning coverage (#3499) by @akoumpa
fix(models): preserve lm-head dtype boundaries (#3491) by @jQizhang
fix(checkpoint): validate PEFT adapter-only state (#3501) by @akoumpa
ci(convergence): fix gemma4 eval setup and re-baseline Qwen3-MoE (#3493) by @athitten
fix(deepseek_v4): avoid TileLang boolx8 backward codegen (#3467) by @HuiyingLi
fix(Qwen3.5): Support split FP8 expert checkpoints for Qwen3.5-MoE (#3495) by @HuiyingLi
fix(checkpoint): preserve FSDP2 mixed precision during recompute (#3513) by @jQizhang
docs(models): add Qwen3.8-Max coverage (#3517) by @HuiyingLi
feat(models): add cohere micro vision (#3518) by @zeryx
fix(test): use the recipe's attention backend for the HF reload parity check (#3500) by @athitten
feat(examples): add Qwen3-32B Tulu-3 finetune recipes (#3104) by @athitten
feat(moe): integrate Mixture-of-Kittens backend (#3422) by @HuiyingLi
fix(deps): resolve Starlette and GitPython CVEs (#3523) by @thomasdhc
docs(cohere): add absolute recipe links and remove dead blog reference (#3526) by @HuiyingLi
ci: AUT-1501 bump SSO preflight to v1.8.10 (#3537) by @svcnemo-autobot
feat(loss): return per-depth MTP losses (#3525) by @Slyne
docs(models): rename Qwen3.8-Max to Qwen3.8-2.4T-A95B (#3549) by @HuiyingLi
fix(retrieval): support canonical Sentence Transformers metadata (#3546) by @qiaochuz-nv
feat(gemma4): add E-series tensor parallelism (#3512) by @HuiyingLi
feat(datasets): Datum record and collate-time THD sequence packing (#3514) by @HuiyingLi
fix(minimax_m3_vl): declare and honour logits_to_keep so fused losses are kept (#3511) by @coderaBruce
ci: route DGX Spark recipes to GB10 (#3538) by @akoumpa
fix(docker): build bitsandbytes for SM121 (#3553) by @akoumpa
feat(qwen): add Qwen3.8-27B fine-tuning recipes (#3560) by @akoumpa
fix(fsdp): uniform reduce dtype and EP-local expert gradients (#3540) by @akoumpa
fix(nemotron_omni): support mixed-resolution image batches in tile path (#3559) by @HuiyingLi
fix(vlm): use FP32 master weights for Nemotron Omni (#3444) by @HuiyingLi
fix(checkpoint): restore Gemma4 Unified HF export keys (#3260) by @khazic
fix(metric_logger): expose the logger buffer size in the factory builder (#3541) by @sahel-sh
fix(training): correct dataloader annotation and modernize Optional syntax (#3534) by @Faisal-Alsrheed
fix(docker): harmonize non-root runtime contract (#3484) by @ko3n1g
fix(deps): resolve msgpack, wandb, and mistune container CVEs (#3563) by @thomasdhc
fix(mtp): support context-parallel caller inputs (#3543) by @Slyne
fix(moe): stabilize async DeepEP checkpointing (#3522) by @pzelasko
fix(recipes): support MTP with context parallelism (#3544) by @Slyne
test(checkpoint): improve resume correctness diagnostics (#3562) by @yuhezhang-ai
fix(model_init): stream Devstral FP8 checkpoints (#3413) by @yuhezhang-ai
fix(checkpoint): restore single-GPU custom-model DCP loading (#3533) by @HuiyingLi
feat(nemotron): allow skipping logits materialization (#3555) by @pzelasko
fix(kimi_linear): make long-context recipe CP8 (#3571) by @HuiyingLi
perf(glm): skip top-k pipeline carry without IndexShare (#3573) by @akoumpa
docs(dllm): align DiffusionGemma guides with the SFT recipe (#3550) by @kashif
fix(minimax): declare packed-sequence and CP attention ownership (#3547) by @athitten
test: add functional coverage for under-tested recipes (#3584) by @akoumpa
fix(perf): honor configured peak in MFU logging (#3569) by @yuhezhang-ai
feat(datasets): support SynthTraces agent SFT (#3371) by @khazic
fix(kimi_k3): apply the HF renames to expert LoRA keys on save (#3435) by @stanley1208
refactor(checkpoint): gate direct loads by adapter capability (#3574) by @yuhezhang-ai
fix(pp): keep accumulated grads across stage re-initialization (#3530) by @athitten
fix(vlm): truncate only the token axis in gemma4 collate (#3531) by @athitten
perf(Gemma4): reduce comms in sliding window, FA implementation, (#2984) by @nathan-az
feat(vlm): add CP vision shape floor (#3581) by @yuhezhang-ai
test(pp): add gemma4 parallelism parity tests [1/4] (#3529) by @athitten
fix(diffusion): apply compute_dtype via autocast when FSDP2 skips (#3589) by @zyzhou5
feat(examples): Qwen3-32B CoderForge long-context CP data + SFT recipe (#3594) by @athitten
feat(glm): add cuDNN DSA backend for GLM-5.2 (#3127) by @HuiyingLi
chore(lint): enforce PEP 604 optional annotations (#3535) by @akoumpa
fix(lint): update Qwen3 prefilter optional annotations (#3602) by @akoumpa
fix(deps): resolve 26.08 rc9 container CVEs (#3607) by @thomasdhc
fix(deps): align pyproject DeepEP pin with the container build (#3608) by @akoumpa
feat(dflash): add DFlash 2 draft model, trainer, and recipe (#3605) by @kashif
docs: announce DFlash 2 support in the README (#3624) by @khazic
fix(perf): recognize RTX PRO 6000 for MFU (#3598) by @ralovets
fix(ci): AUT-1630 route autobot through internal queue (#3629) by @svcnemo-autobot
fix: remove unused RUN_ID env var from CICD summary job (#3536) by @andrewwhitecdw
fix(checkpoint): load base weights under DDP and MegatronFSDP (#3597) by @ralovets
feat(speculative): add Kimi K3 DFlash draft training (#3287) by @khazic
feat(streaming): async producer + SharedDirFeatureStore (#3094) by @kashif
feat(speculative): add Ulysses context parallelism for the EAGLE-3 draft (#3356) by @khazic
fix: EP collective deadlock with variable-length token counts (LoRA flavor) (#3631) by @akx
feat(laguna): add Laguna XS 2.1 training recipe (#3639) by @akoumpa
fix(kimi_k25): map lm_head to model.language_model.lm_head (#3632) by @akx
refactor(checkpoint): StateDictAdapter method target-module rename (#3591) by @stanley1208
test(pp): add deepseek_v4 PP and EP parity tests [2/4] (#3613) by @athitten
feat(dllm): add SCDD training and sampling (#3354) by @kashif
fix(deps): resolve 26.08 rc10 container CVEs (#3647) by @thomasdhc
docs(examples): add Gemma4-31B SWE-bench eval steps + base-vs-SFT agentic result (#3585) by @athitten
perf(checkpoint): reduce allocating grouped MoE load overhead (#3580) by @yuhezhang-ai
test(checkpoint): expand parity metrics and phase coverage (#3567) by @yuhezhang-ai
perf(checkpoint): avoid full GC scans during MoE export (#3621) by @yuhezhang-ai
chore: bump lint tools + fix lint issues (#3634) by @akx
test(ci): report Jensen-Shannon checkpoint parity metrics (#3620) by @yuhezhang-ai
fix(test): unblock the Kimi-Linear vanilla-HF parity reference (#3659) by @yuhezhang-ai
docs(examples): add Qwen3-32B SWE-bench Verified eval (Phase 3) (#3671) by @athitten
fix(moe): equalize and align per-rank token counts for HybridEP dispatch (#3641) by @HuiyingLi
fix(peft): support mixed-dtype memory-efficient LoRA backward (#3675) by @akoumpa
docs(models): add Qwen3.8-Flash-Next coverage (#3691) by @HuiyingLi
fix(retrieval): register Llama classifier auto class (#2905) by @oliverholworthy
ci(convergence): fix gemma4 gate tolerance (#3696) by @athitten
ci: merge L2 matrix entries so the e2e stage fits max-parallel: 10 (#3695) by @akoumpa
fix(fsdp): preserve internal module output dtype (#3657) by @HuiyingLi
feat(qwen3-8-flash-next): add Engram training (#3690) by @HuiyingLi
fix(minimax): repair HF parity reference and gate at the measured envelope (#3674) by @yuhezhang-ai
test(pp): fail parity tests when neither run trained (#3701) by @athitten
ci: AUT-2165 trigger approval queue when CICD completes (#3710) by @svcnemo-autobot
fix(ci): AUT-2165 serialize approval queue runs (#3712) by @svcnemo-autobot
ci: AUT-2161 route external jobs through ephemeral v2 (#3697) by @svcnemo-autobot
fix: catch FileExistsError when checkpointing to distributed fs (#3686) by @akx
feat(glm5-next): add GLM-5.3-Flash training support (#3699) by @HuiyingLi
fix(mtp): enable activation checkpointing for repeated dense MTP blocks (#3660) by @pzelasko
test(pp): add qwen3_5_moe PP and EP parity tests [3/4] (#3649) by @athitten
fix(glm): align and diagnose cross-framework router parity (#3635) by @yuhezhang-ai
perf(qwen3.5): reuse packed GDN metadata across layers (#3705) by @yuhezhang-ai
docs(models): expand GLM-5.3-Flash coverage (#3744) by @HuiyingLi
fix(moe): Kimi-K3 correctness fixes (routing, NCCL warmup, PP, per-GPU tps) (#3731) by @yisongbetter
feat(glm): add GLM-5.3 cuDNN training support (#3747) by @HuiyingLi
docs(glm): add GLM-5.3 training coverage (#3752) by @HuiyingLi
ci(build): bump FA4, quack and ffpa-attn (#3733) by @akoumpa
perf(magi): avoid TE padding for magi (#3757) by @HuiyingLi
feat: support generic freeze-config selectors (#3681) by @pstjohn
feat(registry): add public architecture registration and entry-point discovery (#3645) by @pstjohn
fix(retrieval): support Sentence Transformers v6 metadata (#3600) by @oliverholworthy
fix(release): set version to 0.7.0 (#3770) by @thomasdhc
docs: repair links left dangling by the Fern docs migration (#3769) by @MohammedAlkindi
perf(moe): Kimi-K3 memory and throughput optimizations (#3732) by @yisongbetter
fix(retrieval): do not lose metric records on preemption or error (#3583) by @sahel-sh
fix(config): read YAML config files as UTF-8 (#3782) by @MohammedAlkindi
build: bump ffpa-attn to 0.2.4 (#3788) by @akoumpa
fix(checkpoint): support 64 GB RTX Spark training (#3766) by @akoumpa
docs: correct stale CLI signature and PR title rules in agent docs (#3762) by @kabirvashisht4-glitch
fix(kimi_k25): honour the quantization flag (#3633) by @akx
feat(diffusion): periodic validation loss during training (#3579) by @beccohov
fix(kimi-k3): correct MFU accounting (FLOPs formula, config defaults) (#3792) by @yisongbetter
fix(loss): prevent TP KD gradient over-reduction (#3777) by @mikemikimike
fix: preserve selected routing weights in FP32 in deepseek (#3746) by @edjson
perf(kimi-k3): RMSNorm compile island and benchmark static-routing sync pins (#3779) by @yisongbetter
build(container): bump PyTorch base to 26.08 (#3771) by @thomasdhc
fix(qwen3_8_flash_next): keep flex QSA membership lookup in int64 (#3802) by @HuayiJin
feat: support DeepSeek V4 Flash Vision (#3772) by @HuiyingLi
fix(fsdp): restore full-layer activation checkpointing for KV-shared models (#3677) by @athitten
feat(gemma4): support D512 FFPA attn on SM100 (#3742) by @Butterfingrz
build(deps): Transformers upgrade to 5.15.1 (#3781) by @athitten
test(deepseek): update router parity for transformers 5.15 (#3811) by @akoumpa
feat(moe): add MoK MXFP8 expert support (#3606) by @HuiyingLi
fix(deepseek-v4): advance PP media cursor for text-only microbatches (#3821) by @ZenAlexa
perf(distributed): ring-pooled PP recv buffers and Kimi-K3 2k benchmark shape (#3780) by @yisongbetter
feat(dataset): add mask_generation_prompt to keep prompt tokens out of loss (#3739) by @khazic
feat(recipe): use HybridEP for Qwen3.5 35B (#3827) by @HuiyingLi
fix(checkpoint): synchronize async daemon initialization (#3824) by @HuiyingLi
fix(moe): export fused expert lora in the corrected peft >= 0.19.1 layout (#3626) by @stanley1208
fix(ci): calibrate checkpoint robustness recipes (#3832) by @yuhezhang-ai
fix(ci): honor checkpoint tokenizer in vLLM deploy (#3805) by @yuhezhang-ai
fix(fsdp): shard untied embeddings as separate units (#3708) by @yuhezhang-ai
fix(checkpoint): skip size estimation on non-rank-0 processes (#3826) by @kabirvashisht4-glitch
fix: AUT-2228 route partial reruns to the current attempt (#3833) by @svcnemo-autobot
fix(ci): AUT-2247 use NCCL socket transport on GCP runners (#3843) by @svcnemo-autobot
feat(vlm): add Nemotron 3.5 Super VL (#3801) by @HuiyingLi
perf(checkpoint): expand low-memory DCP support across MoE adapters (#3610) by @yuhezhang-ai
build(deps): make mlflow and wandb optional extras (#3785) by @piyushumate
docs(vlm): add Nemotron 3.5 Super VL CORD-v2 fine-tuning and evaluation guide (#3806) by @HuiyingLi
fix(vlm): drop removed freeze_embeddings from Nemotron 3.5 Super VL recipes (#3851) by @HuiyingLi
fix(checkpoint): shard extracted exports by size (#3423) by @oliverholworthy
feat(tokenizer): add explicit backend selection (#3102) by @oliverholworthy
ci: drop the redundant -rA PASSES report from test runs (#3858) by @akoumpa
perf(checkpoint): load standard HF safetensors with DCP (#3616) by @yuhezhang-ai
fix(qwen3_omni_moe): put the thinker prefix inside the peft prefix on lora saves (#3630) by @stanley1208
fix(peft): export Nemotron adapters in the Transformers v5 namespace (#3866) by @yuhezhang-ai
feat(vlm): Nemotron 3.5 Super VL text-to-SQL LoRA recipe and guide (#3852) by @HuiyingLi
feat: add DeepSeek-V4.1-Flash model training support (#3855) by @khazic
fix(ci): stabilize FlexAttention checkpoint parity (#3868) by @yuhezhang-ai
test: enforce a 5s per-test timeout (#3859) by @akoumpa
fix(transformers): honor dtype and retain torch_dtype compatibility (#2379) by @yuhezhang-ai
fix(datasets): log the VLM pre-filter threshold that is actually applied (#3799) by @kabirvashisht4-glitch
docs: add DeepSeek-V4.1-Flash to README news (#3876) by @HuiyingLi
feat(deepseek-v41): add attention and expert LoRA recipe (#3872) by @HuiyingLi
feat(minimax-m3): add SM100 MSA sparse attention with custom backward (#3809) by @Butterfingrz
test(pp): add mistral3p5 PP and TP parity tests [4/4] (#3709) by @athitten
docs(examples): fix gemma4 SWE-bench tool-call wire format (#3885) by @athitten
chore: stop eagerly importing the transformers model zoo (#3880) by @akoumpa
fix(mistral4): align scaling, routing, and reference precision (#3865) by @yuhezhang-ai
fix(ling): fp32 default router projection and selected weights (#3861) by @edjson
fix(tests): drop the shadowed TestKimiK25VLForConditionalGenerationForward (#3873) by @Anai-Guo
fix(ci): restore Mistral3 routing and bound import probe (#3888) by @akoumpa
fix(distributed): synchronize tensor-parallel replicas (#3822) by @yuhezhang-ai
perf(kimi-k3): manual GC every 10 steps in the GB200 2k benchmark yaml (#3896) by @yisongbetter
perf(checkpoint): bound quantized DCP load memory (#3619) by @yuhezhang-ai
fix(optim): apply checkpoint weight decay and step count on scheduler resume (#3883) by @kabirvashisht4-glitch
perf(moe): static-routing pad-size pin and Kimi-K3 shared-expert overlap (#3895) by @yisongbetter
fix(qwen3_5): keep vision tower rotary inv_freq in fp32 across the dt… (#3856) by @haowen-han
fix(loss): reject InfoNCE directions without candidates to score (#3870) by @kabirvashisht4-glitch
fix(benchmark): avoid full GPT-OSS logits allocation (#3909) by @janbernloehr
ci: separate unit-test budgets from hard timeouts (#3900) by @akoumpa
feat(training): add weighted domain mixtures (#3303) by @akoumpa
fix: deterministic implementation of _float32_rms_norm_fwd (#2971) by @ashors1
perf(training): fused multi-tensor gradient-norm reduction (#3685) by @akoumpa
fix(recipes): normalize seq-cls loss and gradients by a global label count (#3830) by @kabirvashisht4-glitch
fix(recipe): update Step-3.5-Flash LoRA topology (#3910) by @janbernloehr
feat(recipe): add Super 3.5 VL continued pretraining on FineWeb (#3889) by @HuiyingLi
fix(checkpoint): shard CP-only models before loading weights (#3908) by @HuiyingLi
fix(init): initialize DTensor-sharded models whose embedding has a padding_idx (#3808) by @yisongbetter
fix(datasets): drop NeMo-only branch that crashes multi-source blend_per_split (#3919) by @ralovets
perf: add Super 3.5 VL GB200 benchmark config (#3905) by @HuiyingLi
fix(loss): align MTP ignore index with configured loss (#3897) by @akoumpa
fix(nemotron-omni): keep native RADIO names in convert_single_tensor_to_hf (#3924) by @yuekaizhang
feat(deepseek-v41): add context parallelism and packed SFT (#3879) by @HuiyingLi
fix(nemotron_v3): keep the generation config on consolidated exports (#3891) by @stanley1208
perf(checkpoint): bound GPT-OSS MXFP4 loading (#3623) by @yuhezhang-ai
feat(dspark): add DeepSeek V4.1 training support (#3875) by @kashif
fix(models): initialize attention projection biases (#3926) by @janbernloehr
perf(moe): reuse the DeepEP dispatch layout on activation-checkpoint recompute (#3684) by @akoumpa
fix(peft): export merged grouped-expert adapters (#3834) by @pstjohn
feat(deepseek-v41): enable MedPix vision SFT (#3916) by @HuiyingLi
fix(qwen2_5_omni): put the thinker prefix inside the peft prefix on lora saves (#3672) by @stanley1208
perf(qwen3.5): coalesce packed metadata transfers (#3750) by @akoumpa
fix(ci): allow measured GPT-OSS PEFT HF numerical drift (#3937) by @yuhezhang-ai
refactor(distributed): replace globals() monkey-patch, add no-globals lint gate (#3938) by @akoumpa
perf(training,moe): pipeline-stage compile prewarm, HybridEP shared JIT cache (#3893) by @yisongbetter
perf(kimi_k3): opt-in KDA kernel knobs (conv backend, recompute, state layout) (#3932) by @yisongbetter
fix(deepseek-v41): register a FLOPs formula so BenchmarkingRecipe can run (#3955) by @yisongbetter
fix(ci): capture GLM routing with the current HF gate API (#3951) by @yuhezhang-ai
fix(ci): stabilize GPT-OSS and Mamba checkpoint parity (#3953) by @yuhezhang-ai
fix(ci): serialize convergence model config preparation (#3941) by @athitten
fix(mistral): freeze text-only visual branches and enable resume (#3964) by @yuhezhang-ai
test(peft): add a shared HF PEFT reload suite (#3945) by @stanley1208
fix(fsdp): preserve already-sharded embedding units (#3936) by @yuhezhang-ai
fix(examples): rename README_kimi_k3.md and repoint stale links (#3972) by @MohammedAlkindi
feat(flops): register Qwen3.8-Flash-Next formula (#3963) by @Agoniii
fix(data): support Mistral tokenizers in the distillation collator (#3849) by @MrCapricornLiu
fix(datasets): stop default_collater reshaping the caller's example tensors (#3934) by @kabirvashisht4-glitch
fix(deps): update DeepEP for Rubin C++20 builds (#3973) by @akoumpa
fix(inkling): preserve raw checkpoint expert widths (#3977) by @yuhezhang-ai
fix(ci): match Nemotron remote reloads with legacy PEFT exports (#3952) by @yuhezhang-ai
fix(metrics): require explicit text FLOPs accounting (#3901) by @HuiyingLi
fix(ci): align Kimi GB200 expert groups with NVLink blocks (#3914) by @yuhezhang-ai
docs(model-coverage): fix GLM-5.3-Flash install command and transformers pin (#3939) by @waynel96
perf(kimi-k3): Triton SiTU + attn-residual kernels, fp32 glue compile islands (#3894) by @yisongbetter
docs(fern): use gray code backgrounds (#3981) by @akoumpa
fix(recipes): use HybridEP for Qwen3 MoE benchmark [AMINT-370] (#3988) by @HuiyingLi
fix(recipes): use HybridEP for DeepSeek V3.2 LoRA [AMINT-369] (#3987) by @HuiyingLi
fix(dflash): apply activation checkpointing to the draft (#3946) by @kashif
perf(kimi_k3): fast-math Triton SiTU kernels (situ_backend: triton_fast_math) (#3985) by @yisongbetter
fix(launcher): reject nemo_run overrides the executor has no attribute for (#3949) by @kabirvashisht4-glitch
ci: bump preflight workflows to v1.11.0 (#3997) by @ko3n1g
fix(ci): use compatible Transformers for Gemma 4 eval (#3969) by @athitten
refactor(moe): use native grouped mm experts (#3991) by @akoumpa
build(container): include FlashMLA for GLM-5.3 CI [AMINT-347] (#4000) by @akoumpa
fix: update Nemotron 3.5 VL (#4001) by @HuiyingLi
fix(deepseek-v41): honor BackendConfig.fake_balanced_gate (#3996) by @yisongbetter
ci: align CUDA base image with 26.08 (#4004) by @akoumpa
fix(ci): normalize inactive top-p before saving merged adapters (#3998) by @yuhezhang-ai
perf(kimi_k3): opt-in fused KDA kernels (kda_chunk_impl=fused) (#3954) by @yisongbetter
feat: add MiMo V2.6 support (#3992) by @HuiyingLi
fix(distributed): ignore non-zero-dim FSDP copy-out ops in selective AC replay (#3961) by @linmuchuiyang
fix(ci): restore MoE benchmark activation checkpointing [AMINT-372] (#4011) by @akoumpa
feat(deepseek-v41): add H100-80GB recipes, Tulu3 data prep, and coverage notes (#3962) by @linmuchuiyang
fix(ci): raise Qwen3 MoE and Step 3.5 Flash benchmark timeouts (#3966) by @janbernloehr
feat(dflash): support top-p and top-k in speculative decoding (#3627) by @kashif
fix(loss): apply per-token gradients in parallel cross entropy (#3853) by @MrCapricornLiu
perf(qwen3-8-flash-next): vectorize QSA block selection, size chunks by memory (#4006) by @Agoniii
docs: update Super 3.5 VL checkpoint names for GA (#4013) by @HuiyingLi
perf(qwen3.8): add FA4 QSA backend for H100 (#3925) by @HuiyingLi
perf(docker): add build parallelism opt-in build args (#4028) by @ko3n1g
fix(docs): use incremental Fern publishing (#4024) by @akoumpa
feat(mimo_v25): support MiMo-V2.5-Pro (#2514) by @Simar-malhotra09
fix(transformers): pin config and weights to the resolved Hub snapshot (#4019) by @akoumpa
fix(deps): make MoE source builds work with CUDA 13 (#4029) by @akoumpa
fix(nemotron_v3): accept singular embedding HF key alias (#3999) by @dundysm
fix(distributed): retain TE context-parallel token indices [AMINT-353] (#3984) by @HuiyingLi
docs(fern): organize model coverage by provider and model ID (#3974) by @akoumpa
fix(nemotron-flash): support Transformers 5.15 layer types [AMINT-360] (#4002) by @akoumpa
fix(distributed): prepare runtimes before pipeline traffic [AMINT-362] (#4007) by @akoumpa
fix(ci): normalize recipe paths and git decoding in coverage table sync (#3993) by @MohammedAlkindi
fix(benchmark): repair nightly benchmark configs (#4033) by @akoumpa
fix(test): restore Step-3.7 HF PEFT checkpoint reload (#4030) by @yuhezhang-ai
fix(checkpoint): preserve nested HF key renames (#3982) by @yuhezhang-ai
fix(training): reduce grad-clip norms across the expert-parallel axis (#3817) by @stanley1208
perf(benchmark): disable MTP for Qwen3.5 MoE LoRA (#4034) by @akoumpa
fix(ci): accept identical short greedy outputs [AMINT-15] (#4003) by @akoumpa
fix(benchmark): use HybridEP for Kimi K2 [AMINT-385] (#4040) by @akoumpa
fix(optim): exclude empty local shards from TE FusedAdam updates (#4035) by @pzelasko
fix(init): support HF initialization of DTensor-sharded models (#3983) by @yuhezhang-ai
test(peft): close the remaining reload-suite coverage gaps (#4026) by @stanley1208
ci(review): AUT-2439 migrate legacy Claude commands to /review (#4044) by @svcnemo-autobot
fix(nemotron_omni): handle PEFT key conversion for adapter save and reload (#3986) by @taran-dev4u
fix(dataset): reject pickle in Megatron index caches (#4047) by @chtruong814
fix(datasets): handle unanswerable SQuAD rows in the chat-template path (#4038) by @MohammedAlkindi
fix: support native RADIO video projection in Nemotron Omni (#4014) by @HuiyingLi
fix(launcher): make SkyPilot multi-node torchrun rendezvous at the head (#4037) by @MohammedAlkindi
fix: skip None modules during parameter freezing (#3970) by @athitten
fix(qwen3.5): avoid MedPix allocator fragmentation [AMINT-377] (#4057) by @akoumpa
fix(loss): scale InfoNCE logits before masking padded negatives (#3850) by @MrCapricornLiu
feat(diffusion): add Qwen-Image-2.1 text-to-image finetuning (#4027) by @HuiyingLi
fix(ci): route GB200 DeepSeek-V4.1 recipes to GB200 runners [AMINT-358] (#4066) by @HuiyingLi
fix(recipes): load DeepSeek-V4.1 Tulu3 inputs without env vars [AMINT-365] (#4067) by @HuiyingLi
fix(data): load C++ helpers through PyTorch JIT (#4062) by @akoumpa
fix(ci): use NCCL internal IB path for Qwen3.8 Engram recipes [AMINT-344] (#4069) by @HuiyingLi
fix(moe): reduce peak expert MLP memory [AMINT-351] (#4010) by @akoumpa
refactor(retrieval): configure text attention mode (#1837) by @oliverholworthy
refactor(distributed): move parallelization into model sidecars (#4032) by @akoumpa
fix(loss): chunk output projection in cross entropy (#4046) by @akoumpa
test(minimax_m3_vl): raise lazy-import subprocess budget to 30s (#4074) by @HuiyingLi
docs(glm-5.3-flash): state the Transformers reference and proxy for logit parity (#4077) by @waynel96
fix(docs): AUT-2456 preview approved PR mirror branches (#4063) by @svcnemo-autobot
fix(optim): keep per-group lr and weight decay under the LR scheduler (#4050) by @MohammedAlkindi
fix(qwen3.5): support MTP with VLM context parallelism [AMINT-352] (#3990) by @HuiyingLi
fix(loss): fp32 hidden-grad accumulation and torch_compile impl for fused CE (#4048) by @HuiyingLi
perf(qwen3-8-flash-next): replay QSA routes on activation-checkpoint recompute (#4005) by @Agoniii
feat(moe): mxfp4-resident MoE experts for DeepSeek-V4-Flash LoRA (#2548) by @excepshenal
fix(qwen2_5_omni): accept v4_compatible in map_peft_target_module_to_hf (#4042) by @MohammedAlkindi
fix(peft): stop layer wildcards like .layers.1. matching layers 10-19 (#4045) by @MohammedAlkindi
refactor(packing): replace global HF monkeypatch with typed metadata (#3577) by @beccohov
fix(ci): relax GPT-OSS PEFT checkpoint resume tolerance (#4071) by @yuhezhang-ai
fix(deps): AUT-2465 raise security floors and pin patched Nsight Systems (#4084) by @svcnemo-autobot
fix(docker): backport DeepEP NVSHMEM queue-pair compatibility (#3968) by @janbernloehr
ci: install VLM media dependencies during the image build (#4119) by @akoumpa
fix(recipes): reduce seq-cls validation loss over all samples and DP ranks (#4043) by @MohammedAlkindi
fix(deps): install MoK through pyproject and uv sync (#4121) by @akoumpa
feat(mimo): add MiMo-V2.6-Pro support (#4054) by @HuiyingLi
ci: raise test timeouts from 60 to 70 seconds (#4125) by @akoumpa
fix(glm5_next, kimi_linear): always use chunk_kda (#4110) by @waynel96
fix(moe): avoid retaining routed inputs in chunked checkpoints (#4103) by @yh0903
fix(minimax-m3): plan MSA block selection once, outside checkpointed blocks (#4123) by @akoumpa
feat(diffusion): native MoE transformers with expert parallelism (#4113) by @HuiyingLi
perf(moe): add HybridEP compact routing (#3404) by @akoumpa
feat(attention): add native FA4 and model-aware packed batches (#3734) by @akoumpa
feat(retrieval): add Mistral3 VL embedding and reranking (#3915) by @oliverholworthy
build: require torchao>=0.16.0 so peft can inject LoRA adapters (#4131) by @kabirvashisht4-glitch
feat(diffusion): HunyuanImage-3.0 text-to-image fine-tuning with EP (#4129) by @khazic
fix(qwen3-8-flash-next): keep PLE hash and conv inside packed documents (#4132) by @rouchenzi
docs: add HunyuanImage-3.0 to README news (#4139) by @HuiyingLi
fix(hunyuan-image3): import prompt batch keys from the model package (#4144) by @HuiyingLi
fix(glm): honor checkpoint indexer rotary layout (#4142) by @yuvalnissan
fix(gemma4): align unpacked vision masks with Transformers (#4124) by @akoumpa
feat(retrieval): context aware reranker training (#3786) by @sahel-sh
fix(ci): use accessible Super benchmark config (#3980) by @athitten
docs: link Nemotron end-to-end embedding recipe (#4159) by @akoumpa
fix(retrieval): align Mistral prompts and vLLM template detection (#4153) by @oliverholworthy
fix(deps): include SQLite support in the MLflow extra (#4161) by @akoumpa
feat(glm5_next): training chat template that supervises the end-of-turn token (#4165) by @waynel96
fix(peft): groundwork for boundary-owned PEFT prefix handling (#4115) by @stanley1208
perf(moe): fuse bounded weighted expert bias addition (#4166) by @akoumpa
fix(deepseek-v32): align indexer RoPE layout and precision (#4146) by @HuiyingLi
fix(diffusion): restore inference device placement [AMINT-388] (#4136) by @HuiyingLi
fix(loss): inspect DDP model metadata for chunked cross entropy (#4167) by @akoumpa
fix(deps): AUT-2490 enforce CVE floors and remove nsys samplers (#4172) by @svcnemo-autobot
fix(fsdp): preserve model input and output dtypes (#4150) by @HuiyingLi
fix(checkpoint): preserve HF root renames through FSDP wrapping (#4163) by @yuhezhang-ai
fix(peft): freeze router biases in LoRA recipes (#4076) by @yuhezhang-ai
fix(deepseek-v4): make LoRA checkpoints reload and resume (#4072) by @yuhezhang-ai
fix(moe): initialize MOK expert weights on FSDP shards (#4174) by @yuhezhang-ai
docs: refresh release log and prepare 26.10 container references (4162) (#4184) by @svcnvidia-nemo-ci
test: fix reranker persistent worker timeout and cleanup (4183) (#4193) by @svcnvidia-nemo-ci
fix(distributed): use CP-local pipeline token capacity [AMINT-399] (4137) (#4192) by @svcnvidia-nemo-ci
fix(distributed): restore specialized Hugging Face parallelizers (4180) (#4200) by @svcnvidia-nemo-ci
fix(retrieval): export portable Mistral3 reranker checkpoints (4188) (#4197) by @svcnvidia-nemo-ci
fix(moe): restore bias-free throughput and GPT-OSS weight loading (4155) (#4199) by @svcnvidia-nemo-ci
fix(ci): relax intermittent checkpoint resume loss tolerances (4212) (#4222) by @svcnvidia-nemo-ci
cp: backport DeepSeek-V4.1 CP8 gradient sync fix to r0.7.0 (#4203) (#4221) by @akoumpa
fix(llm): release unused model outputs before backward (4201) (#4220) by @svcnvidia-nemo-ci
ci: Update transformers to latest version 5.18.0 (4116) (#4225) by @svcnvidia-nemo-ci
fix(training): validate domain mixture corpus bindings (4195) (#4226) by @svcnvidia-nemo-ci
perf(moe): raise biased expert MLP chunk threshold to 512 MiB (4234) (#4235) by @svcnvidia-nemo-ci
fix(container): AUT-2510 use patched aiohttp in Ray runtime-env agent (4223) (#4237) by @svcnvidia-nemo-ci
fix(ci): keep pyarrow compatible with CUDA build numpy (4244) (#4247) by @svcnvidia-nemo-ci
fix(ci): include recipes in import contract checks (4224) (#4248) by @svcnvidia-nemo-ci
docs(fern): unpublish Nemotron 3.5 Super VL guides (4253) (#4254) by @svcnvidia-nemo-ci
fix(ci): relax DeepSeek V4 LoRA resume loss tolerance (4240) (#4251) by @svcnvidia-nemo-ci
perf(minimax): optimize M3 attention and checkpoint recomputation (4176) (#4249) by @svcnvidia-nemo-ci
fix(retrieval): backport Pixtral FlashAttention image boundaries (4204) (#4267) by @svcnvidia-nemo-ci
fix(deps): raise security floors for audited Python packages (4246) (#4272) by @svcnvidia-nemo-ci
test: keep L0 Hub access opt-in and fix HellaSwag mocks (4268) (#4273) by @svcnvidia-nemo-ci
build(docker): AUT-2515 skip redundant rdma-core installation (4255) (#4274) by @svcnvidia-nemo-ci
fix(ci): reuse verified wheelhouses and container dependencies (4269) (#4280) by @svcnvidia-nemo-ci
fix(ci): relax customizer GPT-OSS resume loss tolerance (4278) (#4287) by @svcnvidia-nemo-ci
fix(qwen3.8): materialize hybrid layer types before model construction (4206) (#4239) by @svcnvidia-nemo-ci
fix(packing): validate TE masks and fix Qwen3-VL packed inputs (4241) (#4259) by @svcnvidia-nemo-ci
fix(diffusion): retain validation tails and weight loss by samples (4227) (#4291) by @svcnvidia-nemo-ci
ci: restore wheelhouse caches directly in pip and UV jobs (4289) (#4301) by @svcnvidia-nemo-ci
fix(ci): stabilize DeepSeek-V4 MXFP4 validation (4245) (#4262) by @svcnvidia-nemo-ci
fix(distributed): reject unsupported MTP CP PP from mesh topology (4252) (#4296) by @svcnvidia-nemo-ci
fix(ci): restore device hooks after HF adapter reload (4277) (#4297) by @svcnvidia-nemo-ci
fix(qwen3.5): support packed vision training with TE attention (4242) (#4303) by @svcnvidia-nemo-ci
fix(vlm): pad variable-resolution images in Ministral recipes (4304) (#4309) by @svcnvidia-nemo-ci
fix(qwen3.5-moe): isolate packed text metadata from vision (4306) (#4316) by @svcnvidia-nemo-ci
fix(vlm): fit Muse Glimmer FP32 optimizer state on two nodes (4307) (#4321) by @svcnvidia-nemo-ci
fix(checkpoint): apply Qwen MTP FP8 scales (AM-864) (4250) (#4318) by @svcnvidia-nemo-ci
fix(deps): AUT-2528 raise Apache Thrift security floor to 0.25.0 (4315) (#4319) by @svcnvidia-nemo-ci
fix(mimo): include the query token in sliding attention windows (4343) (#4344) by @svcnvidia-nemo-ci
fix(benchmark): correct MoK CI topology and DeepSeek V4 FLOPs (4302) (#4345) by @svcnvidia-nemo-ci
fix(ci): use compatible Nemotron Nano 9B deployment settings (4339) (#4350) by @svcnvidia-nemo-ci
docs(fern): add recipe-linked model performance tables (4256) (#4361) by @svcnvidia-nemo-ci
cp: backport Triton RoPE #4214 and rotary fusion #4346 to r0.7.0 (#4360) by @akoumpa
docs: apply jgerh copyedits from PR 4159 (4191) (#4383) by @svcnvidia-nemo-ci
cp: reduce routing and GLM overhead for r0.7.0 (#4400) by @akoumpa
fix(peft): eliminate FP32 storage for frozen expert weights (4388) (#4392) by @svcnvidia-nemo-ci
fix(qwen): use FA4 to avoid cuDNN SDPA backward NaNs (4423) (#4426) by @svcnvidia-nemo-ci
docs: point release documentation links to r0.7.0 (#4431) by @akoumpa
cp: backport homepage and training guides (#4425) to r0.7.0 (#4432) by @akoumpa
cp: backport recipe deprecation metadata to r0.7.0 (#4436) (#4440) by @akoumpa
fix(mimo): pin HybridEP domains for GB200 SFT recipes (4445) (#4447) by @svcnvidia-nemo-ci
fix(deps): AUT-2606 backport Pillow 13 to r0.7.0 (#4458) by @svcnemo-autobot
docs: add 0.6.0 and 0.7.0 release notes (4471) (#4472) by @svcnvidia-nemo-ci

Navigation Menu

Repository navigation

NVIDIA NeMo-Automodel 0.7.0

Choose a tag to compare

Sorry, something went wrong.