Skip to content

Merge main into dev#3865

Open
ilml wants to merge 144 commits intoNVIDIA:devfrom
ilml:merge-main-into-dev
Open

Merge main into dev#3865
ilml wants to merge 144 commits intoNVIDIA:devfrom
ilml:merge-main-into-dev

Conversation

@ilml
Copy link
Contributor

@ilml ilml commented Mar 13, 2026 •
edited
Loading

What does this PR do ?

Sync the latest main branch changes into the development line and preserve ongoing dev-only work through the required conflict resolutions.

Summary

  • Pull the latest upstream main changes into the merge branch, including broad updates across CI/workflows, docs, examples, inference, FSDP/resharding, and test coverage.
  • Preserve active dev work around emerging optimizers and layer-wise optimizer refactoring, HyperConnection, Dynamic-CP / THD handling, MoE, MLA/MTP, and related attention and training paths.
  • Resolve merge conflicts in the key overlap areas so dev behavior is retained while adopting the latest main changes.
  • Keep the existing dev uv.lock during the merge because regenerating it with the installed uv currently fails on upstream nemo-run metadata.

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact the @mcore-oncall.

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

For MRs into `dev` branch The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

maanug-nv and others added 30 commits February 25, 2026 00:59
@maanug-nv
Move config src files into a dedicated dir (NVIDIA#3570)
b17248a 
Signed-off-by: Maanu Grover <maanug@nvidia.com>
@github-actions
chore: rotate oncall schedule
9bb2375
@ko3n1g
Revert "remove encoder_and_decoder from enums (NVIDIA#3406)" (NVIDIA#…
0dba6a0 
…3579)
@yobibyte
Fix default cuda graph persist arg. Add persist to rl common.sh. (NVI…
b359981 
…DIA#3584)
@sidsingh-nvidia
Optimize away add request overheads in dummy ep cuda-graphed forward …
60a25aa 
…passes (NVIDIA#3525)
@ko3n1g
ci: Test docs build (NVIDIA#3583)
e7d80bd 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Fix docs build for release (NVIDIA#3597)
80ef2ae 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Remove secrets (NVIDIA#3598)
2561a59 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Define secrets (NVIDIA#3599)
19444b9 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: gh-release-from-tag (NVIDIA#3600)
a08ad25 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
Ko3n1g/ci/remove twine username (NVIDIA#3601)
ac449c4 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@maanug-nv
Add training code to MCore wheel (NVIDIA#3573)
9088d4f 
Signed-off-by: Maanu Grover <maanug@nvidia.com>
@vasunvidia @yaox12
FP8 attention knob for nvFP4 recipe (NVIDIA#3363)
dd4c0b7 
Co-authored-by: Xin Yao <xiny@nvidia.com>
@guyueh1 @yaox12
Fix error with --load-main-params-from-ckpt (NVIDIA#3569)
c2782fd 
Signed-off-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster>
Co-authored-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster>
Co-authored-by: Xin Yao <xiny@nvidia.com>
@ko3n1g
ci: Create comment (NVIDIA#3610)
c921891 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Skip cleanup-taint-node jobs during deployments (NVIDIA#3612)
9100119 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: No comment for release workflow (NVIDIA#3615)
6161f7a 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Re-add release tag prefix (NVIDIA#3619)
36d8a9d 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@chtruong814
docs: Fix version picker urls (NVIDIA#3621)
1b1f5c4 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
ci: Increase changelog generation max PRs fetched (NVIDIA#3620)
027e0f3 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@yobibyte
Add debug info to an assert. (NVIDIA#3588)
18c94d2
@sbak5
fix: async_utils: explicit GC in persistent checkpoint worker loop (N…
e1a9ac9 
…VIDIA#3591)
@CodersAcademy006 @yaox12
Fix: Perform sigmoid calculation in fp32 for aux loss stability (NVID…
5f668c1 
…IA#2765)

Co-authored-by: Xin Yao <xiny@nvidia.com>
@saakshigupta2002 @yaox12
fix: forward use_te_activation_func flag in non-MoE GPT layer spec (N…
d3c10df 
…VIDIA#3300)

Co-authored-by: Xin Yao <xiny@nvidia.com>
@faradawn
Multimodal: Limit transformer version in Dockerfile (NVIDIA#3448)
7e3f670 
Signed-off-by: nvidia <nvidia@TRY-64956-gpu01.nvidialaunchpad.com>
@tdene
Track and plot per-token off-policy in RL (NVIDIA#3515)
6c0b9c6
@ko3n1g
Revert "Add single-process checkpoint save to avoid forked multiproce… (
a7c207f 
NVIDIA#3630)
@faradawn
Multimodal: fix VQA dataset selection (NVIDIA#3464)
6503bf8 
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
@faradawn
Multimodal: Fix multimodal training example - tokenizer, Triton Cache…
2f549e5 
… Manager patch, docs (NVIDIA#3507)

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
@santhnm2
Support TP > GQA for inference (NVIDIA#3627)
afbce84 
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
@ilml ilml requested review from a team as code owners March 13, 2026 19:54
@copy-pr-bot
Copy link

copy-pr-bot bot commented Mar 13, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci marked this pull request as draft March 13, 2026 19:54
@github-actions
Copy link
Contributor

github-actions bot commented Mar 13, 2026

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@ilml ilml changed the base branch from main to dev March 13, 2026 20:01
@ilml ilml removed request for a team March 13, 2026 21:21
@ilml ilml marked this pull request as ready for review March 13, 2026 23:08
@ilml ilml requested a review from a team as a code owner March 13, 2026 23:09
This was referenced Mar 16, 2026
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

complexity: high

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.