chore: bumpup Megatron-Bridge submodule to main by ZhiyuLi-Nvidia · Pull Request #2039 · NVIDIA-NeMo/RL

ZhiyuLi-Nvidia · 2026-03-01T10:57:04Z

What does this PR do ?

Oncall responsibility, regularly bump up Megatron-Bridge/Megatron-LM dependency:

Megatron-Bridge bump 45dbf473 → a2bb70b9 (137 commits)
Megatron-LM bump 8318b809 → 9e2810417 (127 commits) — synced with what Megatron-Bridge expects

NeMo-RL
├── Megatron-LM           @ 9e2810417
└── Megatron-Bridge        @ a2bb70b9
    └── Megatron-LM        @ 9e2810417  (expected by Bridge)

add `gradient_accumulation_fusion: false` to address the following issue:

RuntimeError: ColumnParallelLinear was called with gradient_accumulation_fusion set to True but the custom CUDA extension fused_weight_gradient_mlp_cuda module is not found. To use gradient_accumulation_fusion you must install APEX with --cpp_ext and --cuda_ext. For example: pip install --global-option="--cpp_ext" --global-option="--cuda_ext ." Note that the extension requires CUDA>=11. Otherwise, you must turn off gradient accumulation fusion.

dsv3 style model defaults gradient_accumulation_fusion to true in mbridge:

fixed once removed gradient_accumulation_fusion=true: https://github.com/NVIDIA-NeMo/Megatron-Bridge/pull/2696/commits
appeared again after reversion: NVIDIA-NeMo/Megatron-Bridge@821aa6a

Issues

List issues that this PR closes (syntax):

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

...

Summary by CodeRabbit

Chores
- Updated Megatron-Bridge submodule reference.

github-actions · 2026-03-01T10:58:08Z

✅ Submodule Fast-Forward Check Results

Check based on commit: bedab04 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-03-01T10:58:41Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 3a26b67 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

coderabbitai · 2026-03-01T10:59:03Z

📝 Walkthrough

Walkthrough

Updates the Megatron-Bridge submodule pointer from commit f91542b90908ad08b7e13672feea03e27bedee27 to commit 383b610dca1fe553346b37fc0172f54917b02069. No functional code changes—only the submodule reference is updated.

Changes

Cohort / File(s)	Summary
Submodule Update `3rdparty/Megatron-Bridge-workspace/Megatron-Bridge`	Submodule pointer updated to a newer commit reference.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Possibly related PRs

feat: Update mbridge with cache support #1187: Updates the same Megatron-Bridge submodule pointer with a different target commit.
chore: update megatron dev (11/21/2025) / mbridge (11/28/2025) #1568: Updates the same Megatron-Bridge submodule pointer to a different commit.
fix: Fix checkpoint conversion error for qwen 30b-a3b #1335: Updates the Megatron-Bridge submodule pointer in the same location to a different commit.

Suggested labels

CI:L1

Suggested reviewers

terrykong
yaoyu-33
yfw

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: bumping up the Megatron-Bridge submodule, which is the sole purpose of this pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Test Results For Major Changes	✅ Passed	PR contains only a minor submodule pointer update with no code or logic changes, qualifying as a routine dependency bump.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch zhiyul/bump_up_mbridge

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-03-01T14:29:26Z

✅ Submodule Fast-Forward Check Results

Check based on commit: ac9fa35 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

copy-pr-bot · 2026-03-25T02:09:28Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-03-25T02:10:28Z

❌ Submodule Fast-Forward Check Failed

Check based on commit: dd5f479 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

❌ Submodules that need attention:

Megatron-Bridge: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/45dbf4735ec3a2034be66ed4183604642a84af0d/
CURRENT (PR #2039 from zhiyul/bump_up_mbridge): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/a2bb70b91b827bd6b085a77442c7cf60cfdb59fe/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

It is as expected as current branch uses cherry-picked fix commits.

ZhiyuLi-Nvidia · 2026-03-25T02:16:53Z

/ok to test dd5f479

ZhiyuLi-Nvidia · 2026-03-25T03:56:08Z

/ok to test a522794

github-actions · 2026-03-25T03:56:13Z

❌ Submodule Fast-Forward Check Failed

Check based on commit: a522794 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

❌ Submodules that need attention:

Megatron-Bridge: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/45dbf4735ec3a2034be66ed4183604642a84af0d/
CURRENT (PR #2039 from zhiyul/bump_up_mbridge): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/a2bb70b91b827bd6b085a77442c7cf60cfdb59fe/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

github-actions · 2026-03-25T04:07:51Z

❌ Submodule Fast-Forward Check Failed

Check based on commit: a8fa0ff (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

❌ Submodules that need attention:

Megatron-Bridge: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/45dbf4735ec3a2034be66ed4183604642a84af0d/
CURRENT (PR #2039 from zhiyul/bump_up_mbridge): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/a2bb70b91b827bd6b085a77442c7cf60cfdb59fe/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

ZhiyuLi-Nvidia · 2026-03-25T04:08:47Z

/ok to test a8fa0ff

ZhiyuLi-Nvidia · 2026-03-25T04:13:13Z

/ok to test 56b8508

github-actions · 2026-03-25T04:13:42Z

❌ Submodule Fast-Forward Check Failed

Check based on commit: 56b8508 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

❌ Submodules that need attention:

Megatron-Bridge: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/45dbf4735ec3a2034be66ed4183604642a84af0d/
CURRENT (PR #2039 from zhiyul/bump_up_mbridge): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/a2bb70b91b827bd6b085a77442c7cf60cfdb59fe/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

ZhiyuLi-Nvidia · 2026-03-25T04:23:44Z

/ok to test ab73597

github-actions · 2026-03-25T04:24:38Z

❌ Submodule Fast-Forward Check Failed

Check based on commit: ab73597 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

❌ Submodules that need attention:

Megatron-Bridge: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/45dbf4735ec3a2034be66ed4183604642a84af0d/
CURRENT (PR #2039 from zhiyul/bump_up_mbridge): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/a2bb70b91b827bd6b085a77442c7cf60cfdb59fe/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

…t-llama3.1-8b-1n8g-fsdp2tp4-dynamicbatch Signed-off-by: Yuki Huang <yukih@nvidia.com>

Signed-off-by: Yuki Huang <yukih@nvidia.com>

github-actions · 2026-03-31T00:27:49Z

❌ Submodule Fast-Forward Check Failed

Check based on commit: 32220d2 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

❌ Submodules that need attention:

Megatron-Bridge: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/45dbf4735ec3a2034be66ed4183604642a84af0d/
CURRENT (PR #2039 from zhiyul/bump_up_mbridge): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/a2bb70b91b827bd6b085a77442c7cf60cfdb59fe/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

github-actions · 2026-03-31T00:28:22Z

❌ Submodule Fast-Forward Check Failed

Check based on commit: 77a4b2c (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

❌ Submodules that need attention:

Megatron-Bridge: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/45dbf4735ec3a2034be66ed4183604642a84af0d/
CURRENT (PR #2039 from zhiyul/bump_up_mbridge): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/a2bb70b91b827bd6b085a77442c7cf60cfdb59fe/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

yuki-97 · 2026-03-31T00:29:07Z

/ok to test 77a4b2c

Signed-off-by: Yuki Huang <yukih@nvidia.com>

yuki-97 · 2026-03-31T05:03:12Z

/ok to test b8f5647

github-actions · 2026-03-31T05:04:01Z

❌ Submodule Fast-Forward Check Failed

Check based on commit: b8f5647 (PR #2039 from zhiyul/bump_up_mbridge)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

❌ Submodules that need attention:

Megatron-Bridge: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/45dbf4735ec3a2034be66ed4183604642a84af0d/
CURRENT (PR #2039 from zhiyul/bump_up_mbridge): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/a2bb70b91b827bd6b085a77442c7cf60cfdb59fe/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

yuki-97

all nightly tests passed, could be merged.

CI:L1 passed in 77a4b2c in https://github.com/NVIDIA-NeMo/RL/actions/runs/23774491454/job/69279367064.

only scripts changes in b8f5647 after that, so run with CI:docs

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com> Signed-off-by: Yuki Huang <yukih@nvidia.com> Co-authored-by: Yuki Huang <yukih@nvidia.com>

ZhiyuLi-Nvidia requested a review from a team as a code owner March 1, 2026 10:57

ZhiyuLi-Nvidia force-pushed the zhiyul/bump_up_mbridge branch from bedab04 to 3a26b67 Compare March 1, 2026 10:57

ZhiyuLi-Nvidia added CI:L2 Run doctests, unit tests, functional tests, and convergence tests Run CICD labels Mar 1, 2026

ZhiyuLi-Nvidia had a problem deploying to nemo-ci March 1, 2026 10:58 — with GitHub Actions Failure

ZhiyuLi-Nvidia force-pushed the zhiyul/bump_up_mbridge branch from ac9fa35 to dd5f479 Compare March 25, 2026 02:09

copy-pr-bot Bot had a problem deploying to nemo-ci March 25, 2026 02:17 Failure

ZhiyuLi-Nvidia requested review from a team as code owners March 25, 2026 03:55

copy-pr-bot Bot had a problem deploying to nemo-ci March 25, 2026 03:56 Error

copy-pr-bot Bot had a problem deploying to nemo-ci March 25, 2026 04:09 Error

copy-pr-bot Bot had a problem deploying to nemo-ci March 25, 2026 04:13 Error

copy-pr-bot Bot had a problem deploying to nemo-ci March 25, 2026 04:24 Error

ZhiyuLi-Nvidia and others added 9 commits March 31, 2026 08:27

align cached dependencies

0c95f4d

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

lint

3c83d54

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

aligned libraries

117bdfb

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

bump up transformers 5.3.0

ca6f0e8

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

fix assertion error tests

d7fd1c9

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

fix nm5 in transformers 5

9961324

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

loosen thershold for dpo-nanov3-30B3AB-1n8g-fsdp8ep8-automodel and sf…

7d23335

…t-llama3.1-8b-1n8g-fsdp2tp4-dynamicbatch Signed-off-by: Yuki Huang <yukih@nvidia.com>

bump automodel to fix several nightly fails

653fe54

Signed-off-by: Yuki Huang <yukih@nvidia.com>

check_mbridge_deps

77a4b2c

Signed-off-by: Yuki Huang <yukih@nvidia.com>

yuki-97 force-pushed the zhiyul/bump_up_mbridge branch from 32220d2 to 77a4b2c Compare March 31, 2026 00:27

copy-pr-bot Bot temporarily deployed to nemo-ci March 31, 2026 00:29 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci March 31, 2026 01:44 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci March 31, 2026 03:25 Inactive

fix nightly after rebase

b8f5647

Signed-off-by: Yuki Huang <yukih@nvidia.com>

yuki-97 added CI:docs Run doctest and removed CI:L1 Run doctests, unit tests, and functional tests Run CICD labels Mar 31, 2026

copy-pr-bot Bot temporarily deployed to nemo-ci March 31, 2026 05:03 Inactive

yuki-97 approved these changes Mar 31, 2026

View reviewed changes

terrykong approved these changes Mar 31, 2026

View reviewed changes

terrykong merged commit bc8aa39 into main Mar 31, 2026
25 of 26 checks passed

terrykong deleted the zhiyul/bump_up_mbridge branch March 31, 2026 05:36

isomap pushed a commit to isomap/RL that referenced this pull request Mar 31, 2026

chore: bumpup Megatron-Bridge submodule to main (NVIDIA-NeMo#2039)

0b814d1

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com> Signed-off-by: Yuki Huang <yukih@nvidia.com> Co-authored-by: Yuki Huang <yukih@nvidia.com>

This was referenced Apr 7, 2026

fix: fix h100 release/performance #2184

Merged

fix: fix gb200 release/performance #2189

Merged

Conversation

ZhiyuLi-Nvidia commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

add gradient_accumulation_fusion: false to address the following issue:

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Mar 1, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Mar 1, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

coderabbitai Bot commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

Uh oh!

github-actions Bot commented Mar 1, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

copy-pr-bot Bot commented Mar 25, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026 • edited by ZhiyuLi-Nvidia Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Submodule Fast-Forward Check Failed

✅ Submodules that are properly updated:

❌ Submodules that need attention:

Uh oh!

ZhiyuLi-Nvidia commented Mar 25, 2026

Uh oh!

ZhiyuLi-Nvidia commented Mar 25, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026

❌ Submodule Fast-Forward Check Failed

✅ Submodules that are properly updated:

❌ Submodules that need attention:

Uh oh!

github-actions Bot commented Mar 25, 2026

❌ Submodule Fast-Forward Check Failed

✅ Submodules that are properly updated:

❌ Submodules that need attention:

Uh oh!

ZhiyuLi-Nvidia commented Mar 25, 2026

Uh oh!

ZhiyuLi-Nvidia commented Mar 25, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026

❌ Submodule Fast-Forward Check Failed

✅ Submodules that are properly updated:

❌ Submodules that need attention:

Uh oh!

ZhiyuLi-Nvidia commented Mar 25, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026

❌ Submodule Fast-Forward Check Failed

✅ Submodules that are properly updated:

❌ Submodules that need attention:

Uh oh!

github-actions Bot commented Mar 31, 2026

❌ Submodule Fast-Forward Check Failed

✅ Submodules that are properly updated:

❌ Submodules that need attention:

Uh oh!

github-actions Bot commented Mar 31, 2026

❌ Submodule Fast-Forward Check Failed

✅ Submodules that are properly updated:

❌ Submodules that need attention:

Uh oh!

ZhiyuLi-Nvidia commented Mar 1, 2026 •

edited

Loading

add `gradient_accumulation_fusion: false` to address the following issue:

coderabbitai Bot commented Mar 1, 2026 •

edited

Loading

github-actions Bot commented Mar 25, 2026 •

edited by ZhiyuLi-Nvidia

Loading