Skip to content

fix(cuda_graphs): handle TE 2.15 removal of FP8GlobalStateManager.set_skip_fp8_weight_update_tensor#4874

Merged
balasaajay merged 6 commits into
NVIDIA:mainfrom
balasaajay:fix-4707
May 22, 2026
Merged

fix(cuda_graphs): handle TE 2.15 removal of FP8GlobalStateManager.set_skip_fp8_weight_update_tensor#4874
balasaajay merged 6 commits into
NVIDIA:mainfrom
balasaajay:fix-4707

Conversation

@balasaajay
Copy link
Copy Markdown
Contributor

@balasaajay balasaajay commented May 19, 2026

What does this PR do ?

Fixes - #4707

After the CI base image bumped to TE 2.15, every FP8-enabled cudagraph runner construction crashed with:
AttributeError: type object 'FP8GlobalStateManager' has no attribute 'set_skip_fp8_weight_update_tensor'

Route the three calls in cuda_graphs.py through a new helper _set_skip_fp8_weight_update_tensor(skip) that:

  • on TE ≤ 2.14: calls the legacy classmethod (unchanged behavior).
  • on TE ≥ 2.15: writes FP8GlobalStateManager.quantization_state.skip_fp8_weight_update_tensor directly, lazily allocating the persistent 1-element float32 CUDA tensor and updating it in place via fill_(skip) — preserving the data pointer that captured cudagraphs read from, so the flag can still be toggled between microbatches without re-capturing.
    No behavior change on supported TE versions; unblocks CI on TE ≥ 2.15.

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact @NVIDIA/mcore-oncall.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

For MRs into `dev` branch The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 19, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@balasaajay balasaajay requested a review from santhnm2 May 19, 2026 18:41
@balasaajay
Copy link
Copy Markdown
Contributor Author

/ok to test a0cd6d7

Comment thread megatron/core/transformer/cuda_graphs.py
@balasaajay
Copy link
Copy Markdown
Contributor Author

/ok to test 5a661e2

@balasaajay balasaajay requested a review from a team as a code owner May 20, 2026 06:17
@svcnvidia-nemo-ci svcnvidia-nemo-ci requested a review from a team May 20, 2026 06:18
@balasaajay balasaajay requested a review from a team as a code owner May 20, 2026 21:07
@balasaajay
Copy link
Copy Markdown
Contributor Author

/ok to test a94a352

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Final Review PR is in the "final review" stage label May 21, 2026
@balasaajay balasaajay enabled auto-merge May 22, 2026 02:04
@svcnvidia-nemo-ci svcnvidia-nemo-ci added Approved All necessary approvals have been made and removed Final Review PR is in the "final review" stage labels May 22, 2026
@balasaajay balasaajay added this pull request to the merge queue May 22, 2026
@svcnvidia-nemo-ci
Copy link
Copy Markdown

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/26301573343

Merged via the queue into NVIDIA:main with commit 0beaa53 May 22, 2026
206 of 215 checks passed
@balasaajay balasaajay deleted the fix-4707 branch May 22, 2026 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved All necessary approvals have been made complexity: low Run functional tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants