Skip to content

Batch-invariant train/inference logprob parity#5897

Merged
wdykas merged 1 commit into
NVIDIA:mainfrom
wdykas:batch-invariant-logprob-parity-main
Jul 23, 2026
Merged

Batch-invariant train/inference logprob parity#5897
wdykas merged 1 commit into
NVIDIA:mainfrom
wdykas:batch-invariant-logprob-parity-main

Conversation

@wdykas

@wdykas wdykas commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

making batch invariance more robust and fixing a few other corner cases

  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact @NVIDIA/mcore-oncall.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

@wdykas
wdykas requested review from a team as code owners July 20, 2026 16:39
@copy-pr-bot

copy-pr-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft July 20, 2026 16:39
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@wdykas
wdykas marked this pull request as ready for review July 20, 2026 16:44
@wdykas

wdykas commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test de53103

@wdykas
wdykas force-pushed the batch-invariant-logprob-parity-main branch from de53103 to 0762c9f Compare July 20, 2026 19:36
@wdykas

wdykas commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 0762c9f

yobibyte
yobibyte reviewed Jul 21, 2026
View reviewed changes
Comment thread megatron/rl/sequence_packing_utils.py
yobibyte
yobibyte reviewed Jul 21, 2026
View reviewed changes
Comment thread megatron/core/transformer/custom_layers/batch_invariant_kernels.py Outdated
yobibyte
yobibyte reviewed Jul 21, 2026
View reviewed changes
Comment thread megatron/core/transformer/custom_layers/batch_invariant_kernels.py Outdated
yobibyte
yobibyte approved these changes Jul 21, 2026
View reviewed changes

@yobibyte yobibyte left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, left some small comments!

@wdykas
wdykas force-pushed the batch-invariant-logprob-parity-main branch from 0762c9f to 661f235 Compare July 21, 2026 13:33
@wdykas
Batch-invariant train/inference logprob parity
aacf1bd 
Make --batch-invariant-mode produce train-side logprobs that match the
dynamic inference engine bitwise, so importance-sampling / off-policy RL
sees zero train-vs-inference logprob mismatch. Four fixes:

1. Pin the FlashAttention generation. The training (TransformerEngine) and
   inference (mcore dynamic-batching) attention paths select a
   FlashAttention version independently based on what is installed. When
   they diverge -- e.g. on Blackwell, where inference runs FA4 while TE
   falls back to FA2 -- their logits differ bitwise past ~48 KV tokens
   (different tile sizes and softmax accumulation orders). Add a
   flash_attention_version TransformerConfig field (--flash-attention-version,
   choices 2/3/4) that pins the generation for both paths: training disables
   the other versions via TransformerEngine's NVTE_FLASH_ATTN_V2/V3/V4
   selection env vars in LanguageModule._set_attention_backend, and inference
   honors it directly in Attention.flash_decode_and_prefill. Batch-invariant
   mode now requires --attention-backend flash and --flash-attention-version
   3 or 4 (FA2 lacks the fixed num_splits schedule the batch-invariant
   kernels need).

2. Route TransformerEngine's fused-module normalization through the
   batch-invariant RMSNorm. TE's fused LayerNormLinear / LayerNormMLP call
   apply_normalization instead of RMSNorm.forward, bypassing the existing
   batch-invariant patch; TE's tex RMSNorm kernel changes its within-row
   reduction with the total row count (observed: identical rows, 1 bf16 ulp
   different output at 928 vs 2274 rows), which amplifies across depth. The
   decode CUDA graph pads every step to max_requests (~4548 rows from the KV
   buffer), so this fired at scale even though small forwards matched.

3. Compare packed inference logprobs in the training dtype. The packed
   inference-vs-train stats compared bf16 training logprobs against fp32
   engine logprobs, showing rounding noise even when both sides were
   bitwise identical.

4. Only compare engine-reported positions. pack_inference_logprobs
   zero-filled positions the engine never reported (e.g. the train-side EOD
   append), each contributing a spurious |p_old - 1| term to the mismatch
   stats. Return a filled-position mask and restrict the stats to it.

Enable the GPT batch-invariance unit tests on FlashAttention-4 (previously
skipped without FA3) so they run on Blackwell.

Signed-off-by: wdykas <wdykas@nvidia.com>
@wdykas
wdykas force-pushed the batch-invariant-logprob-parity-main branch from 661f235 to aacf1bd Compare July 21, 2026 14:08
santhnm2
santhnm2 approved these changes Jul 22, 2026
View reviewed changes
Comment thread megatron/core/models/common/language_module/language_module.py
# which honors config.flash_attention_version directly.
if self.config.flash_attention_version is not None:
for version in (2, 3, 4):
if version != self.config.flash_attention_version:

@santhnm2 santhnm2 Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to have the else case explicitly set version = 1? So

else:
     check_and_set_env_variable(
         f"NVTE_FLASH_ATTN_V{version}", 1, self.config.attention_backend
     )

@wdykas

wdykas commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test aacf1bd

@wdykas
wdykas enabled auto-merge July 22, 2026 21:07
@wdykas
wdykas added this pull request to the merge queue Jul 23, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/29982749312

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/29983602174

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/29988909160

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 23, 2026
@wdykas
wdykas added this pull request to the merge queue Jul 23, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30009974612

Merged via the queue into NVIDIA:main with commit 51e915a Jul 23, 2026
92 of 93 checks passed
@wdykas
wdykas deleted the batch-invariant-logprob-parity-main branch July 23, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yobibyte yobibyte yobibyte approved these changes
@santhnm2 santhnm2 santhnm2 approved these changes
@kvareddy kvareddy kvareddy approved these changes
@shanmugamr1992 shanmugamr1992 shanmugamr1992 approved these changes

Assignees

No one assigned

Labels

complexity: medium

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@wdykas @svcnvidia-nemo-ci @yobibyte @santhnm2 @kvareddy @shanmugamr1992