Skip to content

[fix] Fix num_padding_microbatches metric for Megatron backend#1856

Merged
SumanthRH merged 1 commit into
mainfrom
fix-num-padding
Jul 1, 2026
Merged

[fix] Fix num_padding_microbatches metric for Megatron backend#1856
SumanthRH merged 1 commit into
mainfrom
fix-num-padding

Conversation

@SumanthRH

Copy link
Copy Markdown
Member

What does this PR do?

Fixes num_padding_microbatches metric calculation for the Megatron backend when there are real sequences with all zero loss masks (example with apply_overlong_filtering=true)

The number of padding microbatches should not count all sequences with zero loss mask because some of these are real sequences but masked out for loss calculation.

The fix is to read the num_padding_microbatches attribute from the microbatch_iterator when available.

Signed-off-by: SumanthRH <sumanthrh@anyscale.com>
@SumanthRH SumanthRH requested a review from erictang000 July 1, 2026 02:57

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the calculation of real and padding microbatches in megatron_worker.py. Instead of inferring the number of real microbatches by checking if the loss_mask sum is greater than zero, it now retrieves the padding count directly from the microbatch_iterator via num_padding_microbatches. This prevents real microbatches with an all-zero loss mask (e.g., due to DAPO overlong filtering) from being incorrectly classified as padding. Additionally, the status reporting is updated to use this padding count directly. There are no review comments, so no feedback is provided.

@SumanthRH SumanthRH merged commit dca8f8f into main Jul 1, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant