[Fix] extra ops is called in _fsdp_state of pytorch2.8#1237
Merged
CyCle1024 merged 3 commits intoInternLM:mainfrom Nov 15, 2025
Merged
[Fix] extra ops is called in _fsdp_state of pytorch2.8#1237CyCle1024 merged 3 commits intoInternLM:mainfrom
CyCle1024 merged 3 commits intoInternLM:mainfrom
Conversation
HAOCHENYE
approved these changes
Nov 10, 2025
CyCle1024
force-pushed
the
fix_pt28_extra_ops_fsdp_state
branch
3 times, most recently
from
9a1e068 to
10b6689
Compare
CyCle1024
force-pushed
the
fix_pt28_extra_ops_fsdp_state
branch
from
10b6689 to
fd6adfc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fix extra ops in
SequenceContext.__init__being called in_fsdp_state _pre_forwardof pytorch 2.8.Extra ops is introduced in https://github.com/pytorch/pytorch/blob/v2.8.0/torch/distributed/fsdp/_fully_shard/_fsdp_state.py#L249, and further in https://github.com/pytorch/pytorch/blob/v2.8.0/torch/distributed/utils.py#L227,
dataclasses.replace(x)would callx.__init__in the case ofxis a dataclass object.Note: this is a workaround for SequenceContext, force it not being a dataclass. We should find a way to clean up dataclass object's
__init__method if it's passed into FSDPModule forward method as a param.