-
Notifications
You must be signed in to change notification settings - Fork 25.6k
[Distributed/Profiler] Fix input/output dimension overflow #134360
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/134360
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 8a038dc with merge base 74a9001 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This pull request was exported from Phabricator. Differential Revision: D61728747 |
…34360) Summary: Pull Request resolved: pytorch#134360 When using ParamCommsDebugInfo, the input elements and output elements are stored in `int` instead of `int64_t` Test Plan: Run HTA with new outputted values and make sure overflow does not occur Reviewed By: fengxizhou Differential Revision: D61728747
This pull request was exported from Phabricator. Differential Revision: D61728747 |
10ec3ef
to
8a038dc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR resolves the outNelems overflow issue for collective gather operator when group_size is large. Changing the data type from int32_t to int64_t can solve the problem and will not affect other component. In my opinion, it's safe to land.
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
This PR needs to be approved by an authorized maintainer before merge. |
@pytorchbot merge -f 'landed internally' |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Summary: When using ParamCommsDebugInfo, the input elements and output elements are stored in
int
instead ofint64_t
Test Plan: Run HTA with new outputted values and make sure overflow does not occur
Reviewed By: fengxizhou
Differential Revision: D61728747
cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o