[Gradient Compression] Add an index field to GradBucket for PowerSGD #48757
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/) [ghstack-poisoned]
wayi1
requested review from
mingzhe09088,
mrshenli,
pritamdamania87,
rohan-varma and
zhaojuanmao
as code owners
facebook-github-bot
added
cla signed
oncall: distributed
wayi1
pushed a commit
that referenced
this pull request
Dec 3, 2020
Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/) ghstack-source-id: 117719173 Pull Request resolved: #48757
💊 CI failures summary and remediationsAs of commit c05592d (more details on the Dr. CI page): ✅ None of the CI failures appear to be your fault 💚
🚧 1 ongoing upstream failure:These were probably caused by upstream breakages that are not fixed yet:
This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group. This comment has been revised 26 times. |
…r PowerSGD" Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/) [ghstack-poisoned]
wayi1
pushed a commit
that referenced
this pull request
Dec 3, 2020
Pull Request resolved: #48757 Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 ghstack-source-id: 117748471 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/)
…r PowerSGD" Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/) [ghstack-poisoned]
wayi1
pushed a commit
that referenced
this pull request
Dec 4, 2020
Pull Request resolved: #48757 Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. The replacement will be done in a separate diff, as the definition of this new method somehow couldn't be recognized in the OSS version. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 ghstack-source-id: 117911015 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/)
rohan-varma
approved these changes
Dec 4, 2020
…r PowerSGD" Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/) [ghstack-poisoned]
wayi1
pushed a commit
that referenced
this pull request
Dec 4, 2020
Pull Request resolved: #48757 Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. The replacement will be done in a separate diff, as the definition of this new method somehow couldn't be recognized in the OSS version. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 ghstack-source-id: 117916685 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/)
…r PowerSGD" Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/) [ghstack-poisoned]
…r PowerSGD" Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/) [ghstack-poisoned]
wayi1
pushed a commit
that referenced
this pull request
Dec 5, 2020
Pull Request resolved: #48757 Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor. The replacement will be done in a separate diff, as the definition of this new method somehow couldn't be recognized in the OSS version. Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202 ghstack-source-id: 117939208 Differential Revision: [D25288496](https://our.internmc.facebook.com/intern/diff/D25288496/)
|
This pull request has been merged in 7439bc4. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
Add an index field to GradBucekt, so error_dict is keyed by this index instead of the hashcode of input tensor.
Howevever, sometimes the buckets can be rebuilt in the forward pass. In this case, the shape of the bucket with the same index will not be consistent with the one in the previous iteration, and hence the error tensor will be re--initialized as a zero tensor of the new shape.
Original PR issue: Investigate Applying PowerSGD to Communication Hook for Gradient Compression #47202
Differential Revision: D25288496