[Gradient Compression] Refactor tensor grouping in PowerSGD #52981
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in #52541 (comment) Differential Revision: [D26713689](https://our.internmc.facebook.com/intern/diff/D26713689/) [ghstack-poisoned]
wayi1
requested review from
mingzhe09088,
mrshenli,
pritamdamania87,
rohan-varma and
zhaojuanmao
as code owners
facebook-github-bot
added
cla signed
oncall: distributed
💊 CI failures summary and remediationsAs of commit f997deb (more details on the Dr. CI page):
❄️ 1 failure tentatively classified as flakybut reruns have not yet been triggered to confirm:
|
|
@tvogels Can you take a look? Thanks! |
No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in #52541 (comment) Differential Revision: [D26713689](https://our.internmc.facebook.com/intern/diff/D26713689/) [ghstack-poisoned]
wayi1
pushed a commit
that referenced
this pull request
Feb 28, 2021
Pull Request resolved: #52981 No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in #52541 (comment) ghstack-source-id: 122703391 Differential Revision: [D26713689](https://our.internmc.facebook.com/intern/diff/D26713689/)
|
Looks good to me. |
No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in #52541 (comment) Differential Revision: [D26713689](https://our.internmc.facebook.com/intern/diff/D26713689/) [ghstack-poisoned]
No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in #52541 (comment) Differential Revision: [D26713689](https://our.internmc.facebook.com/intern/diff/D26713689/) [ghstack-poisoned]
wayi1
pushed a commit
that referenced
this pull request
Feb 28, 2021
Pull Request resolved: #52981 No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in #52541 (comment) ghstack-source-id: 122712478 Differential Revision: [D26713689](https://our.internmc.facebook.com/intern/diff/D26713689/)
No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in #52541 (comment) Differential Revision: [D26713689](https://our.internmc.facebook.com/intern/diff/D26713689/) [ghstack-poisoned]
rohan-varma
approved these changes
Mar 3, 2021
|
This pull request has been merged in b59075e. |
aocsa
pushed a commit
to Quansight/pytorch
that referenced
this pull request
Mar 15, 2021
…52981) Summary: Pull Request resolved: pytorch#52981 No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in pytorch#52541 (comment) ghstack-source-id: 122997032 Test Plan: waitforbuildbot Already LGTMed by PowerSGD paper author. Ads1x (completed): https://www.internalfb.com/intern/tupperware/details/job/?handle=priv3_global%2Fmast_hpc%2Ftsm_hpc-wayi_ads_10x_POWER_SGD_gpu8_2021-02-28_15-29.trainer&tatwTabs=tasks&task_id=0&task_tab=TASK_LOGS Detectron2: 1) Before refactoring: f254353864 Accuracy: 39.972 Overall training speed: 67498 iterations in 6:15:42 (0.3340 s / it) 2) After refactoring: f254353380 Accuracy: 39.944 Overall training speed: 67498 iterations in 6:09:41 (0.3286 s / it) Reviewed By: rohan-varma Differential Revision: D26713689 fbshipit-source-id: 12cfcb65feaa2a2d94e3c7793073031f13828305
xsacha
pushed a commit
to xsacha/pytorch
that referenced
this pull request
Mar 31, 2021
…52981) Summary: Pull Request resolved: pytorch#52981 No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to `_should_compress` function. Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in pytorch#52541 (comment) ghstack-source-id: 122997032 Test Plan: waitforbuildbot Already LGTMed by PowerSGD paper author. Ads1x (completed): https://www.internalfb.com/intern/tupperware/details/job/?handle=priv3_global%2Fmast_hpc%2Ftsm_hpc-wayi_ads_10x_POWER_SGD_gpu8_2021-02-28_15-29.trainer&tatwTabs=tasks&task_id=0&task_tab=TASK_LOGS Detectron2: 1) Before refactoring: f254353864 Accuracy: 39.972 Overall training speed: 67498 iterations in 6:15:42 (0.3340 s / it) 2) After refactoring: f254353380 Accuracy: 39.944 Overall training speed: 67498 iterations in 6:09:41 (0.3286 s / it) Reviewed By: rohan-varma Differential Revision: D26713689 fbshipit-source-id: 12cfcb65feaa2a2d94e3c7793073031f13828305
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
No need to create a hard boundary between rank-1 tensors and high-rank tensors, since some high-rank tensors will not be compressed if the compression cannot save enough bandwidth, according to
_should_compressfunction.Therefore, refactor and simplify the tensor grouping logic, which addresses the comment in #52541 (comment)
Differential Revision: D26713689