Diasble distributed per-layer clipping with hooks grad sample mode #747
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rch#720) Summary: register_backward_hook is deprecated and may lead to errors in gradient calculation. We switch to the supported register_full_backward_hook. Differential Revision: D68562558
Summary: We disable support for distributed per-layer clipping with "hooks" grad sample mode, since it raises an error when using `register_full_backward_hook`. Distributed per-layer clipping with "ew" grad sample mode can still be used. The issue arises because DistributedPerLayerOptimizer uses per-parameter hooks on top of the per-module hooks. During the backward pass, the per-parameter hooks fire before the per-module hooks. Per-sample gradients are computed when the per-module hooks fire, and an error occurs when the per-parameter hooks try to access the per-sample gradients before they are computed. Forcing the order in which hooks are called is not possible with PyTorch. In the future, we may consider an approach that does not use per-parameter hooks. Given the limited use of per-layer clipping, we disable the faulty case of distributed data parallel with "hooks". Differential Revision: D71706681
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D71706681 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
We disable support for distributed per-layer clipping with "hooks" grad sample mode, since it raises an error when using
register_full_backward_hook. Distributed per-layer clipping with "ew" grad sample mode can still be used.The issue arises because DistributedPerLayerOptimizer uses per-parameter hooks on top of the per-module hooks. During the backward pass, the per-parameter hooks fire before the per-module hooks. Per-sample gradients are computed when the per-module hooks fire, and an error occurs when the per-parameter hooks try to access the per-sample gradients before they are computed. Forcing the order in which hooks are called is not possible with PyTorch.
In the future, we may consider an approach that does not use per-parameter hooks. Given the limited use of per-layer clipping, we disable the faulty case of distributed data parallel with "hooks".
Differential Revision: D71706681