update ShardedEmbeddingBagCollection to be use registered EBCs with shardedTensors as registered modules #758
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
2 similar comments
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
…hardedTensors as registered modules (#88026) Summary: X-link: pytorch/pytorch#88026 Pull Request resolved: #758 update ShardedEmbeddingBagCollection to be composable according to https://docs.google.com/document/d/1TBJSd5zgEg6cRcXv3Okuj7bBkqQwGS2IPh4TLWNNzFI/edit this works with DMP named_parameters() behavior changes -> use include_fused as temporary flag to gate this behavior note that due to ShardedTensor not supporting grads directly, this won't work for Dense compute kernels when non data parallel. This is not used today, and will add a TODO but is low pri Differential Revision: D40458625 fbshipit-source-id: 9135216ac67c828d8532d5c251cd6b8d170c058b
|
This pull request was exported from Phabricator. Differential Revision: D40458625 |
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
Nov 17, 2022
…e registered EBCs with shardedTensors as registered modules (#758) (#88026) Summary: X-link: meta-pytorch/torchrec#758 This PR fixes a bug in FSDP/DDP, where ShardedTensors are not supported even if passed in as params to ignore. this is important for composability because TorchRec named_parameters() will return FQN of shardedTensors (as defined in goals) It defines device of ShardedTensor to be None when local_tensor() does not exist on rank update ShardedEmbeddingBagCollection to be composable according to https://docs.google.com/document/d/1TBJSd5zgEg6cRcXv3Okuj7bBkqQwGS2IPh4TLWNNzFI/edit Differential Revision: D40458625 Pull Request resolved: #88026 Approved by: https://github.com/wanchaol, https://github.com/rohan-varma
kulinseth
pushed a commit
to kulinseth/pytorch
that referenced
this pull request
Dec 10, 2022
…e registered EBCs with shardedTensors as registered modules (pytorch#758) (pytorch#88026) Summary: X-link: meta-pytorch/torchrec#758 This PR fixes a bug in FSDP/DDP, where ShardedTensors are not supported even if passed in as params to ignore. this is important for composability because TorchRec named_parameters() will return FQN of shardedTensors (as defined in goals) It defines device of ShardedTensor to be None when local_tensor() does not exist on rank update ShardedEmbeddingBagCollection to be composable according to https://docs.google.com/document/d/1TBJSd5zgEg6cRcXv3Okuj7bBkqQwGS2IPh4TLWNNzFI/edit Differential Revision: D40458625 Pull Request resolved: pytorch#88026 Approved by: https://github.com/wanchaol, https://github.com/rohan-varma
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
update ShardedEmbeddingBagCollection to be composable according to https://docs.google.com/document/d/1TBJSd5zgEg6cRcXv3Okuj7bBkqQwGS2IPh4TLWNNzFI/edit
this works with DMP
named_parameters() behavior changes -> use include_fused as temporary flag to gate this behavior
note that due to ShardedTensor not supporting grads directly, this won't work for Dense compute kernels when non data parallel. This is not used today, and will add a TODO but is low pri
Differential Revision: D40458625