Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix distributed store to use add for the counter of DL shared seed #80348

Closed
wants to merge 2 commits into from

Conversation

ejguan
Copy link
Contributor

@ejguan ejguan commented Jun 27, 2022

In order to get the result of _shared_seed_recv_cnt properly, switch from store.get to store.add(key, 0).

See the comment from distributed team for the reason:

# Use 'add' instead of 'get' since for some store implementations 'add'
# doesn't work well with 'get'. Ideally the store implementations should
# be fixed, but for backward compatiblity reasons it is risky to change
# the store implementations. Once, we completely migrate away from these
# legacy stores, we can use 'get' here instead.

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Jun 27, 2022

🔗 Helpful links

✅ No Failures (0 Pending)

As of commit 0de7982 (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚


This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

torch/utils/data/dataloader.py Outdated Show resolved Hide resolved
torch/utils/data/dataloader.py Outdated Show resolved Hide resolved
torch/utils/data/dataloader.py Outdated Show resolved Hide resolved
@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@ejguan
Copy link
Contributor Author

ejguan commented Jun 27, 2022

@pytorchbot merge -g

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a merge job. Check the current status here

@github-actions
Copy link

Hey @ejguan.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

@ejguan ejguan added release notes: dataloader release notes category topic: bug fixes topic category labels Jun 28, 2022
facebook-github-bot pushed a commit that referenced this pull request Jun 30, 2022
…80348) (#80348)

Summary:
In order to get the result of `_shared_seed_recv_cnt` properly, switch from `store.get` to `store.add(key, 0)`.

See the comment from distributed team for the reason:
https://github.com/pytorch/pytorch/blob/590d3e5774110e4657dcaa6acdb387ef69e41b47/torch/distributed/distributed_c10d.py#L242-L246

Pull Request resolved: #80348
Approved by: https://github.com/VitalyFedyunin, https://github.com/NivekT

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/3ec9d34f21edfb330076a2e57dd9b30649070e80

Reviewed By: NivekT

Differential Revision: D37458370

Pulled By: ejguan

fbshipit-source-id: 386457bef43dbb47e3c5b8bb4524d456b5f4343a
@ejguan ejguan added this to the 1.12.1 milestone Jul 8, 2022
atalman pushed a commit to atalman/pytorch that referenced this pull request Jul 21, 2022
…ytorch#80348) (pytorch#80348)

Summary:
In order to get the result of `_shared_seed_recv_cnt` properly, switch from `store.get` to `store.add(key, 0)`.

See the comment from distributed team for the reason:
https://github.com/pytorch/pytorch/blob/590d3e5774110e4657dcaa6acdb387ef69e41b47/torch/distributed/distributed_c10d.py#L242-L246

Pull Request resolved: pytorch#80348
Approved by: https://github.com/VitalyFedyunin, https://github.com/NivekT

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/3ec9d34f21edfb330076a2e57dd9b30649070e80

Reviewed By: NivekT

Differential Revision: D37458370

Pulled By: ejguan

fbshipit-source-id: 386457bef43dbb47e3c5b8bb4524d456b5f4343a
atalman added a commit that referenced this pull request Jul 21, 2022
…80348) (#80348) (#81860)

Summary:
In order to get the result of `_shared_seed_recv_cnt` properly, switch from `store.get` to `store.add(key, 0)`.

See the comment from distributed team for the reason:
https://github.com/pytorch/pytorch/blob/590d3e5774110e4657dcaa6acdb387ef69e41b47/torch/distributed/distributed_c10d.py#L242-L246

Pull Request resolved: #80348
Approved by: https://github.com/VitalyFedyunin, https://github.com/NivekT

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/3ec9d34f21edfb330076a2e57dd9b30649070e80

Reviewed By: NivekT

Differential Revision: D37458370

Pulled By: ejguan

fbshipit-source-id: 386457bef43dbb47e3c5b8bb4524d456b5f4343a

Co-authored-by: erjia (Meta Employee) <erjia@fb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants