Skip to content

[4/n] Thread PG: add reduce_scatter to threaded pg #89442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[4/n] Thread PG: add reduce_scatter to threaded pg #89442

wants to merge 2 commits into from

Conversation

@wanchaol
Copy link
Collaborator

@wanchaol wanchaol commented Nov 21, 2022 •
edited
Loading

@pytorch-bot
Copy link

pytorch-bot bot commented Nov 21, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/89442

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit 997b32a:

The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@wanchaol wanchaol requested a review from kwen2501 as a code owner November 21, 2022 17:58
This was referenced Nov 21, 2022
Closed
Closed
Closed
Closed
@wanchaol wanchaol added the topic: not user facing topic category label Nov 21, 2022
yhcharles
yhcharles approved these changes Nov 21, 2022
View reviewed changes
Copy link
Contributor

@yhcharles yhcharles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pytorchmergebot
Copy link
Collaborator

Rebased gh/wanchaol/217/orig onto refs/remotes/origin/viable/strict because #89443 was rebased, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/89442)

fduwjj
fduwjj approved these changes Nov 21, 2022
View reviewed changes
Copy link
Contributor

@fduwjj fduwjj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just have a n00b question.

torch/testing/_internal/distributed/multi_threaded_pg.py
start_reduction[i] = True
else:
with torch.no_grad():
dest_tensor_on_rank_i[0].add_(to_scatter[i])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, does this mean the overall aggregation logic is executed in one single thread?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, it's not actually running in multi threads, only PG is mocked with multiple threads to assume it's a world.

@wanchaol
Copy link
Collaborator Author

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 21, 2022
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 2 additional jobs have failed, first few of them are: trunk ,trunk / linux-bionic-cuda11.7-py3.10-gcc7 / test (jit_legacy, 1, 1, linux.4xlarge.nvidia.gpu)

Details for Dev Infra team Raised by workflow job

@wanchaol
Copy link
Collaborator Author

@pytorchbot merge -f "failures not related"

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

kulinseth pushed a commit to kulinseth/pytorch that referenced this pull request Dec 10, 2022
@wanchaol @kulinseth
[4/n] Thread PG: add reduce_scatter to threaded pg (pytorch#89442)
ebdb23d 
Pull Request resolved: pytorch#89442
Approved by: https://github.com/yhcharles, https://github.com/fduwjj
@facebook-github-bot facebook-github-bot deleted the gh/wanchaol/216/head branch June 8, 2023 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fduwjj fduwjj fduwjj approved these changes

@mrshenli mrshenli Awaiting requested review from mrshenli

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@rohan-varma rohan-varma Awaiting requested review from rohan-varma

@H-Huang H-Huang Awaiting requested review from H-Huang

@awgu awgu Awaiting requested review from awgu

@kwen2501 kwen2501 Awaiting requested review from kwen2501

+1 more reviewer

@yhcharles yhcharles yhcharles approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wanchaol @pytorchmergebot @yhcharles @fduwjj