[Elastic] Skip store barrier and store get in host assign #136579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

fduwjj wants to merge 1 commit into pytorch:main from fduwjj:export-D63351662

Contributor

fduwjj commented Sep 24, 2024 •

edited by pytorch-bot bot

Loading

Summary: Sometimes, when the worker and role are the same, users want to skip TCPStore in _assign_worker_ranks and barrier in rendezvous

Test Plan: unit test

Differential Revision: D63351662

cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @wz337 @wconstab @d4l3k @c-p-i-o

pytorch-bot bot added oncall: distributed release notes: distributed (torchelastic) labels

pytorch-bot bot commented Sep 24, 2024 •

edited

Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/136579

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 1 Unrelated Failure

As of commit eba30de with merge base f0a9254 ():

NEW FAILURES - The following jobs have failed:

pull / linux-docs / build-docs-python-false (gh)
Process completed with exit code 2.
pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, lf.linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, lf.linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.11-clang10 / test (default, 2, 4, lf.linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.11-clang10 / test (default, 3, 4, lf.linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.11-clang10 / test (default, 4, 4, lf.linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.11-clang10 / test (dynamo, 2, 3, lf.linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, lf.linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (docs_test, 1, 1, lf.linux.2xlarge) (gh)
##[error]Process completed with exit code 2.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no runner)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Contributor

facebook-github-bot commented Sep 24, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

facebook-github-bot added the fb-exported label

fduwjj requested a review from d4l3k

September 25, 2024 00:39

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

fduwjj force-pushed the export-D63351662 branch from 95c24f7 to 413e204 Compare

September 25, 2024 01:02

fduwjj force-pushed the export-D63351662 branch from 413e204 to 6d9b37f Compare

September 25, 2024 01:11

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

d4l3k approved these changes

View reviewed changes

Member

d4l3k left a comment

LGTM

pytorch-bot bot added the ciflow/trunk label

d4l3k reviewed

View reviewed changes

test/distributed/elastic/agent/server/test/api_test.py Outdated Show resolved Hide resolved

fduwjj force-pushed the export-D63351662 branch from 6d9b37f to 272345d Compare

September 25, 2024 18:01

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

fduwjj force-pushed the export-D63351662 branch from 272345d to 49f89b3 Compare

September 25, 2024 18:09

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

fduwjj force-pushed the export-D63351662 branch from 49f89b3 to e0065ba Compare

September 25, 2024 18:14

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

fduwjj force-pushed the export-D63351662 branch from e0065ba to 3d4518a Compare

September 25, 2024 18:58

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

fduwjj added a commit to fduwjj/pytorch that referenced this pull request


          [Elastic] Skip store barrier and store get in host assign (pytorch#13…

c0e9eb5

…6579)

Summary:
Pull Request resolved: pytorch#136579

Sometimes, when the worker and role are the same, users want to skip TCPStore in `_assign_worker_ranks` and barrier in rendezvous

Test Plan: unit test

Reviewed By: d4l3k

Differential Revision: D63351662

fduwjj force-pushed the export-D63351662 branch from 3d4518a to c0e9eb5 Compare

September 25, 2024 19:04

Contributor

facebook-github-bot commented Sep 26, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

fduwjj force-pushed the export-D63351662 branch from c0e9eb5 to 5c09c58 Compare

September 26, 2024 17:54


          [Elastic] Skip store barrier and store get in host assign (pytorch#13…

eba30de

…6579)

Summary:
Pull Request resolved: pytorch#136579

Sometimes, when the worker and role are the same, users want to skip TCPStore in `_assign_worker_ranks` and barrier in rendezvous

Test Plan: unit test

Reviewed By: d4l3k

Differential Revision: D63351662

fduwjj force-pushed the export-D63351662 branch from 5c09c58 to eba30de Compare

September 26, 2024 18:01

Contributor

facebook-github-bot commented Sep 26, 2024

This pull request was exported from Phabricator. Differential Revision: D63351662

Contributor

facebook-github-bot commented Sep 27, 2024

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot added the merging label

Collaborator

pytorchmergebot commented Sep 27, 2024

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Collaborator

pytorchmergebot commented Sep 27, 2024

Merge failed

Reason: 9 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

pytorchmergebot removed the merging label

fduwjj added a commit that referenced this pull request


          [reland][Elastic] Skip store barrier and store get in host assign

0edbaff

As title this is to reland #136579 as it broke some OSS CI

Differential Revision: [D63542918](https://our.internmc.facebook.com/intern/diff/D63542918/)

[ghstack-poisoned]

fduwjj mentioned this pull request

[reland][Elastic] Skip store barrier and store get in host assign #136865

Closed

Contributor Author

fduwjj commented Sep 27, 2024

reland in #136865

fduwjj closed this

fduwjj added a commit that referenced this pull request


          Update on "[reland][Elastic] Skip store barrier and store get in host…

363dec0

… assign"

As title this is to reland #136579 as it broke some OSS CI

Differential Revision: [D63542918](https://our.internmc.facebook.com/intern/diff/D63542918/)

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o

[ghstack-poisoned]

fduwjj added a commit that referenced this pull request


          [reland][Elastic] Skip store barrier and store get in host assign

23fc88f

Pull Request resolved: #136865

As title this is to reland #136579 as it broke some OSS CI
ghstack-source-id: 245131066

Differential Revision: [D63542918](https://our.internmc.facebook.com/intern/diff/D63542918/)

fduwjj added a commit that referenced this pull request


          Update base for Update on "[reland][Elastic] Skip store barrier and s…

d375483

…tore get in host assign"

As title this is to reland #136579 as it broke some OSS CI

Differential Revision: [D63542918](https://our.internmc.facebook.com/intern/diff/D63542918/)

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o

[ghstack-poisoned]

fduwjj added a commit that referenced this pull request


          Update on "[reland][Elastic] Skip store barrier and store get in host…

8561f56

… assign"

As title this is to reland #136579 as it broke some OSS CI

Differential Revision: [D63542918](https://our.internmc.facebook.com/intern/diff/D63542918/)

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o

[ghstack-poisoned]

pytorchmergebot pushed a commit that referenced this pull request


          [reland][Elastic] Skip store barrier and store get in host assign (#1…

f42e88f

…36865)

As title this is to reland #136579 as it broke some OSS CI

Differential Revision: [D63542918](https://our.internmc.facebook.com/intern/diff/D63542918/)

Pull Request resolved: #136865
Approved by: https://github.com/atalman

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk fb-exported oncall: distributed release notes: distributed (torchelastic)