[c10d] Barrier uses stream sync instead of device sync #117804

kwen2501 · 2024-01-18T23:45:43Z

Resubmitting #96785

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225

pytorch-bot · 2024-01-18T23:45:46Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/117804

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit 3cc481e with merge base c317bf2 ():

NEW FAILURES - The following jobs have failed:

pull / linux-focal-py3.11-clang10 / test (dynamo, 2, 3, linux.2xlarge) (gh)
test_weak.py::WeakTest::test_make_weak_keyed_dict_from_weak_keyed_dict
pull / linux-focal-py3.8-clang10 / test (dynamo, 2, 3, linux.2xlarge) (gh)
test_weak.py::WeakTest::test_make_weak_keyed_dict_from_weak_keyed_dict

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-focal-py3-clang9-android-ndk-r21e-gradle-custom-build-single-full-jit / build-and-test (default, 1, 1, linux.2xlarge) (gh)
Process completed with exit code 127.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

kwen2501 · 2024-01-19T02:56:03Z

@pytorchbot merge

pytorchmergebot · 2024-01-19T02:58:05Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-01-19T03:37:55Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

kwen2501 · 2024-01-19T14:51:51Z

@pytorchbot merge

pytorchmergebot · 2024-01-19T14:55:49Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-01-19T14:56:26Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

pull / linux-focal-py3-clang9-android-ndk-r21e-gradle-custom-build-single / build-and-test (default, 1, 1, linux.2xlarge)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

awgu · 2024-01-19T17:08:34Z

I think this change might be BC breaking. Should we call this out?

I have previously used dist.barrier() for timing code where I relied on it doing a device sync, not just a stream sync.

kwen2501 · 2024-01-22T20:11:29Z

@awgu Good idea. I added a note to the documentation. Thanks!

kwen2501 · 2024-01-22T20:12:47Z

@pytorchbot merge -f "Previous CI trunk failure does not seem related"

pytorchmergebot · 2024-01-22T20:14:36Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

clee2000 · 2024-01-22T21:50:40Z

@pytorchbot revert -m "sorry the docs test failure is real, I think it wants the lines after the .. note to be indented https://hud.pytorch.org/pytorch/pytorch/commit/0f6bbb1c070c3a9713893659377e20e147c125f6 https://github.com/pytorch/pytorch/actions/runs/7616827874/job/20745016788. There are also some libtorch builds failing. This looks to be some combination of ignoredsignal (docs build) and nosignal (Dr CI classification for docs_test was wrong, pending libtorch jobs)" -c nosignal

pytorchmergebot · 2024-01-22T21:53:57Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

…)" This reverts commit 0f6bbb1. Reverted #117804 on behalf of https://github.com/clee2000 due to sorry the docs test failure is real, I think it wants the lines after the .. note to be indented https://github.com/pytorch/pytorch/actions/runs/7616827874/job/20745016788. Marking as nosignal due to bad Dr. CI categorization ([comment](#117804 (comment)))

pytorchmergebot · 2024-01-22T21:54:08Z

@kwen2501 your PR has been successfully reverted.

kwen2501 · 2024-01-24T18:40:12Z

@pytorchbot merge -f "Failures were from main and unrelated"

pytorchmergebot · 2024-01-24T18:41:58Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2024-01-26T00:50:32Z

@kwen2501 why this PR updated PocketFFT submodule?

Accidentally downgraded by force merge of #117804

Accidentally downgraded by force merge of #117804 Pull Request resolved: #118348 Approved by: https://github.com/kit1980

Resubmitting #96785 Pull Request resolved: #117804 Approved by: https://github.com/wconstab

…)" This reverts commit 0f6bbb1. Reverted #117804 on behalf of https://github.com/clee2000 due to sorry the docs test failure is real, I think it wants the lines after the .. note to be indented https://github.com/pytorch/pytorch/actions/runs/7616827874/job/20745016788. Marking as nosignal due to bad Dr. CI categorization ([comment](#117804 (comment)))

Resubmitting #96785 Pull Request resolved: #117804 Approved by: https://github.com/wconstab

pytorch-bot bot added the release notes: distributed (c10d) release notes category label Jan 18, 2024

github-actions bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Jan 18, 2024

wconstab approved these changes Jan 19, 2024

View reviewed changes

kwen2501 marked this pull request as ready for review January 19, 2024 02:55

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 19, 2024

pytorchmergebot added the merging label Jan 19, 2024

pytorchmergebot removed the merging label Jan 19, 2024

kwen2501 and others added 3 commits January 18, 2024 20:40

Make barrier sync against current stream instead of whole device

9e8c30e

Add comment

3640312

Silent test_ddp_hook_pickling_powerSGD

27d5884

kwen2501 force-pushed the remove_dev_sync_from_barrier branch from 81b2693 to 27d5884 Compare January 19, 2024 04:41

pytorchmergebot added the merging label Jan 19, 2024

pytorchmergebot removed the merging label Jan 19, 2024

Update doc

6f7975b

pytorchmergebot added the merging label Jan 22, 2024

pytorchmergebot closed this in 0f6bbb1 Jan 22, 2024

pytorchmergebot added Merged and removed merging labels Jan 22, 2024

pytorchmergebot added the Reverted label Jan 22, 2024

pytorchmergebot reopened this Jan 22, 2024

Fix doc build issue

3cc481e

pytorchmergebot added the merging label Jan 24, 2024

pytorchmergebot closed this in 1e185c7 Jan 24, 2024

pytorchmergebot removed the merging label Jan 24, 2024

malfet added a commit that referenced this pull request Jan 26, 2024

Update PocketFFT submodule

0c44acf

Accidentally downgraded by force merge of #117804

malfet mentioned this pull request Jan 26, 2024

Update PocketFFT submodule #118348

Closed

pytorchmergebot pushed a commit that referenced this pull request Jan 26, 2024

Update PocketFFT submodule (#118348)

385d8b3

Accidentally downgraded by force merge of #117804 Pull Request resolved: #118348 Approved by: https://github.com/kit1980

wconstab pushed a commit that referenced this pull request Jan 26, 2024

[c10d] Barrier uses stream sync instead of device sync (#117804)

5bdaa07

Resubmitting #96785 Pull Request resolved: #117804 Approved by: https://github.com/wconstab

wconstab pushed a commit that referenced this pull request Jan 26, 2024

[c10d] Barrier uses stream sync instead of device sync (#117804)

2a29b07

Resubmitting #96785 Pull Request resolved: #117804 Approved by: https://github.com/wconstab

github-actions bot deleted the remove_dev_sync_from_barrier branch February 25, 2024 01:51

[c10d] Barrier uses stream sync instead of device sync #117804

[c10d] Barrier uses stream sync instead of device sync #117804

Uh oh!

Conversation

kwen2501 commented Jan 18, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jan 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/117804

❌ 2 New Failures, 1 Unrelated Failure

Uh oh!

kwen2501 commented Jan 19, 2024

Uh oh!

pytorchmergebot commented Jan 19, 2024

Merge started

Uh oh!

pytorchmergebot commented Jan 19, 2024

Merge failed

Uh oh!

kwen2501 commented Jan 19, 2024

Uh oh!

pytorchmergebot commented Jan 19, 2024

Merge started

Uh oh!

pytorchmergebot commented Jan 19, 2024

Merge failed

Uh oh!

awgu commented Jan 19, 2024

Uh oh!

kwen2501 commented Jan 22, 2024

Uh oh!

kwen2501 commented Jan 22, 2024

Uh oh!

pytorchmergebot commented Jan 22, 2024

Merge started

Uh oh!

clee2000 commented Jan 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorchmergebot commented Jan 22, 2024

Uh oh!

pytorchmergebot commented Jan 22, 2024

Uh oh!

kwen2501 commented Jan 24, 2024

Uh oh!

pytorchmergebot commented Jan 24, 2024

Merge started

Uh oh!

malfet commented Jan 26, 2024

Uh oh!

Uh oh!

kwen2501 commented Jan 18, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jan 18, 2024 •

edited

Loading

clee2000 commented Jan 22, 2024 •

edited

Loading