[NCCL] enable p2p tests #47797
Closed
[NCCL] enable p2p tests #47797
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NCCL p2p tests had hang issues before, the reason is that there were some unexpected context switches. For example, process 1 which is supposed to only use GPU1 could use GPU0 as a result of missing explicitly setting device. Differential Revision: [D24863808](https://our.internmc.facebook.com/intern/diff/D24863808/) [ghstack-poisoned]
mingzhe09088
requested review from
apaszke,
mrshenli,
pietern,
pritamdamania87,
rohan-varma and
zhaojuanmao
as code owners
facebook-github-bot
added
cla signed
oncall: distributed
mingzhe09088
pushed a commit
that referenced
this pull request
Nov 11, 2020
NCCL p2p tests had hang issues before, the reason is that there were some unexpected context switches. For example, process 1 which is supposed to only use GPU1 could use GPU0 as a result of missing explicitly setting device. Differential Revision: [D24863808](https://our.internmc.facebook.com/intern/diff/D24863808/) ghstack-source-id: 116461969 Pull Request resolved: #47797
💊 CI failures summary and remediationsAs of commit 8b00330 (more details on the Dr. CI page):
ci.pytorch.org: 1 failedThis comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group. This comment has been revised 1 time. |
Codecov Report
@@ Coverage Diff @@
## gh/mingzhe09088/15/base #47797 +/- ##
========================================================
Coverage 64.58% 64.58%
========================================================
Files 1680 1680
Lines 166945 166946 +1
========================================================
+ Hits 107815 107822 +7
+ Misses 59130 59124 -6 |
|
This pull request has been merged in 66f9b1d. |
tugsbayasgalan
pushed a commit
to tugsbayasgalan/pytorch
that referenced
this pull request
Nov 16, 2020
Summary: Pull Request resolved: pytorch#47797 NCCL p2p tests had hang issues before, the reason is that there were some unexpected context switches. For example, process 1 which is supposed to only use GPU1 could use GPU0 as a result of missing explicitly setting device. ghstack-source-id: 116461969 Test Plan: waitforsandcastle Reviewed By: jiayisuse Differential Revision: D24863808 fbshipit-source-id: 92bd3a4874be8334210c7c8ee6363648893c963e
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
NCCL p2p tests had hang issues before, the reason is that there were some unexpected context switches. For example, process 1 which is supposed to only use GPU1 could use GPU0 as a result of missing explicitly setting device.
Differential Revision: D24863808