[NCCL] Add Error log when ProcessGroupNCCL takes down process upon timeout/error #44988
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
timeout/error The new NCCL async error handling feature throws an exception from the workCleanup Thread if one of the NCCL operations encounters an error or times out. This PR adds an error log to make it more clear to the user why the training process crashed. Differential Revision: [D23794801](https://our.internmc.facebook.com/intern/diff/D23794801/) [ghstack-poisoned]
osalpekar
requested review from
mrshenli,
pietern,
pritamdamania87,
rohan-varma and
zhaojuanmao
as code owners
osalpekar
added a commit
that referenced
this pull request
Sep 18, 2020
timeout/error The new NCCL async error handling feature throws an exception from the workCleanup Thread if one of the NCCL operations encounters an error or times out. This PR adds an error log to make it more clear to the user why the training process crashed. Differential Revision: [D23794801](https://our.internmc.facebook.com/intern/diff/D23794801/) ghstack-source-id: 112419640 Pull Request resolved: #44988
💊 CI failures summary and remediationsAs of commit 4217efa (more details on the Dr. CI page):
ci.pytorch.org: 1 failedThis comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group. This comment has been revised 7 times. |
Comment on lines
+313
to
+318
| auto exceptionMsg = c10::str( | ||
| "Some NCCL operations have failed or timed out. Due to the ", | ||
| "asynchronous nature of CUDA kernels, subsequent GPU operations ", | ||
| "might run on corrupted/incomplete data. To avoid this inconsistency, ", | ||
| "we are taking the entire process down."); | ||
| LOG(ERROR) << exceptionMsg; |
There was a problem hiding this comment.
Can we add a unit test for this case using multiprocessing?
pritamdamania87
suggested changes
Sep 21, 2020
…ess upon timeout/error" timeout/error** timeout/error The new NCCL async error handling feature throws an exception from the workCleanup Thread if one of the NCCL operations encounters an error or times out. This PR adds an error log to make it more clear to the user why the training process crashed. Differential Revision: [D23794801](https://our.internmc.facebook.com/intern/diff/D23794801/) [ghstack-poisoned]
osalpekar
added a commit
that referenced
this pull request
Oct 8, 2020
timeout/error Pull Request resolved: #44988 The new NCCL async error handling feature throws an exception from the workCleanup Thread if one of the NCCL operations encounters an error or times out. This PR adds an error log to make it more clear to the user why the training process crashed. ghstack-source-id: 113876146 Differential Revision: [D23794801](https://our.internmc.facebook.com/intern/diff/D23794801/)
pritamdamania87
approved these changes
Oct 9, 2020
…ess upon timeout/error" timeout/error** timeout/error** timeout/error The new NCCL async error handling feature throws an exception from the workCleanup Thread if one of the NCCL operations encounters an error or times out. This PR adds an error log to make it more clear to the user why the training process crashed. Differential Revision: [D23794801](https://our.internmc.facebook.com/intern/diff/D23794801/) [ghstack-poisoned]
osalpekar
added a commit
that referenced
this pull request
Oct 9, 2020
timeout/error Pull Request resolved: #44988 The new NCCL async error handling feature throws an exception from the workCleanup Thread if one of the NCCL operations encounters an error or times out. This PR adds an error log to make it more clear to the user why the training process crashed. ghstack-source-id: 114002493 Differential Revision: [D23794801](https://our.internmc.facebook.com/intern/diff/D23794801/)
|
This pull request has been merged in 172036a. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
timeout/error
timeout/error**
timeout/error**
timeout/error
The new NCCL async error handling feature throws an exception from the
workCleanup Thread if one of the NCCL operations encounters an error or times
out. This PR adds an error log to make it more clear to the user why the
training process crashed.
Differential Revision: D23794801