Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Blocking Wait if both Blocking Wait and Async Error Handling Are Set #47926

Closed
wants to merge 1 commit into from
Closed

Use Blocking Wait if both Blocking Wait and Async Error Handling Are Set #47926

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 8 additions & 0 deletions torch/lib/c10d/ProcessGroupNCCL.cpp
View file
Open in desktop
Expand Up @@ -453,6 +453,14 @@ ProcessGroupNCCL::ProcessGroupNCCL(
blockingWait_ = parseEnvVarFlag(NCCL_BLOCKING_WAIT);
asyncErrorHandling_ = parseEnvVarFlag(NCCL_ASYNC_ERROR_HANDLING);

if (blockingWait_ && asyncErrorHandling_) {
LOG(INFO) << "[Rank " << rank_
<< "] NCCL_BLOCKING_WAIT and NCCL_ASYNC_ERROR_HANDLING "
<< "should not both be enabled. "
<< "Only NCCL_BLOCKING_WAIT is being used in this process.";
asyncErrorHandling_ = false;
Comment on lines +456 to +461
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@osalpekar Sorry a bit late on this, but I think we should instead throw an error if both NCCL_BLOCKING_WAIT and NCCL_ASYNC_ERROR_HANDLING is set and ask user to set only one. It is usually confusing for users if we have such behavior where we unset one option. User's might not look closely at the logs and would feel that there might be a bug.

}

#ifdef ENABLE_NCCL_ERROR_CHECKING
ncclCommWatchdogThread_ =
std::thread(&ProcessGroupNCCL::ncclCommWatchdog, this);
Expand Down