Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1.14-rc1 cherry-pick request: NCCL broadcast bug #29231

Merged
merged 2 commits into from May 31, 2019
Merged

1.14-rc1 cherry-pick request: NCCL broadcast bug #29231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
74376de
Fixed a nccl broadcast bug
kaixih May 28, 2019
85a9202
Remove the unnecessary const var
kaixih May 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion tensorflow/core/kernels/nccl_ops.cc
View file
Open in desktop
Expand Up @@ -248,7 +248,7 @@ class NcclBroadcastRecvKernel : public NcclAsyncOpBase {
compute_stream->parent(), compute_stream, gpu_info->event_mgr,
gpu_info->gpu_id, /*input=*/nullptr, output, /*global_rank=*/-1,
std::move(actual_done));
NcclManager::instance()->AddBroadcastSend(
NcclManager::instance()->AddBroadcastRecv(
std::move(participant), {GetCollectiveKey(c),
/*num_local_devices=*/num_devices(),
/*num_global_devices=*/num_devices(),
Expand Down
7 changes: 6 additions & 1 deletion tensorflow/core/nccl/nccl_manager.cc
View file
Open in desktop
Expand Up @@ -410,7 +410,12 @@ void NcclManager::AddParticipant(std::unique_ptr<Participant> participant,
CollectiveType collective_type,
ncclRedOp_t reduction_op) {
Collective* to_run = nullptr;
const DataType data_type = participant->input->dtype();
DataType data_type;
if (participant->input != nullptr) {
data_type = participant->input->dtype();
} else {
data_type = participant->output->dtype();
}
{
mutex_lock l(mu_);
auto collective_it = collectives_.find(context.collective_key);
Expand Down