Skip to content

Commit

Permalink
Fix warning if backend registers timer
Browse files Browse the repository at this point in the history
currently logger timer is registered default for
cpu/cuda. for other backends, it may or may not
registers this timer. It reports warning for other
backends and return which is not expected.
The above may fail, if the backends has have registered
this timer. For example, HPU(habana) backend registers
this timer. so, in this case it reports a warning and
return which is incorrect.

Other case is where lazy backend timer is never registered.
so, this returns a warning, and this is the reason the check
was added, but it fails for other cases.

Add a generic check if the timer is registered, then
don’t report warning.

Signed-off-by: Jeeja <jeejakp@habana.ai>
  • Loading branch information
jeejakp12 committed Feb 21, 2023
1 parent 062380d commit 83fb0de
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion torch/csrc/distributed/c10d/logger.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,9 @@ void Logger::set_runtime_stats_and_log() {
"Cuda time stats are not collected for multi-device modules.");
return;
}
if (!reducer_->params_[0].is_cuda() && !reducer_->params_[0].is_cpu()) {

if (!reducer_->timer_ &&
(!reducer_->params_[0].is_cuda() && !reducer_->params_[0].is_cpu())) {
TORCH_WARN_ONCE(
"Time stats are currently only collected for CPU and CUDA devices. "
"Please refer to CpuTimer or CudaTimer for how to register timer "
Expand Down

0 comments on commit 83fb0de

Please sign in to comment.