Skip to content

Commit

Permalink
warn user once for possible unnecessary find_unused_params (#50133)
Browse files Browse the repository at this point in the history
Summary:
Pull Request resolved: #50133

`find_unused_parameters=True` is only needed when the model has unused parameters that are not known at model definition time or differ due to control flow.

Unfortunately, many DDP users pass this flag in as `True` even when they do not need it, sometimes as a precaution to mitigate possible errors that may be raised (such as the error we raise with not using all outputs).While this is a larger issue to be fixed in DDP, it would also be useful to warn once if we did not detect unused parameters.

The downside of this is that in the case of flow control models where the first iteration doesn't have unused params but the rest do, this would be a false warning. However, I think the warning's value exceeds this downside.
ghstack-source-id: 119707101

Test Plan: CI

Reviewed By: pritamdamania87

Differential Revision: D25411118

fbshipit-source-id: 9f4a18ad8f45e364eae79b575cb1a9eaea45a86c
  • Loading branch information
rohan-varma authored and facebook-github-bot committed Jan 12, 2021
1 parent 8c5b024 commit 78e71ce
Showing 1 changed file with 13 additions and 0 deletions.
13 changes: 13 additions & 0 deletions torch/lib/c10d/reducer.cpp
Expand Up @@ -1036,6 +1036,19 @@ void Reducer::prepare_for_backward(
unused_parameters_.end(), indices.begin(), indices.end());
}
}

// Warn user about unnecessary perf hit if all parameters were used.
if (unused_parameters_.empty()) {
TORCH_WARN_ONCE(
"find_unused_parameters=True was specified in DDP constructor, "
"but did not find any unused parameters. This flag results in an extra "
"traversal of the autograd graph every iteration, which can adversely "
"affect performance. If your model indeed never has any unused "
"parameters, consider turning this flag off. Note that this warning may "
"be a false positive your model has flow control causing later iterations "
"to have unused parameters."
);
}
}

void Reducer::copy_bucket_to_grad(
Expand Down

0 comments on commit 78e71ce

Please sign in to comment.