From 78e71ce62700f3382deae3e24b14e76c0fba4673 Mon Sep 17 00:00:00 2001 From: Rohan Varma Date: Tue, 12 Jan 2021 02:50:38 -0800 Subject: [PATCH] warn user once for possible unnecessary find_unused_params (#50133) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/50133 `find_unused_parameters=True` is only needed when the model has unused parameters that are not known at model definition time or differ due to control flow. Unfortunately, many DDP users pass this flag in as `True` even when they do not need it, sometimes as a precaution to mitigate possible errors that may be raised (such as the error we raise with not using all outputs).While this is a larger issue to be fixed in DDP, it would also be useful to warn once if we did not detect unused parameters. The downside of this is that in the case of flow control models where the first iteration doesn't have unused params but the rest do, this would be a false warning. However, I think the warning's value exceeds this downside. ghstack-source-id: 119707101 Test Plan: CI Reviewed By: pritamdamania87 Differential Revision: D25411118 fbshipit-source-id: 9f4a18ad8f45e364eae79b575cb1a9eaea45a86c --- torch/lib/c10d/reducer.cpp | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/torch/lib/c10d/reducer.cpp b/torch/lib/c10d/reducer.cpp index 56d427909155..ad0497724cfa 100644 --- a/torch/lib/c10d/reducer.cpp +++ b/torch/lib/c10d/reducer.cpp @@ -1036,6 +1036,19 @@ void Reducer::prepare_for_backward( unused_parameters_.end(), indices.begin(), indices.end()); } } + + // Warn user about unnecessary perf hit if all parameters were used. + if (unused_parameters_.empty()) { + TORCH_WARN_ONCE( + "find_unused_parameters=True was specified in DDP constructor, " + "but did not find any unused parameters. This flag results in an extra " + "traversal of the autograd graph every iteration, which can adversely " + "affect performance. If your model indeed never has any unused " + "parameters, consider turning this flag off. Note that this warning may " + "be a false positive your model has flow control causing later iterations " + "to have unused parameters." + ); + } } void Reducer::copy_bucket_to_grad(