-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix data type in PartialDistributedGradientTape #3985
Conversation
Signed-off-by: Chen Zhu <chzhu@linkedin.com>
Signed-off-by: Chen Zhu <chzhu@linkedin.com>
Thanks for the fix @supercharleszhu, seems it's for the indexedslice branch. |
horovod/_keras/__init__.py
Outdated
@@ -182,7 +182,7 @@ def __filtered_reduce_grads(grads, vars): | |||
rg.append(grad) | |||
|
|||
rg = self._allreduce_grads(rg, rv) | |||
horovod_size = size_op(process_set_id=self.process_set.process_set_id) if int(os.environ.get("HOROVOD_ELASTIC", 0)) else self.process_set.size() | |||
horovod_size = tf.cast(size_op(process_set_id=self.process_set.process_set_id), tf.float32) if int(os.environ.get("HOROVOD_ELASTIC", 0)) else self.process_set.size() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having a size of type float
is a bit surprising, given the else branch is still an int. I'd rather have float32(horovod_size)
wherever division by horovod_size
occurs, or introduce a variable horovod_size_float = float32(horovod_size)
and use that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@EnricoMi Agreed. I just updated to float(horovod_size) wherever division by horovod_size occurs. Thanks!
Signed-off-by: Chen Zhu <chzhu@linkedin.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Unit Test Results 971 files + 403 971 suites +403 12h 11m 7s ⏱️ + 3h 12m 2s Results for commit 22ff283. ± Comparison against base commit ca21d8d. ♻️ This comment has been updated with latest results. |
Unit Test Results (with flaky tests) 1 117 files + 487 1 117 suites +487 12h 56m 16s ⏱️ + 3h 30m 21s Results for commit 22ff283. ± Comparison against base commit ca21d8d. ♻️ This comment has been updated with latest results. |
[re-triggered the pipeline: Build and Test macOS (test-cpu-openmpi-py3_7-tf1_15_5-keras2_2_4-torch1_6_0-mxnet1_5_1_p0-macos)] |
Checklist before submitting
Description
In horovod elastic mode we are seeing
which is a type mismatch from size_op and the gradient values. This PR fixes this issue.
Review process to land