New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[hybird optim] reduce rend/recv times for recompute #34248
Conversation
Thanks for your contribution! |
'out_shape': var_shape, | ||
'dtype': var.dtype, | ||
self._op_device_key: cur_dev, | ||
self._op_role_key: op_role, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
之后加一个断言,recompute这个情况下op_role为backward
@@ -4867,15 +4900,13 @@ def _insert_send_recv(cur_id, prev_id): | |||
}) | |||
extra_index_info['index'] += 1 | |||
insert_index = None | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
咋把这几个换行删了。。
@@ -17,6 +17,7 @@ list(APPEND DIST_TEST_OPS test_parallel_dygraph_sparse_embedding) | |||
list(APPEND DIST_TEST_OPS test_parallel_dygraph_sparse_embedding_over_height) | |||
list(APPEND DIST_TEST_OPS test_parallel_dygraph_transformer) | |||
list(APPEND DIST_TEST_OPS test_fleet_pipeline_meta_optimizer) | |||
list(APPEND DIST_TEST_OPS test_fleet_pipeline_meta_optimizer_with_recompute) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
单测加到原来test_fleet_pipeline_meta_optimizer会出错吗
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR types
Performance optimization
PR changes
Others
Describe
观察到recompute的var在forward pass与recompute pass中均被send,会造成stage0 Backward --send--> stage1 Forward(recompute) 的情况。 优化过后,只需在前向时一次send,反向时将前向的var值assgin给 recompute的var即可。