force sync batch norm grad sequential #52268
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
你的PR提交成功，感谢你对开源项目的贡献! |
|
❌ The PR is not created using PR's template. You can refer to this Demo. |
JiabinYang
reviewed
Mar 29, 2023
| // Node Construction | ||
| auto grad_node = | ||
| std::shared_ptr<SyncBatchNormGradNode>(new SyncBatchNormGradNode(6, 5)); | ||
| egr::Controller::Instance().PushBackForceSequentialNodes(grad_node.get()); |
There was a problem hiding this comment.
AddForceSequentialNodes?
There was a problem hiding this comment.
我觉得还是PushBack好，代表了有序的~
| force_sequential_nodes_.pop(); | ||
| } | ||
| } | ||
| void PushBackForceSequentialNodes(GradNodeBase* node) { |
There was a problem hiding this comment.
AddForceSequentialNodes
| @@ -111,6 +111,22 @@ std::vector<paddle::Tensor> RunBackward( | |||
| const std::vector<paddle::Tensor>& no_grad_vars = {}) { | |||
| VLOG(3) << "Start Backward"; | |||
|
|
|||
| std::queue<GradNodeBase*> force_sequential_nodes_forward_queue = | |||
There was a problem hiding this comment.
Seal this into one function
| } | ||
| }; | ||
|
|
||
| if (force_sequential_nodes_set.count(next_node)) { |
There was a problem hiding this comment.
how about give a quick path for empty force_sequential_nodes_set
| } else { | ||
| queue.push_back(std::move(next_node)); | ||
| ready_force_sequential_nodes.insert(next_node); | ||
| continue; |
JiabinYang
approved these changes
Mar 30, 2023
chenwhql
approved these changes
Mar 30, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Bug fixes
PR changes
Others
Describe
本PR用于修复SyncBatchNorm的前向执行顺序和反向的执行顺序错乱，导致的hang或者精度不对的问题。
由于控制流原因，多卡时，每个卡执行的逻辑可能是不一致的，算子数量肯定也是不一致的。
而前向算子的执行顺序，包括SyncBN的执行顺序都是由Python逻辑来控制的。
而反向图算子的执行顺序，是DAG的拓扑序，具体逻辑是框架通过“广度优先搜索（在算子的input全部准备好的情况下）”来确定顺序的。那么每个SyncBNGrad的执行顺序，由他举例Loss的“距离”决定。
而由于前向控制流原因，导致每张卡SyncBNGrad的顺序可能是不一致的，这可能导致hang或者精度不对。