Fix deadlock in instruction done #8897
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lixinqi
commented
Aug 10, 2022
oneflow/core/vm/instruction.cpp
Outdated
Comment on lines
47
to
52
| INTRUSIVE_FOR_EACH_PTR(edge, mut_in_edges()) { | ||
| Instruction* in_instruction = edge->mut_src_instruction(); | ||
| CHECK(in_instruction->Done()); | ||
| in_instruction->mut_out_edges()->Erase(edge); | ||
| mut_in_edges()->Erase(edge); | ||
| } |
There was a problem hiding this comment.
StreamWait由于超前调度的作用，会出现这种情况。这是正常的，因为其上游指令一定也已经完成了。
| } | ||
|
|
||
| bool Instruction::Done() const { | ||
| return stream_policy().QueryInstructionStatusDone(stream(), status_buffer()) | ||
| && in_edges().empty(); |
There was a problem hiding this comment.
There was a problem hiding this comment.
OOMHandler那里的代码基本上是：
for instruction in stream.running_instruction_list:
wait until instruction->Done()
在stream_wait的https://github.com/Oneflow-Inc/oneflow/pull/8571/files#diff-ade37933f507aeea6c610837a09657ef820c915ec7d6d2bd0134c4750b900295R53 这里，Done行为被重构为考虑输入边为空才算完成。这在正常的流程里是没有问题的。但OOMHandler里却会在多卡的情况下有问题。我们观察到的是layer_norm op在上述的 wait until instruction->Done()过程中死锁。因为layer_norm的weight 输入来自于nccl stream，而layer_norm op本身在compute stream上，如果上述for instruction in stream.running_instruction_list先遍历到compute stream，就会出现这个BUG。
daquexian
reviewed
Aug 10, 2022
daquexian
approved these changes
Aug 10, 2022
lixinqi
added
the
need-highest-priority
Flowingsun007
approved these changes
Aug 10, 2022
Speed stats: |
|
CI failed when running job: cuda-speed-test. PR label automerge has been removed |
Speed stats: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
解决近期Instruction->Done的错误修改导致的死锁。