updates teardown to account for ddp #2389
Conversation
| if self.global_rank == 0: | ||
| for proc in self.interactive_ddp_procs: | ||
| subprocess.Popen.kill(proc) |
There was a problem hiding this comment.
shouldn't it go into the run_training_teardown method?
There was a problem hiding this comment.
no because you still want certain things to happen with every process.
There was a problem hiding this comment.
oh wait, i see what you meant. i changed it. better now?
There was a problem hiding this comment.
not sure actually. I guess the "killing" should be the very last thing, after teardown has fully completed?
anyway not an expert here, so whatever works for you :)
There was a problem hiding this comment.
ah yes, i see now. That indeed should be the last thing since it will interrupt every other processes teardown. makes sense!
Codecov Report
@@ Coverage Diff @@
## master #2389 +/- ##
======================================
Coverage 88% 88%
======================================
Files 69 69
Lines 5452 5454 +2
======================================
+ Hits 4818 4820 +2
Misses 634 634 |
|
@awaelchli good to go with this? |
3 participants
No description provided.