New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Delete DDP hooks in Reducer destructor #21591
Closed
Closed
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
fb09803
Delete DDP hooks in Reducer destructor
mrshenli eb90ba2
address comments
mrshenli 5064522
enable c10d test in multi-gpu environment
mrshenli 0780b59
Merge remote-tracking branch 'upstream/master' into hook
mrshenli c8c41f0
fix tests
mrshenli 99847ac
Merge remote-tracking branch 'origin/master' into HEAD
08c0c8b
Get rid of remove_all_hooks and do it in Reducer's destructor
mrshenli File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Python binding can be removed I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Kutta asked for exposing this API, just in case Python GC does not deterministically kick in. @kuttas do you still need this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pietern Even if we call
__del__
on theDistributedDataParallel
object I am not totally sure that the C++ destructor (Reducer::~Reducer) will get called immediately or lazily (GC time). If this is 100% deterministic, then we don't need this call. Otherwise, we need to implement__del__
and explicitly clean up hooks in DDP. I searched pybind documentation and can't figure out what the expected behavior is if you don't have referential cycles - is ref count update immediate in python and does that immediately trigger the C++ destructor of the py-bound class?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't matter if it is lazily deleted. The hooks may still be called, but only as long as the reducer object is still alive (per the destructor). If the
prepare_for_backward
function isn't called by the DDP class that wraps the reducer, all the hooks will be nops.Regarding referential cycles -- IIRC this is where the GC comes in. Objects can be destructed immediately if their refcount drops to zero and then there is a separate GC pass for the cycles.