Fix cloudpickle serialization of wrapped Module methods.#1475
Merged
Conversation
Calling _context inside a closure will make cloudpickle attempt to serialize. It will fail because it cannot serialize ThreadLocals. Factoring out the body of method_wrapper into a seperate method hides the _context for cloudpickle because it can store library methods by name.
Contributor
|
The change LGTM. Can you please file a new issue to keep track of the challenge of testing this? So that we could in the future find a solution? |
Collaborator
|
LGTM - thanks for figuring out what cloudpickle was doing here! |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Calling _context inside a closure will make cloudpickle attempt to serialize.
It will fail because it cannot serialize ThreadLocals.
Factoring out the body of method_wrapper into a
separate method hides the _context for cloudpickle
because it can store library methods by name.
This PR does not come with a test because it requires us to depend on cloudpickle.
More importantly some tests indirectly uses dill which causes a clash in the pickle registry between dill & cloudpickle.
If we decide to make (cloud)pickle a first class citizen in the future we will need some clever test setup and elaborate to check for leaks of library internals through closures.
Fixes #1467