-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-1087] Fix inconsistent pickle from dill when dumping the session #1485
Conversation
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
4bb4148
to
54b969f
Compare
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
54b969f
to
3658793
Compare
Refer to this link for build results (access rights to CI server needed): |
R: @robertwb PTAL |
Ideally we shouldn't be pickling any live objects from apiclient.py... can this be avoided? |
There is a dangerous looking solution in uqfoundation/dill#66 which could be useful in ignoring the objects in apiclient but I think that might add complexity compared to the adding reduce methods to the two classes. Thoughts ? |
I think this is good for now, and in general it'd be better to have more robust session (or at least main module) pickling. LGTM |
Thanks. Yeah, having a more robust pickling story would be great. |
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull request
mvn clean verify
. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
<Jira issue #>
in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.
This PR tries to fix GoogleCloudPlatform/DataflowPythonSDK#31 with adding a reduce hook to the Step and Job classes. We also need to modify the dump session so that the pickling is consistent due to uqfoundation/dill#195.
R: @robertwb PTAL