BUG: Fix crash when pickling dynamic class cycles. #102
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #102 +/- ##
==========================================
+ Coverage 80.62% 81.21% +0.58%
==========================================
Files 2 2
Lines 542 559 +17
Branches 111 112 +1
==========================================
+ Hits 437 454 +17
Misses 75 75
Partials 30 30
Continue to review full report at Codecov.
|
ssanderson
force-pushed
the
make-super-work
branch
from
3844b7a
to
9fa3692
Compare
llllllllll
reviewed
Jun 17, 2017
tests/cloudpickle_test.py
Outdated
| # Pickle and unpickle the class. | ||
| UnpickledDerived = pickle_depickle(Derived) | ||
| self.assertEqual(UnpickledDerived().method(), 2) | ||
| self.assertEqual(UnpickledDerived.__doc__, "Derived Docstring") |
There was a problem hiding this comment.
can you comment that __doc__ is special on pypy?
llllllllll
reviewed
Jun 17, 2017
cloudpickle/cloudpickle.py
Outdated
| # instructions to "rehydrate" the skeleton class by restoring the | ||
| # attributes from the __dict__. | ||
| # | ||
| # A type can appear in a cycle with it's __dict__ if an instance of the |
There was a problem hiding this comment.
your using the wrong its
|
+1 from me |
Fixes a bug where we would fail to pickle a class created inside a function if that class participated in a cycle with its own __dict__. Such cycles occur, for example, when a class defines a method that makes a Python 2-style super call, because we have a cycle from class -> __dict__ -> function -> __closure__ -> class. The fix for this is to use the same technique we use to dynamically-created functions: we first pickle an empty "skeleton class", which we memoize before pickling the rest of the class' __dict__. We then invoke a reduce function that re-attaches the class' attributes from the __dict__.
ssanderson
force-pushed
the
make-super-work
branch
from
9fa3692
to
f8fffa5
Compare
rgbkrk
approved these changes
Jun 28, 2017
|
Thank you! |
Closed
ghost
pushed a commit
to dbtsai/spark
that referenced
this pull request
Aug 22, 2017
## What changes were proposed in this pull request? Based on apache#18282 by rgbkrk this PR attempts to update to the current released cloudpickle and minimize the difference between Spark cloudpickle and "stock" cloud pickle with the goal of eventually using the stock cloud pickle. Some notable changes: * Import submodules accessed by pickled functions (cloudpipe/cloudpickle#80) * Support recursive functions inside closures (cloudpipe/cloudpickle#89, cloudpipe/cloudpickle#90) * Fix ResourceWarnings and DeprecationWarnings (cloudpipe/cloudpickle#88) * Assume modules with __file__ attribute are not dynamic (cloudpipe/cloudpickle#85) * Make cloudpickle Python 3.6 compatible (cloudpipe/cloudpickle#72) * Allow pickling of builtin methods (cloudpipe/cloudpickle#57) * Add ability to pickle dynamically created modules (cloudpipe/cloudpickle#52) * Support method descriptor (cloudpipe/cloudpickle#46) * No more pickling of closed files, was broken on Python 3 (cloudpipe/cloudpickle#32) * ** Remove non-standard __transient__check (cloudpipe/cloudpickle#110)** -- while we don't use this internally, and have no tests or documentation for its use, downstream code may use __transient__, although it has never been part of the API, if we merge this we should include a note about this in the release notes. * Support for pickling loggers (yay!) (cloudpipe/cloudpickle#96) * BUG: Fix crash when pickling dynamic class cycles. (cloudpipe/cloudpickle#102) ## How was this patch tested? Existing PySpark unit tests + the unit tests from the cloudpickle project on their own. Author: Holden Karau <holden@us.ibm.com> Author: Kyle Kelley <rgbkrk@gmail.com> Closes apache#18734 from holdenk/holden-rgbkrk-cloudpickle-upgrades.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes a bug where we would fail to pickle a class created inside a
function if that class participated in a cycle with its own
__dict__.Such cycles occur, for example, when a class defines a method that makes
a Python 2-style super call, because we have a cycle from
class -> __dict__ -> function -> __closure__ -> class.The fix for this is to use the same technique we use to
dynamically-created functions: we first pickle an empty "skeleton
class", which we memoize before pickling the rest of the class'
__dict__. We then invoke a reduce function that re-attaches the class'attributes from the
__dict__.Fixes #99.