Pickling of generic annotations/types in 3.5+ #318

valtron · 2019-12-22T01:21:56Z

This PR adds support for pickling annotations on 3.5+, and fixes some problems with generic annotations on 3.7+.

TODO

Backport for 3.5
Test that fails with TypeError: type() doesn't support MRO entry resolution; use types.new_class() if not using types.new_class for reconstructing classes
Remove typing_extensions dependency
Prefix privates with _
Add test for pickle_depickle'ing annotated functions/classes

Details

The types.new_class change (in _make_skeleton_class) is because of a TypeError: type() doesn't support MRO entry resolution; use types.new_class() error on 3.7+, similar to this issue. Also see python/cpython#6319.

I'm not sure if there are any downsides to TypeVars being __reduce__'d now. Previously, they were only supported as globals (so always imported, I think).

The functions try_decompose_generic and get_bases are brittle the way they are written, because they check for attributes. There might be a better way.

Tests

Passing, and added some new ones.

codecov · 2020-02-10T16:06:54Z

Codecov Report

Merging #318 into master will decrease coverage by 0.04%.
The diff coverage is 97.22%.

@@            Coverage Diff             @@
##           master     #318      +/-   ##
==========================================
- Coverage   93.33%   93.29%   -0.05%     
==========================================
  Files           2        2              
  Lines         765      790      +25     
  Branches      156      162       +6     
==========================================
+ Hits          714      737      +23     
  Misses         26       26              
- Partials       25       27       +2

Impacted Files	Coverage Δ
cloudpickle/cloudpickle.py	`92.26% <97.22%> (-0.02%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8452cc...554a4c6. Read the comment docs.

ogrisel · 2020-02-10T18:26:46Z

Don't worry about the failing CI. It should be resolved as soon as we merge: #342.

ogrisel · 2020-02-10T18:27:17Z

Maybe the lint failures should be fixed though.

pierreglaser · 2020-03-07T13:03:05Z

You may want to rebase, since we recently dropped Python 2 support.

valtron · 2020-03-07T20:23:51Z

@pierreglaser Done. I had to struggle a bit to get the coverage test passing, but it's good now :p

pierreglaser · 2020-03-08T14:42:51Z

Using a potentially pathological example inspired from python/typing#511, I get:

In [7]: from cloudpickle import loads, dumps
   ...: from typing import Generator
   ...: g = Generator[int, None, None]
   ...: loads(dumps(g))
Out[7]: typing.Generator[int, NoneType, NoneType]

Could you look into that @valtron ?

valtron · 2020-03-08T16:49:20Z

Could you elaborate? Do you mean I should add a test for typing.Generator or...?

pierreglaser · 2020-03-08T16:53:13Z

My bad, I did not check the repr of g. All good on that end, sorry @valtron

pierreglaser · 2020-03-08T17:06:55Z

Out of curiosity, do we need typing-extensions to support pickling for recent typing addition in Python < 3.7, or do we need typing-extensions because it includes bugfixes that make pickling typing constructs possible on Python < 3.7?

My concern comes from this comment (from python/typing#511), suggesting that pickling typing constructs is impossible by design.

The problem is, both GenericMeta.__new__ and GenericMeta.__getitem__ do destructive conversion and fixup on inputs, and you can't derive the original inputs from them, because __extra__, __bases__ and friends only give you the converted form. The internal design seems actually hostile to any kind of instance rebuilding.

(From a quick skim at the diff between CPython 3.6 and 3.7, it seems that the problematic class, GenericMeta, was deleted)

pierreglaser · 2020-03-08T17:41:20Z

I'm not sure if there are any downsides to TypeVars being reduce'd now. Previously, they were only supported as globals (so always imported, I think).

Thank you for this remark, we (at least I) was not aware of that. Indeed, on Python 3.7 with the latest cloudpickle, this currently fails:

In [14]: from cloudpickle import loads, dumps
    ...: import typing
    ...: T = typing.TypeVar('T', int, float, complex)
    ...: s = dumps(T)
    ...: del T
    ...: depickled_T = loads(s)  # raises AttributeError

Which is, to me, a bug (as we did not stated explicitly that we do not support it)

valtron · 2020-03-08T19:25:06Z

Out of curiosity, do we need typing-extensions to support pickling for recent typing addition in Python < 3.7, or do we need typing-extensions because it includes bugfixes that make pickling typing constructs possible on Python < 3.7?

Just for the typing additions, Literal and Final. It could be done without it (by making try_decompose_special_generic work differently depending on whether typing_extensions is installed or not) but I couldn't figure out how to get test coverage on both branches.

My concern comes from this comment (from python/typing#511), suggesting that pickling typing constructs is impossible by design.

The problem is, both GenericMeta.__new__ and GenericMeta.__getitem__ do destructive conversion and fixup on inputs, and you can't derive the original inputs from them, because __extra__, __bases__ and friends only give you the converted form. The internal design seems actually hostile to any kind of instance rebuilding.

Yeah, those two methods do a lot of stuff I don't understand, but I don't think it loses any information, so in principle it can be done. I just wrote the most straightforward code that worked, and only a small fixup was needed to account for the "conversion" that it does. I know it's wonky to try to write this without having a full understanding of what GenericMeta is doing, but I tried really hard to find failing examples and couldn't. (I also have a project that makes heavy use of annotations, and whereas before it failed with plain dill/cloudpickle, everything works with this PR.)

Thank you for this remark, we (at least I) was not aware of that. Indeed, on Python 3.7 with the latest cloudpickle, this currently fails:

...

Which is, to me, a bug (as we did not stated explicitly that we don't not support it)

Glad you found that bug! I should add that as a test.

pierreglaser · 2020-03-08T19:49:30Z

The types.new_class change (in _make_skeleton_class) is because of a TypeError: type() doesn't support MRO entry resolution; use types.new_class()

I'd have to double check, but I'm in favour of switching to type.new_class as a reconstructor for dynamic classes as IIRC, this should be the "official way" of creating a class (especially when this class comes from custom metaclasses). But class reconstruction is one of the most important and sensitive features of cloudpickle: that would thus be great to have an extra unit-test for this change, reproducing TypeError: type() doesn't support MRO entry resolution; use types.new_class() that does not involve typing constructs.

pierreglaser · 2020-03-08T19:52:43Z

Just for the typing additions, Literal and Final. It could be done without it (by making try_decompose_special_generic work differently depending on whether typing_extensions is installed or not) but I couldn't figure out how to get test coverage on both branches.

Let's not introduce typing_extensions as a new hard dependency for cloudpickle. Don't worry about the coverage being red for now, by the time this PR is ready to merge I guarantee you it will be green :)

pierreglaser · 2020-03-08T20:02:00Z

One other question: why is it that this feature is not backported to Python 3.5?

valtron · 2020-03-08T20:32:14Z

One other question: why is it that this feature is not backported to Python 3.5?

I left it for later, so I'll be doing it soon.

I'd have to double check, but I'm in favour of switching to type.new_class as a reconstructor for dynamic classes as IIRC, this should be the "official way" of creating a class (especially when this class comes from custom metaclasses). But class reconstruction is one of the most important and sensitive features of cloudpickle: that would thus be great to have an extra unit-test for this change, reproducing TypeError: type() doesn't support MRO entry resolution; use types.new_class() that does not involve typing constructs.

Will do.

valtron · 2020-03-08T23:58:14Z

Seems to already work for 3.5.

valtron · 2020-03-09T01:34:37Z

Added test for types.new_class.

I couldn't get a test that failed without the PR's changes, because of this: obj.__bases__ is always a list of types, and that error only gets thrown when calling type on bases that include a non-type. The PR changes that line to use obj.__orig_bases__ for generic types, which is what necessitates the type -> types.new_class change.

ogrisel

Also +1 for not adding a hard dependency on typing_extensions. We need to install typing_extensions on some CI workers though to make sure the integration works as expected.

Here are some comments about testing:

cloudpickle/generic_shims.py

ogrisel · 2020-03-10T09:15:59Z

tests/cloudpickle_test.py

+        ]
+
+        for obj in objs:
+            _ = pickle_depickle(obj, protocol=self.protocol)


We also need a new test (or extend this one to check that classes and functions defined using such type annotations are pickled as expected. However because of the class tracking feature of cloudpickle, using pickle_depickle + checks might not be enough to properly test this (#313). We probably need to use a subprocess in a similar way to the way we do insubprocess_pickle_echo.

For instance we could have a subprocess_pickle_and_run_checks helper and use it as follows:

for type_hint in type_hints: class MyClass: attribute: type_hint type_hint_repr: str def __init__(self): # executed in the parent process self.type_hint_repr = repr(self.__annotations__["attribute"]) def run_checks(self): # executed in the child process assert self.type_hint_repr == repr(self.__annotations__["attribute"]) instance = MyClass() subprocess_pickle_run_checks(instance)

Right, I don't know how I missed that. I'll add those tests.

About class tracking: it sounds like all tests should be checking both ways. Maybe pickle_depickle can be a test parameter (like self.protocol)?

Here's a POC of that idea: 873e556

I am not opposed to use functools.partial on pickle_echo to remove the redundant protocol argument but I am not sure how it relates to my original concern.

I should have done a better job linking to the relevant part of the PR :p Here it is: 873e556#diff-ad0a048ef2f4c0dc3eaf2baa553b134eR2097.

By parametrizing pickle_depickle, it becomes possible to add a subclass that runs all the tests using subprocess_pickle_echo.

pierreglaser · 2020-03-10T13:30:37Z

I couldn't get a test that failed without the PR's changes, because of this: obj.bases is always a list of types, and that error only gets thrown when calling type on bases that include a non-type. The PR changes that line to use obj.orig_bases for generic types, which is what necessitates the type -> types.new_class change.

About this - the contract of cloudpicklew/ regards to type annotations was to silently drop them on Python < 3.7 when pickling, and support them on Python >=3.7. But actually this snippet:

from typing import Generic, TypeVar
from cloudpickle import loads, dumps
T = TypeVar('T')
G = Generic[T]
class MyClass(G):  # this semantic allows people to use MyClass[int] as a type hint
    pass
loads(dumps(MyClass))

errors out on cloudpickle master for both Python versions (while it should not, for both Python versions). So I'm close to consider this code snippet as a the proof of a bug, which the types.new_class change in this PR solves. In such a case I believe this should be added as a separate test. WDYT @ogrisel?

PS: I needed some time to get around the meaning of this semantic (class E(Generic[T])). Please refer to https://www.python.org/dev/peps/pep-0484/#id20 for more detail.

valtron · 2020-03-10T14:38:16Z

@pierreglaser

About this - the contract of cloudpicklew/ regards to type annotations was to silently drop them on Python < 3.7 when pickling, and support them on Python >=3.7. But actually this snippet:

...

errors out on cloudpickle master for both Python versions (while it should not, for both Python versions).

As far as I can tell, that snippet fails because of the global lookup for typevar T, which is fixed here, not by types.new_class.

Here's the errors I got:

Python < 3.7:

cloudpickle\cloudpickle.py:776: in save_global
    self.save_dynamic_class(obj)
...
AttributeError: module 'abc' has no attribute '_get_dump'

Python >= 3.7:

cloudpickle\cloudpickle.py:774: in save_global
    Pickler.save_global(self, obj, name=name)
...
_pickle.PicklingError: Can't pickle ~T: it's not found as tests.cloudpickle_test.T

pierreglaser · 2020-03-10T14:41:11Z

My bad, I was using 1.2.2and not 1.3. Both error come from an attempt to pickle T actually.
Reproducing an error related to types.new_class is doable in 3.7 by simply putting T in a module and importing it in the snippet I wrote.

It does not change my statement though: this error is a breach of cloudpickle's contract (to me), and such a snippet should be added as a non-regression test.

In addition, this error on 3.6 is a little bit worrying. It comes from an issue in this part of the code that is revealed by a combination of subclassing and meta-classes.

valtron · 2020-03-10T17:44:54Z

Ah, thanks for the explanation. So there are two small changes that can be factored out of here:

types.new_class
changing how TypeVars are pickled

Should I make a PR for each?

pierreglaser · 2020-03-10T19:03:49Z

Sure :)

valtron · 2020-03-11T01:46:28Z

I tried making a PR just for this test, but types.new_class doesn't fix that issue: it fails for Python < 3.7 because of abc._get_dump, and for Python >= 3.7 it fails with:

Traceback (most recent call last):
  File "foo.py", line 7, in <module>
    loads(dumps(MyClass))
  File "C:\Users\default\cloudpickle\cloudpickle\cloudpickle.py", line 1136, in _make_skeleton_class
    skeleton_class = types.new_class(
  File "D:\dev\python38\lib\types.py", line 77, in new_class
    return meta(name, resolved_bases, ns, **kwds)
  File "D:\dev\python38\lib\typing.py", line 905, in __init_subclass__
    raise TypeError("Cannot inherit from plain Generic")
TypeError: Cannot inherit from plain Generic

... which is because it's calling make_skeleton_class with the bases from __bases__, which contains a bare Generic (it needs to use __orig_bases__).

I think maybe this PR should be split up into two parts:

One to handle Generic and such
One to handle annotations

What do you think?

pierreglaser · 2020-04-15T08:16:56Z

A few updates:

it turns out that the typing module does destructive conversion of inputs - I don't think there is any way of distinguishing List[(str,)] and List[str], but the two objects differ in identity.
In anyways, I believe identity is not a part of typing's contract: List[int] is List[int] only works because typing added a LRU cache for the corresponding __getitem__ method. But, typically:

In [28]: from typing import TypeVar, Generic
    ...: tvs = []
    ...: gvs = []
    ...: for i in range(200):
    ...:     t = TypeVar('t'+str(i))
    ...:     g = Generic[t]
    ...:     tvs.append(t)
    ...:     gvs.append(g)
    ...:
    ...: print(gvs[0] is Generic[tvs[0]])

prints False.

For these two reasons I wonder if cloudpickle should try to guarantee identity checks, e.g
pickle_depickle(type_hint) is type_hint. Note that pickle_depickle(type_hint) == type_hint will work though.

The last commit disabled identity checks in tests, but It can be re-enabled by leveraging cloudpickle's class tracking feature in the reducer.

Also, I need to investigate why codecov complains.

I'm not merging this yet -- I'd like @ogrisel to take a look at it before that, but it might take a couple extra weeks given the current circumstances.

ogrisel

LGTM but it would be great to understand what's going on with code coverage.

ogrisel · 2020-04-21T10:41:27Z

cloudpickle/cloudpickle.py

@@ -941,6 +955,31 @@ def inject_addons(self):
        """Plug in system. Register additional pickling functions if modules already loaded"""
        pass

+    if sys.version_info < (3, 7):  # pragma: no branch


This should not be enabled if typing_extensions is not available, right?

Suggested change

if sys.version_info < (3, 7): # pragma: no branch

if sys.version_info < (3, 7) and _typing_extensions is not None: # pragma: no branch

save_parametrized_type_hints is useful to pickle:

typing_extensions construct such as Literal and Final

some stdlib typing construct, such as ClassVar, Union, Tuple, Generic...

But do want to enable a half broken feature in cloudpickle when typing_extensions is not installed?

I would rather have the type annotations to be dropped or raise an explicit error message that states that installing typing_extensions on Python 3.6 an earlier.

I don't think this code breaks if typing_extensions is not installed: for instance, the CI tests the pickling of typing constructs in all entries of the CI whether typing_extensions is installed or not (typing_extensions is installed in only one entry in the CI).

All typing_extensions objects in this branch are created through this process, which include import guards.

Can you precise what the "half-broken feature" is?

Typing extensions is only an extension of the typing module in Python 3.5-3.6. It only exposes new type annotations and does not impact the behavior of cloudpickle when pickling type annotations from the typing module.

tests/cloudpickle_test.py

pierreglaser · 2020-04-21T12:44:36Z

I guess that the fact that we deleted more lines than we added in this PR can potentially make the overall coverage percentage decrease even though the coverage diff is 100% (as we can potentially increase the proportion of uncovered lines by deleting only covered lines).

We can try to have smarter (or at least a little bit looser) codecov threshold, or maybe only rely on the coverage diff, but I'm open to suggestions.

ogrisel · 2020-04-22T07:24:31Z

No worries. The diff coverage looks good.

ogrisel · 2020-04-22T07:24:55Z

Maybe launch CI downstream before merging this PR?

pierreglaser · 2020-04-22T07:49:01Z

Yes. I'll also do a last pass on the tests before merging.

Co-Authored-By: Olivier Grisel <olivier.grisel@gmail.com>

ogrisel · 2020-04-23T11:28:04Z

Merged! Thank you very much @valtron and @pierreglaser!

pierreglaser · 2020-04-23T11:39:55Z

Thanks a lot @valtron for this sequence of contributions!

ahirner · 2020-05-18T16:14:38Z

Thanks from me too @valtron et al, it solves a production issue with dask for us!

2.0.0 ===== - Python 3.5 is no longer supported. - Support for registering modules to be serialised by value. This allows code defined in local modules to be serialised and executed remotely without those local modules installed on the remote machine. ([PR #417](cloudpipe/cloudpickle#417)) - Fix a side effect altering dynamic modules at pickling time. ([PR #426](cloudpipe/cloudpickle#426)) - Support for pickling type annotations on Python 3.10 as per [PEP 563]( https://www.python.org/dev/peps/pep-0563/) ([PR #400](cloudpipe/cloudpickle#400)) - Stricter parametrized type detection heuristics in _is_parametrized_type_hint to limit false positives. ([PR #409](cloudpipe/cloudpickle#409)) - Support pickling / depickling of OrderedDict KeysView, ValuesView, and ItemsView, following similar strategy for vanilla Python dictionaries. ([PR #423](cloudpipe/cloudpickle#423)) - Suppressed a source of non-determinism when pickling dynamically defined functions and handles the deprecation of co_lnotab in Python 3.10+. ([PR #428](cloudpipe/cloudpickle#428)) 1.6.0 ===== - `cloudpickle`'s pickle.Pickler subclass (currently defined as `cloudpickle.cloudpickle_fast.CloudPickler`) can and should now be accessed as `cloudpickle.Pickler`. This is the only officially supported way of accessing it. ([issue #366](cloudpipe/cloudpickle#366)) - `cloudpickle` now supports pickling `dict_keys`, `dict_items` and `dict_values`. ([PR #384](cloudpipe/cloudpickle#384)) 1.5.0 ===== - Fix a bug causing cloudpickle to crash when pickling dynamically created, importable modules. ([issue #360](cloudpipe/cloudpickle#354)) - Add optional dependency on `pickle5` to get improved performance on Python 3.6 and 3.7. ([PR #370](cloudpipe/cloudpickle#370)) - Internal refactoring to ease the use of `pickle5` in cloudpickle for Python 3.6 and 3.7. ([PR #368](cloudpipe/cloudpickle#368)) 1.4.1 ===== - Fix incompatibilities between cloudpickle 1.4.0 and Python 3.5.0/1/2 introduced by the new support of cloudpickle for pickling typing constructs. ([issue #360](cloudpipe/cloudpickle#360)) - Restore compat with loading dynamic classes pickled with cloudpickle version 1.2.1 that would reference the `types.ClassType` attribute. ([PR #359](cloudpipe/cloudpickle#359)) 1.4.0 ===== **This version requires Python 3.5 or later** - cloudpickle can now all pickle all constructs from the ``typing`` module and the ``typing_extensions`` library in Python 3.5+ ([PR #318](cloudpipe/cloudpickle#318)) - Stop pickling the annotations of a dynamic class for Python < 3.6 (follow up on #276) ([issue #347](cloudpipe/cloudpickle#347)) - Fix a bug affecting the pickling of dynamic `TypeVar` instances on Python 3.7+, and expand the support for pickling `TypeVar` instances (dynamic or non-dynamic) to Python 3.5-3.6 ([PR #350](cloudpipe/cloudpickle#350)) - Add support for pickling dynamic classes subclassing `typing.Generic` instances on Python 3.7+ ([PR #351](cloudpipe/cloudpickle#351)) 1.3.0 ===== - Fix a bug affecting dynamic modules occuring with modified builtins ([issue #316](cloudpipe/cloudpickle#316)) - Fix a bug affecting cloudpickle when non-modules objects are added into sys.modules ([PR #326](cloudpipe/cloudpickle#326)). - Fix a regression in cloudpickle and python3.8 causing an error when trying to pickle property objects. ([PR #329](cloudpipe/cloudpickle#329)). - Fix a bug when a thread imports a module while cloudpickle iterates over the module list ([PR #322](cloudpipe/cloudpickle#322)). - Add support for out-of-band pickling (Python 3.8 and later). https://docs.python.org/3/library/pickle.html#example ([issue #308](cloudpipe/cloudpickle#308)) - Fix a side effect that would redefine `types.ClassTypes` as `type` when importing cloudpickle. ([issue #337](cloudpipe/cloudpickle#337)) - Fix a bug affecting subclasses of slotted classes. ([issue #311](cloudpipe/cloudpickle#311)) - Dont pickle the abc cache of dynamically defined classes for Python 3.6- (This was already the case for python3.7+) ([issue #302](cloudpipe/cloudpickle#302)) 1.2.2 ===== - Revert the change introduced in ([issue #276](cloudpipe/cloudpickle#276)) attempting to pickle functions annotations for Python 3.4 to 3.6. It is not possible to pickle complex typing constructs for those versions (see [issue #193]( cloudpipe/cloudpickle#193)) - Fix a bug affecting bound classmethod saving on Python 2. ([issue #288](cloudpipe/cloudpickle#288)) - Add support for pickling "getset" descriptors ([issue #290](cloudpipe/cloudpickle#290)) 1.2.1 ===== - Restore (partial) support for Python 3.4 for downstream projects that have LTS versions that would benefit from cloudpickle bug fixes. 1.2.0 ===== - Leverage the C-accelerated Pickler new subclassing API (available in Python 3.8) in cloudpickle. This allows cloudpickle to pickle Python objects up to 30 times faster. ([issue #253](cloudpipe/cloudpickle#253)) - Support pickling of classmethod and staticmethod objects in python2. arguments. ([issue #262](cloudpipe/cloudpickle#262)) - Add support to pickle type annotations for Python 3.5 and 3.6 (pickling type annotations was already supported for Python 3.7, Python 3.4 might also work but is no longer officially supported by cloudpickle) ([issue #276](cloudpipe/cloudpickle#276)) - Internal refactoring to proactively detect dynamic functions and classes when pickling them. This refactoring also yields small performance improvements when pickling dynamic classes (~10%) ([issue #273](cloudpipe/cloudpickle#273)) 1.1.1 ===== - Minor release to fix a packaging issue (Markdown formatting of the long description rendered on pypi.org). The code itself is the same as 1.1.0. 1.1.0 ===== - Support the pickling of interactively-defined functions with positional-only arguments. ([issue #266](cloudpipe/cloudpickle#266)) - Track the provenance of dynamic classes and enums so as to preseve the usual `isinstance` relationship between pickled objects and their original class defintions. ([issue #246](cloudpipe/cloudpickle#246)) 1.0.0 ===== - Fix a bug making functions with keyword-only arguments forget the default values of these arguments after being pickled. ([issue #264](cloudpipe/cloudpickle#264)) 0.8.1 ===== - Fix a bug (already present before 0.5.3 and re-introduced in 0.8.0) affecting relative import instructions inside depickled functions ([issue #254](cloudpipe/cloudpickle#254)) 0.8.0 ===== - Add support for pickling interactively defined dataclasses. ([issue #245](cloudpipe/cloudpickle#245)) - Global variables referenced by functions pickled by cloudpickle are now unpickled in a new and isolated namespace scoped by the CloudPickler instance. This restores the (previously untested) behavior of cloudpickle prior to changes done in 0.5.4 for functions defined in the `__main__` module, and 0.6.0/1 for other dynamic functions. 0.7.0 ===== - Correctly serialize dynamically defined classes that have a `__slots__` attribute. ([issue #225](cloudpipe/cloudpickle#225)) 0.6.1 ===== - Fix regression in 0.6.0 which breaks the pickling of local function defined in a module, making it impossible to access builtins. ([issue #211](cloudpipe/cloudpickle#211)) 0.6.0 ===== - Ensure that unpickling a function defined in a dynamic module several times sequentially does not reset the values of global variables. ([issue #187](cloudpipe/cloudpickle#205)) - Restrict the ability to pickle annotations to python3.7+ ([issue #193]( cloudpipe/cloudpickle#193) and [issue #196]( cloudpipe/cloudpickle#196)) - Stop using the deprecated `imp` module under Python 3. ([issue #207](cloudpipe/cloudpickle#207)) - Fixed pickling issue with singleton types `NoneType`, `type(...)` and `type(NotImplemented)` ([issue #209](cloudpipe/cloudpickle#209)) 0.5.6 ===== - Ensure that unpickling a locally defined function that accesses the global variables of a module does not reset the values of the global variables if they are already initialized. ([issue #187](cloudpipe/cloudpickle#187)) 0.5.5 ===== - Fixed inconsistent version in `cloudpickle.__version__`. 0.5.4 ===== - Fixed a pickling issue for ABC in python3.7+ ([issue #180]( cloudpipe/cloudpickle#180)). - Fixed a bug when pickling functions in `__main__` that access global variables ([issue #187]( cloudpipe/cloudpickle#187)). 0.5.3 ===== - Fixed a crash in Python 2 when serializing non-hashable instancemethods of built-in types ([issue #144](cloudpipe/cloudpickle#144)). - itertools objects can also pickled ([PR #156](cloudpipe/cloudpickle#156)). - `logging.RootLogger` can be also pickled ([PR #160](cloudpipe/cloudpickle#160)). 0.5.2 ===== - Fixed a regression: `AttributeError` when loading pickles that hold a reference to a dynamically defined class from the `__main__` module. ([issue #131]( cloudpipe/cloudpickle#131)). - Make it possible to pickle classes and functions defined in faulty modules that raise an exception when trying to look-up their attributes by name. 0.5.1 ===== - Fixed `cloudpickle.__version__`. 0.5.0 ===== - Use `pickle.HIGHEST_PROTOCOL` by default. 0.4.4 ===== - `logging.RootLogger` can be also pickled ([PR #160](cloudpipe/cloudpickle#160)). 0.4.3 ===== - Fixed a regression: `AttributeError` when loading pickles that hold a reference to a dynamically defined class from the `__main__` module. ([issue #131]( cloudpipe/cloudpickle#131)). - Fixed a crash in Python 2 when serializing non-hashable instancemethods of built-in types. ([issue #144](cloudpipe/cloudpickle#144)) 0.4.2 ===== - Restored compatibility with pickles from 0.4.0. - Handle the `func.__qualname__` attribute. 0.4.1 ===== - Fixed a crash when pickling dynamic classes whose `__dict__` attribute was defined as a [`property`](https://docs.python.org/3/library/functions.html#property). Most notably, this affected dynamic [namedtuples](https://docs.python.org/2/library/collections.html#namedtuple-factory-function-for-tuples-with-named-fields) in Python 2. (cloudpipe/cloudpickle#113) - Cloudpickle now preserves the `__module__` attribute of functions (cloudpipe/cloudpickle#118). - Fixed a crash when pickling modules that don't have a `__package__` attribute (cloudpipe/cloudpickle#116). 0.4.0 ===== * Fix functions with empty cells * Allow pickling Logger objects * Fix crash when pickling dynamic class cycles * Ignore "None" mdoules added to sys.modules * Support WeakSets and ABCMeta instances * Remove non-standard `__transient__` support * Catch exception from `pickle.whichmodule()` 0.3.1 ===== * Fix version information and ship a changelog 0.3.0 ===== * Import submodules accessed by pickled functions * Support recursive functions inside closures * Fix `ResourceWarnings` and `DeprecationWarnings` * Assume modules with `__file__` attribute are not dynamic 0.2.2 ===== * Support Python 3.6 * Support Tornado Coroutines * Support builtin methods

valtron mentioned this pull request Dec 22, 2019

Pickling generic annotations errors #317

Closed

This was referenced Feb 20, 2020

[Bug] sporadic segfaults from joblib tests in CircleCI facebookresearch/hydra#421

Closed

_pickle.PicklingError: Can't pickle typing.Union[str, NoneType]: it's not the same object as typing.Union facebookresearch/hydra#428

Closed

valtron changed the title ~~Pickling of generic annotations/types in 3.6+~~ Pickling of generic annotations/types in 3.5+ Mar 9, 2020

ogrisel reviewed Mar 10, 2020

View reviewed changes

cloudpickle/generic_shims.py Outdated Show resolved Hide resolved

ogrisel reviewed Mar 10, 2020

View reviewed changes

valtron mentioned this pull request Mar 11, 2020

Pickle typing.TypeVars not as globals #350

Merged

valtron and others added 2 commits April 4, 2020 19:47

Handle generic special forms in dispatch

bc0c368

refactor a bit the type-saving logic

55489f4

test pickling typing-extensions constructs in CI

ef4a263

pierreglaser added the ci python-nightly Signal the CI to run the test suite of cloudpickle against the master branch of CPython label Apr 15, 2020

CI use single quotes (?)

333d6b0

pierreglaser removed the ci python-nightly Signal the CI to run the test suite of cloudpickle against the master branch of CPython label Apr 15, 2020

pierreglaser added 2 commits April 15, 2020 11:33

correct typos in coverage pragmas

26e06f5

add a few more pytest pragmas

eb1cebd

ogrisel approved these changes Apr 21, 2020

View reviewed changes

comment rephrasings

63bc551

Co-Authored-By: Olivier Grisel <olivier.grisel@gmail.com>

pierreglaser added 3 commits April 22, 2020 09:59

CI trigger

f889731

rephrase CHANGES.md

236b339

check equality of depickled type hints

554a4c6

ogrisel merged commit 1ba10a5 into cloudpipe:master Apr 23, 2020

LachlanStuart mentioned this pull request Oct 21, 2020

Generic type annotation error due to old version of CloudPickle lithops-cloud/lithops#392

Closed

	if sys.version_info < (3, 7): # pragma: no branch
	if sys.version_info < (3, 7) and _typing_extensions is not None: # pragma: no branch

Pickling of generic annotations/types in 3.5+ #318

Pickling of generic annotations/types in 3.5+ #318

Conversation

valtron commented Dec 22, 2019 • edited

TODO

Details

Tests

codecov bot commented Feb 10, 2020 • edited

Codecov Report

ogrisel commented Feb 10, 2020

ogrisel commented Feb 10, 2020

pierreglaser commented Mar 7, 2020

valtron commented Mar 7, 2020

pierreglaser commented Mar 8, 2020

valtron commented Mar 8, 2020

pierreglaser commented Mar 8, 2020

pierreglaser commented Mar 8, 2020 • edited

pierreglaser commented Mar 8, 2020 • edited

valtron commented Mar 8, 2020 • edited by pierreglaser

pierreglaser commented Mar 8, 2020 • edited

pierreglaser commented Mar 8, 2020

pierreglaser commented Mar 8, 2020

valtron commented Mar 8, 2020

valtron commented Mar 8, 2020

valtron commented Mar 9, 2020

ogrisel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pierreglaser commented Mar 10, 2020 • edited

valtron commented Mar 10, 2020

pierreglaser commented Mar 10, 2020 • edited

valtron commented Mar 10, 2020

pierreglaser commented Mar 10, 2020

valtron commented Mar 11, 2020

pierreglaser commented Apr 15, 2020 • edited

ogrisel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pierreglaser Apr 22, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pierreglaser commented Apr 21, 2020 • edited

ogrisel commented Apr 22, 2020 • edited

ogrisel commented Apr 22, 2020

pierreglaser commented Apr 22, 2020

ogrisel commented Apr 23, 2020

pierreglaser commented Apr 23, 2020

ahirner commented May 18, 2020

valtron commented Dec 22, 2019 •

edited

codecov bot commented Feb 10, 2020 •

edited

pierreglaser commented Mar 8, 2020 •

edited

pierreglaser commented Mar 8, 2020 •

edited

valtron commented Mar 8, 2020 •

edited by pierreglaser

pierreglaser commented Mar 8, 2020 •

edited

pierreglaser commented Mar 10, 2020 •

edited

pierreglaser commented Mar 10, 2020 •

edited

pierreglaser commented Apr 15, 2020 •

edited

pierreglaser Apr 22, 2020 •

edited

pierreglaser commented Apr 21, 2020 •

edited

ogrisel commented Apr 22, 2020 •

edited