Typing: Specialized subclasses of generics cannot be unpickled

[maatt]

BPO	25472
Nosy	@gvanrossum, @pitrou, @avassalotti, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gvanrossum'
closed_at = <Date 2015-11-19.05:20:46.402>
created_at = <Date 2015-10-25.03:09:33.179>
labels = ['type-bug', 'library']
title = 'Typing: Specialized subclasses of generics cannot be unpickled'
updated_at = <Date 2015-11-19.05:20:46.400>
user = 'https://bugs.python.org/maatt'

bugs.python.org fields:

activity = <Date 2015-11-19.05:20:46.400>
actor = 'gvanrossum'
assignee = 'gvanrossum'
closed = True
closed_date = <Date 2015-11-19.05:20:46.402>
closer = 'gvanrossum'
components = ['Library (Lib)']
creation = <Date 2015-10-25.03:09:33.179>
creator = 'maatt'
dependencies = []
files = []
hgrepos = []
issue_num = 25472
keywords = []
message_count = 7.0
messages = ['253421', '253514', '253515', '254877', '254878', '254879', '254880']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'pitrou', 'alexandre.vassalotti', 'python-dev', 'serhiy.storchaka', 'maatt']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25472'
versions = ['Python 3.5', 'Python 3.6']

[maatt]

If I try to pickle and unpickle an object of a class that has specialized a generic superclass, when I try to unpickle I get this error:

TypeError: descriptor '__dict__' for 'A' objects doesn't apply to 'B' object

Test case:

from typing import Generic, TypeVar
import pickle

T = TypeVar("T")


class A(Generic[T]):
    def __init__(self, x: T):
        self.x = x


class B(A[str]):
    def __init__(self, x: str):
        self.x = x


b = B("hello")
z = pickle.dumps(b)
print(z)
_ = pickle.loads(z)

[gvanrossum]

FWIW the pickle produced looks like this, decoded with pickletools.dis():

0: \\x80 PROTO      3
2: c    GLOBAL     '\_\_main__ B'

14: q BINPUT 0
16: ) EMPTY_TUPLE
17: \x81 NEWOBJ
18: q BINPUT 1
20: } EMPTY_DICT
21: q BINPUT 2
23: X BINUNICODE 'x'
29: q BINPUT 3
31: X BINUNICODE 'hello'
41: q BINPUT 4
43: s SETITEM
44: b BUILD
45: . STOP

[gvanrossum]

The issue seems to be the line

            inst_dict = inst.__dict__

in _Unpickler.load_build(). (I found this out by forcing sys.modules['_pickle'] = None, so the pure-Python pickle.py code gets used.)

This leads to a simpler repro:

# Use the same class definitions for A and B
b = B("hello")
print(b.__dict__)  # Same error message as before

That is, specialized subclasses of generics don't have a working __dict__ attribute! Interestingly, A("hello").__dict__ works.

I have to ponder this more.

[gvanrossum]

Update: I have not forgotten this, but it's a tough case and I haven't made much progress. I don't think I'll make the deadline for 3.5.1.

[gvanrossum]

Actually, I just found an embarrassingly simple fix:

diff --git a/src/typing.py b/src/typing.py
index d900036..49c4a06 100644
--- a/src/typing.py
+++ b/src/typing.py
@@ -981,7 +981,7 @@ class GenericMeta(TypingMeta, abc.ABCMeta):
                         "Cannot substitute %s for %s in %s" %
                         (_type_repr(new), _type_repr(old), self))
 
-        return self.__class__(self.__name__, self.__bases__,
+        return self.__class__(self.__name__, (self,) + self.__bases__,
                               dict(self.__dict__),
                               parameters=params,
                               origin=self,

I'll go check this in now, together with a test.

[python-dev]

New changeset 1d37f7dbbb37 by Guido van Rossum in branch '3.5':
Issue bpo-25472: In B[<type>], insert B in front of __bases__, to make the __dict__ descriptor work.
https://hg.python.org/cpython/rev/1d37f7dbbb37

New changeset 63c261185fc7 by Guido van Rossum in branch 'default':
Issue bpo-25472: In B[<type>], insert B in front of __bases__, to make the __dict__ descriptor work. (Merge 3.5->3.6)
https://hg.python.org/cpython/rev/63c261185fc7

[gvanrossum]

Thanks for the report! Made it before the 3.5.1 (rc1) deadline.