dataclass MRO entry resolution for type variable metaclasses: TypeError

[Ricyteach]

BPO	33188
Nosy	@ericvsmith, @ilevkivskyi, @Ricyteach
PRs	bpo-33188: Allow dynamic creation of generic dataclasses #6319 [3.7] bpo-33188: Allow dynamic creation of generic dataclasses (GH-6319) #6320
Files	dataclass_metaclass_issue.py: demo of problem

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-04-06.21:31:36.900>
created_at = <Date 2018-03-30.18:03:43.623>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'dataclass MRO entry resolution for type variable metaclasses: TypeError'
updated_at = <Date 2018-04-06.21:31:36.899>
user = 'https://github.com/Ricyteach'

bugs.python.org fields:

activity = <Date 2018-04-06.21:31:36.899>
actor = 'levkivskyi'
assignee = 'none'
closed = True
closed_date = <Date 2018-04-06.21:31:36.900>
closer = 'levkivskyi'
components = ['Library (Lib)']
creation = <Date 2018-03-30.18:03:43.623>
creator = 'Ricyteach'
dependencies = []
files = ['47508']
hgrepos = []
issue_num = 33188
keywords = ['patch']
message_count = 11.0
messages = ['314703', '314704', '314708', '314709', '314710', '314721', '314724', '314726', '314728', '314729', '314738']
nosy_count = 3.0
nosy_names = ['eric.smith', 'levkivskyi', 'Ricyteach']
pr_nums = ['6319', '6320']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33188'
versions = ['Python 3.7', 'Python 3.8']

[Ricyteach]

I'm getting the following error at when attempting to create a new dataclass with any base class that is supplied a type variable:

TypeError: type() doesn't support MRO entry resolution; use types.new_class()

In principle, it seems like this shouldn't cause any problems, since this works as expected:

    @dataclass
    class MyClass(Generic[MyTypeVar]):
        pass

The problem seems to be the call to type in make_dataclass for instantiating the class object, since type doesn't support type variables. If I replace the dataclass line that produces the error with the following code block, it seems to work:

try:
    cls = type(cls_name, bases, namespace)
except TypeError:
    cls = types.new_class(cls_name, bases, namespace)

I haven't tested, but it might be possible to just remove the call to type altogether.

I've attached a file demonstrating the issue.

[Ricyteach]

Sorry: to be clear, the error only occurs when attempting to create the class using make_dataclass.

[ericvsmith]

You can also cause this same error without dataclasses:

from typing import TypeVar, Generic
from types import new_class
MyTypeVar = TypeVar("MyTypeVar")
MyParent = new_class("MyParent", (Generic[MyTypeVar],), {})
c = type('MyChild', (MyParent[int],), {})  # error in 3.8

I assume you get the same error in 3.7, but I can't compile it just now.

The above code works in 3.6. And the code in dataclass_metaclass_issue.py works in 3.6, using the backported dataclasses.py from PyPI.

So it seems this is a typing problem, not a dataclasses problem.

+Ivan, in case he has some insights.

[Ricyteach]

Same error on 3.7.

Probably getting beyond my knowledge here but from the error message, it seems like the answer is simply that:

type('MyChild', (MyParent[int],), {})

...is just the wrong way to make a new type when utilizing type variables.

[ilevkivskyi]

This is not a bug but an explicit design decision. Generic classes are static typing concept and therefore are not supposed to work freely with dynamic class creation. During discussion of PEP-560 it was decided that there should be at least one way to dynamically create generic classes, types.new_class was chosen for this, see https://www.python.org/dev/peps/pep-0560/#dynamic-class-creation-and-types-resolve-bases

Also the exception message is quite clear about this. Unfortunately, PEPs 560 and 557 were discussed in parallel so not every possible interactions where thought out. But is it critical for dataclasses to call type? I believe there should be no other differences with types.new_class. I would say the latter is even better than type because it correctly treats __prepare__ on the metaclass IIRC. So I would propose to switch from type() to types.new_class() for dynamic creation of dataclasses.

[Ricyteach]

Eric: see also Ivan's comment on bpo-33190, which will factor into the solution to this I think. It seems you can't just pass the namespace to the kwds argument (as I have done in my solution above).

[ericvsmith]

If we're going to call new_class in make_dataclass, then we should change the signature of make_dataclass to have the new_class parameters.

[ilevkivskyi]

If we're going to call new_class in make_dataclass, then we should change the signature of make_dataclass to have the new_class parameters.

Why? I think we should care about what API/signature is reasonable/typical for dataclasses use cases, while new_class is really just an implementation detail. IMO passing keyword arguments to metaclass will be much more rare for dataclasses than passing a ready namespace.

[Ricyteach]

passing keyword arguments to metaclass will be much more rare for dataclasses than passing a ready namespace

The impetus of my running into these issues was assuming that things like Generic[MyTypeVar] would "just work" with make_dataclass, which seemed like a valid assumption since the class creation approach made heavy use of by dataclasses implies this:

@dataclass
class MyDclass(Generic[MyTypeVar]):
    var: MyTypeVar

The fact that I cannot do this, then, without error is surprising:

MyDclass = make_dataclass("MyDclass", (("var", MyTypeVar),), bases=(Generic[MyTypeVar],))

I'm not stating it HAS to be fixed. Maybe it doesn't have to. But to me, the above seems like the reason to do it if it's going to be done.

[Ricyteach]

I'll also say: one of the biggest reasons I was excited to read the dataclasses PEP was because I thought "AWESOME! Now I can delete all of the stupid boilerplate init and repr methods for those typing.Sequences, typing.Mapping, etc etc subclasses!"

[ericvsmith]

Yeah, no need to change the API. I was over-generalizing.