Skip to content

TypeError when registering ForwardRefs in python 3.10.2 #206

New issue
New issue
Open
#215
Open
TypeError when registering ForwardRefs in python 3.10.2#206
#215
Labels
enhancement
@JWCook

Description

@JWCook
JWCook
opened on Jan 15, 2022 · edited by JWCook
  • cattrs version: 1.10.0
  • Python version: 3.10.2
  • Operating System: Ubuntu 20.04.3

Description

This is related to #201, but probably worth making a separate issue for, since it affects one of the (previously working) workarounds mentioned in that issue. Feel free to close this if you feel this is redundant with #201 or #94.

I have a converter that registers a ForwardRef. In python 3.6.x through 3.10.1, this has worked as intended. As of python 3.10.2, this is now throwing a TypeError:

TypeError: Invalid first argument to `register()`. ForwardRef('MyClass') is not a class.

Strangely, I don't see anything in the python 3.10.2 changelog that would indicate changed behavior with ForwardRef or register().

What I Did

Minimal example:

from attr import define, field
from cattr import Converter  # Note: same behavior with GenConverter
from typing import ForwardRef, List

@define()
class MyClass:
    history: List['MyClass'] = field(factory=list)

converter = Converter()
converter.register_structure_hook(
    ForwardRef('MyClass'), lambda obj, _: converter.structure(obj, MyClass)
)

Traceback:

TypeError                                 Traceback (most recent call last)
Input In [17], in <module>
----> 1 converter.register_structure_hook(
      2     ForwardRef("MyClass"), (lambda obj, _: converter.structure(obj, MyClass))
      3 )

File ~/.virtualenvs/rc-310/lib/python3.10/site-packages/cattr/converters.py:269, in Converter.register_structure_hook(self, cl, func)
    267     self._structure_func.clear_cache()
    268 else:
--> 269     self._structure_func.register_cls_list([(cl, func)])

File ~/.virtualenvs/rc-310/lib/python3.10/site-packages/cattr/dispatch.py:57, in MultiStrategyDispatch.register_cls_list(self, cls_and_handler, direct)
     55         self._direct_dispatch[cls] = handler
     56     else:
---> 57         self._single_dispatch.register(cls, handler)
     58         self.clear_direct()
     59 self.dispatch.cache_clear()

File ~/.pyenv/versions/3.10.2/lib/python3.10/functools.py:856, in singledispatch.<locals>.register(cls, func)
    854 else:
    855     if func is not None:
--> 856         raise TypeError(
    857             f"Invalid first argument to `register()`. "
    858             f"{cls!r} is not a class."
    859         )
    860     ann = getattr(cls, '__annotations__', {})
    861     if not ann:

TypeError: Invalid first argument to `register()`. ForwardRef('MyClass') is not a class.

Activity

JWCook

JWCook commented on Jan 15, 2022

@JWCook
JWCook
on Jan 15, 2022 · edited by JWCook
ContributorAuthor

Note that one of the other solutions posted in #201 still works in python 3.10.2: #201 (comment)

So the above example would become:

from attr import define, field
from cattr import GenConverter
from typing import ForwardRef, List

@define()
class MyClass:
    history: List['MyClass'] = field(factory=list)

converter = GenConverter()
converter .register_structure_hook_func(
    lambda t: t.__class__ is typing.ForwardRef,
    lambda v, t: converter.structure(v, t.__forward_value__),
)

So this may be a non-issue, but probably still useful for someone else out there googling for the same error.

JWCook
mentioned this on Jan 15, 2022
aha79

aha79 commented on Jan 15, 2022

@aha79
aha79
on Jan 15, 2022 · edited by aha79

I think the error is due to this commit (a change in functools.py), which was a fix due to bpo-46032.

It seems that the newly added check _is_valid_dispatch_type() returns false and thus we get the TypeError. (I cannot run the example as only Python 10.2 for macos is out).

It is a bit hard to follow the logic, but ForwardRef("MyClass") technically is an instance and not a type (which may explain the behavior).

However, I do not understand is why the solution in your last comment works.

Also what the change to singledispatch means for cattrs. I have the feeling that this will bite us again.

JWCook

JWCook commented on Jan 15, 2022

@JWCook
JWCook
on Jan 15, 2022
ContributorAuthor

Good to know, thanks for tracking that down. It looks like this may be an unintended side effect of that bugfix rather than an intentional change, then.

It is a bit hard to follow the logic, but ForwardRef("MyClass") technically is an instance and not a type (which may explain the behavior).

True, but an instance of ForwardRef also looks like a class:

>>> from typing import ForwardRef
>>> type(ForwardRef("MyClass"))
<class 'typing.ForwardRef'>

However, I do not understand is why the solution in your last comment works.

Tinche would be able to give a more thorough explanation, but I believe it works because Converter.register_structure_hook_func() takes a different path than register_structure_hook(), via FunctionDispatch.dispatch():

cattrs/src/cattr/dispatch.py

Lines 95 to 131 in 22b24c2

@attr.s(slots=True)
class FunctionDispatch:
"""
FunctionDispatch is similar to functools.singledispatch, but
instead dispatches based on functions that take the type of the
first argument in the method, and return True or False.
objects that help determine dispatch should be instantiated objects.
"""
_handler_pairs: list = attr.ib(factory=list)
def register(
self, can_handle: Callable[[Any], bool], func, is_generator=False
):
self._handler_pairs.insert(0, (can_handle, func, is_generator))
def dispatch(self, typ):
"""
returns the appropriate handler, for the object passed.
"""
for can_handle, handler, is_generator in self._handler_pairs:
# can handle could raise an exception here
# such as issubclass being called on an instance.
# it's easier to just ignore that case.
try:
ch = can_handle(typ)
except Exception:
continue
if ch:
if is_generator:
return handler(typ)
else:
return handler
raise StructureHandlerNotFoundError(
f"unable to find handler for {typ}", type_=typ
)

We're using the function we provide (in this case testing t.__class__ is typing.ForwardRef) instead of the checks in functools.singledispatch.register(), so _is_valid_dispatch_type() is never called on a ForwardRef instance.

Tinche

Tinche commented on Jan 16, 2022

@Tinche
Tinche
on Jan 16, 2022
Member

Yeah, as you folks found out, singledispatch is very inadequate for a bunch of use cases using more abstract types (i.e. not actual classes). That's why after checking singledispatch cattrs has a list of predicates that it'll check in order, and that's what you're enabling with register_structure_hook_func. I'm actually surprised ForwardRefs worked before with singledispatch.

So the predicate approach would be preferred, yeah.

ermshiperete
mentioned this on Jan 18, 2022
ermshiperete

ermshiperete commented on Jan 18, 2022

@ermshiperete
ermshiperete
on Jan 18, 2022

Same bug occurs with Python 3.9.10 (on Debian/Ubuntu).

ermshiperete
added 2 commits that reference this issue on Jan 18, 2022

fix(linux): Add workaround for Python bug

ec3355c

fix(linux): Add workaround for Python bug

412b6c6
ermshiperete
mentioned this on Jan 18, 2022
ermshiperete
added 3 commits that reference this issue on Jan 18, 2022

fix(linux): Add workaround for Python bug

59b28a0

fix(linux): Add workaround for Python bug

10e1697

fix(linux): Add workaround for Python bug

87d2361
ermshiperete
mentioned this on Jan 18, 2022
ermshiperete

ermshiperete commented on Jan 19, 2022

@ermshiperete
ermshiperete
on Jan 19, 2022

See also the bug in requests-cache: requests-cache/requests-cache#501

20 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @ex-nerd@ermshiperete@JWCook@Tinche@alekseiloginov

      Issue actions
        TypeError when registering ForwardRefs in python 3.10.2 · Issue #206 · python-attrs/cattrs