Refactor typing._GenericAlias

[serhiy-storchaka]

BPO	40397
Nosy	@gvanrossum, @serhiy-storchaka, @ilevkivskyi, @JelleZijlstra
PRs	bpo-40397: Refactor typing._GenericAlias #19719 bpo-40397: Remove __args__ and __parameters__ from _SpecialGenericAlias #19984 bpo-40397: Fix subscription of nested generic alias without parameters. #20021

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-02.20:34:09.483>
created_at = <Date 2020-04-26.16:35:15.063>
labels = ['type-feature', 'library', '3.9']
title = 'Refactor typing._GenericAlias'
updated_at = <Date 2021-05-02.20:34:09.482>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-05-02.20:34:09.482>
actor = 'JelleZijlstra'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-02.20:34:09.483>
closer = 'JelleZijlstra'
components = ['Library (Lib)']
creation = <Date 2020-04-26.16:35:15.063>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 40397
keywords = ['patch']
message_count = 12.0
messages = ['367312', '368317', '368367', '368463', '368535', '368556', '368570', '368571', '368578', '368583', '368586', '392727']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'serhiy.storchaka', 'levkivskyi', 'JelleZijlstra']
pr_nums = ['19719', '19984', '20021']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40397'
versions = ['Python 3.9']

[serhiy-storchaka]

typing._GenericAlias represents two different types: user defined (like List[int]) and special (like List). They have different behavior, and common methods contain special cases.

The proposed PR rewrites the implementation in more object-oriented paradigm: different classes for different behavior. _GenericAlias is split on three classes: user defined (it may be replaced with GenericAlias in future), special, and the base class for common methods. Its subclasses are also split on classes for special types Tuple and Callable and for parametrized Callable[] and Annotated[]. The work is not finished yet and the code is still complex.

[serhiy-storchaka]

New changeset c1c7d8e by Serhiy Storchaka in branch 'master':
bpo-40397: Refactor typing._GenericAlias (GH-19719)
c1c7d8e

[serhiy-storchaka]

The next PR removes __args__ and __parameters__ from _SpecialGenericAlias.

• No existing test is failed. It is an evidence that these attributes are not intended, but a side effect of mixing two different kinds in one class.

• get_args() ignores __args__ and returns () for _SpecialGenericAlias (after resolving bpo-40398).

• Nested _SpecialGenericAlias is not considered a generic class containing type variables:

>>> from typing import *
>>> T = TypeVar('T')
>>> Dict[int, List[T]][str]
typing.Dict[int, typing.List[str]]
>>> Dict[int, List][str]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 244, in inner
    return func(*args, **kwds)
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 695, in __getitem__
    _check_generic(self, params)
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 194, in _check_generic
    raise TypeError(f"{cls} is not a generic class")
TypeError: typing.Dict[int, typing.List] is not a generic class

It also fixes the following error:

>>> from typing import *
>>> T = TypeVar('T')
>>> Dict[T, List][str]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 241, in inner
    return cached(*args, **kwds)
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 703, in __getitem__
    subargs = tuple(subst[x] for x in arg.__parameters__)
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 703, in <genexpr>
    subargs = tuple(subst[x] for x in arg.__parameters__)
KeyError: ~T

and allows to simplify __eq__ and __hash__ for _SpecialGenericAlias. Currently it has a weird behavior in corner case:

>>> from typing import *
>>> from typing import T
>>> List == List[T]
True

[JelleZijlstra]

The most recent change here caused a regression. The following file:

from typing import Generic, TypeVar, Union

class CannotTransform(Exception): pass

T = TypeVar("T")
E = TypeVar("E", bound=Exception)

class Ok(Generic[T]): pass
class Err(Generic[E]): pass

Result = Union[Ok[T], Err[E]]
TResult = Result[T, CannotTransform]
TMatchResult = TResult[int]

Now fails with:

Traceback (most recent call last):
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/../../black/black.py", line 22, in <module>
    TMatchResult = TResult[int]
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 244, in inner
    return func(*args, **kwds)
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 704, in __getitem__
    arg = arg[subargs]
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 244, in inner
    return func(*args, **kwds)
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 695, in __getitem__
    _check_generic(self, params)
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 194, in _check_generic
    raise TypeError(f"{cls} is not a generic class")
TypeError: __main__.Err[__main__.CannotTransform] is not a generic class

Before commit c1c7d8e it was fine. This was found when we added 3.9-dev to CI for Black (psf/black#1393).

[gvanrossum]

@serhiy can you look at this? A possible fix might be:

diff --git a/Lib/typing.py b/Lib/typing.py
index 681ab6d21e..adcef1e82b 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -701,7 +701,8 @@ class _GenericAlias(_BaseGenericAlias, _root=True):
                 arg = subst[arg]
             elif isinstance(arg, (_BaseGenericAlias, GenericAlias)):
                 subargs = tuple(subst[x] for x in arg.__parameters__)
-                arg = arg[subargs]
+                if subargs:
+                    arg = arg[subargs]
             new_args.append(arg)
         return self.copy_with(tuple(new_args))
 

The immediate cause of the problem seems to be that we call arg[subargs] with subargs being (). So arg is a _GenericAlias but its __parameters__ is (). I'm not sure what path was taken previously. :-(

[gvanrossum]

I think I have a shorter repro, not involving unions:

from typing import *
T = TypeVar("T")
S = TypeVar("S")
U = TypeVar("U")
class A(Generic[T]): ...
class B(Generic[T]): ...
class C(Generic[T, S]): ...
print(C[A[U], B[int]][str])

Fails in the same place, works in 3.8 (I didn't check just before the offending commit), and the same fix works.

[serhiy-storchaka]

New changeset fcb2856 by Serhiy Storchaka in branch 'master':
bpo-40397: Remove __args__ and __parameters__ from _SpecialGenericAlias (GH-19984)
fcb2856

[serhiy-storchaka]

Thank you Jelle for your report. There is even simpler example:

Dict[Tuple[T], List[int]][str]

[serhiy-storchaka]

Even simpler: Dict[T, List[int]][str].

Dict[T, list[int]][str] also fails. But dict[T, list[int]][str] works as expected (and there are tests).

[serhiy-storchaka]

New changeset 0122d48 by Serhiy Storchaka in branch 'master':
bpo-40397: Fix subscription of nested generic alias without parameters. (GH-20021)
0122d48

[JelleZijlstra]

Thanks Serhyi! I can confirm that the issue I posted is fixed.

[JelleZijlstra]

Looks like there's nothing left to do here.