Bug or new behavior of generics parameterization?

[Eclips4]

from typing import TypeVar, Generic, Callable


T = TypeVar("T")
P = TypeVar("P")

class Test(Generic[T]): ...


S = Callable[[Test], P]

def test(arg: S[T]): ...

On python3.11 i can't run this code. Code falls on an attempt to parameterize S
Traceback:

Traceback (most recent call last):
  File "C:\Users\KIRILL-1\PycharmProjects\dataclass_factory\exp.py", line 12, in <module>                  
    def test(arg: S[T]): ...                                                                               
                  ~^^^                                                                                     
  File "C:\Users\KIRILL-1\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 360, in inner       
    return cached(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\KIRILL-1\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 1391, in __getitem__
    new_args = self._determine_new_args(args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\KIRILL-1\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 1440, in _determine_new_args
    subargs.append(new_arg_by_param[x])
                   ~~~~~~~~~~~~~~~~^^^
KeyError: ~T

How i understand, in 3.11, an attempt is being made to parameterize Test in S, but in 3.10 - not
Is it new behavior or a bug?

• PR: gh-98852: Fix subscription of type aliases #98920

• PR: [3.11] gh-98852: Fix subscription of type aliases (GH-98920) #98952

• PR: [3.10] gh-98852: Fix subscription of types.GenericAlias instances (GH-98920) #98969

[sobolevn]

This is how it used to work in 3.10: https://github.com/python/cpython/blame/85f88f63d96b07208c98a725391af7cb710fe06b/Lib/typing.py#L1060-L1081

>>> from typing import TypeVar, Generic, Callable
>>> 
>>> 
>>> T = TypeVar("T")
>>> P = TypeVar("P")
>>> 
>>> class Test(Generic[T]): ...
... 
>>> 
>>> S = Callable[[Test], P]
>>> 
>>> def test(arg: S[T]): ...
... 
>>> test.__annotations__
{'arg': typing.Callable[[__main__.Test], ~T]}

Clearly looks like a bug to me. I am investigating what is going on :)

[Eclips4]

I know about how it works in 3.10 =)
My simple solution:

...
            if substfunc:
                new_arg = substfunc(new_arg_by_param[old_arg])
            elif not isinstance(old_arg, (_GenericAlias, GenericAlias, types.UnionType)):
                new_arg = old_arg
            else:
 ...

With that, code above works fine.

>>> test.__annotations__
{'arg': typing.Callable[[__main__.Test], ~T]}

What you think about this?

[sobolevn]

Yes, looks like it might work. Previously the same logic was guarded by this check: https://github.com/python/cpython/blame/85f88f63d96b07208c98a725391af7cb710fe06b/Lib/typing.py#L1071

Would you like to send a PR with this proposal? :)

[gvanrossum]

Could we first find which PR introduced the bug?

Also, @JelleZijlstra ^^

[Eclips4]

It is bpo-44796 (#31143)

[serhiy-storchaka]

Indeed, there is a bug in _collect_parameters which causes to skipping generic types.

Yet one interesting example:

>>> class Test(Generic[T]): ...
... 
>>> Test.__parameters__
(~T,)
>>> class Test2(Test): ...
... 
>>> Test2.__parameters__
()

But after fixing this code I got other test failure. And it seems that we have contradictory tests. Test in test_complex_subclasses expects that we can successfully create a class inheriting from generic base and Generic[].

cpython/Lib/test/test_typing.py

Lines 3898 to 3900 in 018b248

	# see gh-94607: this fails in that bug
	class Sub(Base, Generic[T]):
	...

But tests in test_generic_errors and test_generic_inheritance expect that Generic[] should include all typing variables, otherwise the class creation should fail.

cpython/Lib/test/test_typing.py

Lines 3047 to 3048 in 018b248

	with self.assertRaises(TypeError):
	class MyGeneric(List[T], Generic[S]): ...

cpython/Lib/test/test_typing.py

Lines 6492 to 6494 in 018b248

	with self.assertRaises(TypeError):
	class Point3D(Point2DGeneric[T], Generic[KT]):
	c: KT

The only difference between these examples is that in the former case the generic base is a generic class, and in the latter cases it is a generic alias. I do not think that there should be difference between them in this context. We should either require Generic[] containing all typing variables or allow it only containing some of them.

See also #94607.

[JelleZijlstra]

I don't see a contradiction. In class Sub(Base, Generic[T]):, Base should be equivalent to Base[Any], which is the general behavior for generics missing a subscription.

[serhiy-storchaka]

It makes sense if you look at it this way. The problem is with distinguishing free and bound variables. Both are represented in __parameters__.

[serhiy-storchaka]

The issue is more serious and is not completely new.

In the following code:

from typing import *
T = TypeVar("T")
T2 = TypeVar("T2")
class A(Generic[T2]): ...

Tuple[A, T][int]
tuple[A, T][int]
Tuple[TypeVar, T][int]
tuple[TypeVar, T][int]

3.11 raises exception for the 1st and 3rd examples, produces incorrect result for the 2nd example, and crashes on the last example.

3.10 raises exception for the 2nd examples.

[hauntsaninja]

Is there anything left to do here?

[serhiy-storchaka]

It is completed.