Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[serhiy-storchaka]

BPO	44801
Nosy	@gvanrossum, @ambv, @serhiy-storchaka, @miss-islington, @Fidget-Spinner
PRs	bpo-44801: Check arguments in substitution of ParamSpec in Callable #27585 [3.10] bpo-44801: Check arguments in substitution of ParamSpec in Callable (GH-27585) #27598
Dependencies	bpo-44793: Arguments ignored in substitution in typing.Callable

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-04.21:00:34.993>
created_at = <Date 2021-08-01.07:08:58.391>
labels = ['type-bug', 'library', '3.10', '3.11']
title = 'Type expression is coerced to a list of parameter arguments in substitution of ParamSpec'
updated_at = <Date 2021-08-04.21:00:34.992>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-08-04.21:00:34.992>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-04.21:00:34.993>
closer = 'lukasz.langa'
components = ['Library (Lib)']
creation = <Date 2021-08-01.07:08:58.391>
creator = 'serhiy.storchaka'
dependencies = ['44793']
files = []
hgrepos = []
issue_num = 44801
keywords = ['patch']
message_count = 5.0
messages = ['398687', '398688', '398842', '398925', '398953']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'lukasz.langa', 'serhiy.storchaka', 'miss-islington', 'kj']
pr_nums = ['27585', '27598']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44801'
versions = ['Python 3.10', 'Python 3.11']

[serhiy-storchaka]

Type expression is coerced to a list of parameter arguments in substitution of ParamSpec. For example:

>>> from typing import *
>>> T = TypeVar('T')
>>> P = ParamSpec('P')
>>> C = Callable[P, T]
>>> C[int, str]
typing.Callable[[int], str]

int becomes [int]. There is even a dedicated test for this.

But it is not followed from PEP-612. Furthermore, it contradicts one of examples in the PEP:

>>> class X(Generic[T, P]):
...     f: Callable[P, int]
...     x: T
... 
>>> X[int, int]  # Should be rejected
__main__.X[int, int]

It makes the implementation (at least the code in bpo-44796) more complex and makes the user code more errorprone.

[Fidget-Spinner]

Type expression is coerced to a list of parameter arguments in substitution of ParamSpec.

It's not, only the repr is like that. Internally it's not coerced.

>>> C[int, str]
typing.Callable[[int], str]
>>> C[int, str].__args__
(<class 'int'>, <class 'str'>)

Because Callable's correct form is Callable[[type], type] (where type is not ParamSpec or Concatenate) so the repr reflects that.

Internally, Callable also flattens the list of args:

>>> Callable[[int, str], int].__args__
(<class 'int'>, <class 'str'>, <class 'int'>)

Because __args__ *must* be hashable and immutable. Otherwise, it will not work with Union or Literal. Some time ago we tried converting to nested tuple. But that was too difficult (and backwards incompatible) because every other typing operation expected __args__ to contain types, not a tuple.

[serhiy-storchaka]

Thank you Ken Jin. So the problem is the __args__ (x, str) is interpreted differently depending on x, and after substituting x the interpretation can be changed. If x was ParamSpec, it was interpreted in one way, but if it becomes int, it is now interpreted in other way.

The solution is to forbid substitution of P with wrong values (not parameters expression). Some normalization is also needed, before and after substitution.

Other related example is:

>>> from typing import *
>>> P = ParamSpec("P")
>>> class Z(Generic[P]): pass
... 
>>> A = Z[[int]]
>>> B = Z[int]
>>> A
__main__.Z[(<class 'int'>,)]
>>> B
__main__.Z[int]
>>> A.__args__
((<class 'int'>,),)
>>> B.__args__
(<class 'int'>,)

It is expected that A and B should the same.

[ambv]

New changeset 3875a69 by Serhiy Storchaka in branch 'main':
bpo-44801: Check arguments in substitution of ParamSpec in Callable (GH-27585)
3875a69

[miss-islington]

New changeset 536e35a by Miss Islington (bot) in branch '3.10':
bpo-44801: Check arguments in substitution of ParamSpec in Callable (GH-27585)
536e35a