Generic types that use ParamSpec aren't callable

[NeilGirdhar]

from functools import partial
from typing import Callable, Generic, TypeVar

from typing_extensions import ParamSpec

R = TypeVar('R')
P = ParamSpec('P')

class custom_vjp(Generic[P, R]):
    def __init__(self, fun: Callable[P, R], x: int = 0):
        self.fun = fun

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
        return self.fun(*args, **kwargs)

partial(custom_vjp, x=0)  # error: Argument 1 to "partial" has incompatible type "Type[custom_vjp[Any, Any]]"; expected "Callable[..., custom_vjp[P, R]]"  [arg-type]

[A5rocks]

Types are callable in the general case: https://mypy-play.net/?mypy=latest&python=3.10&gist=a18ffb3451fb8341f3f89341edebab5b

I'm not sure what's going on here; I suspect some weird interaction with ParamSpec, though.

[achimnol]

Maybe due to existence of self? In such cases, we need to use typing.Concatenate introduced in Python 3.10, but mypy does not yet support it.

[NeilGirdhar]

@achimnol How would you use Concatenate here?

[achimnol]

Hm... I think I'm wrong about using Concatenate here.

The following code is well accepted and type-checked by mypy,

def myfun(x: int) -> str:
    return 'b' + str(x)

print('a' + custom_vjp(myfun, x=0)(123))  # accepted
print(1 + custom_vjp(myfun, x=0)(123))    # rejected

but here the problem seems to lie in the type inference of functools.partial against the class constructor.

pyright accepts your example while mypy shows an error as you reported.

[achimnol]

Though, your example may be converted to:

def custom_vjp(func: Callable[P, R], x: int = 0) -> Callable[P, R]:
    def inner(*args: P.args, **kwargs: P.kwargs) -> R:
        return func(*args, **kwargs)
    return inner

partial(custom_vjp, x=0)  # accepted & type-checked well

unless you are using other states and methods in custom_vjp.

[A5rocks]

OK, investigated a bit and it looks like it's due to ParamSpecs not being treated right in subtype checking: (*Any, **Any) -> Any should probably be a supertype of an arbitrary P, I think.

I believe this is due to a missing special case in subtype checking here (is_subtype(left=def (*Any, **Any) -> Any, right=ParamSpec)):

mypy/mypy/subtypes.py

Lines 333 to 362 in ef43416

	def visit_callable_type(self, left: CallableType) -> bool:
	right = self.right
	if isinstance(right, CallableType):
	if left.type_guard is not None and right.type_guard is not None:
	if not self._is_subtype(left.type_guard, right.type_guard):
	return False
	elif right.type_guard is not None and left.type_guard is None:
	# This means that one function has `TypeGuard` and other does not.
	# They are not compatible. See https://github.com/python/mypy/issues/11307
	return False
	return is_callable_compatible(
	left, right,
	is_compat=self._is_subtype,
	ignore_pos_arg_names=self.ignore_pos_arg_names)
	elif isinstance(right, Overloaded):
	return all(self._is_subtype(left, item) for item in right.items)
	elif isinstance(right, Instance):
	if right.type.is_protocol and right.type.protocol_members == ['__call__']:
	# OK, a callable can implement a protocol with a single `__call__` member.
	# TODO: we should probably explicitly exclude self-types in this case.
	call = find_member('__call__', right, left, is_operator=True)
	assert call is not None
	if self._is_subtype(left, call):
	return True
	return self._is_subtype(left.fallback, right)
	elif isinstance(right, TypeType):
	# This is unsound, we don't check the __init__ signature.
	return left.is_type_obj() and self._is_subtype(left.ret_type, right.item)
	else:
	return False

Compare what will happen for is_subtype(left=typing.Any, right=TypeVar):

mypy/mypy/subtypes.py

Lines 220 to 221 in ef43416

	def visit_any(self, left: AnyType) -> bool:
	return True

This might be an easy first PR for someone wanting that :P (just check that right is ParamSpecType and that left is def (*Any, **Any) -> Any, I believe?)

[A5rocks]

I ran into this in some code I was writing myself and can probably fix it (once my existing mypy PR gets merged, I really don't want to have to think about multiple PRs at once). If someone beats me to it, great!

[ilevkivskyi]

Original example works on master, likely fixed by #15837