mypy is unable to assign Callable taking Concatenate to Union of that Callable and another using the same ParamSpec

[schuelermine]

Bug Report

mypy is unable to assign Callable[Concatenate[B, C], D] to Union[Callable[Concatenate[B, C], D], Callable[C, D].

To Reproduce

from typing import TypeVar, ParamSpec, Concatenate, cast
from collections.abc import Callable

A = TypeVar("A")
B = TypeVar("B")
C = ParamSpec("C")
D = TypeVar("D")


def precall_guard_none(
    a: A, b: B | None, f: Callable[Concatenate[A, B, C], D]
) -> Callable[Concatenate[B, C], D] | Callable[C, D]:
    r: Callable[Concatenate[B, C], D] | Callable[C, D]
    if b is None:

        def g(b: B, *args: C.args, **kwargs: C.kwargs) -> D:
            return f(a, b, *args, **kwargs)

        r = g  # This should be permitted because the type of g is one of the variants of the type of r, a Union
    else:

        def h(*args: C.args, **kwargs: C.kwargs) -> D:
            return f(a, cast(B, b), *args, **kwargs)  # This cast is surprisingly necessary

        r = h

    return r

Expected Behavior

I expect no error.

Actual Behavior

mypy produces this error:

main.py:19: error: Incompatible types in assignment (expression has type "Callable[[B, **C], D]", variable has type "Union[Callable[[B, **C], D], Callable[C, D]]")  [assignment]

Your Environment

mypy playground

• Mypy version used: 1.2.0
• Mypy command-line flags: --strict
• Python version used: 3.11

[nikochiko]

The problem seems to be with using Unions inside some other type.

I ran into the same issue, but with Dict.

... has incompatible type "Dict[str, str]"; expected "Dict[str, Union[str, bool]]"

[JelleZijlstra]

@nikochiko that's an unrelated issue. Mypy is correct here; see https://mypy.readthedocs.io/en/stable/common_issues.html#invariance-vs-covariance. Let's keep this issue focused on the ParamSpec problem.

[nikochiko]

Ah, TIL. Thanks for the pointer.

[mat-xc]

I may have the same issue but with Optional

from typing import Concatenate, ParamSpec, Callable, Optional


P = ParamSpec("P")

def func1(func: Callable[Concatenate[str, P], None], *args):
    pass

def func2(func: Optional[Callable[Concatenate[str, P], None]], *args):
    pass

def func3(a: str, b: str):
    pass

func1(func3, "hello")
func2(func3, "hello")

test.py:18: error: Argument 1 to "func2" has incompatible type "Callable[[str, str], Any]"; expected "Optional[Callable[[str, VarArg(<nothing>), KwArg(<nothing>)], None]]"  [arg-type]

func1 works, func2 not.

[mat-xc]

I have done some research, those are my findings.

The issue seems to be originated here

mypy/mypy/constraints.py

Lines 256 to 288 in 7fe1fdd

	# Now the potential subtype is known not to be a Union or a type
	# variable that we are solving for. In that case, for a Union to
	# be a supertype of the potential subtype, some item of the Union
	# must be a supertype of it.
	if direction == SUBTYPE_OF and isinstance(actual, UnionType):
	# If some of items is not a complete type, disregard that.
	items = simplify_away_incomplete_types(actual.items)
	# We infer constraints eagerly -- try to find constraints for a type
	# variable if possible. This seems to help with some real-world
	# use cases.
	return any_constraints(
	[infer_constraints_if_possible(template, a_item, direction) for a_item in items],
	eager=True,
	)
	if direction == SUPERTYPE_OF and isinstance(template, UnionType):
	# When the template is a union, we are okay with leaving some
	# type variables indeterminate. This helps with some special
	# cases, though this isn't very principled.
	result = any_constraints(
	[
	infer_constraints_if_possible(t_item, actual, direction)
	for t_item in template.items
	],
	eager=False,
	)
	if result:
	return result
	elif has_recursive_types(template) and not has_recursive_types(actual):
	return handle_recursive_union(template, actual, direction)
	return []
	# Remaining cases are handled by ConstraintBuilderVisitor.
	return template.accept(ConstraintBuilderVisitor(actual, direction))

While for func1 it checks constraints with

mypy/mypy/constraints.py

Line 288 in 7fe1fdd

return template.accept(ConstraintBuilderVisitor(actual, direction))

For func2 it enters here

mypy/mypy/constraints.py

Lines 270 to 285 in 7fe1fdd

	if direction == SUPERTYPE_OF and isinstance(template, UnionType):
	# When the template is a union, we are okay with leaving some
	# type variables indeterminate. This helps with some special
	# cases, though this isn't very principled.
	result = any_constraints(
	[
	infer_constraints_if_possible(t_item, actual, direction)
	for t_item in template.items
	],
	eager=False,
	)
	if result:
	return result
	elif has_recursive_types(template) and not has_recursive_types(actual):
	return handle_recursive_union(template, actual, direction)
	return []

then, in infer_constraints_if_possible, here

mypy/mypy/constraints.py

Lines 301 to 304 in 7fe1fdd

	if direction == SUPERTYPE_OF and not mypy.subtypes.is_subtype(
	actual, erase_typevars(template)
	):
	return None

because the two functions (func1 Callable[[str, str], Any] and Callable[[str, *Any, **Any], None]) are not one the subtypes of the other (I don't know if this is correct but it seems reasonable). In is_subtype the function is checked with

mypy/mypy/subtypes.py

Line 332 in 7fe1fdd

return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))

To sum up, func1 is checked using ConstraintBuilderVisitor while func2 is checked with SubtypeVisitor

I have tried with this change and it works good, but it seems like an hack to me.

diff --git a/mypy/constraints.py b/mypy/constraints.py
index 33230871b..2de4ea1e9 100644
--- a/mypy/constraints.py
+++ b/mypy/constraints.py
@@ -298,8 +298,10 @@ def infer_constraints_if_possible(
     """
     if direction == SUBTYPE_OF and not mypy.subtypes.is_subtype(erase_typevars(template), actual):
         return None
-    if direction == SUPERTYPE_OF and not mypy.subtypes.is_subtype(
-        actual, erase_typevars(template)
+    if (
+        direction == SUPERTYPE_OF
+        and not isinstance(get_proper_type(template), CallableType)
+        and not mypy.subtypes.is_subtype(actual, erase_typevars(template))
     ):
         return None
     if (

I hope that this would be useful for someone

[ilevkivskyi]

The original example works on master, likely fixed by #15837