Type variables in Callables don't unify

[maxrothman]

Issue type: Bug
Python version: 3.8
mypy version: 0.760
mypy flags: whatever the defaults are on https://mypy-play.net/

I discovered this issue when playing around with a monad-y Either thing (and here's it working in Typescript)

Here is (I think) a minimal repro:

from typing import Callable, TypeVar, Generic

def foo() -> int:
    return 1

T = TypeVar('T')
bar: Callable[[], T] = foo

main.py:7: error: Incompatible types in assignment (expression has type "Callable[[], int]", variable has type "Callable[[], T]")

I would expect that in the type declaration of bar, T would unify with int, but clearly it does not.

[JukkaL]

The TypeScript and Python examples look different to me, in particular the Error class is generic in the Python version, which results in some of the errors, and at least some of the errors seem legitimate. I'd recommend trying a direct translation of the TypeScript example first. I also don't understand how the minimal repro related to the original example. Maybe it would be helpful if you could show how you'd represent the minimal repro in TypeScript?

[maxrothman]

The error container in the Typescript example is generic, the names were confusingly inconsistent. I've modified them to match, here they are:

• Typescript
• Mypy

I'm not at all confident that my repro is correct. Now that I'm going back and attempting to do a repro more closely related to my actual code, I can't seem to get a typecheck error to occur. I'll continue poking at it, but regardless, we've got a direct translation to Typescript that typechecks and a version in with mypy that does not. Do you have a sense of why mypy's having an issue with this code?

[Michael0x2a]

Here is a smaller repro of the first error.

from typing import TypeVar, Generic, Union

T = TypeVar('T')
class Wrapper(Generic[T]):
    pass

T1 = TypeVar('T1')
T2 = TypeVar('T2')

def passthrough(f: Union[Wrapper[T1], T2]) -> Union[Wrapper[T1], T2]:
    return f
    
x: Union[Wrapper[int], str]

# E: Argument 1 to "passthrough" has incompatible type "Union[Wrapper[int], str]"; expected "Union[Wrapper[<nothing>], str]"
passthrough(x)

It seems mypy infers odd results if we include two TypeVars within a union. We get similarly poor results if we define passthrough to accept and return a Union[T1, T2] -- though perhaps it's more reasonable for mypy to choke in that case.

---

Here's a smaller repro for the second error.

from typing import TypeVar, Generic, Union, Callable

class Parent: pass
class Child1(Parent): pass
class Child2(Parent): pass

T = TypeVar('T')
class Wrapper(Generic[T]):
    inner: T

TParent = TypeVar('TParent', bound=Parent)
class Pipe(Generic[TParent]):
    def __init__(self, x: Wrapper[TParent]) -> None:
        pass

    def __or__(self) -> Pipe[Union[Child1, Child2]]:
        # E: Argument 1 to "Pipe" has incompatible type "Union[Wrapper[Child1], Wrapper[Child2]]";
        #    expected "Wrapper[Union[Child1, Child2]]"
        x: Union[Wrapper[Child1], Wrapper[Child2]]
        return Pipe(x)

In this case, the root cause is that mypy does not consider Union[Wrapper[A], Wrapper[B]] to be the same thing as Wrapper[Union[A, B]].

In this case, I believe mypy is actually correct here. If we have a w: Wrapper[Union[A, B]], it would actually be sound to do w.inner = A(); w.inner = B(). This wouldn't be sound for the former: if you happen to actually have, say, a Wrapper[A], doing w.inner = B() would introduce a bug.

I'm not sure why TypeScript isn't detecting this particular issue, even with a similar simplified example. You do get some errors if you switch to doing new Pipe(...) instead of new Pipe<...>(...), but that error feels unrelated. Maybe it could be due to TypeScript's decision to check generics bivariantly by default? Maybe I didn't translate the simplified example correctly? Not sure, I'm not really a TypeScript expert.

The remaining errors seem to be the same as the first.

[JukkaL]

The first error seems like mypy bug. The second example type checks cleanly if we make T a covariant type variable. There is a potential usability improvement here -- mypy should perhaps suggest that invariance is the cause of the error, like it does in some other examples with invariant generics.

[maxrothman]

Thanks for the help with the repro @Michael0x2a! I completely agree with your assessment.

In the meantime until this bug is addressed, is there a way to work around the issue using casts or something, or is the only option a mypy extension?

I also tried using a fully-tagged union (rather than only tagging errors), and while either and Pipe typecheck, exhaustiveness checking on the resulting union does not work properly.

Here's a first attempt where Ok and Error inherit from the same class

Here's a second attempt where I had no parent class

[maxrothman]

Any thoughts on my previous comment?

[anentropic]

I ran into this case recently and I wonder if it is the same/related (apologies if not):

from typing import TypeVar

T = TypeVar("T")

def do_something(a: T) -> T:
    return a

map(do_something, [1])

This gives:

error: Argument 1 to "map" has incompatible type "Callable[[T], T]"; expected "Callable[[int], T]"  [arg-type]

I can see in typeshed the type for map looks like:

class map(Iterator[_S], Generic[_S]):
    @overload
    def __init__(self, __func: Callable[[_T1], _S], __iter1: Iterable[_T1]) -> None: ...

So the problem seems to be that map uses two TypeVars in its callable arg spec, and mypy has failed to unify them if you give it a callable which uses the same TypeVar for both places?

[hauntsaninja]

@anentropic I think your example passes on mypy master using the --new-type-inference flag

[anentropic]

@hauntsaninja indeed it does... thank you 🎉