Generic decorators don't work with generic functions

[alexsydell]

UPDATE: Although the original example is now special-cased in a plugin, the underlying problem is not solved, see #1317 (comment)

The following:

from typing import *
from contextlib import *

T = TypeVar("T")

@contextmanager
def foo(x):
  # type: (T) -> Iterator[T]
  yield x

y = 1
with foo(2) as f:
  y = f

gives this error:

error: Incompatible types in assignment (expression has type "T", variable has type "int")

It should recognize the input as an int which would lead to the output being an int as well, not a T. This is probably due to the way the contextmanager stub is defined in typeshed.

[gvanrossum]

Yeah, contextmanager seems to bring out the worst in mypy, because it's a decorator and wraps a generator. :-( Note this comment in contextlib.pyi:

# TODO this doesn't capture the relationship that the returned function's args are the same as func's.
def contextmanager(func: Callable[..., Iterator[_T]]) -> Callable[..., ContextManager[_T]]: ...

Fixing that will probably require some way to describe the relationship between the args of the Callable used to describe contextmanager's 'func' argument and the args of the Callable describing its return type. (Note that the return type of the two Callables are different.)

[JukkaL]

Variadic type variables (python/typing#193) would be useful here, assuming that they could preserve all the details of function argument lists such as keyword argument names. They aren't in PEP 484 and mypy doesn't support them, so they won't help us right now.

Another way to go about this would be to support type checker plugins that can special case the checking of certain library functions. Mypy doesn't support them either, so the current signature is about the best we can do.

[rwbarton]

Variadic type variables would certainly be the way to go here, but there's actually a latent mypy bug here as well, which you can expose by changing the stub type of contextlib.contextmanager to the type that should make this program type check, namely

def contextmanager(func: Callable[[_T], Iterator[_T]]) -> Callable[[_T], ContextManager[_T]]: ...

Recall that

@contextmanager
def foo(x):
  # type: (T) -> Iterator[T]
  yield x

really means something like

def foo_(x):
  # type: (T) -> Iterator[T]
  yield x
foo = contextmanager(foo_)

foo_ is a generic function with a type variable T, and in the call to the generic function contextmanager we determine that _T = T. But foo does not end up being a generic function in mypy's internal representation. It is just a function whose type involves a type variable T that is not bound anywhere. So the call foo(2) then already fails to type check, since the type of 2 is int and not T.

With the current type of contextmanager, foo gets the type Callable[..., ContextManager[T]], so foo(2) is accepted, but the result type ContextManager[T] still has a nonsensical reference to an unbound type variable.

When inferring the type of the result of a call to a generic function, we need to take care when that type still contains type variables. If the result is a callable, then we should promote it to a generic callable ("generalization"). If the result is not callable, then mypy seems to already give an error message error: Need type annotation for variable, which is sensible.

With such a change, we should almost be able to type check the original program. After all, the more specific type for contextmanager using variadic type variables should only serve to reject ill-typed programs, such as if you passed foo something other than an int.

[rwbarton]

("almost be able to type check" because mypy actually refuses to infer a type for f, and AFAIK you cannot specify one (aside from writing a cast on foo(2)). See #1356.)

[rwbarton]

Actually, type generalization as I described above is technically unsound. (TODO: Insert realistic example here.) The result of the decorator could be a closure that refers to a mutable variable of a type that depends on the type variables of the closure. What matters for soundness of generalization is whether the value is syntactically defined as a function, not whether it has a function type.

For the purposes of mypy it probably makes most sense not to worry about this point and generalize all values of function type.

[JukkaL]

This may require fixing #603 first, which is not scheduled until 0.4.x. If that's the case, perhaps we should postpone this until 0.4.x?

[gvanrossum]

Maybe, or we could move #603 up, or we could make both stretch goals for 0.4. It's really up to Reid.

[JukkaL]

Moved to 0.4.x.

[JukkaL]

This is still happening, even though mypy now special cases type inference for contextmanager. The signature of foo has a type variable, but it's not treated as a generic function (the type is def (x: T'-1) -> contextlib.GeneratorContextManager[T'-1]).

[gvanrossum]

Are you saying this is a bug in the plugin for contextmanager?

[ethanhs]

The provided example works on 0.540, so I am closing this.

[ilevkivskyi]

@ethanhs The underlying issue is still not fixed however:

def transform(f: Callable[..., T]) -> Callable[..., Iterator[T]]:
    ...

def func(x: T) -> T:
    ...

reveal_type(transform(func))  # Revealed type is 'def (*Any, **Any) -> typing.Iterator[T`-1]'

The last line shows that transform(func) is not generic and has an unbound type variable in it. The same happens if one uses transform as a decorator. I think this works for contextmanager because we have a plugin.

I therefore prefer to keep this issue open, since it has the discussion and a possible solution by Reid.

[ilevkivskyi]

#1515 contains another important example of this bug in stdlib stubs.

[ilevkivskyi]

#3193 shows another example in stdlib.

[ilevkivskyi]

Related #1506 and #1507 (I am not sure we need to keep all three open)

[ssbarnea]

I think it worth keeping this open until mypy documentation will contain an example on how to use typing with a simple @contextmanager, like one that takes care of chdir maybe?

[JukkaL]

Lowering priority since arguably the most common case is now supported by using ParamSpec:

from typing import Callable, TypeVar, Iterator
from typing_extensions import ParamSpec

T = TypeVar("T")
P = ParamSpec("P")

def transform(f: Callable[P, T]) -> Callable[P, Iterator[T]]:
    ...

def func(x: T) -> T:
    ...

reveal_type(transform(func))  # Revealed type is 'def [T] (x: T`-1) -> typing.Iterator[T`-1]'