Make Callable more flexible

[gvanrossum]

We keep hearing requests for a way to declare a callback with either specific keyword arguments or just optional arguments. It would be nice if we could come up with a way to describe these.

Some suggestions:

• a magical decorator (@JukkaL)
• a way to point to an existing function as a template (@guido)

[ilevkivskyi]

I could add another option (actually rather a variant of the second option) of allowing a signature object to be used, as here:

from inspect import signature

def func(x: int, *, opt: str = 'full') -> int:
    ...

sign = signature(func)

def another_func(callback: Callable[sign]) -> None:
    ...

In this way, a user could either use the above form (pointing to existing function as a template) or construct an arbitrary signature in a programmatic way:

from inspect import Signature, Parameter

sign = Signature([Parameter('x', Parameter.POSITIONAL_OR_KEYWORD, annotation=int),
                  Parameter('opt', Parameter.KEYWORD_ONLY, default='full', annotation=str)],
                  return_annotation=int)

MyCallbackType = Callable[sign]

[gvanrossum]

What's the advantage of using Signature() over a template function? It would require mypy to learn a lot of stuff about the Signature and Parameter API, and it would probably encourage people to mistakenly think that mypy understands dynamically constructed signatures (it would only understand your example because all the arguments are compile-time constants). Requiring an actual function leverages mypy's existing understanding of function definitions, and makes it clear(er) that the template must be statically understood. I don't think the worry that people might accidentally try to call the template should be a big concern.

[JukkaL]

There are a few additional things to consider (the latter 2 only apply to @gvanrossum 's approach) that we've discussed offline:

1. What if we don't care about the argument names? We could use a double underscore prefix for an argument name that shouldn't be considered part of a signature.
2. How would we write a template function in Python 2, as ... is not valid there? Just using pass is an option.
3. How does a type checker know not to complain about a missing return statement (for functions that don't return None) in a template function? Looking at whether the function is used in Callable[sig] anywhere in a file could work but feels pretty magical. Alternatively, ... as a function body in Python 3 could mark a function as something that can't be called. However, this wouldn't trivially extend to Python 2.

[gvanrossum]

(1) the double underscore sounds fine.
(2), (3) Use raise NotImplementedError().

[ilevkivskyi]

I agree that allowing Signature will make people mistakenly think that mypy understands dynamically constructed signatures, while using templates clarifies that they are static. I also like the NotimplemetedError idea.

If only templates are going to be supported I don't think that a template should be wrapped in some kind of special Signature type or a dummy decorator. If you agree, then it looks like the only thing that should be changed is to allow Callable to be indexed with only one argument that is a function

def template(__x: int, *, option: str='verbose') -> int:
    raise NotImplementedError()

def func(callback: Callable[template]):
    ...

The remaining question is does Callable need to support only function templates, or an arbitrary callable? (for example a class, mypy could already deduce the signature from all present constructors)

[ilevkivskyi]

@gvanrossum I have noticed here
python/mypy#1886 (comment)
another interesting use case for Callable that you proposed.

Indeed, it is quite common for decorators to preserve all the variables (independently of their types) and only change the type of the return value of a function. In the mentioned comment, it was proposed to allow yet another use of Callable with two arguments, where the first argument is not a list, not an ellipsis, but a type variable (such type variable representing types of all variables is very similar to type varargs proposed in #193).

It looks like there is an agreement on the initial issue (allowing single argument use for Callable with template functions), please correct me if I am wrong. If this is the case, then there are two options:

1. We close this issue, I make corresponding PRs to python/peps and python/typing, and open another issue for even more flexible behavior of Callable
2. We continue discussion here, until there is an agreement on all additional use cases that Callable needs to support.

[sixolet]

I think I have a possible solution to this problem. It draws on the notion of a variadic generic we've been discussing over here: #193 (comment) (link is to the closest-to-complete proposal as a comment, but the entire thread is worth reading, if you're getting there from this thread). Some of what I'm saying here has been already stated in this thread; I may be restating some of it to lay out a proposal in a way I hope is understandable.

There are three problems we need to solve, if we want to be able to describe higher-order functions in enough generality to be able to specify the type of most decorators:

1. Argument lists in Python function signatures are more complicated than just a sequence of types. Arguments have names (usually), and what mypy calls kinds (a kind is one of positional|optional|star|star2), alongside their type. Names are important because callers can specify values by formal argument name at the call site. Kinds are important because they determine the rest of how actual arguments are mapped to formal arguments. We need a way to talk about all these things in Callable types.
2. If we can talk about arguments (with names and all) in Callable types, we also need a way to transfer information about these arguments between callable types. If we were only carrying type information, a TypeVar would be sufficient. Since we need to carry name and kind information as well, we need an extension to the idea of a TypeVar. In this proposal I'll call it an ArgVar.
3. To specify the operations most decorators perform, it's not only necessary to specify particular arguments, it's necessary to specify arbitrary sequences of arguments in the input, and talk about how those entire sequences appear in the output of a function.

1. Specifying complete arguments

To talk about functions with full accuracy in our type system, we need our type system to be able to accurately spell their signatures. The design space here has a number of degrees of freedom, but here is one possibility, described by example:

# type: Callable[[Arg('a')[int], Arg('b')[int]], int]
def add(a: int, b: int) -> int:
    return a + b

# type: Callable[[Arg('a')[int], StarArg('more_ints')[int]], int]
def add(a: int, *more_ints: int) -> int:
    sum = a
    for i in more_ints:
        sum += i
    return sum

# type: Callable[[Arg('a')[str], StarArg('args')[str], KeywordArg('kwargs')[str]], str]
def sprintf(fmt: str, *args: str, **kwargs: str) -> str: ...

T1 = TypeVar('T1')
T2 = TypeVar('T2')
# type: Callable[[Arg('a')[T1], Arg('b')[T2]], Tuple[T1, T2]]
def two_things(a: T1, b: T2) -> Tuple[T1, T2]:
    return (a, b)

T = TypeVar('T')
# type: Callable[[Arg('a')[T], Arg('b')[T], OptArg('c')[T]], Tuple[T, ...]]
def two_or_three_things(a: T, b: T, c: T = None) -> Tuple[T, ...]:
    if c is not None: 
        return (a, b, c)
    return (a, b)

The simple way (without arg names and such) of specifying a Callable type results in a callable without argument names, and all arguments positional.

Note that for types T1, T2, and R, Callable[[Arg('a')[T1], Arg('b')[T2]], R] is a subtype of Callable[[T1, T2], R]. To prove this to yourself, consider: two_things(b=1, a=3).

2. Capturing arguments for use in higher-order functions

We're used to using type variables to capture types in our function signatures, and show how the various argument and return types of a function relate to each other:

R = TypeVar('R')
T1 = TypeVar('T1')
T2 = TypeVar('T2')

# Yes, I know this example is silly
def switch_args(f: Callable[[T1, T2], R]) -> Callable[[T2, T1], R]:
    def ret(a, b):
        return f(b, a)
    return ret

But for higher-order functions, that's not quite enough! We need to be able to capture the argument kinds and names, too.

def cheer(name: str, punctuation: str) -> str:
    return "Hooray for %s%s" % (name, punctuation)

switch_cheer = switch_args(cheer)
switch_cheer("!!!", "Sally") # "Hooray for Sally!!!"
switch_cheer(name=Joseph, punctuation="?") # THIS IS AN ERROR

Let's have a new idea, similar to a TypeVar, but this one captures kinds and names. Call it an ArgVar.

R = TypeVar('R')
A1 = ArgVar('A1')
A2 = ArgVar('A2')

def switch_args(f: Callable[[A1, A2], R]) -> Callable[[A2, A1], R]:
    ... # The implementation of this particular example preserving names is quite tricky.  
        # It is possible, but this margin blah blah you know the drill.  It involves the inner
        # function using `*args` and `**kwargs`, and I don't think there's an easier way

def cheer(name: str, punctuation: str) -> str:
    return "Hooray for %s%s" % (name, punctuation)

switch_cheer = switch_args(cheer)
switch_cheer("!!!", "Sally") # "Hooray for Sally!!!"
switch_cheer(name=Joseph, punctuation="?") # "Hooray for Joseph?"

3. Sequences of arguments.

Steps 1. and 2. are still not enough. We need to talk about arbitrary sequences of arguments. That's where the whole variadic thing comes in, as presaged above in the link to the other thread (running out of time at the computer here, so this section will be a little sketchier for now).

As = ArgVar('As', variadic=True)

def cast_to_int_decorator(f: Callable[[As, ...], SupportsInt]) -> Callable[[As, ...], int]:
    def ret(*args, **kwargs):
        return int(f(*args, **kwargs))
    return ret

Commentary

• Like the variadic type variable proposal, the typechecking of the decorator bodies would be made possible with liberal use of Any substitution. The goal would be to require extra care on the part of the decorator writer, to the benefit of being able to typecheck all use sites of the decorator.
• This is a pretty complicated proposal. I'm not sure we should do it, but I think it does solve the problem, and produces all the power you'd need.

I have to run now, but I'm happy to expand on any part of this that's unclear when I get back to a computer.

[ilevkivskyi]

@sixolet Thanks! Your idea looks interesting, but the proposal is quite complicated indeed. However, I could not propose anything simpler. I don't think that we need to support all the possible situations/use cases, but rather have a simple proposal that covers some typical situations with decorators (like capturing all arguments completely).

I am not yet sure what part of this we need. There are two questions that could be discussed at this point:

• What are the relation between TypeVar and ArgVar, should the latter be subclass of the former? Will ArgVar accept the bound keyword? (probably yes) Will it also accept covariant etc? (probably no)
• How to decorate methods of generic classes? Maybe there should be some way to wrap a TypeVar inside an ArgVar? (like A = ArgVar('A', typevar=T))

[sixolet]

Cool! I hadn't thought of putting TypeVar inside ArgVar. Being able to put a type variable inside an arg variable could help in spelling things like "these two arguments must be the same type, which must be the same as the return type, but can have different name and/or kind" but I'm not clear on how it helps decorate methods of generic classes. Does the decorator have to care about the fact that the class is generic, or can it just accept the arguments of the function it's passed?

If we can put a type var into an arg var, there's no reason not to put a specific known type in there too, for when you want to capture name/kind info, but you know the type -- you could have ArgVar('I', type=int) or ArgVar('T_arg', type=T). I'd love to see an example of a decorator that needs such a thing, if you have one.

While considering your comment this morning, I thought for a moment you'd narrowed down the two-and-a-half new concepts in my post to two, because of the conceit that if you put a variadic TypeVar into an ArgVar the ArgVar would itself just become variadic naturally, but I don't think it's quite so pleasant -- it's possible the natural meaning (without making it a special case) of that composition would be to have a series of arguments, each with the same name and kind, but a different type variable binding (which would be useless!). Instead, you want your series of arguments to have a series of ArgVar, A_1 through A_n, each composed over T_1 through T_n, and allowing themselves separate names and kinds. That's a meaning that we could give to "you put a variadic type var into an arg var" but it's got to be a bit of a special case.

One thing that I don't like about putting type variables into argument variables is that it "hides" the type vars in the actual signature of the callable -- ideally you'd want them all visible. Let's say A1 and A2 are both based on R -- when you write Callable[[A1, A2], R] you'd probably rather have some way of knowing that the result type is the same as the argument types.

Here's an idea that fixes the hidden-type-variable problem: What if ArgVar('A') creates something that knows how to capture name and kind, but needs further parameterization by type (and, as above, when you parameterize it by a variadic type, the name-and-kind capture magically becomes variadic too). Then a function that takes two (named) arguments of the same type and produces that type becomes Callable[[A1[R], A2[R]], R]. If you just wrote Callable[[A1, A2], R], then the inputs would be unrelated to the output, and that would be equivalent to writing Callable[[A1[Any], A2[Any]], R]. To get the previously proposed sense of an ArgVar that has the type built in, you'd write:

T = TypeVar('T')
A = ArgVar('A')
A_T = A[T]

And then Callable[[A_T], T] would be equivalent to the hidden-type-in-your-arg-var world above.

[sixolet]

I'd previously thought that ArgVar is a subtype of TypeVar, but writing the comment above convinced me otherwise. They're different things. In the last structure I proposed above, the ArgVar doesn't even have a TypeVar necessarily -- every ArgVar gets a type parameter, the default one is Any (just like any other parameterized type), and you can give it a type variable if you like.

[sixolet]

As for simplicity:

I think the vast majority of use cases want "please just capture all the arguments of this function". To that end, I think the typing module should export AllArgs, defined like so:

Ts = TypeVar('Ts', variadic=True)
A = ArgVar('A')
AllArgs = Expand[A[Ts]]

This uses the last form of ArgVar in the comment above, the one that has to be further parameterized by type, and @gvanrossum's simplified Expand syntax for expanding variadic type variables, since if we want to be able to use it in an exported alias that seems best.

That way you can write the majority of decorator cases only having the vague idea that AllArgs means "some series of arguments, I don't care what". Hell, all the ink I spilled above could just end up as some kind of formalized definition of what AllArgs "really means", and the only thing that's actually part of the API is AllArgs.

from typing import AllArgs, TypeVar
R = TypeVar('R')

def add_initial_int_arg(f: Callable[[AllArgs], R]) -> Callable[[int, AllArgs], R]: 
    ...

[ilevkivskyi]

AllArguments is a good idea. But, maybe call it OtherArgs, since people sometimes want not only preserve, or add arguments, but also remove/replace some of them. Examples:

from typing import OtherArgs, TypeVar, Callable
R = TypeVar('R')

def change_ret_type(f: Callable[[OtherArgs], R]) -> Callable[[OtherArgs], int]: ...

def add_initial_int_arg(f: Callable[[OtherArgs], R]) -> Callable[[int, OtherArgs], R]: ...

def fix_initial_str_arg(f: Callable[[str, OtherArgs], R]) -> Callable[[OtherArgs], int]:
    def ret(*args, **kwargs):
        f('Hello', *args, **kwargs)
    return ret

It looks like your proposal also covers the last example, maybe this is exactly what we need? @sixolet what do you think?

[sixolet]

Yeah OtherArgs is a better name than AllArgs. Another possibility is CapturedArgs? But yes, the concept can be used to add arguments, remove arguments, or pass arguments unchanged.

The places you need the more powerful ArgVar idea is to reorder, restrict, change the type of, or otherwise mess with arguments in a deeper way. I haven't yet done a survey of decorators, but I suspect this is super uncommon, and you can mostly just write the decorators using OtherArgs (or whatever we end up calling it)

[sixolet]

Here's a challenge: can we formalize the meaning of OtherArgs without reference to an ArgVar, or without reference to a variadic TypeVar, or without either of those concepts? If so, is that cleaner?

[ilevkivskyi]

I think that some of the less common cases of decorators could be typed with function templates discussed earlier in this thread #239 (comment)
I could speculate that function templates and OtherArgs will together cover vast majority of use cases for Callable (callbacks, decorators, etc.)

I think it could be possible to formalize OtherArgs without reference to ArgVar, but we need variadic TypeVar (in general, I think variadic type variables is a useful concept on its own).

For example, a short definition: OtherArgs is equivalent to

Ts = TypeVar('Ts', variadic=True)
OtherArgs = Expand[Ts]

with an exception that expanded type variables T_1, T_2, ... capture not only types but also kinds and names of function arguments.

@gvanrossum what do you think about this idea?

[sixolet]

Hmm. Personally I find explicitly spelling name and kind in a signature (a la OptArg('foo')[int] or similar) to be more pleasant than using a template function. I think that's because I like the idea that the ways you have to write function signatures in the type language are all compatible with each other, rather than one weird-but-more-powerful way you write a function signature when you need it for a decorator (OtherArgs) and another weird-but-more-powerful way you write a function signature when you need to specify a callback (templates) but they're not compatible.

Unless they are compatible and I am failing to see how.

[sixolet]

The other thing to do if we do go the way of template functions is to be very careful about documentation -- "template" is a word that means something quite like "generic" in C++ land, and we run a risk of confusing people.

[ilevkivskyi]

@sixolet I agree that the name "template function" could be changed to avoid confusions, maybe "stub function"? In fact, stub functions are much less verbose than full "systematic" signatures (see discussion above, and this is a relatively simple function, real life functions will have verbose signatures even with your shorter syntax). Moreover, people could already have functions to use as stub functions in their code. I think there is already certain agreement on "stub functions".

Still, the OtherArgs depends on #193, so that I propose to first focus on that issue, and then come back here.

[sixolet]

Made a self-contained proposal for only the part about specifying argument names and kinds here: #264

[gvanrossum]

How does this issue differ from #264?

[ilevkivskyi]

How does this issue differ from #264?

This issue is a parallel discussion more focused on template-based syntax and variadic-type-variable-like approach (OtherArgs). I think we can keep only one issue open (I propose #264 since we went with the syntax proposed there) until the actual PEP is written and accepted.

[rominf]

OK, sorry if I misread this issue comments, but how do I annotate these functions in one annotation:

def f0(x: int) -> int:
    pass

def f1(x: int, y: int = 0) -> int:
    pass

?

I want something like:

Callable[[int, [int]], int]

[JelleZijlstra]

You can now use a callable protocol (https://mypy.readthedocs.io/en/latest/protocols.html#callback-protocols).

[rominf]

@JelleZijlstra, thank you! That's what I needed.