Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Generalize Callable to be able to specify argument names and kinds #264

Closed
sixolet opened this issue Aug 19, 2016 · 48 comments
Closed

Comments

@sixolet
Copy link

@sixolet sixolet commented Aug 19, 2016

Right now you can specify callables with two patterns of arguments (shown here by example):

  • Callable[..., int] takes in any arguments, any number.
  • Callable[[int, str, bool], int] takes in a predetermined number of required positional arguments, none of which have names specified.

These don't cleanly match the actual types of callable objects in Python. Argument names, whether arguments are optional, and whether arguments are *args or **kwargs do affect the type of a callable. We should be able to spell these things in the type language. Doing so would enable us to correctly write the types of callback functions, for example.

Callable should take two arguments: an argument list and a return type. The return type is exactly as currently described in PEP484. The argument list is either:

  • ..., indicating the function can take any arguments at all.
  • Square brackets around a comma separated series of argument specifiers (argspec for short), indicating particulars about the functions arguments

An argument specifier is one of:

  • A bare type TYP. This has the same meaning as Arg(TYP)
  • Arg(type, name=None), indicating a positional argument. If the name is specified, the argument must have that name.
  • OptionalArg(type, name=None), indicating an optional positional argument. If the name is specified, the argument must have that name. (alternate name possibility OptArg(type, name=None)
  • StarArg(type), indicating a "star argument" like *args
  • KwArg(type), indicating a "double star argument" like **kwargs. (an alternate name here would be Star2Arg(type).

The round parens are an indication that these are not actual types but rather this new argument specifier thing.

Like the rules for python function arguments, all positional argspecs must come before all optional argspecs must come before zero or one star argspecs must come before zero or one kw argspecs.

This should be able to spell all function types you can make in python by defining single functions or methods, with the exception of functions that need SelfType to be properly specified, which is an orthogonal concern.

Some statements I think are true:

  • Callable[[Arg(T1, name='foo'), Arg(T2, name='bar')], R] is a subtype of Callable[[T1, T2], R]
  • Callable[[T1, OptionalArg(T2)], R] is a subtype of Callable[[T1], R]
  • Callable[[StarArg(T1)], R] is a subtype of Callable[[], R] and is also a subtype of Callable[[T1], R] and is also a subtype of Callable[[T1, T1], R] and so on.
  • Callable[[T1, StarArg(T1)], R] is a subtype of Callable[[T1], R] and is also a subtype of Callable[[T1, T1], R] and so on, but is not a subtype of Callable[[], R]
  • If we want to be able to spell overloaded function types we'll need a specific way of combining callable types in the specific way overloading works; this proposal doesn't address that.
@JukkaL
Copy link
Contributor

@JukkaL JukkaL commented Aug 19, 2016

Another thing to consider is keyword-only arguments in Python 3. Keyword-only arguments can be optional or non-optional. Maybe use Arg(t, name='foo', keyword_only=True), and similarly for OptionalArg?

@sixolet
Copy link
Author

@sixolet sixolet commented Aug 19, 2016

@JukkaL Yep seems good.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Aug 19, 2016

@sixolet Thankyou for your proposal!
I proposed a similar idea in #239 based on inspect.Signature and Guido rejected it in favor of function stubs. His argument (and I agree with him) is readability. Compare:

def cb_stub(x: T, *args, option: str='verbose') -> int:
    raise NotImplementedError

def func(callback: Callable[cb_stub]):
    ...

and

Args = [Arg(T, name='x'), StarArg(Any), OptionalArg(str, name='option')]

def func(callback: Callable[Args, int]):
    ...

I think the first one is more readable and allows to quickly grasp the function signature. Also it is not clear from your proposal how to type annotate a decorator that preserves the types of all arguments in *args (propbably you also need to discuss how this interacts with variadic generics etc).

I would like to reiterate, function stubs together with OtherArgs discussed in #239 cover vast majority of use cases, so that I think practicality beats purity here.

@JukkaL what do you think about the magic (variadic) type variable OtherArgs? I copy here use cases in decorators from previous discussions:

from typing import OtherArgs, TypeVar, Callable
R = TypeVar('R')

def change_ret_type(f: Callable[[OtherArgs], R]) -> Callable[[OtherArgs], int]: ...

def add_initial_int_arg(f: Callable[[OtherArgs], R]) -> Callable[[int, OtherArgs], R]: ...

def fix_initial_str_arg(f: Callable[[str, OtherArgs], R]) -> Callable[[OtherArgs], int]:
    def ret(*args, **kwargs):
        f('Hello', *args, **kwargs)
    return ret
@sixolet
Copy link
Author

@sixolet sixolet commented Aug 19, 2016

Regarding OtherArgs, I think it's separable and so we should consider it separately; being able to spell argument names and kinds is useful in the absence of such a thing. I'll write it up separately once we know what the fate of variadic type variables is likely to be (since the exact definition/meaning of OtherArgs depends on whether we can explain it with variadic typevars).

@sixolet
Copy link
Author

@sixolet sixolet commented Aug 19, 2016

It's true that this proposal is similar to your proposal based on inspect.Signature, but it differs in a few important ways:

  • It's more concise and readable (I know, a "simple matter of syntax", but I think it actually makes a difference here). It's concise enough that you're likely to spell the whole type of a callable in just one line.
  • It's exactly the same as the current syntax in the common/simple case. This also aids readability.
  • It's specialized to the purpose of specifying what arguments are like, instead of being intended for runtime introspection and dynamic function creation/metaprogramming. "Do one job well" is UNIX philosophy, not python philosophy, but I think it applies here.
@JukkaL
Copy link
Contributor

@JukkaL JukkaL commented Aug 19, 2016

@ilevkivskyi I agree with @sixolet that OtherArgs should be discussed separately from this. This proposal (or the alternative proposed syntax) would be useful as such. Callbacks with other than positional arguments have been a pretty common mypy feature request.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Aug 19, 2016

OK, if you want to proceed with this, I would propose to try to make it more concise. Here are some tweaks, apart from already mentioned OpArgs:

  • make name a first argument,
  • it could be positional,
  • no one probably will use Arg(T) it is just equivalent to T, so that better make the type optional:
Arg('x', T)
Arg('y') # same as Arg('y', Any)
  • probably use shorter keyword for keyword only kw_only=True,
  • allow omitting kw_only in obvious places, e.g., after StarArg,
  • missing type in StarArg and KwArg should also mean Any (like actually everywhere in PEP 484, missing type means Any).

With all these tweaks it will look like this:

def fun1(x: List[T], y: int = 0, *args, **kwargs: str) -> None:
    ...

has type

Callable[[Arg('x', List[T]), OptArg('y', int), StarArg(), KwArg(str)], None]

Second example:

def fun2(__s: str, __z: T, *, tmp: str, **kwargs) -> int:
    ...

has type

Callable[[str, T, Arg('tmp', str, kw_only=True), KwArg()], int]

It looks more concise, and still readable. What do you think?

@sixolet
Copy link
Author

@sixolet sixolet commented Aug 19, 2016

@ilevkivskyi I like most of that.

The only bit I think I disagree with is

allow omitting kw_only in obvious places, e.g., after StarArg

And that's because I think in this particular case consistency is more important than brevity.

Huh. Can we omit the parens when they're not being used for anything? Why not

Callable[[str, T, Arg('tmp', str, kw_only=True), KwArg], int]
@sixolet
Copy link
Author

@sixolet sixolet commented Aug 19, 2016

(Answering my own question, the round parens remind us it's an argspec not a type. Right.)

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Aug 19, 2016

@sixolet OK
Now that you have +1 from Jukka, you should get an approval from Guido and then add this to the PEP and typing.py.

By the way I was thinking a bit about how this will interoperate with variadic generics, and I think the solution is simple: StarArg and KwArg should be allowed to accept them as an argument and that's it. Then you could easily type a decorator with something like

As = TypeVar('As', variadic=True)
Ks = TypeVar('Ks', variadic=True)

Input = Callable[[int, StarArg(As), KwArg(Ks)], None]
Output = Callable[[StarArg(As), KwArg(Ks)], None]
Deco = Callable[[Input], Output]

Moreover OtherArgs could be easily defined as [StarArg(As), KwArg(Ks)], so that one can write (unpacking just unpacks two items here):

Input = Callable[[int, *OtherArgs], None]
Output = Callable[*OtherArgs], None]

But now it is probably not needed in the specification. So that now I give your proposal a solid +1.

@elazarg
Copy link

@elazarg elazarg commented Oct 21, 2016

I don't understand what's wrong with something like Callable[lambda a: int, *, b: str =...:...]

It will be much easier for type checkers to learn, and the meaning is obvious. Why invent new ways of saying the same thing?

@JukkaL
Copy link
Contributor

@JukkaL JukkaL commented Oct 21, 2016

It's not syntactically valid. Lambdas can't have type annotations.

@sixolet
Copy link
Author

@sixolet sixolet commented Oct 25, 2016

@Lazarg Also, we should be careful to avoid confusing people regarding "how do you write a function" vs "how do you write the type of a function"

@elazarg
Copy link

@elazarg elazarg commented Oct 25, 2016

The type (the interface) of the function is the thing that appears right after its name - the signature.

@elazarg
Copy link

@elazarg elazarg commented Oct 25, 2016

Oh. Consequences of using : instead of => ?

That's very unfortunate.

@elazarg
Copy link

@elazarg elazarg commented Oct 26, 2016

How about Callable[lambda x, *y, **z: {x:int, y:str, z:str}] ?

@sixolet
Copy link
Author

@sixolet sixolet commented Oct 27, 2016

@elazarg I suspect we'll be better off and have a clearer language if we steer clear of situations where people can confuse an object of a type (a function) with the type itself (the callable). Here it's not exactly what you're suggesting, though, I understand. You're providing a function as the type parameter to a Callable type to demonstrate what type the callable will be. It might be even more confusing, especially if the function that you're providing as a type parameter produces a very special kind of dict object, and that has nothing to do with the return type of the Callable.

JukkaL added a commit to python/mypy that referenced this issue May 2, 2017
…thing you can do (#2607)

Implements an experimental feature to allow Callable to have any kind of signature an actual function definition does.

This should enable better typing of callbacks &c.

Initial discussion: python/typing#239
Proposal, v. similar to this impl: python/typing#264
Relevant typeshed PR: python/typeshed#793
@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented May 3, 2017

I think we should turn the finalized proposal into a new PEP, rather than amending PEP 484. Who wants to draft it?

@sixolet
Copy link
Author

@sixolet sixolet commented May 3, 2017

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented May 3, 2017

PEP 1 describes the general process: https://www.python.org/dev/peps/pep-0001/
PEP 12 is a template: https://www.python.org/dev/peps/pep-0012/

I'm not sure if these are 100% up to date, but basically you should create a PR for https://github.com/python/peps and assign it PEP number 9999; a reviewer will assign a number before merging the initial draft. Note that the initial draft usually gets merged as soon as it conforms to the form guidelines -- after that the discussion starts, typically with a post to the python-dev mailing list.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented May 4, 2017

@sixolet
You know that I was quite cold to this syntax (I prefer prototype based syntax), so that here I have some ideas that could make the future PEP look better from my point of view:

  • We need a good name for this. Since Python already has several mini-languages (like format mini-language), maybe we could call this callable type mini-language, or function specification mini-language.
  • I would prefer if the PEP will encourage using type aliases (including generic aliases):
    Func = Callable[[T, VarArg(S), KwArg(S)], T]
    
    def apply_str(fun: Func[T, str]) -> T:
        ...
    def apply_int_and_bytes(fun: Func[int, bytes]) -> None:
        ...
  • I would prefer if the PEP will recommend multi-line-with-comments syntax for particularly complex cases:
    F = Callable[[int,               # number of processes to run
                  Arg(int, 'wait'),  # waiting time in msec
                  DefaultArg(str, 'foo'),
                  VarArg(int),
                  NamedArg(int, 'bar'),
                  DefaultNamedArg(int, 'baz'),  # some other comment
                  KwArg(int)],
                 int]  # return code of last process
  • It would be nice to have a detailed explanation about why other options (status quo, prototype based, variadic generics) were rejected/postponed.
@mjpieters
Copy link

@mjpieters mjpieters commented May 25, 2017

Are there any plans to make it possible to use a placeholder (like a typevar) for the whole argument list? That's helpful for decorators where the wrapped function signature is preserved; contrived e.g.:

A = TypeVar('A')      # arguments generic
RT = TypeVar('RT')  # Return type 
def singleton_list(func: Callable[A, RT]) -> Callable[A, List[RT]):
    def wrapper(*args, **kwargs):
        return [func(*args, **kwargs)]
    return wrapper

I used TypeVar('A') here to represent the argument list; this is probably violating a whole slew of principles, but it should serve to illustrate the idea nicely.

@JelleZijlstra
Copy link
Member

@JelleZijlstra JelleZijlstra commented May 25, 2017

I proposed something quite similar in python/mypy#3028. I think we want to move forward with something like this, but it needs somebody to actually do the work.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Mar 27, 2018

Yes, an overload can be given where a union is expected. But note that if someone will give a union there, it will crash at runtime (while being uncaught statically).

The problem is that union is much wider type than an overload, each element of the union will be actually better.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Mar 28, 2018

Another (simpler) interesting example is the ambiguity about what should this mean:

def fun() -> Callable[[T], T]:
    ...

Currently this creates a non-generic function that returns a generic function. This is probably the right thing to do, but this is too implicit, for each given signature mypy tries to guess what the author of the function means (and I am not sure there are any strict rules). It would be more unambiguous if one could write:

def same(x: T) -> T:
    ...
def fun() -> Callable[same]:
    ...

It is clear now that fun is not intended to be generic.

@howinator
Copy link

@howinator howinator commented Jul 10, 2018

What's the status of this? Has anyone taken the initiative to draft a PEP? If none of the core devs have the time, I can distill the discussion here down into a PEP.

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Jul 10, 2018

@howinator
Copy link

@howinator howinator commented Jul 10, 2018

Wait you're saying I have to have original thought? :P

But seriously, I've dealt with this lack of expressiveness across a couple different projects now, so I do have some thoughts about how this should work. I'll use your "PEP how-to" links above. Thanks!

@devxpy
Copy link

@devxpy devxpy commented Oct 16, 2018

Can we please have something like this?

Callable[[dict, ...], dict]

Which means the first argument to the callable must be a dict, and any number of arguments are accepted after that..


This is quite useful for type annotating wrapper functions.

def print_apples(fn: Callable[[dict, ...], dict]) -> Callable[[dict, ...], dict]:
    
	def wrapper(fruits, *args, **kwargs):
		print('Apples:', fruits['apples'])

		return fn(fruits, *args, **kwargs)

	return wrapper

@print_apples
def get(fruits, key):
	return fruits[key]
@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Oct 17, 2018

@devxpy Note that you can already express lots of things using callback protocols

@Seanny123
Copy link

@Seanny123 Seanny123 commented Nov 6, 2018

@howinator did you end up making this PEP?

@devxpy
Copy link

@devxpy devxpy commented Nov 7, 2018

@ilevkivskyi This does not seem to work with *args and **kwargs.

Am I doing something wrong?

from typing import Callable, Any
from typing_extensions import Protocol



class ApplesWrapper(Protocol):
    def __call__(self, fruits: dict, *args, **kwargs) -> Any: ...


def print_apples(fn: ApplesWrapper) -> ApplesWrapper:    
    def wrapper(fruits: dict, *args, **kwargs):
        print('Apples:', fruits['apples'])

        return fn(fruits, *args, **kwargs)

    return wrapper

@print_apples
def get(fruits: dict, key):
    return fruits[key]
$ mypy test.py
test.py:18: error: Argument 1 to "print_apples" has incompatible type "Callable[[Dict[Any, Any], Any], Any]"; expected "ApplesWrapper"
@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Nov 7, 2018

@Seanny123 Now that we have callback protocols, the PEP will be less valuable (if needed at all). The only missing thing is variadic generics, but this is a separate question, and the problem with them is not who will write the PEP, but who will implement it.

@devxpy Well, mypy correctly says that type of get() is not a subtype of ApplesWrapper, there are many possible calls to ApplesWrapper that will fail on get().

I think what you want is OtherArgs proposed in #239 and also discussed a bit above. This is more about variadic generics which, as I explained above, is a separate issue.

msullivan pushed a commit to python/mypy_extensions that referenced this issue Jan 15, 2019
…thing you can do (#2607)

Implements an experimental feature to allow Callable to have any kind of signature an actual function definition does.

This should enable better typing of callbacks &c.

Initial discussion: python/typing#239
Proposal, v. similar to this impl: python/typing#264
Relevant typeshed PR: python/typeshed#793
@JukkaL
Copy link
Contributor

@JukkaL JukkaL commented May 30, 2019

Should we close this issue now that callback protocols are described in PEP 544 (which was accepted)?

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented May 30, 2019

I seem to be in the minority, but I find callback protocols too verbose, and not very intuitive.

I'd rather have a decorator you can put on a dummy def. E.g. instead of

class ApplesWrapper(Protocol):
    def __call__(self, fruits: dict, *args, **kwargs) -> Any: ...

I'd prefer

@some_decorator
def ApplesWrapper(fruits: dict, *args, **kwargs) -> Any: ...

My reasons are hard to explain -- I find the need to use class, Protocol, __call__ and self for the current solution rather noisy.

@JukkaL
Copy link
Contributor

@JukkaL JukkaL commented May 31, 2019

I agree that the current syntax is overly verbose. On the other hand complex callables don't seem to be required more than occasionally by most programmers. Maybe somebody will come up with a nicer and more concise syntax, though.

@antonagestam
Copy link

@antonagestam antonagestam commented May 31, 2019

To find similarity to another thing within the language, async def, it might be interesting to explore a "def" keyword like below. Since the spec can't have a body, the colon and ellipsis/pass should be omitted in my opinion.

argspec def ApplesWrapper(fruits: dict, *args, **kwargs) -> Any

Or, if we'd like to be even less verbose with an even shorter syntax, making this it's entirely "own" thing.

argspec ApplesWrapper(fruits: dict, *args, **kwargs) -> Any
@Michael0x2a
Copy link
Contributor

@Michael0x2a Michael0x2a commented May 31, 2019

Maybe we could add in some decorator that when present will promote some function into a callable protocol? Basically, make it so that type checkers treat Guido's second example as a shorthand for the first.

The decorator could just be the identity function at runtime. We could also make it literally construct the protocol class if the distinction between a function vs a callable object is important to preserve.

At least for mypy, I feel we could pretty easily implement this via the plugin system.

@jdelic
Copy link

@jdelic jdelic commented May 31, 2019

@antonagestam
I haven't fully thought this through, but right now I like your idea... as it also specifies return types and not just args I would go for

typed def callback(a: str) -> bool:
    ...
# or
typespec callback(...) -> bool:
    ...
# or
typedef callback(...) -> bool:
    ...

instead of argspec.

And then only declare the implementation's intention of fulfilling the contract using a decorator:

@typeimpl(callback)
def mycallback(...) -> bool:
    ...

As the language already introduced class variable annotations and return type annotations I'd find a syntax extension more natural for "first class support" of function types than a decorator syntax that leads to typing being implemented counter-intuitively through a mixture of new language keywords and special code.

@antonagestam
Copy link

@antonagestam antonagestam commented May 31, 2019

@jdelic As for naming of this hypothetical keyword, would signature be even better? I agree that argspec is not an accurate name.

signature Greet(name: str) -> str

(I hope this is not the wrong forum for hypothetical discussions like this)

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented May 31, 2019

Note that such syntactical changes are hard to get accepted -- adding a reserved word to Python requires a PEP, a from __future__ import ... statement, and several releases of deprecation warnings. (That's what it cost to introduce async def and await.)

In contrast, a decorator is trivial to add -- you can just import it.

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Jun 2, 2019

I am fine with the current way. Some reasons:

  • It already works
  • There should be only one way to do it
  • It is not an ad-hoc solution, and therefore is easy to understand after one understood protocols in general
  • Some people don't like camel case in def
  • Finally, IMO the current way looks better for generics. One can easily indicate the binding of type variables explicitly. For example these are two different things:
class One(Protocol):
    def __call__(self, x: T) -> T: ...
class Other(Protocol[T]):
    def __call__(self, x: T) -> T: ...

with the proposed syntax this would require supporting something like:

@some_decorator(bind=(T,))
def Other(x: T) -> T: ...

and add some tricky runtime machinery to allow Other[int]. I would propose to just close this and move on. I think there are more important things to do than minor syntactic improvements for relatively rare use cases.

@ambv
Copy link
Contributor

@ambv ambv commented Nov 11, 2019

I always sided with the line of thinking described by Guido here. I didn't even think we'd need a special decorator for it. I always thought of function objects as "types with callable signatures". I've seen many cases where being able to pass an example function as a type would make things clearer instead of using Callable[].

In practice I don't mind specifying a Protocol with __call__() much but the requirement to add self as the first argument there adds confusion on top of the overall verbosity of this.

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Nov 11, 2019

It's a tough call. In the end I think we should just try to live with the Protocol solution for a while before we decide that it's too onerous. So I'm closing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet