Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define functools.partial as overloaded function instead of its own class #2878

Merged
merged 3 commits into from May 6, 2019

Conversation

@berdario
Copy link
Contributor

commented Mar 19, 2019

So, I opened this issue 3 years ago, I see that some use cases are now handled, but there're still issues with properly checking simple snippets like:

from functools import partial

def inc(a:int) -> int:
    return a + 1

partial(inc, 1)(1)

(thanks for the all the work in Mypy and Typeshed since then, btw! I haven't touched much Python in the meanwhile, and it's good to see the improvements)

I thought a bit more about what can be done. One possibility could be to define different overloads for __init__ and __call__, like:

_T = TypeVar('_T')
_T2 = TypeVar('_T2')
_OT = TypeVar('_OT')


class partial(Generic[_T, _T2, _OT]):
    @overload
    def __init__(self, func: Callable[[_T], _OT], a: _T) -> None: ... 
    @overload
    def __init__(self, func: Callable[[_T, _T2], _OT], a: _T) -> None: ...
    @overload
    def __init__(self, func: Callable[[_T, _T2], _OT], a: _T, b: _T2) -> None: ...

    @overload
    def __call__(self) -> _OT: ...
    @overload
    def __call__(self, a: _T2) -> _OT: ...

But this would have the same problem with the snippet above, since there'd be no relationship between how the partial had been constructed (which __init__) had been invoked, and its __call__ (the wrong arity for the Callable could be invoked). Also, every partial would have to be parametrized on the number of types of the max arity supported by the stub (even for the types that aren't used)

Ideally, we'd want to "overload the whole class", something like:

@overload
class partial(Generic[_OT]:
    def __init__(self, func: Callable[[_T], _OT], a: _T) -> None: ...
    
    def __call__(self) -> _OT: ...


@overload
class partial(Generic[_T2, _OT]):
    @overload
    def __init__(self, func: Callable[[_T, _T2], _OT], a: _T) -> None: ...
    @overload
    def __init__(self, func: Callable[[_T, _T2], _OT], a: _T, b: _T2) -> None: ...

    @overload
    def __call__(self) -> _OT: ...
    @overload
    def __call__(self, a: _T2) -> _OT: ...

but this obviously cannot work.

This is when I thought of defining partial's type as a simple function.

Ideally we'd want to return a different class (with different type parameters) in each case. I don't think we'd be able to properly type that class, since the only 2 type constructors that I've seen with an arbitrary number of type parameters are Tuple and Callable (besides other stuff like Generic), and neither can be subclassed (though that's a detail that might change). It'd probably have to be defined as a SpecialForm

So, that's why I settled on it simply returning a Callable.

This is obviously not ideal, since this way we'd lose the ability to type-check inspections of func, args, keywords. But when people use functools.partial, that's to obtain some callables and call them, so I think the most common use case can actually be covered with the overloading, at the expense of these 3 other attributes.

I picked a maximum arity of 5, as that's also the arity used to type map (above it'll just rely on Any). This also won't help in the case of partial application with keyword arguments.

I don't particularly like this approach, and I'm sure that it had already been thought of when first implementing the current type stub for functools.partial, but I thought worthwhile to open this PR since I haven't seen anyone else proposing this.

On the other hand, I think this approach can also be justified by the other precedents we have, like itertools.cycle and itertools.tee, which are implemented as returning a custom type and yet are defined in Typeshed as simply being a function that returns an abstract type like Iterator (even if the actual implementation provides a couple more implementation details, namely {'__setstate__'} and {'__setstate__', '__copy__'} respectively)

Another advantage for doing things this way, is that it wouldn't require any special type checker implementation, so the stub could improve the current checking in both mypy and pytype.

Again: I'm not particularly fond of this change, so I understand if the current approach will continue to be preferred.

@berdario berdario force-pushed the berdario:overloaded_partial branch from 4ff51af to 9322a05 Mar 25, 2019

@JelleZijlstra
Copy link
Collaborator

left a comment

Thanks for your PR, and sorry for the long wait!

I think this approach makes sense, but just have one comment.

stdlib/2/_functools.pyi Outdated Show resolved Hide resolved

@JelleZijlstra JelleZijlstra merged commit e45f443 into python:master May 6, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@JukkaL

This comment has been minimized.

Copy link
Contributor

commented May 13, 2019

Note that this PR generates a small number of false positives in our internal repositories since keyword arguments can no longer be used with the result of partial(). This is probably still worth it, as this seems to work in the vast majority of cases.

@jekbradbury

This comment has been minimized.

Copy link

commented May 29, 2019

This change also causes false positives in situations where functools.partial is used as a class (e.g. isinstance or subclassing). Not sure about the right way to handle that.

@JelleZijlstra

This comment has been minimized.

Copy link
Collaborator

commented Jun 20, 2019

With the new mypy release this also caused a couple of new errors in our codebase: two from isinstance() checks and a dozen or so more from keyword arguments. Perhaps it's better to undo this change, unfortunately. To get better types for partial we'll have to add a mypy plugin.

@wkschwartz

This comment has been minimized.

Copy link

commented Jun 20, 2019

This change caused errors in my code base where I had used partial's instance attributes func and keywords.

JelleZijlstra added a commit that referenced this pull request Jun 20, 2019

JelleZijlstra added a commit that referenced this pull request Jun 21, 2019

JelleZijlstra added a commit that referenced this pull request Jun 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.