A way to make writing overloads on complex signatures easier

[Daverball]

I've come across this many times where a function has maybe four or five optional arguments that can either be positional or keyword arguments and the return type depends on one or two of the optional arguments in arbitrary positions. Now you're often forced to write at least two versions of every call where the optional argument is specified, to both match the positional and keyword case, since required positional arguments can't follow optional positional arguments.

But for overload matching this requirement shouldn't really matter, the rules for positional arguments still apply, so the type checker knows all the arguments preceding the required argument have to be set as well, if it's called positionally, but may be omitted if the argument is instead specified as a keyword argument, so it can generate the two possible overloads from a single overload itself.

So if we could violate the positional argument rules for specifying overloads we could write the following

@overload
def foo(x: int, y: None = None, z: None = None) -> int: ...
@overload
def foo(x: int, y: None = None, z: Callable[[int], _T]) -> _T: ...

instead of having to write

@overload
def foo(x: int, y: None = None, z: None = None) -> int: ...
@overload
def foo(x: int, y: None, z: Callable[[int], _T]) -> _T: ...
@overload
def foo(x: int, y: None = None, *, z: Callable[[int], _T]) -> _T: ...

If you define further overloads for y that affect the result of the function you can see how this very quickly can get out of hand and you end up with many more overloads than you would like. The obvious solution would be to change the order of arguments or to make them keyword-only, in order to minimize complexity, but you don't always have the luxury of changing the implementation.

Now of course you can't just change the parameter rules for overloads, since they're built into the parser. But a pragmatic workaround would be to add something like ValueRequired as a sentinel value to mark an argument in an overload as required in order to match that overload, but it may either match positionally or by keyword:

@overload
def foo(x: int, y: None = None, z: None = None) -> int: ...
@overload
def foo(x: int, y: None = None, z: Callable[[int], _T] = ValueRequired) -> _T: ...

Or maybe as a generic type so the type checker does not have to special case the default value, similar to NotRequired for TypedDict.

@overload
def foo(x: int, y: None = None, z: None = None) -> int: ...
@overload
def foo(x: int, y: None = None, z: Required[Callable[[int], _T]] = ...) -> _T: ...

[JelleZijlstra]

@srittau suggested a version of this on the mailing list: https://mail.python.org/archives/list/python-dev@python.org/message/7W5RMJVBRN6NE6A3LUP5Y4BMOZKQRYWH/. It didn't really go anywhere, and I'm not sure a PEP to make this change would be accepted, as the use case might be too narrow to justify new syntax.

Your workaround seems reasonable though, that's something we can add to typing. I think it should still need a PEP.

[Daverball]

I figured it already came up at some point, considering how annoying it can get to deal with, but it's a surprisingly difficult issue to search for, so thanks for digging that one out.

But yes, I would have to agree, that for this narrow of an use-case, it would be difficult to advocate for changing the syntax. I probably like the version with Required the most, since it could also be used in a TypedDict(total=False) as a small QoL improvement, when almost every key in a large TypedDict is optional. So it has more than one potential use-case and can be easily understood in tandem with NotRequired.

[JelleZijlstra]

I don't understand your last sentence: it sounds like you're advocating for the syntax of PEP 655, which is already implemented. Unless there's some variant of that syntax that you'd like to add?

I might vote for z: Callable[[int], _T] = Required, reusing the existing Required name but putting it in the default. That's more concise than combining with a ... default and makes it clearer that there is actually no default.

[Daverball]

Oh, I had no idea Required was already a thing, for some reason I thought NotRequired was all by its lonesome.

The reason why I like Required[Foo] = ... a little bit better than Foo = Required, is because it matches TypedDict, even though it makes the default redundant, it also seems easier to implement for type checkers.

[hmc-cs-mdrissi]

My alternate proposal for this kind of situation that leads to messy overload explosion is have type level map that is only sugar for overload. Using your example,

@overload
def foo(x: int, y: None = None, z: None = None) -> int: ...
@overload
def foo(x: int, y: None = None, z: Callable[[int], _T]) -> _T: ...

can be written like,

FooType = TypeMapping({None: int, Callable[[int], _T]: _T})

T = TypeVar('T')
def foo(x: int, y: None = None, z:  T = None) -> FooType[T]: ...

If you want to add a 3rd overload just add it to map as a key. TypeMapping here takes tuple[type, ...] -> type. Number of elements in tuple is fixed for a given type map but just like you can have one length tuple you could also have something like,

FooType = TypeMapping({(int, str): int, (list[int], list[str]): list[int]})

T1 = TypeVar('T1')
T2 = TypeVar('T2')

def foo(x: T1, y: T2) -> FooType[T1, T2]

would mean exact same thing as,

@overload
def foo(x: int, y: str) -> int:
  ...

@overload
def foo(x: list[int], y: list[str]) -> list[int]:
  ...

And FooType is reusable across multiple functions if you have several functions with similar overload patterns.

There is some ambiguity as to default value although default values can only align with keys in map that match default value's type which should usually narrow it down.

[Daverball]

Yeah I have seen your proposal and think it can be an adequate solution for very complex cases, although it may be less readable than overloads in some cases, since you have to keep referring back to the mapping in order to read the signature. But it's certainly better than having to read a dozen overloads, to find the correct one.

The main issue with TypeMapping are default values, type checkers ignore the default value in a signature for overload resolution, so TypeMap would probably require defaults for TypeVar i.e. PEP696 but considering function signatures have been specifically excluded from this PEP due to inherent complexity for the type solver, this may be a no-go.