problems implementing overloads on some flexible class methods

[cjw296]

Hi All,

I eventually managed to release a py.typed version of testfixtures, but to do so I had to give up on a bunch of overloads which I thought helped document intended methods of use:

simplistix/testfixtures@70283c2

I've tried to get my head around this a few times over the last few months:

• How do I type an overloaded function's keyword arguments that may not be specified?
• mypy complaining about calls to a datetime factory #1961

I even ended up logging a mypy bug:

• erroneous complaint that overloaded function implementation does not accept all possible arguments of signature

...and as well as several attempts myself, LLMs have also been struggling, even once the underlying typing was sorted out:

• Codex attempt 1
• Codex attempt 2
• Claude attempt 1
• Claude attempt 2

At a high level, I'm essentially looking to revert simplistix/testfixtures@70283c2 but with functioning type annotations, essentially providing these three calling patterns for the factory:

@overload
def mock_datetime(
        tzinfo: TZInfo | None = None,
        delta: float | None = None,
        delta_type: str = 'seconds',
        date_type: type[date] = date,
        strict: bool = False
) -> type[MockDateTime]:
    ...


@overload
def mock_datetime(
        year: int,
        month: int,
        day: int,
        hour: int = ...,
        minute: int = ...,
        second: int = ...,
        microsecond: int = ...,
        tzinfo: TZInfo | None = None,
        delta: float | None = None,
        delta_type: str = 'seconds',
        date_type: type[date] = date,
        strict: bool = False
) -> type[MockDateTime]:
    ...


@overload
def mock_datetime(
        default: datetime,
        tzinfo: TZInfo | None = None,
        delta: float | None = None,
        delta_type: str = 'seconds',
        date_type: type[date] = date,
        strict: bool = False
) -> type[MockDateTime]:
    ...


@overload
def mock_datetime(
        default: None,  # explicit None positional
        tzinfo: TZInfo | None = None,
        delta: float | None = None,
        delta_type: str = 'seconds',
        date_type: type[date] = date,
        strict: bool = False
) -> type[MockDateTime]:
    ...


def mock_datetime(
        *args,
        tzinfo: TZInfo | None = None,

...and then these overload patterns for the methods for manipulating the queue of upcoming mock datetimes:

class MockDateTime(MockedCurrent, datetime):

    @overload
    @classmethod
    def add(
            cls,
            year: int,
            month: int,
            day: int,
            hour: int = ...,
            minute: int = ...,
            second: int = ...,
            microsecond: int = ...,
            tzinfo: TZInfo = ...,
    ) -> None:
        ...

    @overload
    @classmethod
    def add(
            cls,
            instance: datetime,
    ) -> None:
        ...

    @classmethod
    def add(cls, *args, **kw):
        """
@@ -170,29 +147,6 @@ def add(cls, *args, **kw):
        """
        return super().add(*args, **kw)

    @overload
    @classmethod
    def set(
            cls,
            year: int,
            month: int,
            day: int,
            hour: int = ...,
            minute: int = ...,
            second: int = ...,
            microsecond: int = ...,
            tzinfo: TZInfo = ...,
    ) -> None:
        ...

    @overload
    @classmethod
    def set(
            cls,
            instance: datetime,
    ) -> None:
        ...

    @classmethod
    def set(cls, *args, **kw):
        """
@@ -205,28 +159,6 @@ def set(cls, *args, **kw):
        """
        return super().set(*args, **kw)

    @overload
    @classmethod
    def tick(
            cls,
            days: float = ...,
            seconds: float = ...,
            microseconds: float = ...,
            milliseconds: float = ...,
            minutes: float = ...,
            hours: float = ...,
            weeks: float = ...,
    ) -> None:
        ...

    @overload
    @classmethod
    def tick(
            cls,
            delta: timedelta,  # can become positional-only when Python 3.8 minimum
    ) -> None:
        ...

    @classmethod
    def tick(cls, *args, **kw) -> None:
        """

...and similar across mock_datetime, mock_time and mock_date and the helper types they construct.

Is Python typing just "not there yet" or am I missing something? Happy to be shown the way, ecstatic if someone feels inspired to throw up a PR 😅

[erictraut]

Can you describe the problem you're seeing in more detail? You mentioned in the other thread that the ellipses make mypy unhappy. When I paste your code into a new document, add the missing import statements, and run mypy on it, I don't see any errors.

Normally, you'd need to supply real default argument values rather than using an ellipsis, but type checkers generally allow you to use an ellipsis as a stand-in for the actual value when you're writing overloads or stubs. It's still a good idea to provide the actual default argument values because it allows language servers to show users these values when they call the function. For example, here's what I see when I start typing a call to MockDateTime.set when using the pyright language server.

It would be better if you supplied the actual default values. I presume they are zero by default? If so, replace the ... with 0.

[cjw296]

@hauntsaninja - thanks, your branch is happy with the minimal examples above, however, is this another related bug?

class Foo:
    pass

@overload
def foo(x: int) -> None: ...

@overload
def foo(*, x: Foo) -> None: ...

@overload
def foo(a: str, /) -> None: ...

def foo(*args: int | str, **kw: int | Foo) -> None:
    pass

...gives:

$ mypy demo.py 
demo.py:40: error: Overloaded function implementation does not accept all possible arguments of signature 1  [misc]

The key bit appears to be the | Foo on kw, so while the above is "more complete" (and relates to the TZInfo type from my real world problem), I can still reproduce the error with the following, more minimal:

class Foo:
    pass

@overload
def foo(x: int) -> None: ...

@overload
def foo(a: str, /) -> None: ...

def foo(*args: int | str, **kw: int | Foo) -> None:
    pass

$ mypy demo.py 
demo.py:37: error: Overloaded function implementation does not accept all possible arguments of signature 1  [misc]

Interestingly, the | str does appear important to trigger the bug, as:

class Foo:
    pass

@overload
def foo(x: int) -> None: ...

@overload
def foo(*, x: Foo) -> None: ...

def foo(*args: int, **kw: int | Foo) -> None:
    pass

$ mypy demo.py 
Success: no issues found in 1 source file

[hauntsaninja]

Yeah, that corresponds to the comment in that diff (got merged faster than I expected). Let's try python/mypy#19619

[cjw296]

Okay, great, thanks, and now on to the next one(s):

class MockedCurrent:

    @classmethod
    def add(cls, *args: int | datetime, **kw: int | TZInfo | None) -> None:
        return None


class MockDateTime(MockedCurrent):

    @overload
    @classmethod
    def add(
            cls,
            year: int,
            month: int,
            day: int,
            hour: int = 0,
            minute: int = 0,
            second: int = 0,
            microsecond: int = 0,
            *,
            tzinfo: TZInfo | None = None,
            fold: int = 0
    ) -> None:
        ...

    @overload
    @classmethod
    def add(
            cls,
            instance: datetime,
            /
    ) -> None:
        ...

    @classmethod
    def add(cls, *args: int | datetime, **kw: int | TZInfo | None) -> None:
        return cls.add(*args, **kw)

On mypy master, I get:

demo.py:14: error: Signature of "add" incompatible with supertype "MockedCurrent"  [override]
demo.py:14: note:      Superclass:
demo.py:14: note:          @classmethod
demo.py:14: note:          def add(cls, *args: int | datetime, **kw: int | tzinfo | None) -> None
demo.py:14: note:      Subclass:
demo.py:14: note:          @overload
demo.py:14: note:          @classmethod
demo.py:14: note:          def add(cls, year: int, month: int, day: int, hour: int = ..., minute: int = ..., second: int = ..., microsecond: int = ..., *, tzinfo: tzinfo | None = ..., fold: int = ...) -> None
demo.py:14: note:          @overload
demo.py:14: note:          @classmethod
demo.py:14: note:          def add(cls, datetime, /) -> None
demo.py:42: error: Argument 1 to "add" of "MockDateTime" has incompatible type "*tuple[int | datetime, ...]"; expected "int"  [arg-type]
demo.py:42: error: Argument 2 to "add" of "MockDateTime" has incompatible type "**dict[str, int | tzinfo | None]"; expected "int"  [arg-type]
demo.py:42: error: Argument 2 to "add" of "MockDateTime" has incompatible type "**dict[str, int | tzinfo | None]"; expected "tzinfo | None"  [arg-type]

Why is the superclass considered incompatible? It can be called in exactly the same way as the subclass.

However, more confusing are the subsequent errors, what are those trying to communicate? Are they further bugs?

[cjw296]

Here's a minimal reproducer for the the confusing stuff above:

class Base:

    @overload
    def add(cls, x: int) -> None: ...

    @overload
    def add(cls, y: str, /) -> None: ...

    def add(cls, *args: int | str, **kw: int) -> None:
        return None


class Sample(Base):

    @overload
    def add(cls, x: int) -> None: ...

    @overload
    def add(cls, y: str, /) -> None: ...

    def add(cls, *args: int | str, **kw: int) -> None:
        return cls.add(*args, **kw)

demo.py:26: error: Argument 1 to "add" of "Sample" has incompatible type "*tuple[int | str, ...]"; expected "int"  [arg-type]

...although if this is a bug, I suspect there's a parallel one for **kw as shown in the previous comment...

[hauntsaninja]

I think both errors are legit:

Incompatible with supertype

This boils down to:

class A:
    def foo(self, *args: int, **kw: str) -> None: ...

class B(A):
    def foo(self, year: int, *, tzinfo: str) -> None: ...

which is unsound because you could do:

def calls_foo(a: A):
    a.foo(1, 2, 3, 4, 5, oops="A.foo handles these args, Liskov substitution principle implies B.foo should as well")

calls_foo(B())  # boom

pyright doesn't complain about the datetime example in your second last comment though, so maybe myself and mypy are missing something

Errors when calling superclass from your subclass

This is because only the overload types matter to the caller. The caller needs to prove to the type checker that the way it is calling the function matches the overloads.

pyright complains about this as well

[Compare-Your-Parking-Deals]

Ambiguity in Method Resolution:

Problem: When methods are overloaded with flexible types (e.g., #seq<_> or #exn in F#, or generic types in Python), the compiler or runtime may struggle to resolve which method to call due to overlapping type hierarchies or ambiguous signatures. For example, in F# (), overloading methods with flexible types like #Uri and #Stream can trigger errors like FS0438: Duplicate method.
Impact: This leads to compilation errors or unexpected runtime behavior, as the compiler cannot determine the most specific method to invoke.

Language-Specific Overloading Limitations:

Python: Python does not support traditional method overloading because it treats methods as attributes, and the last defined method with a given name overwrites previous ones (). This makes it difficult to implement overloads for flexible types without workarounds.
F#: Flexible types (e.g., #IEnumerable) are powerful but can cause issues when overloaded, as the type system may not allow multiple methods with the same name if their signatures are too similar ().
Java/C#: Overloading is supported but requires distinct parameter lists (number, type, or order). Flexible types like generics or interfaces can lead to ambiguous method calls if not carefully designed (,).

Flexible Types and Type Inference Issues:

Problem: Flexible types (e.g., #T in F# or generics in Python with @overload) complicate type inference. For instance, in Python’s mypy with @AbstractMethod and @overload, requiring an implementation for abstract methods can clutter the code ().
Impact: Developers may need to add redundant implementations or use workarounds, reducing code clarity and maintainability.

Code Readability and Maintenance:

Problem: Overloading methods, especially with flexible types, can make code harder to read and maintain. As noted in, overloading can introduce concealed semantics, leading to bugs or misunderstandings in method behavior.
Impact: Teams may struggle to debug or extend code, particularly when overloads handle similar tasks with subtle differences.

Incompatibility with Abstract or Generic Classes:

Problem: In abstract classes or interfaces, overloading methods with flexible types (e.g., Python’s @AbstractMethod with @overload) may require unnecessary implementations or cause type checker errors ().
Impact: This forces developers to write boilerplate code or use alternative approaches, increasing complexity.

Professional Solutions and Best Practices

Resolve Ambiguity with Explicit Signatures:

Solution: Ensure method signatures are distinct in terms of parameter types, number, or order. In Java or C#, use specific types rather than overly flexible ones (e.g., List instead of List<?>) to avoid ambiguity (,).
Example (Java):
javaclass Calculator {
int add(int x, int y) { return x + y; }
double add(double x, double y) { return x + y; }
int add(int x, int y, int z) { return x + y + z; }
}
Ensure no overlap in parameter types to prevent ambiguous calls.
F# Tip: For flexible types, explicitly constrain types or use distinct method names to avoid FS0438 errors ().

Workarounds for Python’s Overloading Limitation:

Solution: Since Python doesn’t support traditional overloading, use a single method with optional parameters, default values, or type checking (). Alternatively, use the @overload decorator with typing for static type checkers like mypy.
Example (Python):
pythonfrom typing import overload, Union

class Calculator:
@overload
def add(self, x: int, y: int) -> int: ...
@overload
def add(self, x: float, y: float) -> float: ...
def add(self, x: Union[int, float], y: Union[int, float]) -> Union[int, float]:
return x + y
This allows mypy to enforce type checking while using a single implementation.

Handle Flexible Types in Abstract Classes:

Solution: For abstract classes with flexible types (e.g., Python’s @AbstractMethod with @overload), define a generic base implementation to satisfy the type checker, as suggested in. Alternatively, use generic types to parameterize the class.
Example (Python):
pythonfrom abc import ABC, abstractmethod
from typing import overload, TypeVar, Generic

T = TypeVar('T')
U = TypeVar('U')

class Base(ABC, Generic[T, U]):
@overload
def process(self, value: int) -> str: ...
@overload
def process(self, value: str) -> int: ...
@AbstractMethod
def process(self, value: T) -> U: ...

class Concrete(Base[int, str]):
def process(self, value: int) -> str:
return str(value)
Using generics avoids redundant implementations and clarifies type relationships.

Improve Readability and Avoid Overloading Pitfalls:

Solution: Limit overloading to cases where it improves usability without introducing complexity. As suggested in, consider using distinct method names if overloads risk confusion or bugs.
Example (Java):
Instead of:
javaclass Cart {
void add(int id) { /* Fetch product by ID / }
void add(Product p) { / Add product directly / }
}
Use descriptive names:
javaclass Cart {
void addById(int id) { / Fetch product by ID / }
void addProduct(Product p) { / Add product directly */ }
}
This avoids hidden semantics and improves maintainability.

F# Workaround for Flexible Types:

Solution: For F#’s flexible type issues (), either enumerate specific implementations for each type or use different method names. If overloads are necessary, constrain flexible types explicitly.
Example (F#):
fsharptype Processor() =
static member Process(d: System.Uri) = 42
static member Process(d: System.IO.Stream) = 52
Avoid flexible types like #Uri and use concrete types to prevent overlap errors.

Leverage Type Systems and Static Analysis:

Solution: Use static type checkers (e.g., mypy for Python, or IDE tools for Java/C#) to catch overloading issues early. In Python, ensure @overload definitions are consistent with the implementation (). website: https://thedifferentlanguages.org/
Tool Tip: Configure mypy with strict mode (--strict) to enforce type safety for overloads.

Refactor to Reduce Overloading:

Solution: If overloading becomes excessive, refactor to use polymorphism, interfaces, or design patterns like Strategy or Factory to handle flexible behavior ().
Example (C#):
csharpinterface IOperation {
double Execute(double x, double y);
}

class AddOperation : IOperation {
public double Execute(double x, double y) => x + y;
}

class Calculator {
public double Compute(IOperation op, double x, double y) => op.Execute(x, y);
}
This avoids overloading by delegating behavior to specific classes.

Recommendations

Choose the Right Tool for the Job: If overloading is problematic in your language (e.g., Python), consider alternative approaches like default parameters or generics. In F#, avoid flexible types in overloads unless absolutely necessary.
Test Thoroughly: Write unit tests to verify that the correct overloaded method is called for each input type. Use tools like JUnit (Java), pytest (Python), or FsUnit (F#).
Document Clearly: Use clear method names and documentation to indicate the purpose of each overload, especially when dealing with flexible types.
Consult Language-Specific Resources: For F#, check the F# documentation or community forums. For Python, refer to typing module docs or mypy guidelines. For Java/C#, review official documentation or style guides (e.g., Oracle’s Java tutorials, Microsoft’s C# guidelines).