Usage of AnyStr yields bad diagnostic "Argument 1 to "endswith" of "bytes" has incompatible type "str"; expected "Union[bytes, Tuple[bytes, ...]]"

[shoffmeister]

Bug Report

An error diagnostic on an AnyStr yields Argument 1 to "endswith" of "bytes" has incompatible type "str"; expected "Union[bytes, Tuple[bytes, ...]]", which does not make sense.

To Reproduce

import os
from typing import AnyStr

def thing(value: AnyStr) -> None:
    with os.scandir(value) as it:
        for file in it:
            if file.name.endswith('.xml'): ...

thing("hello")
#thing(b"world")

Expected Behavior

a) text output

Argument 1 to "endswith" of "bytes" was confusing for me (new to mypy, Python typing). What the diagnostic really seems to try saying is

Argument 1 of "endswith", of type "bytes", ...

or even

"endswith": Declared argument 1 of type "bytes" ...

With the adjusted wording, I am able to ergonomically parse what mypy is trying to tell me; I'd therefore recommend tweaking the text of the diagnostic.

b) individual type checking

Given that AnyStr == [str | bytes], it follows from the typeshed that file.name is [str | bytes], too. The type checker now faces the awkward problem of determining whether the invocation of endswith() is sound:

• [str].endswith('.xml') - happy
• [bytes].endswith('.xml') - unhappy

mypy (IMHO) now has the following options to proceed:

• accept that at least one element out of the set of allowable types satisfies the type check; this is the case in my sample code - in this case, no diagnostic is in order
• require that all allowable types satisfy the type check - from a total correctness point of view it could be debatable that mypy ought to assert this property. Personally, I'd really support this for fresh strongly dynamically typed code (sort of light generics). Alas, here I am invoking something inside the Python Standard Library, and the typeshed is a retrofit, so this behaviour would be getting very awkward for libraries which do not subscribe to a clean typing story ...
• iterate over all allowable types, determine type check passing for each type and emitting a warning in case mypy cannot successfully reason about validity of the set of types (e.g. "Out of the set of allowable types, [bytes] does not pass the type-check; [str] does pass the type-check")
• pick some arbitrary type out of the TypeVar (e.g.: [bytes]) and run the check - and it seems as if mypy is doing exactly that. It seems as if the choice is made predictably ("first listed type in TypeVar"), but still this is an arbitrary choice which is not emitted as a diagnostic ("Assumption: Validating dispatch of endswith on name with type [bytes] " info level?)

So, IOW, IMHO, and depending on the design intents of the implementation, the behaviour of mypy should change to either compute the type check in a different way, or to be more talkative (more diagnostic output) about the decisions that mypy makes under the hood.

IMHO, a short term fix could be emitting additional diagnostic output about the decisions that mypy currently makes silently. A longer term fix could be iterating over all allowable types and emitting different diagnostics.

Your Environment

Fedora 34, Python 3.9.7, mypy 0.9.10 (via pipenv)

[shoffmeister]

FWIW, I understand that I can resolve the diagnostic simply by

- def thing(value: AnyStr) -> None:
+ def thing(value: str) -> None:

but that's not my point ;)

[JelleZijlstra]

Mypy is correct here. It checks that your code works for both of the allowed types (str and bytes) and finds that it doesn't work for bytes, so it emits an error.

I'm having trouble understanding why you think this should be a false positive. Your function throws an error at runtime if you pass a bytes object, exactly as mypy says.

[shoffmeister]

I see two challenges:

a) the diagnostic text as emitted by mypy was difficult for me to mentally parse, hence my proposal to tweak it

b) right now, mypy behaves as if it does "require that all allowable types satisfy the type check" within the scope of the checked unit (the function?). From context / whole program analysis it would be clear that out of the possible resolutions of AnyStr

    def thing(value: bytes) -> None:
    def thing(value: str) -> None:

only the basic type str is used, hence that eventually execution would sort of type-check successfully.

I understand that whole program analysis might be infeasible in larger context.

FWIW, I am just probing around a bit, being new to mypy and Python type checking. I may not be making sense a lot ;)