Surprising error involving bytes duck-type compatibility

[dvarrazzo]

Mypy seems to fail to narrow down union with not conditions. I couldn't find an issue already open on the topic.

To Reproduce

Running mypy on the following sample, an error is reported:

from typing import Union

Buffer = Union[bytes, bytearray, memoryview]

def f(data: Buffer) -> str:
    if not isinstance(data, bytes):
        data = bytes(data)

    return data.decode()

$ mypy testunion.py 
testunion.py:9: error: Item "memoryview" of "Union[bytes, bytearray, memoryview]" has no attribute "decode"  [union-attr]

This is unexpected because, after the if, the type of data is definitely bytes.

If a test if isinstance(data, (bytearray, memoryview)) is used instead there is no error. However this has likely a lower performance and requires more maintenance should other types be added to the union.

Your Environment

• Mypy version used: 0.941
• Python version used: 3.8.10
• In pyproject.toml:

[tool.mypy]
warn_unused_ignores = true
show_error_codes = true
strict = true

[AlexWaygood]

Unfortunately I believe this may actually be expected behaviour, despite the fact that it is somewhat baffling and counter-intuitive. It likely stems from the documentation for typing.ByteString:

class typing.ByteString(Sequence[int])
A generic version of collections.abc.ByteString.

This type represents the types bytes, bytearray, and memoryview of byte sequences.

As a shorthand for this type, bytes can be used to annotate arguments of any of the types mentioned above.

In other words (although it is not stated in a PEP anywhere), the typing module docs state that type checkers should treat bytearray and memoryview as implicit subtypes of bytes. This causes mypy to make false assumptions about reachability, as we can see if we run mypy on your snippet with the --warn-unreachable flag:

from typing import Union

Buffer = Union[bytes, bytearray, memoryview]

def f(data: Buffer) -> str:
    if not isinstance(data, bytes):
        data = bytes(data)  # error: Statement is unreachable

    return data.decode()  # error: Item "memoryview" of "Union[bytes, bytearray, memoryview]" has no attribute "decode

We can also see that mypy is able to quite happily do type narrowing using not conditions if we use different types in the Union; it's only the weird nexus of {bytes, bytearray, memoryview} in this snippet that causes confusion for mypy. Mypy emits no errors if run on the following code:

from typing import Union

Buffer = Union[bytes, int, list[int]]

def f(data: Buffer) -> str:
    if not isinstance(data, bytes):
        data = bytes(data)

    return data.decode()

[JelleZijlstra]

Was just going to write the same thing @AlexWaygood said :)

We maybe should reconsider this ByteString behavior. It's confusing, underspecified, and not useful as frequently as with int/float.

[AlexWaygood]

We maybe should reconsider this ByteString behavior. It's confusing, underspecified, and not useful as frequently as with int/float.

Agree 100%.

[dvarrazzo]

As a shorthand for this type, bytes can be used to annotate arguments of any of the types mentioned above.

You are going to send lady Liskov to an early grave 😄 This looks like a bug to me: how can bytes be a a valid annotation for memoryview, if the latter doesn't have methods such as decode()?

I believe I have been fighting mypy issues for two years in psycopg 3 codebase thanks to this helpful shortcut.

[JelleZijlstra]

Let's fix it: https://github.com/JelleZijlstra/peps/blob/bufferpep/pep-9999.rst.

[CarliJoy]

The problem with duck typing is even worse: #12824

[CarliJoy]

Let's fix it: https://github.com/JelleZijlstra/peps/blob/bufferpep/pep-9999.rst.
@JelleZijlstra the link gives a 404 not found

[AlexWaygood]

Let's fix it: https://github.com/JelleZijlstra/peps/blob/bufferpep/pep-9999.rst.

@JelleZijlstra the link gives a 404 not found

https://peps.python.org/pep-0688/

[ilevkivskyi]

FWIW the original example doesn't give an error in the current master (and probably also in 0.990)