Type stub incompatiblity with tarfile.ExFileObject (i.e. tarfile.TarFile.extractfile() result)

[mgorny]

Please consider the following snippet:

import tomli
import tarfile

with tarfile.open("test.tar") as tar:
    f = tar.extractfile("test.toml")
    assert f is not None
    tomli.load(f)

When I run mypy on it, I get:

test.py:7: error: Argument 1 to "load" has incompatible type "IO[bytes]"; expected "BinaryIO"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

To be honest, typing docs don't really say what the difference between IO[bytes] and BinaryIO is. Besides type stubs, the code works. FWICS, tomllib would suffer from the same problem if not for the fact that mypy overrides its type stub to:

def load(__fp: SupportsRead[bytes], *, parse_float: Callable[[str], Any] = ...) -> dict[str, Any]: ...

[hukkin]

Thanks for bringing this up!

I'd like to hear what @hauntsaninja has to say here, as they are far more knowledgeable in Python typing than I am.

It seems to me that BinaryIO should be pretty much equivalent to IO[bytes] (even the __enter__ method override seems redundant to me, but there must be a good reason for it). But it is a subtype of IO, hence the incompatibility (I think).

Perhaps tarfile should move to using BinaryIO instead of IO[bytes] or mypy should not error?

We can consider moving to using a custom SupportsRead[bytes] here instead of BinaryIO, which is too strict a requirement anyways but was convenient to use because it's in the stdlib.

[hauntsaninja]

I think using something like SupportsRead[bytes] might end up being the most user friendly thing.

I think BinaryIO should be avoided. There's not really any difference between it and IO[bytes] and it mainly just serves to confuse. In general, typeshed tries to avoid use of BinaryIO and TextIO, and even of IO. These things were added to the stdlib before protocols were a thing. In practice they often don't work well because it turns out there's a tonne of variation in what "file-like" actually means, and they don't capture well the difference between readers and writers. There's more discussion at python/typing#829

[mgorny]

Thanks for the fix. Would you also submit it to tomllib or should I try?

[hauntsaninja]

We don't need to change tomllib, since nothing looks at type hints in the standard library; everything just looks at typeshed

[hukkin]

Yup, and typeshed already uses SupportsRead so you shouldn't have this issue with tomllib.

[mgorny]

We don't need to change tomllib, since nothing looks at type hints in the standard library; everything just looks at typeshed

Well, I did look at types in the stdlib at first and took me quite a while to figure out why mypy complains about tomli and not the "identical" types in tomllib.