Check file type annotations against open mode #2337

Closed
flother opened this Issue Oct 26, 2016 · 5 comments

Comments

Projects
None yet
4 participants
@flother

flother commented Oct 26, 2016

I have a function argument whose type I want to limit to a file object that's open for binary reading — i.e. I want only an instance of io.BufferedReader as returned by open() when mode="rb".

from typing import IO


def parser(file_obj: IO[bytes]) -> None:
    pass

As I would expect, mypy runs without complaint on this code:

parser(open("example.bin", "rb"))

But it doesn't output an error for this:

parser(open("example.txt", "rt"))

In that case, I would want an error along the lines of Argument 1 to "parser" has incompatible type IO[str]; expected "IO[bytes]".

@gvanrossum

This comment has been minimized.

Show comment
Hide comment
@gvanrossum

gvanrossum Oct 26, 2016

Member
Member

gvanrossum commented Oct 26, 2016

@JukkaL

This comment has been minimized.

Show comment
Hide comment
@JukkaL

JukkaL Oct 26, 2016

Collaborator

I think that special casing open and a few other stdlib functions would be totally reasonable. We already special case the string % operator so there is even a precedent.

Collaborator

JukkaL commented Oct 26, 2016

I think that special casing open and a few other stdlib functions would be totally reasonable. We already special case the string % operator so there is even a precedent.

@gvanrossum gvanrossum added this to the 0.5 milestone Oct 27, 2016

@gvanrossum gvanrossum added the feature label Oct 27, 2016

@illume

This comment has been minimized.

Show comment
Hide comment
@illume

illume Feb 18, 2017

Hello,

I was trying to type check if the file was open in binary mode too, and noticed a few things (and a workaround right at the end of all this)...

mypy -2 doesn't seem to know that io.BufferedReader is there.

io.BufferedReader
afile = io.open('/etc/passwd', 'rb')

def read_data(f):
    # type: (io.BufferedReader) -> bytes
    return f.read()
bytes_issue.py:2: error: "module" has no attribute "BufferedReader"
bytes_issue.py:5: error: Name 'io.BufferedReader' is not defined

If I run mypy in python3 mode, it passes.

Also if I change it to use _io, it works under mypy -2, but not in python3 mode.

_io.BufferedReader
afile = _io.open('/etc/passwd', 'rb')
def read_data(f):
    # type: (_io.BufferedReader) -> bytes
    return f.read()

With mypy in 3 mode I get:

bytes_issue.py:1: error: Cannot find module named '_io'

In python python2.7 and 3.6, the types for io.open are like so.

>>> type(io.open('/etc/passwd', 'r'))
<type '_io.TextIOWrapper'>
>>> type(io.open('/etc/passwd', 'rb'))
<type '_io.BufferedReader'>

Hacking about some more I make this mess below which does let me check that every file obj passed into the object is a BufferedReader. Of course, I need to manually cast every instance... not at all nice.

import io, sys
from typing import cast
PY3 = sys.version_info > (3,)
if PY3:
    from io import BufferedReader
else:
    from _io import BufferedReader
afile = io.open('/etc/passwd', 'rb')
def read_data(f):
    # type: (BufferedReader) -> bytes
    return f.read()
read_data(cast(BufferedReader, afile))
#The following code will give a type error, as I haven't cast it to a binary file.
#another_file = io.open('/etc/passwd', 'r')
#read_data(afile)

Finally I decide to have a separate open function wrapper for every mode. This way I can use a cast inside it, and the type checker works. Any file not using my custom wrappers also fails - which is great because I know where I have to track things down to check them.

import io, sys
from typing import cast
PY3 = sys.version_info > (3,)

if PY3:
    from io import BufferedReader, TextIOWrapper
else:
    from _io import BufferedReader, TextIOWrapper

def my_io_open_r(fname): # type: (str) -> TextIOWrapper
    return cast(TextIOWrapper, io.open(fname, 'r'))
def my_io_open_rb(fname): # type: (str) -> BufferedReader
    return cast(BufferedReader, io.open(fname, 'rb'))

def read_data(f): # type: (BufferedReader) -> bytes
    return f.read()

read_data(my_io_open_rb('/etc/passwd'))

# below fails, as it is opened with 'r'
# read_data(my_io_open_r('/etc/passwd'))

# below also fails type check with a got IO[Any] error.
# read_data(io.open('/etc/passwd', 'r'))

I think this technique would work anywhere you have a factory pattern where the return types are dependent on logic inside the function, or the parameters. As long as number of combinations are not too big that is! (open_rb, open_r, open_w, open_wb)

cheers,

illume commented Feb 18, 2017

Hello,

I was trying to type check if the file was open in binary mode too, and noticed a few things (and a workaround right at the end of all this)...

mypy -2 doesn't seem to know that io.BufferedReader is there.

io.BufferedReader
afile = io.open('/etc/passwd', 'rb')

def read_data(f):
    # type: (io.BufferedReader) -> bytes
    return f.read()
bytes_issue.py:2: error: "module" has no attribute "BufferedReader"
bytes_issue.py:5: error: Name 'io.BufferedReader' is not defined

If I run mypy in python3 mode, it passes.

Also if I change it to use _io, it works under mypy -2, but not in python3 mode.

_io.BufferedReader
afile = _io.open('/etc/passwd', 'rb')
def read_data(f):
    # type: (_io.BufferedReader) -> bytes
    return f.read()

With mypy in 3 mode I get:

bytes_issue.py:1: error: Cannot find module named '_io'

In python python2.7 and 3.6, the types for io.open are like so.

>>> type(io.open('/etc/passwd', 'r'))
<type '_io.TextIOWrapper'>
>>> type(io.open('/etc/passwd', 'rb'))
<type '_io.BufferedReader'>

Hacking about some more I make this mess below which does let me check that every file obj passed into the object is a BufferedReader. Of course, I need to manually cast every instance... not at all nice.

import io, sys
from typing import cast
PY3 = sys.version_info > (3,)
if PY3:
    from io import BufferedReader
else:
    from _io import BufferedReader
afile = io.open('/etc/passwd', 'rb')
def read_data(f):
    # type: (BufferedReader) -> bytes
    return f.read()
read_data(cast(BufferedReader, afile))
#The following code will give a type error, as I haven't cast it to a binary file.
#another_file = io.open('/etc/passwd', 'r')
#read_data(afile)

Finally I decide to have a separate open function wrapper for every mode. This way I can use a cast inside it, and the type checker works. Any file not using my custom wrappers also fails - which is great because I know where I have to track things down to check them.

import io, sys
from typing import cast
PY3 = sys.version_info > (3,)

if PY3:
    from io import BufferedReader, TextIOWrapper
else:
    from _io import BufferedReader, TextIOWrapper

def my_io_open_r(fname): # type: (str) -> TextIOWrapper
    return cast(TextIOWrapper, io.open(fname, 'r'))
def my_io_open_rb(fname): # type: (str) -> BufferedReader
    return cast(BufferedReader, io.open(fname, 'rb'))

def read_data(f): # type: (BufferedReader) -> bytes
    return f.read()

read_data(my_io_open_rb('/etc/passwd'))

# below fails, as it is opened with 'r'
# read_data(my_io_open_r('/etc/passwd'))

# below also fails type check with a got IO[Any] error.
# read_data(io.open('/etc/passwd', 'r'))

I think this technique would work anywhere you have a factory pattern where the return types are dependent on logic inside the function, or the parameters. As long as number of combinations are not too big that is! (open_rb, open_r, open_w, open_wb)

cheers,

@JukkaL

This comment has been minimized.

Show comment
Hide comment
@JukkaL

JukkaL May 31, 2017

Collaborator

#3299 has fixed this for calls where the mode argument is either a literal or not explicitly provided.

Collaborator

JukkaL commented May 31, 2017

#3299 has fixed this for calls where the mode argument is either a literal or not explicitly provided.

@JukkaL JukkaL closed this May 31, 2017

@gvanrossum

This comment has been minimized.

Show comment
Hide comment
@gvanrossum

gvanrossum May 31, 2017

Member

Actually some of @illume's issues (e.g. io.BufferedReader doesn't exist in PY2) are pure typeshed issues. @illume Could you open a new issue in typeshed about this (and the other issues you've found related to missing classes in io.pyi)?

Member

gvanrossum commented May 31, 2017

Actually some of @illume's issues (e.g. io.BufferedReader doesn't exist in PY2) are pure typeshed issues. @illume Could you open a new issue in typeshed about this (and the other issues you've found related to missing classes in io.pyi)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment