make mypy more strict for prototype datasets #4513

pmeier · 2021-09-30T13:56:16Z

Instead of retrofitting more strictness for mypy later, we can start off with strict settings.

NicolasHug · 2021-10-05T07:08:48Z

mypy.ini

+; untyped definitions and calls
+disallow_untyped_defs = True
+
+; None and Optional handling
+no_implicit_optional = True
+
+; warnings
+warn_unused_ignores = True
+warn_return_any = True
+warn_unreachable = True
+
+; miscellaneous strictness flags
+allow_redefinition = True


Are these the default values for those options?
If they're not the default, do we have a strong reason to use them instead of the defaults? Is this going to be clearly beneficial to the code-base and to us as developers?

Are these the default values for those options?

Nope.

If they're not the default, do we have a strong reason to use them instead of the defaults? Is this going to be clearly beneficial to the code-base and to us as developers?

Let's go through them one by one:

disallow_untyped_defs: by default mypy simply accepts untyped functions and uses Any for the input and output annotations. If our ultimate goal is to declare torchvision typed, we should make sure that we don't miss some functions. This flag enforces that.

no_implicit_optional: By default mypy allows this:

def foo(bar: int = None) -> int: pass

With this option enabled, it has to be

def foo(bar: Optional[int] = None) -> int: pass

Given that None is a valid input, we should also explicitly mention it in the annotation.

warn_unused_ignores: Sometimes we use # type: ignore directives on stuff that is actually wrong in other libraries. For example fix annotation for Demultiplexer pytorch#65998 will make some ignore directives obsolete that are needed now. Without this flag, we would never know.

warn_return_any: If a function does something with dynamic types, mypy usually falls back to treating the output as Any. This will warn us if something like this happened, but we specified a more concrete output type.

warn_unreachable: This is more a test functionality, as mypy will now warn us if some code is unreachable. For example, with this flag set, mypy will warn that the if branch is unreachable.

def foo(bar: str) -> str: if isinstance(bar, int): bar = str(bar) return bar

allow_redefinition: See Set allow_redefinition = True for mypy #4531. If we have this globally, we can of course remove it here.

Apart from warn_return_any and warn_unreachable I think these flags are clearly beneficial. For the other two, they were beneficial for me in the past, but I can others object to them.

torchvision/prototype/datasets/_builtin/celeba.py

NicolasHug

Thanks @pmeier !
Just some questions for my own understanding, but LGTM

NicolasHug · 2021-10-21T08:41:15Z

torchvision/prototype/datasets/_builtin/sbd.py

-            for line in lines[1:-1]
-        ]
-        return tuple(zip(*sorted(categories_and_labels, key=lambda category_and_label: int(category_and_label[1]))))[0]
+        categories_and_labels = cast(


just wondering why we need to cast anything here?

pattern.match(line).groups() returns a Tuple[Optional[str], ...]. So we need to cast to tell it that this will be a tuple of length 2 and every group was actually matched.

Do we need to cast because of the Optional bit or because of the exact length of the tuple? Or both?
Would List[Tuple[str, ...]], be enough?

Also can we remove the # type: ignore[union-attr] below now?

Do we need to cast because of the Optional bit or because of the exact length of the tuple? Or both?
Would List[Tuple[str, ...]], be enough?

List[Tuple[str, ...]] seems to work out. I assumed I needed a two element tuple due to the assignment in L177.

Also can we remove the # type: ignore[union-attr] below now?

Nope. re.match returns Optional[Match] and since we don't check for match is None because we are sure that we will always match, mypy complains that None has no attribute groups.

NicolasHug · 2021-10-21T08:42:48Z

torchvision/prototype/datasets/decoder.py

@@ -12,4 +13,4 @@ def raw(buffer: io.IOBase) -> torch.Tensor:


 def pil(buffer: io.IOBase, mode: str = "RGB") -> torch.Tensor:
-    return pil_to_tensor(PIL.Image.open(buffer).convert(mode.upper()))
+    return cast(torch.Tensor, pil_to_tensor(PIL.Image.open(buffer).convert(mode.upper())))


Do we need to call cast because pil_to_tensor is not typed?

Correct. For untyped functions mypy assumes Any and then complains because we return the more specific torch.Tensor here. I've added a warn_redundant_casts = True option that will emit a warning that this cast can be removed as soon as pil_to_tensor is typed.

NicolasHug · 2021-10-21T08:43:42Z

torchvision/prototype/datasets/utils/_resource.py

@@ -8,7 +8,7 @@


 # FIXME
-def compute_sha256(_) -> str:
+def compute_sha256(path: pathlib.Path) -> str:


lol I'm afraid to ask

This file needs heavy refactoring as soon as the torchdata download API is stable-ish. Adding the type was just faster than adding an ignore.

Summary: * make mypy more strict for prototype datasets * fix code format * apply strictness only to datasets * fix more mypy issues * cleanup * fix mnist annotations * refactor celeba * warn on redundant casts * remove redundant cast * simplify annotation * fix import Reviewed By: NicolasHug Differential Revision: D31916328 fbshipit-source-id: 55eac940a3ed5bc3197debeb8b7bdb20ea543578

* make mypy more strict for prototype datasets * fix code format * apply strictness only to datasets * fix more mypy issues * cleanup * fix mnist annotations * refactor celeba * warn on redundant casts * remove redundant cast * simplify annotation * fix import

make mypy more strict for prototype datasets

d47ede9

pmeier added module: datasets prototype labels Sep 30, 2021

pmeier requested a review from fmassa September 30, 2021 13:56

facebook-github-bot added the cla signed label Sep 30, 2021

pmeier mentioned this pull request Oct 4, 2021

Fix annotation of draw_segmentation_masks #4527

Merged

pmeier added 2 commits October 5, 2021 08:27

Merge branch 'main' into prototype-mypy

914ebee

fix code format

be3babf

pytorch-probot bot added the ciflow/default label Oct 5, 2021

pmeier mentioned this pull request Oct 5, 2021

add prototype dataset for CelebA #4514

Merged

NicolasHug reviewed Oct 5, 2021

View reviewed changes

pmeier added 5 commits October 20, 2021 15:28

Merge branch 'main' into prototype-mypy

be44a6e

apply strictness only to datasets

b437fa3

fix more mypy issues

182a4ea

cleanup

7b14618

Merge branch 'main' into prototype-mypy

0fe36e4

NicolasHug reviewed Oct 21, 2021

View reviewed changes

torchvision/prototype/datasets/_builtin/celeba.py Outdated Show resolved Hide resolved

NicolasHug approved these changes Oct 21, 2021

View reviewed changes

pmeier added 10 commits October 21, 2021 11:58

Merge branch 'main' into prototype-mypy

722dbe6

fix mnist annotations

f7943d7

refactor celeba

735e41f

Merge branch 'main' into prototype-mypy

dc85d36

warn on redundant casts

1c83855

remove redundant cast

069d402

simplify annotation

2095cdb

fix import

0a954d3

Merge branch 'main' into prototype-mypy

70ce129

Merge branch 'main' into prototype-mypy

7da45d3

pmeier merged commit 4ba91bf into pytorch:main Oct 21, 2021

pmeier deleted the prototype-mypy branch October 21, 2021 15:24

pmeier mentioned this pull request Nov 3, 2021

remove hard coded input from category file generation #4841

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make mypy more strict for prototype datasets #4513

make mypy more strict for prototype datasets #4513

pmeier commented Sep 30, 2021 •

edited by pytorch-probot bot

Loading

NicolasHug Oct 5, 2021

pmeier Oct 5, 2021

NicolasHug left a comment

NicolasHug Oct 21, 2021

pmeier Oct 21, 2021

NicolasHug Oct 21, 2021

pmeier Oct 21, 2021

NicolasHug Oct 21, 2021

pmeier Oct 21, 2021

NicolasHug Oct 21, 2021

pmeier Oct 21, 2021

make mypy more strict for prototype datasets #4513

make mypy more strict for prototype datasets #4513

Conversation

pmeier commented Sep 30, 2021 • edited by pytorch-probot bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NicolasHug left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmeier commented Sep 30, 2021 •

edited by pytorch-probot bot

Loading