mypy tests: Allow per distribution strictness options

[JelleZijlstra]

We should try turning on the --disallow-incomplete-defs flag from python/mypy#3744. This can catch issues where we forget type annotations for some function arguments in a stub.

[gvanrossum]

Good idea.

[ilinum]

I agree :)

There are a lot of errors (little over 5000).
Here is the gist with all the errors it produces.

Not sure if the effort to fix all of them so we can enable the flag is worth it.

[OddBloke]

Considering only the mypy --python-version 3.6 --strict-optional --disallow-incomplete-defs failures, the breakdown between stdlib/third-party is

$ cut -d/ -f1 < out | sort | uniq -c
     77 stdlib
    895 third_party

so enabling this just for the stdlib initially might make addressing them more tractable?

(Full breakdown by module at https://gist.github.com/OddBloke/dbd78409dcd53bdbb6b3b8571bd29720)

[CraftSpider]

I'm interested in tackling at least the stdlib side of this. Would it be better to do this as individual, more well thought-out PRs for each module, or one much larger commit that only does relatively general types

[srittau]

In general, smaller PRs are easier to review, but too many PRs can also become cumbersome. It might be best to have one PR per package if a package has a few changes needed and a few PRs for packages that only have few changes.

[hauntsaninja]

Note --disallow-untyped-defs is actually stricter than --disallow-incomplete-defs and should be preferred.

[srittau]

I am not sure we should enable either. I prefer to have unannotated types over using Any. I also prefer unannotated types over not having definitions or using "incomplete" markers. Enabling these warnings would raise the bar for contributions, with more complex libraries significantly.

[hauntsaninja]

I agree on both points! (Although it's reasonable to aspire to turning this on as a lint for stdlib one day, since we're close to completion and hopefully the stdlib isn't changing too drastically).

My point with the above was just that --disallow-untyped-defs is more in line with what I want for identifying stubs that need improvement (in the stub context, it can be a surprise that --disallow-incomplete-defs doesn't complain about definitions that are missing types entirely). If someone is using these flags to improve typeshed, it's a good thing to know.

[srittau]

Good point about stdlib.

[JukkaL]

Once we've migrated to modular typeshed, it should be easy to add support for specifying stricness options for each distribution separately in the metadata file. This wouldn't directly help with the stdlib, since stdlib would be distributed as a single entity.

[srittau]

I renamed this ticket to what I believe the consensus of the discussion is. Please revert if you don't agree.

[srittau]

I've removed the "deferred" label as this could now be implemented.

[JelleZijlstra]

Fixed in #5169