Mypy plugin #722

dmontagu · 2019-08-07T10:41:52Z

Change Summary

Add a mypy plugin with a number of features; see the added docs for more details.

Other notable improvements for pydantic developers:

The mypy tests have been reorganized to be easier to work on
- Eventually, it will probably make sense to better organize individual tests, but
  this seemed like the most practical solution for now
Thanks to the reorganization, the mypy tests now run after all others (I think this is preferable)
The mypy tests and codebase check now run much faster on repeated execution because they no longer invalidate each other's caches
- The tests now make use of separate cache directories by plugin settings, all contained as subfolders of the previously used .mypy_cache folder.
- Because they are subfolders of .mypy_cache, the same make clean works to clean up all of the cache directories

Related issue number

Closes #460

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
change description file added to changes/,
see changes/README.md for details
on the format

samuelcolvin · 2019-08-07T10:44:36Z

In principle this looks great, but as you say let's defer it until 1.1 - it shouldn't cause any backwards incompatibility issues I assume?

setup.cfg

dmontagu · 2019-08-07T10:48:03Z

You can always just turn the plugin off (by not incluiding it in mypy.ini/setup.cfg), so it is strictly opt-in. So I don't think there are backwards compatibility issues.

dmontagu · 2019-08-07T10:50:05Z

For what it's worth, if anyone wants this in the interim, you can just copy/paste the contents of the pydantic/mypy.py file into your own project and add it to the mypy plugins in your mypy.ini/setup.cfg and it should just work. (Doesn't require any changes to pydantic.)

It would be nice to be able to distribute it with the library though.

codecov · 2019-09-05T09:53:10Z

Codecov Report

Merging #722 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #722    +/-   ##
======================================
  Coverage     100%   100%            
======================================
  Files          16     17     +1     
  Lines        2800   3142   +342     
  Branches      543    615    +72     
======================================
+ Hits         2800   3142   +342

Impacted Files	Coverage Δ
pydantic/mypy.py	`100% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb2a302...7bf2595. Read the comment docs.

dmontagu · 2019-10-25T11:00:12Z

After some painful edge case exploration and mypy debugging, I've gotten the test coverage of the plugin itself up to 100% (modulo two minor no-covers related to import edge cases, where I've just copied the logic from the official dataclasses plugin).

Still needs:

docs
support for type checking the construct method

I think it might make sense to (eventually) support other config settings and/or finer-grained strictness controls, but I'm more inclined to just release this and see what features people actually request.

Edit: Actually, I'm going to hold off on adding support for construct until I have a chance to work on python/mypy#7301; otherwise we'd have to copy a lot of code right out of mypy.

Edit 2: It turned out to be easier than I expected to get this working. PR opened for the mypy issue, but I was able to drop a single function into place that we can remove once the PR is merged.

tests/test_mypy.py

dmontagu · 2019-10-27T06:21:47Z

@samuelcolvin I think this is basically done! I added docs and updated the description at the top with some more detail.

I think the docs pretty clearly cover everything the plugin adds.

The tests are a little haphazard organized, basically just dumped into the big plugin-success.py and plugin-fail.py files; we can fix that eventually but I personally think this is good enough for a first release. I'm happy to put more effort into cleaning it up though if you want that to happen before a merge.

Special note: these changes should make running the mypy tests and mypy check faster on subsequent runs due to better caching (even though there are now more tests).

dmontagu · 2019-10-27T06:22:53Z

Also, @koxudaxi if you have a chance to review I think you might find it interesting and/or have some inputs.

samuelcolvin · 2019-10-27T08:55:06Z

I'm away from my computer but will look on Tuesday if not before.

Are we caching the mypy cache directory (sorry can't see on my phone)?

dmontagu · 2019-10-27T09:04:26Z

@samuelcolvin Sounds good, no rush.

I’m not sure what you mean by “are we caching the mypy cache directory” — could you clarify?

I put the test caches inside the default one (used during checks). The structure of the caching directories is such that this doesn’t cause any conflicts.

samuelcolvin · 2019-10-27T09:06:39Z

Sorry, I mean there's a .mypy_cache or simpler directory. Have we added that to the list of cached directories in Travis?

dmontagu · 2019-10-27T09:12:16Z

Yeah, the directory is .mypy_cache. I didn’t know you could cache directories in Travis. Actually I don’t know much about Travis in general. I’m not at my computer now either but can try to figure it out when I can if you’d like.

samuelcolvin · 2019-10-27T09:22:01Z

Well, might make things faster. Don't worry if it's a pain.

docs/mypy_plugin.md

samuelcolvin

In general this all looks great. Must have been a massive effort!

More of my comments are questions or trivial corrections.

docs/mypy_plugin.md

pydantic/mypy.py

samuelcolvin · 2019-10-29T14:03:19Z

tests/mypy/test_mypy.py

+
+cases = [
+    ('mypy-plugin.ini', 'plugin_success.py', None),
+    ('mypy-plugin.ini', 'plugin_fail.py', 'plugin-fail.txt'),


are you sure these are really easier as separate files? Surely easier to have the expected output right here with the test even if some are very verbose?

It's extremely annoying to iterate on it without a mechanism for generating the expected output. By putting it in a file, I can set GENERATE=True above and it will write the file with the actual output, and I can look at the git diffs to see if things are as desired.

I initially tried having the results in-line here, but the problem is that a one line changes all the line numbers. And the line numbers actually do matter, so I think we shouldn't just strip them from the expected output. But then every time I tweak a line I had to manually compare all the output lines and fix the line numbers. Then I'd make another minor change and have to do it all again.

I think there might be better ways to manage mypy plugin tests (might be good to look at how the sqlalchemy or django plugins do this). But I thought they might ultimately rely on calling mypy on a separate file for each test, and I was specifically trying to minimize the amount of time spent executing mypy tests during normal test runs, and I think there is a substantial amount of overhead for each file analyzed. I tried to get all the tests into as few files as possible to keep the tests running fast.

dmontagu · 2019-10-30T04:26:31Z

@samuelcolvin I think I've incorporated all your feedback on the non-docs files, or at least responded above where I had concerns.

I renamed the plugin config settings so they could all be False by default (but basically have the same meaning).

I'm going to wait until we reach consensus on the non-docs stuff before updating the docs.

dmontagu · 2019-10-30T08:58:17Z

It looks like this is what django-stubs uses.

Here's a blog article discussing: https://sobolevn.me/2019/08/testing-mypy-types

It looks like a massive improvement over what is used in this PR, but 1) I'm not sure how fast it would run, and 2) I'm not sure how maintained it is.

We could also just run the mypy plugin tests separately, the only downside would be the added complexity of checking for 100% coverage.

tests/mypy/test_mypy.py

samuelcolvin · 2019-10-30T12:35:04Z

Looking good.

Apart from the docs and a few pretty trivial potential fixes, I think this is more or less ready.

dmontagu · 2019-10-31T03:21:40Z

I tried using dataclasses but it caused the cythonized builds to fail due to the __init__ method not getting created properly.

Maybe there's a workaround, but since it's not really giving much benefit here I figured hand-writing the init function was fine.

dmontagu · 2019-10-31T03:22:01Z

Also, I believe I have now addressed all your concerns with regard to the docs. Maybe give it another once over, but I think it's good to go now.

samuelcolvin · 2019-10-31T12:19:17Z

this is awesome.

Thank you very much!

* Add mypy plugin * Make all arguments optional for BaseSettings * Get test coverage up * Add changes * Add type-checking for BaseModel.construct, and checking for from_orm * Fix formatting and linting * Fix the build * Heavy refactor of plugin and mypy tests * Make linting pass * Handle dynamic aliases * Better organize plugin code * Add docs * Add support for error codes * Fix minor docs typo * Rename config settings, add docstrings, and incorporate other feedback * Incorporate feedback * Update docs, remove dataclasses for cython * fix mypy example

dmontagu force-pushed the mypy-plugin branch from 977c4d0 to 04982a7 Compare Aug 7, 2019

dmontagu reviewed Aug 7, 2019

View changes

setup.cfg Outdated Show resolved Hide resolved

dmontagu mentioned this pull request Aug 9, 2019

mypy plugin #460

Closed

koxudaxi mentioned this pull request Aug 14, 2019

Improve default value on fields koxudaxi/pydantic-pycharm-plugin#30

Closed

samuelcolvin added the deferred label Aug 17, 2019

samuelcolvin mentioned this pull request Aug 27, 2019

Too many arguments / Unexpected keyword argument errors with mypy --strict #781

Closed

dmontagu force-pushed the mypy-plugin branch from b884b59 to f8d513d Compare Sep 5, 2019

dmontagu mentioned this pull request Oct 14, 2019

Disable all validations #897

Closed

dmontagu mentioned this pull request Oct 15, 2019

Faux immutability / allow_mutation has no effect on dataclass #639

Open

David Montague added 2 commits Oct 25, 2019

Add mypy plugin

bbdbd88

Make all arguments optional for BaseSettings

0cda68e

dmontagu force-pushed the mypy-plugin branch from f8d513d to 0cda68e Compare Oct 25, 2019

Get test coverage up

98352f3

dmontagu removed the deferred label Oct 25, 2019

Add changes

f232324

David Montague added 3 commits Oct 26, 2019

Add type-checking for BaseModel.construct, and checking for from_orm

9525193

Fix formatting and linting

f190508

Fix the build

97c9b5c

dmontagu reviewed Oct 26, 2019

View changes

tests/test_mypy.py Show resolved Hide resolved

David Montague added 5 commits Oct 27, 2019

Heavy refactor of plugin and mypy tests

f70c8d1

Make linting pass

a9b4a27

Handle dynamic aliases

d3c0f39

Better organize plugin code

09b4815

Add docs

465fb38

Add support for error codes

544a604

koxudaxi reviewed Oct 27, 2019

View changes

docs/mypy_plugin.md Outdated Show resolved Hide resolved

docs/mypy_plugin.md Outdated Show resolved Hide resolved

docs/mypy_plugin.md Outdated Show resolved Hide resolved

Fix minor docs typo

3a07afd

dmontagu mentioned this pull request Oct 27, 2019

Make compatible with pydantic v1 tiangolo/fastapi#646

Merged

samuelcolvin reviewed Oct 29, 2019

View changes

dmontagu force-pushed the mypy-plugin branch from 9e53321 to 8873ea0 Compare Oct 30, 2019

Rename config settings, add docstrings, and incorporate other feedback

8873ea0

samuelcolvin reviewed Oct 30, 2019

View changes

tests/mypy/test_mypy.py Outdated Show resolved Hide resolved

David Montague added 2 commits Oct 31, 2019

Incorporate feedback

ecb54c8

Update docs, remove dataclasses for cython

f56b11c

fix mypy example

7bf2595

samuelcolvin merged commit 0c18619 into samuelcolvin:master Oct 31, 2019
10 checks passed

Mypy plugin #722

Mypy plugin #722

dmontagu commented Aug 7, 2019 •

edited

samuelcolvin commented Aug 7, 2019

dmontagu commented Aug 7, 2019

dmontagu commented Aug 7, 2019 •

edited

codecov bot commented Sep 5, 2019 •

edited

dmontagu commented Oct 25, 2019 •

edited

dmontagu commented Oct 27, 2019 •

edited

dmontagu commented Oct 27, 2019

samuelcolvin commented Oct 27, 2019

dmontagu commented Oct 27, 2019

samuelcolvin commented Oct 27, 2019

dmontagu commented Oct 27, 2019

samuelcolvin commented Oct 27, 2019

samuelcolvin left a comment

samuelcolvin Oct 29, 2019

dmontagu Oct 30, 2019 •

edited

dmontagu commented Oct 30, 2019

dmontagu commented Oct 30, 2019 •

edited

samuelcolvin commented Oct 30, 2019

dmontagu commented Oct 31, 2019

dmontagu commented Oct 31, 2019 •

edited

samuelcolvin commented Oct 31, 2019

Mypy plugin #722

Mypy plugin #722

Conversation

dmontagu commented Aug 7, 2019 • edited

Change Summary

Related issue number

Checklist

samuelcolvin commented Aug 7, 2019

dmontagu commented Aug 7, 2019

dmontagu commented Aug 7, 2019 • edited

codecov bot commented Sep 5, 2019 • edited

Codecov Report

dmontagu commented Oct 25, 2019 • edited

dmontagu commented Oct 27, 2019 • edited

dmontagu commented Oct 27, 2019

samuelcolvin commented Oct 27, 2019

dmontagu commented Oct 27, 2019

samuelcolvin commented Oct 27, 2019

dmontagu commented Oct 27, 2019

samuelcolvin commented Oct 27, 2019

samuelcolvin left a comment

samuelcolvin Oct 29, 2019

Choose a reason for hiding this comment

dmontagu Oct 30, 2019 • edited

Choose a reason for hiding this comment

dmontagu commented Oct 30, 2019

dmontagu commented Oct 30, 2019 • edited

samuelcolvin commented Oct 30, 2019

dmontagu commented Oct 31, 2019

dmontagu commented Oct 31, 2019 • edited

samuelcolvin commented Oct 31, 2019

dmontagu commented Aug 7, 2019 •

edited

dmontagu commented Aug 7, 2019 •

edited

codecov bot commented Sep 5, 2019 •

edited

dmontagu commented Oct 25, 2019 •

edited

dmontagu commented Oct 27, 2019 •

edited

dmontagu Oct 30, 2019 •

edited

dmontagu commented Oct 30, 2019 •

edited

dmontagu commented Oct 31, 2019 •

edited