Add --add/--remove/--update flags to 'dmypy recheck' #5745
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gvanrossum
commented
Oct 6, 2018
mypy/dmypy_server.py
Outdated
| if not self.fine_grained_manager: | ||
| return {'error': "Command 'recheck' is only valid after a 'check' command"} | ||
| return self.check(self.previous_sources) | ||
| print(f"\nadd={add}, remove={remove}, update={update}") |
There was a problem hiding this comment.
This print() still needs to be removed.
gvanrossum
changed the title
|
I don’t believe this is right though. It ignores typeshed. |
ilevkivskyi
reviewed
Oct 11, 2018
| """ | ||
| self.remove_watched_paths(remove) | ||
| self.add_watched_paths(add) | ||
| return self._find_changed(add + update) |
There was a problem hiding this comment.
I didn't do a careful review, but generally this looks good. I just have one question to clarify (and it is probably worth mentioning in the docstring): Is it OK to pass files to --remove that were never in the build?
My motivation is to avoid duplication of work in the daemon and in the runner script that filters sources (what we do internally). If only files that were previously in the build are allowed in --remove, then we will need to keep full list of the checked files also in the runner script (because there is no way to check whether already deleted file should be filtered or not, if the filtering is content based, like presence of type comments/annotations).
I would prefer therefore to allow passing arbitrary files to --remove and explicitly document this.
There was a problem hiding this comment.
To clarify, from the current implementation in this PR it looks like it should work, I just would like to make this explicit.
added 13 commits
October 16, 2018 08:54
However there's still a flaw: it makes a bunch of stat() calls that
'dmypy recheck' (without --update) doesn't make. Example (after
updating a comment in mypy/fswatcher.py):
```
add=['mypy/fswatcher.py'], remove=None, update=None
Good stat('mypy/__init__.py')
LOG: fine-grained increment: cmd_recheck: 0.000s
Good stat('mypy/fswatcher.py')
Real read('mypy/fswatcher.py')
LOG: fine-grained increment: find_changed: 0.001s
LOG: fine-grained: ==== update 'mypy.fswatcher' ====
LOG: fine-grained: --- update single 'mypy.fswatcher' ---
LOG: Metadata not found for mypy.fswatcher
LOG: Parsing mypy/fswatcher.py (mypy.fswatcher)
LOG: fine-grained: triggered: ['<mypy.fswatcher.FileSystemWatcher._find_changed>', '<mypy.fswatcher.FileSystemWatcher.update_changed>', '<mypy.fswatcher.FileSystemWatcher[wildcard]>']
LOG: fine-grained: process: mypy.dmypy_server.Server.update_changed
Good stat('.')
Good stat('/Users/guido/src/mypy')
Good stat('/Users/guido/src/mypy/mypy/typeshed/stdlib/3.6')
Good stat('/Users/guido/src/mypy/mypy/typeshed/stdlib/3.5')
Good stat('/Users/guido/src/mypy/mypy/typeshed/third_party/3.5')
Good stat('/Users/guido/src/mypy/mypy/typeshed/stdlib/3')
Good stat('/Users/guido/src/mypy/mypy/typeshed/third_party/3')
Good stat('/Users/guido/src/mypy/mypy/typeshed/stdlib/2and3')
Good stat('/Users/guido/src/mypy/mypy/typeshed/third_party/2and3')
LOG: fine-grained: triggered: []
LOG: fine-grained: update once: mypy.fswatcher in 0.081s - 0 left
LOG: fine-grained increment: update: 0.001s
Stats for FileSystemCache:
len(stat_cache)=11
len(stat_error_cache)=82
{'mypy/__init__.pyi': FileNotFoundError(2, 'No such file or directory'), '/Users/guido/src/mypy/test-data/packages/typedpkg/mypy-stubs': FileNotFoundError(2, 'No such file or directory'), '/Users/guido/src/mypy/test-data/packages/typedpkg/mypy/py.typed': FileNotFoundError(2, 'No such file or directory'), '/Users/guido/src/mypy/test-data/packages/typedpkg/mypy/fscache/py.typed': FileNotFoundError(2, 'No such file or directory'), '/Users/guido/src/mypy/test-data/packages/typedpkg/mypy/fscache/File...kages/psutil-stubs': FileNotFoundError(2, 'No such file or directory'), '/Library/Python/3.6/site-packages/psutil/py.typed': FileNotFoundError(2, 'No such file or directory'), '/Users/guido/Library/Python/3.6/lib/python/site-packages/psutil-stubs': FileNotFoundError(2, 'No such file or directory'), '/Users/guido/Library/Python/3.6/lib/python/site-packages/psutil/py.typed': FileNotFoundError(2, 'No such file or directory'), '/usr/local/lib/mypy': FileNotFoundError(2, 'No such file or directory')}
len(listdir_cache)=9
len(listdir_error_cache)=18
len(isfile_case_cache)=18
len(read_cache)=1
len(read_error_cache)=0
len(hash_cache)=1
len(fake_package_cache)=0
```
This reverts commit d068bf3. I don't trust the fix (it doesn't take typeshed into account). Also it doesn't really belong in this diff -- the same problem appears with the original `dmypy recheck` command.
msullivan
approved these changes
Oct 16, 2018
mypy/dmypy_server.py
Outdated
| sources = self.previous_sources | ||
| if remove: | ||
| removals = set(remove) | ||
gvanrossum
force-pushed
the
dmypy-recheck-update
branch
from
ed6f019
to
6bc8984
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using
dmypy recheck --add <files> --remove <files> --update <files>will take the previous list of files, apply adds/removes/updates, and then check the resulting list of sources, without callingstat()on any of the unchanged files. (Useful if the caller is using e.g. Watchman.) Each list of<files>may be empty in case there are no files in that category.For backwards compatibility, if none of
--add,--removeor--updateis used,dmypy recheckwill revert to its old behavior (i.e. re-check the exact same list of files, callingstat()on each file to determine whether it has changed). Note that even if the list of<files>is empty, the presence of the new flags triggers the new behavior.There are also some cleanups in the diff. (And some loose ends that I will try to address next week.)