New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Two optimizations for load_graph() #4294

Merged
merged 4 commits into from Nov 30, 2017

Conversation

Projects
None yet
2 participants
@gvanrossum
Member

gvanrossum commented Nov 30, 2017

Profiling load_graph() found that there were some unnecessary stat() calls going on, and, more importantly, that Options.clone_for_module() is very inefficient if you have many sections in your mypy.ini file. I solved the latter by introducing a cache. An alternative design would move the cache into BuildManager -- that would avoid a reference cycle, but it turns out that having the cache in the root Options object means that it survives between dmypy check runs, and that's a nice win there. (This explains at least partly why load_graph() was slow even with everything loaded in memory.)

@JukkaL

gvanrossum added some commits Nov 30, 2017

Speed up clone_for_module() by using a cache
(Alternatively we could put the cache in BuildManager.)
@JukkaL

Just a few minor things. Feel free to merge after you've updated the PR or you can merge as is and create a follow-up PR.

Show outdated Hide outdated mypy/options.py Outdated
@@ -1056,11 +1058,10 @@ def validate_meta(meta: Optional[CacheMeta], id: str, path: Optional[str],
# TODO: Share stat() outcome with find_module()
path = os.path.abspath(path)
# TODO: Don't use isfile() but check st.st_mode
if not os.path.isfile(path):
st = manager.get_stat(path) # TODO: Errors

This comment has been minimized.

@JukkaL

JukkaL Nov 30, 2017

Collaborator

What about adding a is_file method to BuildManager instead and using it here?

@JukkaL

JukkaL Nov 30, 2017

Collaborator

What about adding a is_file method to BuildManager instead and using it here?

This comment has been minimized.

@gvanrossum

gvanrossum Nov 30, 2017

Member

That's a bigger effort, we'd have to give BuildManager a cache results and use that everywhere. There's also an inefficiency in find_module(), which seems to make way to many stat() and listdir() calls, which I haven't tracked down.

@gvanrossum

gvanrossum Nov 30, 2017

Member

That's a bigger effort, we'd have to give BuildManager a cache results and use that everywhere. There's also an inefficiency in find_module(), which seems to make way to many stat() and listdir() calls, which I haven't tracked down.

@gvanrossum gvanrossum merged commit 9779e18 into python:master Nov 30, 2017

0 of 2 checks passed

continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

@gvanrossum gvanrossum deleted the gvanrossum:opt-build branch Nov 30, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment