New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize file system cache memory usage #4902

Merged
merged 2 commits into from Apr 13, 2018

Conversation

Projects
None yet
3 participants
@JukkaL
Collaborator

JukkaL commented Apr 13, 2018

Copy exception objects to get rid of tracebacks and frame objects that
are not needed and consume memory.

Verified memory savings using the recently added memory profiler.

Alternative to #4809 (which didn't seem to reclaim the space in my
experiments).

JukkaL added some commits Apr 11, 2018

Optimize file system cache memory usage
Copy exception objects to get rid of tracebacks and frame objects that
are not needed and consume memory.
@msullivan

Looks good. How much does it save?

@JukkaL

This comment has been minimized.

Collaborator

JukkaL commented Apr 13, 2018

This saves about 4% of total memory use when type checking mypy.

@JukkaL JukkaL merged commit 74c5130 into master Apr 13, 2018

4 checks passed

continuous-integration/appveyor/branch AppVeyor build succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
self.stat_cache[path] = st
return st
def listdir(self, path: str) -> List[str]:
if path in self.listdir_cache:
return self.listdir_cache[path]
if path in self.listdir_error_cache:
raise self.listdir_error_cache[path]
raise copy_os_error(self.listdir_error_cache[path])

This comment has been minimized.

@eric-wieser

eric-wieser Apr 13, 2018

Contributor

Why a new copy here?

@gvanrossum gvanrossum deleted the memory-fscache branch Apr 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment