Don't buffer all the data in memory when writing, reducing peak memory usage #256
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
My one concern here is that Windows semantics might be a problem. I will submit a separate PR to switch CI to using GitHub Actions, then merge this forward. |
|
I ran tests using the GitHub Actions setup in #257. It works fine on Windows, but fails on Python 2.7 where mmap objects don't support the buffer API. There are two solutions, I am happy to implement both:
Please let me know which you prefer and I will submit a PR or update a PR (but I recommend merging #257 first). |
|
@ionrock OK this is now ready for review. |
jjjamie
reviewed
Nov 5, 2021
| @@ -62,8 +82,8 @@ def _close(self): | |||
| # and allows the garbage collector to do it's thing normally. | |||
| self.__callback = None | |||
|
|
|||
| # Closing the BytesIO stream releases memory. Important when caching | |||
| # big files. | |||
| # Closing the tempoary file releases memory and frees disk space. | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #145.
Before:
After:
In both cases running the script from #145 (comment) using
fil-profile run example.py.