This repository has been archived by the owner on Sep 26, 2022. It is now read-only.
make client.downloader to download large file to file #57
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The current implementation will download the whole file into memory and will crash when the file is too large. This patch will fix this issue by: - download larger file (>4M) to a temp file on disk, and rename it to `self.filename` on `self._write_to_file()` - download smaller file to memory, but to a list of blocks the `self.file_binary_data += block` has performace issue - do checksuming while downloading, most CPU can calculate the checksum faster than download speed, this saves a full reading from disk for larger files
|
:/ py33 py34, and coverage :/ Will fix them. |
|
@mayli Those tests fail randomly. I'll add a few retries to the tests. |
|
@mayli Also, thanks for the PR. It's dope! |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#56 The current implementation will download the whole file into memory and
will crash when the file is too large.
This patch will fix this issue by:
and rename it to
self.filenameonself._write_to_file()the
self.file_binary_data += blockhas performace issuemost CPU can calculate the checksum faster than download speed,
this saves a full reading from disk for larger files