Multiple changes #339
Merged
Multiple changes #339
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r config usage * Improved typing somewhat albeit is no subsitiute for those full type generation scripts. * Fixed concurrency issue in _download_part where writing to seekable async file-like objects, the seek and write operations were not guaranteed to happen sequentially. An asyncio.Lock solves this. * Added non-seekable IO support to download_fileobject. There are some issues here in terms of resource consumption/utilisation. The coroutines to download chunks can happen in any order which means chunks at the end of a file could be downloaded first, and thus must be kept in memory until all their preceding chunks have been downloaded, this is fixable by moving the code to something similar to how upload part works. The other thing is unrelated to seekable IO and more that downloading a 1GiB file with an 8MB chunksize results in 128 tasks being gathered, isn't that much of an issue but if the chunksize is lowered or the file is larger then the event loop could get bogged down with lots of tasks. Performance on average downloading a 1GiB file on a somewhat unutilised network is around 16 seconds which matches the original S3 transfer speeds.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.