Support uploading large files #68
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When requests handles files, it reads the entire file into memory and sends in a single payload. Currently the python SSL tooling does not support single payloads larger than 2^31-1 bytes (https://bugs.python.org/issue42853#msg384566) This means that files ~ >= 2GiB cannot be sent using requests This commit allows a switchover to the streaming multipart payload provided by requests_toolbelt (if installed)
jhollowe
added
type:bug 🐞
Until requests/toolbelt#316 is merged, this will allow requests_toolbelt to send all the data needed by Proxmox
* Checks the total file size, not just the size of each file * Throws OverflowError if payload will be too large for SSL * Suggests using `requests_toolbelt` for medium-large files, but uses `requests` with the associated memory hogging * Uses `requests` for uploads under a threshold (currently 10MiB)
morph027
approved these changes
Oct 4, 2021
|
Tests just failing due to coverage reduction. I'll work on tests later. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR allows a switchover to the streaming multipart payload provided by
requests_toolbelt(if installed) for files larger than supported byrequestson its own.depends-on requests/toolbelt#316 for full functionality with Proxmox
resolves #67