Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The reason that uploading file larger than 2GB will be truncated. #5024

Open
kmiku7 opened this issue Mar 18, 2019 · 0 comments
Open

The reason that uploading file larger than 2GB will be truncated. #5024

kmiku7 opened this issue Mar 18, 2019 · 0 comments

Comments

@kmiku7
Copy link

kmiku7 commented Mar 18, 2019

When I upload multipart-encoded file larger than 2GB in python2 environment, the body will be truncated. Server side can not read data completely. But the content-length value in header is expected. At client side, it do not send all data.

After tracing of function calls, it is caused by python2 socket library bug. Requests will use socket.sendall() to send body, but it use int type to keep bytes need be sent and has sent: python2-socket-lib. I also create a issue to report this bug to python team.

Rather than waiting python team fix it, I thought Requests can fix quickly. After search it has also been reported many times.
Requests lib can split encoded string body into many segments each one smaller than 2GB, and send segments one by one. Or wrapping string body as as file-like object is also works, like this:

from cStringIO import StringIO
body=StringIO(request.body)
@kmiku7 kmiku7 changed the title Upload file larger than 2GB will be truncated. The reason that ppload file larger than 2GB will be truncated. Mar 18, 2019
@kmiku7 kmiku7 changed the title The reason that ppload file larger than 2GB will be truncated. The reason that uploading file larger than 2GB will be truncated. Mar 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant