Running out of memory when uploading large files

[mad4linux]

Hello,
When my server tries to upload large files, it runs out of memory. Problem seems to be the read() and write() commands that generate the post string. Would it be possible to send large files in chunks somehow?
Here's the error message (the read error is repeated several times in the logs):

Traceback (most recent call last):
  File "/home/USER/customscripts/backup_pcloud/pCloudBackup.py", line 146, in pCloudUpload
    pDrive.uploadfile(folderid=bBSID, files=fileBuf)
  File "/usr/local/lib/python3.7/dist-packages/pcloud/validate.py", line 19, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pcloud/api.py", line 156, in uploadfile
    return self._upload("uploadfile", files, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pcloud/api.py", line 135, in _upload
    resp = self.session.post(self.endpoint + method, files=files, data=kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 519, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 462, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python3/dist-packages/requests/models.py", line 316, in prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 504, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 169, in _encode_files
    body, content_type = encode_multipart_formdata(new_fields)
  File "/usr/lib/python3/dist-packages/urllib3/filepost.py", line 90, in encode_multipart_formdata
    body.write(data)
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/USER/customscripts/backup_pcloud/pCloudBackup.py", line 134, in pCloudUploadSingle
    pDrive.uploadfile(folderid=uploadFolder, files=[uploadFile])
  File "/usr/local/lib/python3.7/dist-packages/pcloud/validate.py", line 19, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pcloud/api.py", line 156, in uploadfile
    return self._upload("uploadfile", files, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pcloud/api.py", line 135, in _upload
    resp = self.session.post(self.endpoint + method, files=files, data=kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 519, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 462, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python3/dist-packages/requests/models.py", line 316, in prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 504, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 159, in _encode_files
    fdata = fp.read()

[tomgross]

Hi @mad4linux
According to https://stackoverflow.com/a/28198229 the Python requests library reads all files into the memory to build a multi-part post body. This leads to the MemoryError in your case. There is a suggested solution, which might be worth a try to be included into pycloud.
If you are interested in an implementation, which fixes this problem, please contact me for a quote.
A PR is also welcome 😄

[mad4linux]

Hi @tomgross
Fair enough. As I'm doing this for my 2 head company, there is no budget :-( But thanks for the hint. I'll dig into this after the summer break. Looks pretty straight forward on first sight. As always :-)

[matt-wisdom]

Can i fix this issue? I need it fixed for a project.

[tomgross]

Hi @cRyp70s A PR is welcome 😄. Another option is to contact me for a quote so I will fix it.

[tomgross]

@cRyp70s what's the status? Are you working on that?

[tomgross]

Implemented for 1.0b1: https://pypi.org/project/pcloud/1.0b1/