Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3: streaming uses up too much memory #3568

ncw opened this issue Sep 24, 2019 · 1 comment


Copy link

@ncw ncw commented Sep 24, 2019

In #3503 @ivandeex noted:

// The S3 backend additionally triggers extra large upload buffers.
// Namely, multiupload track in the Object.upload() method of S3
// backend (rclone/backends/s3.go) selects PartSize about 512M,
// upload.init() of AWS SDK (vendor/.../s3/s3manager/upload.go)
// allocates upload.bufferPool of that size for each concurrent
// upload goroutine. Given default concurrency of 4, this results
// in 2G buffers persisting until the test executable ends.

This is clearly too much memory for too little benefit! These buffers could be recycled better.

@ncw ncw added the bug label Sep 24, 2019
@ncw ncw added this to the v1.50 milestone Sep 24, 2019
@ncw ncw modified the milestones: v1.50, v1.51 Oct 26, 2019
ncw added a commit that referenced this issue Nov 6, 2019
… size

Before this change rclone would allow the user to stream (eg with
rclone mount, rclone rcat or uploading google photos or docs) 5TB
files.  This meant that rclone allocated 4 * 525 MB buffers per
transfer which is way too much memory by default.

This change makes rclone use the configured chunk size for streamed
uploads.  This is 5MB by default which means that rclone can stream
upload files up to 48GB by default staying below the 10,000 chunks

This can be increased with --s3-chunk-size if necessary.

If rclone detects that a file is being streamed to s3 it will make a
single NOTICE level log stating the limitation.

This fixes the enormous memory usage.

Fixes #3568
@ncw ncw closed this in 9b53081 Nov 9, 2019

This comment has been minimized.

Copy link
Collaborator Author

@ncw ncw commented Nov 9, 2019

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.51

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
1 participant
You can’t perform that action at this time.