-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
For files over 50 GB S3_complete_multipart executes prematurely. #2114
Comments
Note that this only happens when a cache file is required. That generally happens after a redirect from one server to another because the S3 plugin doesn't know the file size which would be necessary to use streaming mode. |
That sounds pretty costly. Can the existing thread pool be reused? |
It is costly but for a file of 50 GiB and 10 upload threads we are talking about 1 re-instantiation. With the change for 5119, it would be 4 re-instantiations. |
right, marginally zero if you're moving 50GiB. |
@JustinKyleJames - Please close if complete. Thanks! |
Closing |
When using streaming mode but with a local cache file, when the cache file is flushed, there are simultaneously S3_MPU_THREADS being executed (if the file is significantly large). Normally there is a one-to-one mapping between part sizes and number of threads.
However, if each thread handles more than S3_MAX_UPLOAD_SIZE megabytes, more parts have to be used so there is not a one-to-one mapping.
With the defaults of S3_MPU_THREADS = 10 and S3_MAX_UPLOAD_SIZE of 5 GB, if the file is over 50 GB this happens.
The logic is roughly as follows:
The problem is that I was reusing an existing thread_pool for the second iteration of 2-4. Because of this the wait returned immediately and the complete multipart executed prematurely.
The fix is to instantiate a new version of thread_pool on each iteration.
As an example, if the file is 55 GB, there will be 11 parts each of size 5 GB. The first 10 will execute and before the 11th part executes S3_complete_multipart() will execute.
The text was updated successfully, but these errors were encountered: