For files over 50 GB S3_complete_multipart executes prematurely.

[JustinKyleJames]

• main
• 4-2-stable

---

When using streaming mode but with a local cache file, when the cache file is flushed, there are simultaneously S3_MPU_THREADS being executed (if the file is significantly large). Normally there is a one-to-one mapping between part sizes and number of threads.

However, if each thread handles more than S3_MAX_UPLOAD_SIZE megabytes, more parts have to be used so there is not a one-to-one mapping.

With the defaults of S3_MPU_THREADS = 10 and S3_MAX_UPLOAD_SIZE of 5 GB, if the file is over 50 GB this happens.

The logic is roughly as follows:

1. Figure out number of required parts.
2. Perform min(S3_MPU_THREADS, parts_left) S3_upload_part calls in parallel.
3. Wait for all active upload threads to complete.
4. If more parts need to be uploaded, continue with 2.

The problem is that I was reusing an existing thread_pool for the second iteration of 2-4. Because of this the wait returned immediately and the complete multipart executed prematurely.

The fix is to instantiate a new version of thread_pool on each iteration.

As an example, if the file is 55 GB, there will be 11 parts each of size 5 GB. The first 10 will execute and before the 11th part executes S3_complete_multipart() will execute.

[JustinKyleJames]

Note that this only happens when a cache file is required. That generally happens after a redirect from one server to another because the S3 plugin doesn't know the file size which would be necessary to use streaming mode.

[korydraughn]

The fix is to instantiate a new version of thread_pool on each iteration.

That sounds pretty costly. Can the existing thread pool be reused?

[JustinKyleJames]

The fix is to instantiate a new version of thread_pool on each iteration.

That sounds pretty costly. Can the existing thread pool be reused?

It is costly but for a file of 50 GiB and 10 upload threads we are talking about 1 re-instantiation. With the change for 5119, it would be 4 re-instantiations.

[trel]

right, marginally zero if you're moving 50GiB.

[alanking]

@JustinKyleJames - Please close if complete. Thanks!

[JustinKyleJames]

Closing