S3TransferManager V2 - Support for Upload Pause/Resume

[sabramson01]

Describe the feature

S3 TransferManager V1 included support for pause/restart of an upload. This feature is critical for certain scenarios within our application. We do see improvement leaps from TM v1 --> v2 in terms of performance and reliability however pause/resume upload is an important part of our user experience.

Is your Feature Request related to a problem?

Our application has a video component which is network transmission intensive. When a user has invoked this component we need to pause any current file uploads(typically 20GB - 60GB files) and allow most if not all network bandwidth to our video component. If we could pause the upload and resume later where we left off this would be a huge advantage for us.

Proposed Solution

Enable V2 S3TransferManager to include pause/resume functionality for multipart upload.

Describe alternatives you've considered

We have considered creating two multipart upload modes. The normal mode running 20 or so threads and 20MG chunk size. This would run when network bandwidth is available. A low resource mode when we are running our video component network bandwidth application. This would use 1 thread with 100k chunk size.

Acknowledge

• I may be able to implement this feature request

AWS Java SDK version used

software.amazon.awssdk:s3:2.17.109

JDK version used

java 8

Operating System and version

ubuntu 18.04.2

[zoewangg]

Hi @sabramson01, thanks for reaching out and glad you see improvements in v2! This feature is definitely on our radar; in fact, we are working on pause/resume for download right now in #3094 and will tackle upload pause/resume next :). Do you have any feedback on v1's pause/resume implementation? Is there any specific change that you would like to see in v2?

[freshmanken]

Hi @zoewangg, any update for the feature progress?

[zoewangg]

@freshmanken Pause/resume for download is done. We are planning to start the implementation for pause/resume for upload soon

[zoewangg]

Just a quick update, PR #3357 to support pause/resume for upload has been merged, but we have not released it yet. We will post it here once the change is released

[zoewangg]

To provide an update, we have released the change in 2.19.0. Closing the issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[puneesh]

hello @zoewangg I am following samples mentioned here to pause and resume file upload to S3 bucket.

Issue I am facing is that the moment upload.pause() is called java.util.concurrent.CancellationException: null is thrown and ``Transfer failed.``` is printed in log.

So, if I catch exception - I am able to persist upload information using resumableFileUpload.serializeToFile(path) but resume is not happening when this line executes FileUpload resumedUpload = transferManager.resumeUploadFile(persistedResumableFileUpload). What am I missing here, can you please advise?

I am using 2.20.0 version of s3-transfer-manager, aws-crt, aws-crt-client under software.amazon.awssdk