Upload not marked as complete after finish event is issued

[MatiasSMd]

Using TransferManager we handle TRANSFER_COMPLETED_EVENT to detect when an upload has finished. However if the upload's state is queried after receiving the event, it still says it is in progress.

The following code reproduces the issue:

import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.event.ProgressEvent;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.transfer.PersistableTransfer;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.Upload;
import com.amazonaws.services.s3.transfer.internal.S3ProgressListener;
import org.junit.Test;

import java.io.File;

public class S3AccessServiceTest {

    @Test
    public void test() throws InterruptedException {
        // File having some content to upload
        File file = new File("/tmp/somefile"); //SET YOUR FILE
        String accessKey = ""; //SET YOUR ACCESS KEY
        String secretKey = ""; //SET YOUR SECRET KEY
        String bucket = "mybucket"; //SET YOUR BUCKET

        final TransferManager transferManager = new TransferManager(new BasicAWSCredentials(accessKey, secretKey));

        class MyListener implements S3ProgressListener {
            private Upload upload;

            public synchronized Upload getUpload() {
                return upload;
            }

            public synchronized void setUpload(Upload upload) {
                this.upload = upload;
            }

            public void onPersistableTransfer(PersistableTransfer persistableTransfer) {}

            public void progressChanged(ProgressEvent progressEvent) {
                System.out.println(progressEvent);
                Upload upload = getUpload();
                if (upload != null) {
                    System.out.println(upload.getState());
                } else {
                    System.out.println("No upload yet");
                }
            }
        }

        MyListener listener = new MyListener();

        Upload upload = transferManager.upload(new PutObjectRequest(bucket, "some_key", file), listener);
        listener.setUpload(upload);
        upload.waitForUploadResult();
    }
}

This is the ouput:

TRANSFER_STARTED_EVENT, bytes: 0
InProgress
REQUEST_CONTENT_LENGTH_EVENT, bytes: 97082
InProgress
CLIENT_REQUEST_STARTED_EVENT, bytes: 0
InProgress
HTTP_REQUEST_STARTED_EVENT, bytes: 0
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 8192
InProgress
HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
InProgress
HTTP_RESPONSE_STARTED_EVENT, bytes: 0
InProgress
HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
InProgress
CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
InProgress
REQUEST_BYTE_TRANSFER_EVENT, bytes: 6970
InProgress
TRANSFER_COMPLETED_EVENT, bytes: 0
InProgress

[MatiasSMd]

Used pom dependency:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>1.10.0</version>
</dependency>

[manikandanrs]

TransferManager internally uses single or multipart uploads depending on the file size being uploaded. The specific issue happens when the TransferManager decides to use single upload for the file.

Progress Events (TRANSFER_STARTED, BYTE_TRANSFER, TRANSFER_COMPLETED) are associated with the AmazonS3 client level while the TransferState (Waiting, InProgress, Completed, Cancelled) is associated with the TransferManager. TRANSFER_COMPLETED_EVENT is fired by the PutObject method in AmazonS3Client. When the put object call is successful , TransferManager then updates the state to Completed. So, there is a slight delay in the State moving to Completed.

To check if the upload is completed, use the Upload.isDone() method.

[manikandanrs]

Closing this issue as we haven't heard back. Feel free to reopen.

[Druckles]

I have the same problem. Having received the TRANSFER_COMPLETED_EVENT, the Transfer object (Upload in this case) returns isDone() == false. We are using the isDone() method to check completion because a MultipleFileUpload fires several TRANSFER_COMPLETED_EVENTs. Therefore, the only reliable way to know that the upload is complete seems to be to check isDone() && progressEvent == TRANSFER_COMPLETED_EVENT.

The docs don't actually mention how to tell if a transfer is complete. Wait for a Transfer to Complete talks about blocking and then says "or receive progress updates asynchronously using a ProgressListener." The rest of the page, however, only talks about general events. Is there a recommended practice for this?

Looking at the code, it appears that the two events may indeed happen in any order and it's only been working for us up till this point by chance. In UploadCallable:

UploadResult result = this.multipartUploadCallable.call();
if (result == null) {
    // ...
} else {
    this.uploadComplete();
}

the Upload actually happens in .call(), when the AmazonS3Client is called and sends its event(s) (as mentioned above). Then after this, the Upload object is marked as done in uploadComplete().

In the MultipleFile and Multipart Uploads, this works because an extra TRANSFER_COMPLETED_EVENT is sent after/during uploadComplete().

So, if I am to use the ProgressListener to avoid blocking and isDone to check whether the Transfer is finished, this does seem like a bug, right?

[eihsu]

For those who arrive here in 2023 or later: it looks like people have moved on to SDK 2, which has an updated TransferManager and also a standalone async client for S3. My org ran into similar problems to the above by trying to add a progress listener to wait for isDone to become true. Sometimes a final TRANSFER_COMPLETED_EVENT would come through, with no more progress events would follow, even though isDone was not yet set to true (seems to be related to re-merging the parts of a multi-part upload). So our code would hang.

We switched to the crt async client and everything is working fine for us!

https://aws.amazon.com/blogs/developer/introducing-crt-based-s3-client-and-the-s3-transfer-manager-in-the-aws-sdk-for-java-2-x/