Upload files to S3 fails

[devgutt]

When I upload a file bigger than 200kb for the first time, using TransferUtility, I received the follow error, and the file fails to upload.

I/AmazonHttpClient: Unable to execute HTTP request: Write error: ssl=0xb73cbbf8: I/O error during system call, Connection reset by peer
javax.net.ssl.SSLException: Write error: ssl=0xb73cbbf8: I/O error during system call, Connection reset by peer
    at com.android.org.conscrypt.NativeCrypto.SSL_write(Native Method)
    at com.android.org.conscrypt.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:740)
    at com.android.okio.Okio$1.write(Okio.java:70)
    at com.android.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:116)
    at com.android.okio.RealBufferedSink.write(RealBufferedSink.java:44)
    at com.android.okhttp.internal.http.HttpConnection$FixedLengthSink.write(HttpConnection.java:287)
    at com.android.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:116)
    at com.android.okio.RealBufferedSink$1.write(RealBufferedSink.java:131)
    at com.amazonaws.http.UrlHttpClient.write(UrlHttpClient.java:172)
    at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:129)
    at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:65)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:196)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4212)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1619)
    at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.uploadSinglePartAndWaitForCompletion(UploadTask.java:194)
    at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:76)
    at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:41)
    at java.util.concurrent.FutureTask.run(FutureTask.java:237)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
    at java.lang.Thread.run(Thread.java:818)` 

But If I try to upload a file smaller than 200kb, the upload works just fine. The upload also works if I START to upload using the smaller file and than upload the bigger ones.

I'm using AWS SDK 2.2.11 and I tested and found this same behavior in the android sample (https://github.com/awslabs/aws-sdk-android-samples - S3TransferUtilitySample).

This is happening ONLY on my device running Android 5.0.2 (Moto G 2).
I tested in other devices and emulators, but didn't find any error on those:

• Android 4.1.1 (Device Prism II) -> OK
• Android 4.4.2 (emulator x86) -> OK
• Android 5.0.2 (emulator x86) -> OK

I tried to change the timeout and the protocol (https and http), but didn't work either.
I did all the tests on Wi-Fi (internet very stable).

The location of the bucket seems irrelevant, because the upload works, if I don't upload the first file bigger than 200kb.

As workaround, I'm uploading a 1kb file, prior any upload, but it's obviously not the correct solution:

    public static TransferUtility getTransferUtility(Context context) {
        if (sTransferUtility == null) {
            String bucketName = context.getResources().getString(R.string.AWS_S3_BUCKET);

            sTransferUtility = new TransferUtility(getS3Client(context.getApplicationContext()),
                    context.getApplicationContext());

            /////////////////////////////////////
            /* BUG FIX - START */
            //Bug fix for lollipop devices (not emulator)
            //First file bigger than 200Kb doesn't work (only device lollipop, kitkat, kellybeans ok)

            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
                String filename = "xfile";
                File file = new File(context.getFilesDir(), filename);
                try {
                    FileOutputStream stream = new FileOutputStream(file);
                    stream.write("G".getBytes());
                    stream.close();
                    sTransferUtility.upload(bucketName, file.getName(), file);

                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            /* END BUG FIX */
            ////////////////////////////////////
        }

        return sTransferUtility;
    }

Any help would be really appreciated.

[fosterzhang]

Thanks for the very detailed report. I've seen a few reports recently, but I couldn't reproduce it. I'll try it on Android 5.0.2 (Moto G 2) and get back to you ASAP. BTW, is your device behind a firewall?

[devgutt]

Thank you @fosterzhang ...No firewall here. Let me know if you need any other information or log. Cheers.

[bdipak]

Any update on this issue ? I am seeing the same issue.

[wdane]

#95 is a duplicate of this issue on "aws-sdk version 2.2.11 on a Nexus 4 with Android 5.1.1 without any firewall. And the S3 bucket is in the eu-west-1 region."

[sergiopantoja]

I also experienced this issue. I'm trying to figure out how to reproduce it. In the meantime, maybe my anecdote will help:

• I had two Samsung Galaxy devices running Android 4.3 and AWS SDK 2.1.1
• I was on a property with a captive wifi portal
• Both devices were uploading files successfully to the us-east-1 S3 region using the deprecated TransferManager
• At some unknown point, this bug hit only one of the devices. Same app, same library versions, same wifi network. Really weird... They were uploading different sets of files, though. I don't have a copy of those files anymore.
• I upgraded both devices to AWS SDK 2.1.11, updated the code to use the new TransferUtility, switched to a different wifi network without any firewall or captive portals, restarted the devices, but the still issue persisted for that one
• I added @devgutt's workaround (without the Build.VERSION check since I was on Android 4.3) and it fixed the issue

[Bitmaximus]

I had the same issue (same logcat print out), it only occurred on one of my devices (running Android 5.1) and not on the device running 4.3. I was originally using a dependency for AWS SDK 2.+ with the us-east-1 S3 region as well and I changed it to 2.2.11 with no effect. I wiped and reset the 5.1 device eventually and the problem stopped, quite bizarre, I'll be interested to understand what was happening if anyone finds out. -Cheers

[fosterzhang]

@devgutt Sorry for the late reply. I am having trouble in reproducing the exact problem. Thanks to the fragmentation of Android devices, it's hard to get the exact same device, either a Nexus 4 running Android 5.1.1 or a Moto G 2nd gen running 5.0.2. I'll keep bugging people for those two devices.

[fosterzhang]

@devgutt Sorry that I still don't have an answer. I did test the sample on several devices except the device you mentioned, but I couldn't reproduce the issue. To narrow it down:

• please try to upload a file with AmazonS3.putObject(String, String, File). See if the same problem occurs.
• try it with different region and/or bucket name
• try it with different network.
• try it with different devices if possible.

[fosterzhang]

Any update guys? I need a reliable way to reproduce the issue. PS: I don't have Nexus 4.

[devgutt]

Sorry the late reply, I will try to upload with putObject this week and return to you shortly.

[eviljames]

I'm seeing this issue happen fairly consistently, but there's no way to truly trigger the behaviour.

This is what happens in the log:

03-15 02:56:33.614 26529-10782/? I/AmazonHttpClient: Unable to execute HTTP request: timeout
                                                     java.net.SocketTimeoutException: timeout
  at com.android.okhttp.okio.Okio$3.newTimeoutException(Okio.java:207)
  at com.android.okhttp.okio.AsyncTimeout.exit(AsyncTimeout.java:250)
  at com.android.okhttp.okio.AsyncTimeout$2.read(AsyncTimeout.java:217)
  at com.android.okhttp.okio.RealBufferedSource.indexOf(RealBufferedSource.java:306)
  at com.android.okhttp.okio.RealBufferedSource.indexOf(RealBufferedSource.java:300)
  at com.android.okhttp.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196)
  at com.android.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:191)
  at com.android.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80)
  at com.android.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:904)
  at com.android.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:788)
  at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:443)
  at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:388)
  at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponseMessage(HttpURLConnectionImpl.java:497)
  at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseMessage(DelegatingHttpsURLConnection.java:109)
  at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseMessage(HttpsURLConnectionImpl.java)
  at com.amazonaws.http.UrlHttpClient.createHttpResponse(UrlHttpClient.java:72)
  at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:66)
  at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:196)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4213)
  at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3304)
  at com.amazonaws.mobileconnectors.s3.transferutility.UploadPartTask.call(UploadPartTask.java:48)
  at com.amazonaws.mobileconnectors.s3.transferutility.UploadPartTask.call(UploadPartTask.java:28)
  at java.util.concurrent.FutureTask.run(FutureTask.java:237)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)
  at java.lang.Thread.run(Thread.java:818)

To trigger this, I launch transferUtility.upload in a Service and just wait. It will happen several times during the transfer of a large file (dozens, maybe). Usually the mid-file Parts will retry and upload correctly. Occasionally the file will complete its upload and notify via TransferListener, and occasionally it will hang at 99% (appears that the API call to complete the transfer is the one that times out, but not exactly sure). Sometimes the transferUtility.resume function resumes at 95-99%, sometimes it restarts from 0 or near 0. I'm going to chase some stuff down with the debugger at some point to see if there's a way I can identify when a transfer has stalled, failed, or is actually in progress. The status provided by the Observer returned by transferUtility.upload is typically not accurate when the requests are failing.

[fosterzhang]

@eviljames It's SocketTimeoutException. Socket timeout means no incoming data within a given time frame. It's usually caused by bad network. To mitigate it:

• Increase socket timeout threshold via ClientConfiguration.setSocketTimeout(int). The default is 15 seconds.
• Choose an S3 region that is closest to end user.
• Choose a more reliable network

[devgutt]

I don't think this is the problem. If I upload a small file before anything else, this problem will never occur.

[eviljames]

@fosterzhang Thanks for the feedback. I'll increase the Socket Timeout to 30, however this happens irrespective of the network I'm on. Across the Canadian mobile providers, and every Wi-Fi network I've tested on.

[fosterzhang]

@devgutt This thread is mixed with different issues.

• "Connection reset by peer" is unresolved. Does putObject have similar problem as TransferUtility?
• "SocketTimeoutException" is definitely caused by bad network and can be relieved by increasing socket timeout.

@eviljames 30 seconds or 30 milliseconds? ClientConfiguration.setSocketTimeout takes number in milliseconds. Network is a vague term. I can be bad cellular or WiFi. It can also be that the S3 bucket is too far away from you, say accessing Frankfurt from Canada.

[devgutt]

Hey @fosterzhang, I updated the sdk to the version 2.2.13 and the issue seems to be solved. I could drop my workaround as well. I noticed that another problem which occurs intermittently (the TransferObserver didn't fire on COMPLETE, just IN_PROGRESS) also was corrected. I will make more tests and report if I find something else. Thanks!

[vishnuganta22]

Unable to execute HTTP request: Read timed out
java.net.SocketTimeoutException: Read timed out
at com.android.org.conscrypt.NativeCrypto.SSL_read(Native Method)
at com.android.org.conscrypt.OpenSSLSocketImpl$SSLInputStream.read(OpenSSLSocketImpl.java:699)
at com.android.okio.Okio$2.read(Okio.java:113)
at com.android.okio.RealBufferedSource.indexOf(RealBufferedSource.java:147)
at com.android.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:94)
at com.android.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:179)
at com.android.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:101)
at com.android.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:628)
at com.android.okhttp.internal.http.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:388)
at com.android.okhttp.internal.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:332)
at com.android.okhttp.internal.http.HttpURLConnectionImpl.getResponseMessage(HttpURLConnectionImpl.java:496)
at com.android.okhttp.internal.http.DelegatingHttpsURLConnection.getResponseMessage(DelegatingHttpsURLConnection.java:109)
at com.android.okhttp.internal.http.HttpsURLConnectionImpl.getResponseMessage(HttpsURLConnectionImpl.java:25)
at com.amazonaws.http.UrlHttpClient.createHttpResponse(UrlHttpClient.java:72)
at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:66)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:196)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4213)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1620)
at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.uploadSinglePartAndWaitForCompletion(UploadTask.java:194)
at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:76)
at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:41)
at java.util.concurrent.FutureTask.run(FutureTask.java:237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:818)
03-31 10:22:17.959 2200-6009/com.yantranet.signware.agent D/UploadTask: Transfer 164 is interrupted by user

frequently i am getting this error when ever i tried to upload files to s3.
i tried on different networks,but getting the same error.
Is their any way to get a call-back when ever i get this error

[fosterzhang]

@vishnuganta22 Socket timeout is a different issue. A few things you can do:

• Increase socket timeout via ClientConfiguration.setSocketTimeout(int). The default is 15 seconds.
• Choose an S3 region close to you (or your customers).

[fosterzhang]

I'll close this issue, as the original problem is resolved. Please open a new issue if other problems occur.

[ox-michaelradionov]

For me the problem was that my S3 was configured for US region, but when uploading I specified EU (eu-west-1). Switching to US (us-east-1) fixed it for me.

The device was Google Nexus 7 with Android 4.4 and I was getting the exact same error.
But on simulator I had another error: "The bucket you are attempting to access must be addressed using the specified endpoint", which means the region was incorrect, it helped me identify the problem.
After I switched the region it worked both on real device and simulator.

[hetang-shah]

still getting this error. Any pointer for this

com.amazonaws.AmazonClientException: Unable to execute HTTP request: Write error: ssl=0x7f1dce3400: I/O error during system call, Connection reset by peer
                                                                                                         at com.amazonaws.http.AmazonHttpClient.executeHelper(Unknown Source)
                                                                                                         at com.amazonaws.http.AmazonHttpClient.execute(Unknown Source)
                                                                                                         at com.amazonaws.services.s3.AmazonS3Client.invoke(Unknown Source)
                                                                                                         at com.amazonaws.services.s3.AmazonS3Client.putObject(Unknown Source)
                                                                                                         at com.fivestars.mma.promotions.services.CreatePromotionIntentService$onHandleIntent$1$1$1.subscribe(Unknown Source)
                                                                                                         at io.b.e.e.b.c.a(Unknown Source)
                                                                                                         at io.b.f.b(Unknown Source)
                                                                                                         at io.b.e.e.b.h$b.a(Unknown Source)
                                                                                                         at io.b.e.e.b.h$b.a_(Unknown Source)
                                                                                                         at io.b.e.e.b.l$a.f(Unknown Source)
                                                                                                         at io.b.e.e.b.l$a.run(Unknown Source)
                                                                                                         at io.b.e.g.f.run(Unknown Source)
                                                                                                         at io.b.e.g.f.call(Unknown Source)
                                                                                                         at java.util.concurrent.FutureTask.run(FutureTask.java:237)
                                                                                                         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:272)
                                                                                                         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
                                                                                                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
                                                                                                         at java.lang.Thread.run(Thread.java:762)
                                                                                                      Caused by: javax.net.ssl.SSLException: Write error: ssl=0x7f1dce3400: I/O error during system call, Connection reset by peer
                                                                                                         at com.android.org.conscrypt.NativeCrypto.SSL_write(Native Method)
                                                                                                         at com.android.org.conscrypt.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:824)
                                                                                                         at com.android.okhttp.okio.Okio$1.write(Okio.java:81)
                                                                                                         at com.android.okhttp.okio.AsyncTimeout$1.write(AsyncTimeout.java:155)
                                                                                                         at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:176)
                                                                                                         at com.android.okhttp.okio.RealBufferedSink.write(RealBufferedSink.java:46)
                                                                                                         at com.android.okhttp.internal.http.HttpConnection$FixedLengthSink.write(HttpConnection.java:350)
                                                                                                         at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:176)
                                                                                                         at com.android.okhttp.okio.RealBufferedSink$1.write(RealBufferedSink.java:198)
                                                                                                         at com.amazonaws.http.UrlHttpClient.write(Unknown Source)
                                                                                                         at com.amazonaws.http.UrlHttpClient.writeContentToConnection(Unknown Source)
                                                                                                         at com.amazonaws.http.UrlHttpClient.execute(Unknown Source)
                                                                                                         at com.amazonaws.http.AmazonHttpClient.executeHelper(Unknown Source)

code:

val clientConfiguration = ClientConfiguration()
// Timeout for file upload is 30 sec
clientConfiguration.connectionTimeout = 3 * 1000
clientConfiguration.socketTimeout = 30 * 1000
clientConfiguration.maxErrorRetry = 3
clientConfiguration.maxConnections = 8

val developerProvider = AwsPromotionAuthenticationProvider(
        openTokenResponse.accountId, identityPoolId,
        region, openTokenResponse)

/**
 * We do not want to use CognitoCacheCredentialsProvider here. As
 * it caches token in shared preferences for sub sequent requests
 */
val credentialsProvider = CognitoCredentialsProvider(
        developerProvider, region, clientConfiguration)

val s3Client = AmazonS3Client(credentialsProvider,
        clientConfiguration)
s3Client.setRegion(Region.getRegion(Regions.US_EAST_1))

s3Client.putObject(PutObjectRequest(bucket,
        openTokenResponse.identityId,
        File(createPromotionRequest.perk.photo)))

[thisismohitgupta]

still facing this issue in

    compile 'com.amazonaws:aws-android-sdk-s3:2.6.3'

I need to upload a small file before uploading a file greater than 200kb, any help on this issue is much appreciated

[sushantchauhan08]

getting TrasferObserver state as progess and then failed

[martinflorek]

Anything new? I am getting the same error SSLException: Write error: ssl=0xaed86300: I/O error during system call, Connection reset by peer but I use plain OkHttp instead of the AWS SDK. So it looks like an issue with the S3 upload URLs or with OkHttp and not with the SDK.

EDIT:
In the end it looks like an issue with OkHttp. I have tried the Java's HttpURLConnection to upload file to a pre-signed URL and it works. Sample code how to to that can be found at the bottom of this documentation page.