fix: make connection resets retryable in OkHttp engine

[aajtodd]

Issue #

fixes aws-sdk-kotlin#905

Description of changes

#829 disabled the OkHttp engine setting retryOnConnectionFailure. The reason for this was sound, we don't want an "inner" retry loop running inside an HTTP engine as it will retry outside the scope of the retry strategy/policy (e.g. not respecting backoffs, etc).

This increases the chance we hit a connection/network related error though. In particular S3 seems to aggressively reset connections under load. The logs show a connection is grabbed from the pool that was used previously for a successful request and fails on getting the response headers (it sends the request at which point S3 closes the connection).

This manifest with an exception like the following:

    java.io.IOException: unexpected end of stream on https://mybucket.s3.us-west-2.amazonaws.com/...                                 at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:209)                                                                     
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:111)                                                                                            at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:95)                                                                           
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)                                                                                      at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)                                                                           
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)                                                                                      at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)                                                                                    
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)                                                                                      at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)                                                                                   
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)                                                                                      at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:65)                                                               
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)                                                                                      at aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:30)                                                          
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)                                                                                      at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:205)                                                                         at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:537)                                                                                                  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)                                                                    
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)                                                                            at java.base/java.lang.Thread.run(Thread.java:829)                                                                                                              
    Caused by: java.io.EOFException: \n not found: limit=0 content=…                                                                                                    
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:332)                                                                                                at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)                                                                                           
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:179)                                                                     
        ... 18 more   

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[ianbotsf]

Nit: "Retry connection reset errors in OkHttp engine"

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication