IO Exception when listing s3 objects in bucket

[Sebo11]

Describe the bug

When executing below code it randomly happens that the call to s3 fails with an IO Exception (exception is inserted below). I cannot put a precise number on it but I guess this happens once every 20 request or so. It can be "fixed" when the attempts are upped to more than 5. Because when this happens it failes always exactly 5 times with the 6th attempt going trough somehow.

I cannot explain why this occurs and why it always fails 5 times with the 6th attempt going trough.

import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.ListObjectsV2Request
import jakarta.inject.Singleton
import io.micronaut.context.annotation.Property


@Singleton
class Service(private val s3: S3Client) : Logger {
    @Property(name = "bucket-name")
    private lateinit var awsbucket: String

    suspend fun listObjects(uuid: UUID): List<Object>? {
        val request = ListObjectsV2Request {
            bucket = awsbucket
            prefix = "documents/$uuid"
        }
        return s3.listObjectsV2(request).contents
    }

with this factory

import aws.sdk.kotlin.services.s3.S3Client
import io.micronaut.context.annotation.Factory
import io.micronaut.context.annotation.Property
import jakarta.inject.Singleton

@Factory
class S3Factory {
    @Singleton
    fun getS3Client(): S3Client = S3Client {
        region = "eu-central-1"
    }
}

the resulting exception

15:41:51.721 [DefaultDispatcher-worker-17] ERROR i.m.http.server.RouteExecutor - Unexpected error occurred: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
aws.smithy.kotlin.runtime.http.HttpException: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
        at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine.roundTrip(OkHttpEngine.kt:158)
        at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine$roundTrip$1.invokeSuspend(OkHttpEngine.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
Caused by: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:209)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:111)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:65)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:30)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:205)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:537)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.EOFException: \n not found: limit=0 content=
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:335)
        at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:179)
        ... 18 common frames omitted

Expected behavior

Not failing randomly exactly 5 times when this error occurs.

Current behavior

Like explained before the error which is thrown.

15:41:51.721 [DefaultDispatcher-worker-17] ERROR i.m.http.server.RouteExecutor - Unexpected error occurred: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
aws.smithy.kotlin.runtime.http.HttpException: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
        at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine.roundTrip(OkHttpEngine.kt:158)
        at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine$roundTrip$1.invokeSuspend(OkHttpEngine.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
Caused by: java.io.IOException: unexpected end of stream on https://example-bucket.s3.eu-central-1.amazonaws.com/...
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:209)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:111)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:65)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:30)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:205)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:537)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.EOFException: \n not found: limit=0 content=
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:335)
        at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:179)
        ... 18 common frames omitted

Here with more attempts set going trough

09:29:19.565 [default-nioEventLoopGroup-1-3] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(i
d=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.566 [default-nioEventLoopGroup-1-3] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 004111cbb877796271fe307182ce84a0aa2ce6414c9a1e5a74ba3cd038d64e0b
09:29:19.574 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 2
09:29:19.574 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=a
ws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.574 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 8bc5844c65f86333f4cb21e7206b3ae7e471c8973789db9f86605d02ddc5f0a4
09:29:19.588 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 3
09:29:19.588 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=a
ws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.588 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 9d2db5c865b79490bb54de1f21d43909e23a0ab9611ebd4f488ad00af09c8f56
09:29:19.609 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 4
09:29:19.609 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=a
ws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.609 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 74a7dffb53a0ab23be7e32814c42dbe526e96b768e71ba406e13b4a374bb3814
09:29:19.621 [DefaultDispatcher-worker-18] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 5
09:29:19.621 [DefaultDispatcher-worker-18] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.621 [DefaultDispatcher-worker-18] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 549aae8fd9ed87f1ffdd3117dbfcf69b9522946925a2a97356a31daac8b58aa0
09:29:19.653 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 6
09:29:19.653 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=a
ws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
09:29:19.653 [DefaultDispatcher-worker-2] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: c1e9643a2ffa94e87ad7d1828c7aaf215374963703785314552aefceb0a4c357
09:29:19.832 [DefaultDispatcher-worker-18] DEBUG httpTraceMiddleware - HttpResponse: 200: OK

Steps to Reproduce

The framework in use is Micronaut in the version 4.2.2 . The s3 bucket itself has several objects in it and returns with the prefix key 10 elements.

The code was started locally via Gradle (version 8.4) and invoked over the micronaut controoler via Postman. Note that the same behavior is present in the cloud environment.

import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.ListObjectsV2Request
import jakarta.inject.Singleton
import io.micronaut.context.annotation.Property


@Singleton
class Service(private val s3: S3Client) : Logger {
    @Property(name = "bucket-name")
    private lateinit var awsbucket: String

    suspend fun listObjects(uuid: UUID): List<Object>? {
        val request = ListObjectsV2Request {
            bucket = awsbucket
            prefix = "documents/$uuid"
        }
        return s3.listObjectsV2(request).contents
    }

with this factory

import aws.sdk.kotlin.services.s3.S3Client
import io.micronaut.context.annotation.Factory
import io.micronaut.context.annotation.Property
import jakarta.inject.Singleton

@Factory
class S3Factory {
    @Singleton
    fun getS3Client(): S3Client = S3Client {
        region = "eu-central-1"
    }
}

with these build.gradle.kts file

plugins {
    id("org.jetbrains.kotlin.jvm") version "1.9.22"
    id("org.jetbrains.kotlin.plugin.allopen") version "1.9.22"
    id("com.google.devtools.ksp") version "1.9.22-1.0.17"
    id("com.github.johnrengelman.shadow") version "8.1.1"
    id("io.micronaut.application") version "4.3.2"
    id("io.micronaut.aot") version "4.3.2"
}

version = "0.1"
group = "com.example"

val kotlinVersion="1.9.22"
repositories {
    mavenCentral()
}

dependencies {
    ksp("io.micronaut:micronaut-http-validation")
    ksp("io.micronaut.serde:micronaut-serde-processor")
    implementation("io.micronaut.kotlin:micronaut-kotlin-runtime")
    implementation("io.micronaut.serde:micronaut-serde-jackson")
    implementation("org.jetbrains.kotlin:kotlin-reflect:${kotlinVersion}")
    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:${kotlinVersion}")
    compileOnly("io.micronaut:micronaut-http-client")
    runtimeOnly("ch.qos.logback:logback-classic")
    runtimeOnly("com.fasterxml.jackson.module:jackson-module-kotlin")
    testImplementation("io.micronaut:micronaut-http-client")
    implementation("aws.sdk.kotlin:s3:1.0.54")

}


application {
    mainClass.set("com.example.ApplicationKt")
}
java {
    sourceCompatibility = JavaVersion.toVersion("21")
}


graalvmNative.toolchainDetection.set(false)
micronaut {
    runtime("netty")
    testRuntime("junit5")
    processing {
        incremental(true)
        annotations("com.example.*")
    }
    aot {
    // Please review carefully the optimizations enabled below
    // Check https://micronaut-projects.github.io/micronaut-aot/latest/guide/ for more details
        optimizeServiceLoading.set(false)
        convertYamlToJava.set(false)
        precomputeOperations.set(true)
        cacheEnvironment.set(true)
        optimizeClassLoading.set(true)
        deduceEnvironment.set(true)
        optimizeNetty.set(true)
    }
}

Possible Solution

No response

Context

I want to show the user the s3 objects which the user has saved in the cloud via a web app. However at some point the request fails randomly (locally and in the cloud).

AWS Kotlin SDK version used

1.0.54

Platform (JVM/JS/Native)

Java 21.0.2 with gradle 8.4

Operating System and version

Windows 10 Pro

[Sebo11]

Hello, I created a new Micronaut project (version 4.3.1) and tested the behavior again. It still fails occasionally with the same error. However, in the fresh application it only needs 2 attempts instead of 6. I can't explain why exactly as the dependencies and the setup are the same in my opinion.

Don't know if it is worth it to look into it.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[lauzadis]

Hi, thanks for the report! I've set up a local application using Micronaut, but I can't replicate this behavior.

Can you please configure logMode = LogMode.LogRequest on the client and share some logs (with sensitive information redacted) demonstrating the problematic requests being retried and then finally succeeding? It will help determine what it changing in the request to make it succeed.

[Sebo11]

Hello, sorry for the late response. Here the requested logs :)

Like always after the sixth request it somehow gets trough. However, like mentioned before it does not happen a lot. Every 20 request or so.

15:24:29.029 [default-nioEventLoopGroup-1-3] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(i
d=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:29.029 [default-nioEventLoopGroup-1-3] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 416437ca1204259b13badc097e54b87aab915e307ddc825c48a9d4efa74b20e3
15:24:29.030 [default-nioEventLoopGroup-1-3] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=1; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142429Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=416437ca1204259b13badc097e54b87aab915e307ddc825c48a9d4efa74b20e3       


15:24:29.449 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 2
15:24:29.449 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:29.450 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 10f692f13983e1b42442bb12f138fbde4ee6f8ea0e1c0fac80bc77b66c6653e1
15:24:29.450 [DefaultDispatcher-worker-13] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=2; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142429Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=10f692f13983e1b42442bb12f138fbde4ee6f8ea0e1c0fac80bc77b66c6653e1       


15:24:29.795 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 3
15:24:29.796 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:29.796 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: f3b55ed40eefbb74e0e41ca9496e1029f8ff351d68fdfaa9cea3c37b1ec427b7
15:24:29.797 [DefaultDispatcher-worker-20] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=3; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142429Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=f3b55ed40eefbb74e0e41ca9496e1029f8ff351d68fdfaa9cea3c37b1ec427b7       


15:24:30.066 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 4
15:24:30.067 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:30.068 [DefaultDispatcher-worker-20] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: c46c59d8f3bc4990336f52ebc36fcc8e3b70ce8e037cdf05fcca13aa7ff81937
15:24:30.068 [DefaultDispatcher-worker-20] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=4; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142430Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=c46c59d8f3bc4990336f52ebc36fcc8e3b70ce8e037cdf05fcca13aa7ff81937       


15:24:30.315 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
15:24:30.315 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Fill pool skipped, pool has sufficient level or currently being filled.
15:24:30.804 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 5
15:24:30.804 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:30.805 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: 001bf3ba8b261c022ce2f040ad2c5073cf30d5841e7ee312a89842c28c1d2657
15:24:30.805 [DefaultDispatcher-worker-13] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=5; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142430Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=001bf3ba8b261c022ce2f040ad2c5073cf30d5841e7ee312a89842c28c1d2657       


15:24:31.931 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.h.middleware.RetryMiddleware - retrying request, attempt 6
15:24:31.933 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=https://https://example-bucket.s3.eu-central-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=
aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=eu-central-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
15:24:31.934 [DefaultDispatcher-worker-13] DEBUG a.s.k.r.a.a.DefaultAwsSignerImpl - Calculated signature: b443cc2ec0581a82ab2cc410678b2e67eef0a617a718837ac31908d5562f7235
15:24:31.935 [DefaultDispatcher-worker-13] DEBUG httpTraceMiddleware - HttpRequest:
GET /?list-type=2&prefix=documents%2Ffc6cb449-326e-4926-b6a7-58599791a0d7
Host: https://example-bucket.s3.eu-central-1.amazonaws.com
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/windows#10.0 lang/kotlin#1.9.22 md/javaVersion#21.0.2 md/jvmName#Java_HotSpotTM_64-Bit_Server_VM md/jvmVersion#21.0.2+13-LTS-58
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 551be614-830f-4bad-a4b8-56e242692085
amz-sdk-request: attempt=6; max=10
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240226T142431Z
X-Amz-Security-Token: securityToken
Authorization: AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAAAAAAAA/20240226/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=b443cc2ec0581a82ab2cc410678b2e67eef0a617a718837ac31908d5562f7235       


15:24:32.099 [DefaultDispatcher-worker-20] DEBUG httpTraceMiddleware - HttpResponse: 200: OK

[cloudshiftchris]

We're experiencing the same exception w/ S3 (in our case, its on GetObject - we aren't doing a ListObject in this use case).

Context:

• Fargate job run on-demand that does a bunch of processing, orchestrating various AWS services - S3, Dynamo, RDS, Secrets Manager. Only S3 client is experiencing these retries
• S3 client (and other AWS clients) are using default configurations (S3Client.fromEnvironment() etc) - 3 retries. These are singletons established at the start of the job and reused
• while there is some coroutine parallelism, the retries occur on most S3.GetObject requests even in initial, sequential sections before hitting the parallelism
• some times it retries once, sometimes twice, and fails when exceeding the 3 try limit.
• latest AWS Kotlin SDK 1.3.x (we upgrade this weekly via Dependant an have been having this issue for a few months now)

Below entire request that retried (TRACE level output); you can see the initial request, the "call failed" (there is no response, just the EOFException), retry setup, establishing a new connection, and a second attempt. Seems like somehow the initial connection is closed.

2024-10-09T15:09:44.204-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation started","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.205-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.http.operation.AuthHandler","message":"resolved endpoint: Endpoint(uri=https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=us-east-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.210-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"Canonical request:\nGET\n\/mssql\/5.0.1-alpha.0.20241009194137961%2Bdev\/mssql-fingerprint.zip\nx-id=GetObject\namz-sdk-invocation-id:ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7\namz-sdk-request:attempt=1; max=3\nhost:marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241009T210944Z\nx-amz-security-token:***\nx-amz-user-agent:aws-sdk-kotlin\/1.3.47\n\namz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.214-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"String to sign:\nAWS4-HMAC-SHA256\n20241009T210944Z\n20241009\/us-east-1\/s3\/aws4_request\n6d82ce770911a8184749db5e349dcca5a2a7c709de9cb8c0ccb39add9a62b390","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.214-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"Calculated signature: cc5865ae501a7f9cb5b0fbd991a9ef0940e332ba1372f373e3c507afbabb692b","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.215-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-4","logger":"httpTraceMiddleware","message":"HttpRequest:\nGET \/mssql\/5.0.1-alpha.0.20241009194137961%2Bdev\/mssql-fingerprint.zip?x-id=GetObject\r\nHost: marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\r\nUser-Agent: aws-sdk-kotlin\/1.3.47 ua\/2.1 api\/s3#1.3.47 os\/linux#5.10.226-214.879.amzn2.x86_64 lang\/kotlin#2.0.20 md\/javaVersion#21.0.4 md\/jvmName#OpenJDK_64-Bit_Server_VM md\/jvmVersion#21.0.4+7-LTS exec-env\/AWS_ECS_FARGATE m\/E\r\nx-amz-user-agent: aws-sdk-kotlin\/1.3.47\r\namz-sdk-invocation-id: ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7\r\namz-sdk-request: attempt=1; max=3\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nX-Amz-Date: 20241009T210944Z\r\nX-Amz-Security-Token: ***\r\nAuthorization: AWS4-HMAC-SHA256 Credential=ASIAY6YOVYHTRPQJL243\/20241009\/us-east-1\/s3\/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=cc5865ae501a7f9cb5b0fbd991a9ef0940e332ba1372f373e3c507afbabb692b\r\n\r\n","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.220-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"call started","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.220-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"connection acquired: conn(id=374268970)=Connection{marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com:443, proxy=DIRECT hostAddress=marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/54.231.224.138:443 cipherSuite=TLS_AES_128_GCM_SHA256 protocol=http\/1.1}; connPool: total=2, idle=1","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.220-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"sending request headers","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.221-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"finished sending request headers","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.222-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"response failed","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"},"stacktrace":"java.io.IOException: unexpected end of stream on https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...\n\tat okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:220)\n\tat okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:114)\n\tat okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:72)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:32)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:203)\n\tat okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:527)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base\/java.lang.Thread.run(Thread.java:1583)\nCaused by: java.io.EOFException: \\n not found: limit=0 content=…\n\tat okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:341)\n\tat okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)\n\tat okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:188)\n\t... 18 common frames omitted\n"}
2024-10-09T15:09:44.222-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"connection released: conn(id=374268970)=Connection{marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com:443, proxy=DIRECT hostAddress=marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/54.231.224.138:443 cipherSuite=TLS_AES_128_GCM_SHA256 protocol=http\/1.1}; connPool: total=1, idle=1","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.223-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"call failed","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"},"stacktrace":"java.io.IOException: unexpected end of stream on https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...\n\tat okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:220)\n\tat okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:114)\n\tat okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:72)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:32)\n\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)\n\tat okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:203)\n\tat okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:527)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base\/java.lang.Thread.run(Thread.java:1583)\nCaused by: java.io.EOFException: \\n not found: limit=0 content=…\n\tat okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:341)\n\tat okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)\n\tat okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:188)\n\t... 18 common frames omitted\n"}
2024-10-09T15:09:44.224-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.awsprotocol.ClockSkewInterceptor","message":"service did not return \"Date\" header, skipping skew calculation","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.314-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware","message":"retrying request, attempt 2","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.315-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.http.operation.AuthHandler","message":"resolved endpoint: Endpoint(uri=https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=us-east-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.317-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"Canonical request:\nGET\n\/mssql\/5.0.1-alpha.0.20241009194137961%2Bdev\/mssql-fingerprint.zip\nx-id=GetObject\namz-sdk-invocation-id:ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7\namz-sdk-request:attempt=2; max=3\nhost:marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241009T210944Z\nx-amz-security-token:***\nx-amz-user-agent:aws-sdk-kotlin\/1.3.47\n\namz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.318-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"String to sign:\nAWS4-HMAC-SHA256\n20241009T210944Z\n20241009\/us-east-1\/s3\/aws4_request\n8593f46cf4df7a5ddff7fd66638b2a5a4a17f1985761725ba771cd852b00b4ea","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.325-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-4","logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"Calculated signature: 441fe3e2d3b0ac6ec81c66da5158737494bbc04586c815ec0705f622f2c34543","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.328-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-4","logger":"httpTraceMiddleware","message":"HttpRequest:\nGET \/mssql\/5.0.1-alpha.0.20241009194137961%2Bdev\/mssql-fingerprint.zip?x-id=GetObject\r\nHost: marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\r\nUser-Agent: aws-sdk-kotlin\/1.3.47 ua\/2.1 api\/s3#1.3.47 os\/linux#5.10.226-214.879.amzn2.x86_64 lang\/kotlin#2.0.20 md\/javaVersion#21.0.4 md\/jvmName#OpenJDK_64-Bit_Server_VM md\/jvmVersion#21.0.4+7-LTS exec-env\/AWS_ECS_FARGATE m\/E\r\nx-amz-user-agent: aws-sdk-kotlin\/1.3.47\r\namz-sdk-invocation-id: ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7\r\namz-sdk-request: attempt=2; max=3\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nX-Amz-Date: 20241009T210944Z\r\nX-Amz-Security-Token: ***\r\nAuthorization: AWS4-HMAC-SHA256 Credential=ASIAY6YOVYHTRPQJL243\/20241009\/us-east-1\/s3\/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=441fe3e2d3b0ac6ec81c66da5158737494bbc04586c815ec0705f622f2c34543\r\n\r\n","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.333-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-6","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"call started","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.334-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"proxy select start: url=https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.334-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"proxy select end: url=https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/; proxies=[DIRECT]","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.334-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"dns query: domain=marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.404-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"dns resolved: domain=marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com; records=[marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/54.231.224.138, marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/52.217.163.226, marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/52.216.210.242, marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/52.216.77.88, marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/16.15.185.14, marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/52.217.100.40, marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/3.5.24.120, marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/3.5.17.103]","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.404-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp connect https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"starting connection: addr=marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/54.231.224.138:443; proxy=DIRECT","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.406-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"initiating TLS connection","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.435-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"TLS connect end: handshake=Handshake{tlsVersion=TLS_1_3 cipherSuite=TLS_AES_128_GCM_SHA256 peerCertificates=[CN=s3.amazonaws.com, CN=Amazon RSA 2048 M01, O=Amazon, C=US, CN=Amazon Root CA 1, O=Amazon, C=US] localCertificates=[]}","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.438-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"connection established: addr=marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/54.231.224.138:443; proxy=DIRECT; protocol=http\/1.1","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.439-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"connection acquired: conn(id=1940407935)=Connection{marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com:443, proxy=DIRECT hostAddress=marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/54.231.224.138:443 cipherSuite=TLS_AES_128_GCM_SHA256 protocol=http\/1.1}; connPool: total=3, idle=0","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.439-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"sending request headers","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.439-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"finished sending request headers","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.573-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"response headers start","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.574-06:00[America/Denver] {"level":"TRACE","thread":"OkHttp https:\/\/marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/...","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"response headers end: contentLengthHeader=0","keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.579-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-2","logger":"httpTraceMiddleware","message":"HttpResponse:\nHTTP 200: OK\r\naccept-ranges: bytes\r\ncontent-length: 276044\r\ncontent-type: application\/octet-stream\r\ndate: Wed, 09 Oct 2024 21:09:45 GMT\r\netag: \"23537892b77b791940be4cbb8e45592e\"\r\nlast-modified: Wed, 09 Oct 2024 20:11:36 GMT\r\nserver: AmazonS3\r\nx-amz-id-2: MQLG0jGFfbzBsW8Jdmgr+5HaBu3MM9f+NaNtq6D0UWcZYg2w1MC395d1ineBuEPQ4+haaOTBRIM=\r\nx-amz-request-id: D4E03NJPDJSP3QDJ\r\nx-amz-server-side-encryption: AES256\r\nx-amz-version-id: sjBG4eRd67vhnynCfxT8DbDP_uOhF8z_\r\n\r\n","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.580-06:00[America/Denver] {"level":"DEBUG","thread":"DefaultDispatcher-worker-2","logger":"aws.smithy.kotlin.runtime.awsprotocol.ClockSkewInterceptor","message":"service did not return \"Date\" header, skipping skew calculation","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.581-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-2","logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation completed successfully","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.582-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-2","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"response body available","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.605-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-2","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"response body finished: bytesConsumed=276044","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.607-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-2","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"connection released: conn(id=1940407935)=Connection{marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com:443, proxy=DIRECT hostAddress=marko-codepipeline-poc-test.s3.us-east-1.amazonaws.com\/54.231.224.138:443 cipherSuite=TLS_AES_128_GCM_SHA256 protocol=http\/1.1}; connPool: total=3, idle=3","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}
2024-10-09T15:09:44.608-06:00[America/Denver] {"level":"TRACE","thread":"DefaultDispatcher-worker-2","logger":"aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine","message":"call complete","mdc":{"tenantId":"8e08ebca-16d4-49fa-baac-cc29a9790305"},"keyValues":{"rpc":"S3.GetObject","sdkInvocationId":"ce8b1d40-2a69-4d91-9c4f-ea883ec72ee7"}}

[cloudshiftchris]

Well, took a bit to find a minimal reproducer, and boy is it minimal - this snippet causes retries due to the connection being closed.

suspend fun main() {

    val s3 = S3Client.fromEnvironment {
        // CUSTOM: for running locally
        region = "us-east-1"
        credentialsProvider = DefaultChainCredentialsProvider(profileName = "***")

        retryStrategy {
            maxAttempts = 1 // make retries fail faster for diagnostic purposes
        }
    }

    repeat(7) {
        val getObjectRequest = GetObjectRequest {
            bucket = "***"
            key = "***"  // ~262k ZIP file
        }

        val outputFile = File(System.getProperty("java.io.tmpdir")).resolve("${UUID.randomUUID()}.zip")
        s3.getObject(getObjectRequest) { resp -> resp.body?.writeToFile(outputFile) }
        outputFile.delete()
        delay(15.seconds) // also fails at 10s, 15s, not at 5s ???
    }
}

[ianbotsf]

I've been able to reproduce the issue with the code you've given @cloudshiftchris and I've begun investigating possible causes and solutions.