EOFException after upgrading AWS SDK Kotlin to 0.22.1-beta

[cj848]

Describe the bug

After upgrading the AWS SDK Kotlin from 0.22.0-beta to 0.22.1-beta in a Spring Boot application, I am encountering an EOFException related to OkHttp when using the SDK to upload files to Amazon S3 while processing multiple concurrent requests using async. Rolling back to 0.22.0-beta resolves the issue.

exception is below.

Caused by: java.io.EOFException: \n not found: limit=0 content=…
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:332) ~[okio-jvm-3.3.0.jar:na]
at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29) ~[okhttp-jvm-5.0.0-alpha.11.jar:na]
at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:179) ~[okhttp-jvm-5.0.0-alpha.11.jar:na]
... 16 common frames omitted

Expected behavior

No EOFException should be thrown, and the application should be able to upload files to Amazon S3 without any issues while processing multiple concurrent requests.

Current behavior

The application throws an EOFException when trying to upload files to Amazon S3 after upgrading the AWS SDK Kotlin to 0.22.1-beta and processing multiple concurrent requests using async.

Steps to Reproduce

Upgrade the AWS SDK Kotlin to 0.22.1-beta.
Configure the S3 client in a Spring Boot application using the provided configuration code.
Use the provided upload logic to upload a file to Amazon S3.
Observe the EOFException being thrown.

@Configuration
@EnableConfigurationProperties(StorageProperties::class)
class StorageConfiguration(private val properties: StorageProperties) {
    @Bean(destroyMethod = "close")
    fun s3Client() = S3Client {
        region = properties.region
        credentialsProvider = StaticCredentialsProvider {
            accessKeyId = properties.accessKey
            secretAccessKey = properties.secretKey
        }
    }
}

@ConfigurationProperties(prefix = "aws.s3")
data class StorageProperties(
    val bucketName: String,
    val accessKey: String,
    val secretKey: String,
    val region: String,
    val keyPrefix: String
)

@Service
class StorageService(
    private val client: S3Client,
    private val properties: StorageProperties,
) {
    suspend fun upload(binary: ByteArray, contentType: String): ByteArray {
        client.putObject {
            bucket = properties.bucketName
            key ="path/to/my/file"
            body = ByteStream.fromBytes(binary)
            this.contentType = contentType
        }
        return binary
    }
}

Possible Solution

No response

Context

No response

AWS Kotlin SDK version used

0.22.1-beta, kotlin 1.8.20 and 1.8.21 both versions occured

Platform (JVM/JS/Native)

JVM

Operating System and version

The application is built using the paketo buildpack with JDK 20 and run as a Docker image.

[mgroth0]

I am having the same issue with 0.27.2-beta

[mgroth0]

And 28.0-beta

[aajtodd]

0.22.1 refactored how certain errors were retried, in particular it added better support for identifying other types of retryable HTTP errors, agnostic of the underlying HTTP engine. We decided to disable the retryOnConnectFailure setting in the OkHttp client. The rationale being that it represents an "internal" retry loop inside of the existing retry strategy for the overall operation. This would potentially violate overall max retry settings for the operation.

We may need to revisit this or try and classify this particular exception as being retryable.

[aajtodd]

This should be fixed by smithy-kotlin#896 and be available in the next release of the SDK.

[cj848]

@aajtodd
thanks for the correction However, we thought it was solved several times, and after updating it, we were frustrated in the operating environment. Could you create a clear reproducible test for this situation? We want to make a regression test and confirm that this error does not occur in the future.

[aajtodd]

This is not easily reproducible unfortunately. We can see the error on S3 PutObject under load. The only way to simulate this is to mock the exception at very low level in the HTTP engine since in this case the exception is related to network issues and comes from OkHttp internals.

If you do start seeing this or similar issues we would be interested to learn more. The main issue is detecting which underlying network issues are retryable. If you find that an exception should be retryable but the SDK doesn't mark it so (either because we haven't identified it or because it isn't safe to generically classify it as retryable) you can always augment the retry policy associated with your client.

e.g.

object MyCustomRetryPolicy: RetryPolicy<Any?> {
    override fun evaluate(result: Result<Any?>): RetryDirective {
        val ex = result.exceptionOrNull()
        return when (ex) {
            is ExceptionICareAbout -> RetryDirective.RetryError(RetryErrorType.ServerSide)
            else -> AwsDefaultRetryPolicy.evaluate(result)
        }
    }
}


fun main() = runBlocking {
     val s3 = S3Client.fromEnvironment{
          retryPolicy = MyCustomRetryPolicy
     }
}

NOTE: you may need to add an explicit dependency on aws.sdk.kotlin:aws-http:<sdk-version> to access the AwsDefaultRetryPolicy

[aajtodd]

Closing as 0.29.0 was released yesterday and should contain a fix to make this exception retryable by default. If you continue to see this issue please re-open or create a new issue with more details.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[pollux-]

I'm still seeing the same issue

[maggie-weber]

I have also seen this issue using 1.0.70. It occurred under a period of high contention within our application.

[SpencerMKSmith]

This is affecting our service and would be a huge help to customers if this could be prioritized.

[pollux-]

There was a fix to do the retry. After that occurrence was significantly reduced. But not completely 👀

[seljabali]

Heads up @aajtodd, still seeing the issue with version 1.1.22 + retry of 4 attempts. Thanks for your help!

[lauzadis]

Hello, I'm unable to reproduce this on v1.2.42. Can you please share a minimal reproduction? Also please ensure you've upgraded to v1.2.x, which included an upgrade to the latest version of OkHttp with various bug fixes.

My (attempted) reproduction code:

fun main(): Unit = runBlocking {
    val client = S3Client.fromEnvironment {
        credentialsProvider = // credentials
        region = "us-east-1"
        httpClient {
            maxConcurrency = 2048u
        }
        logMode = LogMode.LogResponse
    }

    val results = mutableListOf<Deferred<PutObjectResponse>>()
    for (i in 1 until 2048) {
        results.add(async {
            client.putObject {
                bucket = "testbucketmatas"
                key = "test-eofexception/$i.txt"
                body = ByteStream.fromBytes("a".repeat(1024).encodeToByteArray())
                contentType = "text/plain"
            }
        })
    }

    results.awaitAll()
    client.close()
}

[ianbotsf]

Announcement

This issue is closed and, as such, isn't the ideal place to discuss new occurrences based on the latest fixes. Can someone who is still seeing problems with EOFExceptions and retries on a recent version of the SDK please create a new bug report with a minimal repro example? Thanks!