How to set custom timeout duration?

[AlexandreBrown]

Describe the issue

I am hitting a timeout error when performing a put request.
I would like to tweak the timeout duration to be much more than 20 seconds.

Steps to Reproduce

1. Make a put request last more than 20 seconds.

Current behavior

The app crashes

ERROR aws.smithy.kotlin.runtime.http.engine.ktor.KtorEngine - throwing
java.util.concurrent.CancellationException: Timed out waiting for 20000 ms
	at aws.smithy.kotlin.runtime.http.engine.CoroutineUtilsKt$attachToOuterJob$cleanupHandler$1.invoke(CoroutineUtils.kt:39)
	at aws.smithy.kotlin.runtime.http.engine.CoroutineUtilsKt$attachToOuterJob$cleanupHandler$1.invoke(CoroutineUtils.kt:37)
	at kotlinx.coroutines.InvokeOnCancelling.invoke(JobSupport.kt:1457)

AWS Kotlin SDK version used

0.14.0-beta

Platform (JVM/JS/Native)

JVM

Operating System and version

Ubuntu 20.04.3 LTS

[aajtodd]

@AlexandreBrown is there anymore context around what you are trying to do that you can share? Or a minimal reproduction?

Is this a single put object request to s3 or is this part of some concurrent operation? Is it a large object?

[AlexandreBrown]

@aajtodd Here is some code sample.

private suspend fun putS3(jsonRequest: SomeKotlinxSerializableDataClass) = 
 withContext(coroutineContext) {
    val bodyJson = Json.encodeToString(SomeKotlinxSerializableDataClass.serializer(), jsonRequest)
    
    val putRequest = PutObjectRequest {
	    bucket = outputS3Bucket
	    key = "$outputS3Prefix/myfile.json"
	    body = ByteStream.fromString(bodyJson)
    }
    
    s3Client.use {
	    it.putObject(putRequest)
    }
}

Here SomeKotlinxSerializableDataClass contains a FloatArray that might be very large. (eg: 524 288 items).
coroutineContext is Dispatchers.IO

[aajtodd]

Was this something that worked before?

Need to dive into which timeout is triggering. To directly answer your question though it depends which timeout we are talking about, not all of them are configurable yet. Some of the HTTP ones are if you configure an http engine explicitly.

A large request shoudn't matter so this feels like a bug to me but need to reproduce.

[AlexandreBrown]

@aajtodd Sometimes it doesn't work on first try, sometimes it works for 1 request but then the next one fails :

Saving predictions result to S3...
19:39:28.865 [DefaultDispatcher-worker-22] DEBUG httpTraceMiddleware - sdkRequestId: 00ae226d-e8cd-4372-8e09-2d95289bbe02; service: S3; operation: PutObject; - HttpResponse: 200: OK  
Saving predictions result to S3...
19:39:28.940 [DefaultDispatcher-worker-22] ERROR aws.smithy.kotlin.runtime.http.engine.ktor.KtorEngine - throwing
kotlinx.coroutines.JobCancellationException: Parent job is Completed
Exception in thread "main" kotlinx.coroutines.JobCancellationException: Parent job is Completed; job=JobImpl{Completed}@24569dba

[aajtodd]

I've been able to reproduce this with a very large upload (1GB). The 20 sec timeout is coming from the retry middleware which is erroneously applying timeouts to the whole request (including upload time).

I'm a little surprised the size document you are dealing with is causing this though. I couldn't reproduce this under 500MB but that could also of course just be differences in upload speed or possibly if you have a highly concurrent environment the request is getting starved.

[AlexandreBrown]

Thank you for investigating @aajtodd ,
I am pretty sure the json file we upload is much lighter than 500MB (maybe 25MB at most would be my guess) that being said it is in a highly concurrent environment therefore it might be that the request is not given high priority since we have a lot of coroutines running on the Dispatchers.IO.

I will continue to investigate to seek a better understanding but it's good that at least you're able to reproduce, we'll be waiting for a fix in the meantime.

Thanks a lot