Ktor client crashes on service error

[sdhuka]

Describe the bug

Ktor client crashes with stacktrace below when service throws exception.

Expected behavior

Ktor should not crash incase of exception.

Current behavior

Stacktrace:

Caused by: java.lang.IllegalStateException: Trying to call 'getOrThrow' on a failed channel result: Failed
        at kotlinx.coroutines.channels.ChannelResult.getOrThrow-impl(Channel.kt:444)
        at aws.smithy.kotlin.runtime.http.engine.ktor.Waiter.signal(KtorEngine.kt:161)
        at aws.smithy.kotlin.runtime.http.engine.ktor.KtorEngine$execute$2$1.invoke(KtorEngine.kt:121)
        at aws.smithy.kotlin.runtime.http.engine.ktor.KtorEngine$execute$2$1.invoke(KtorEngine.kt:121)
        at kotlinx.coroutines.InvokeOnCompletion.invoke(JobSupport.kt:1391)
        at kotlinx.coroutines.JobSupport.notifyCompletion(JobSupport.kt:1519)
        at kotlinx.coroutines.JobSupport.completeStateFinalization(JobSupport.kt:323) 
        at kotlinx.coroutines.JobSupport.finalizeFinishingState(JobSupport.kt:240) 
        at kotlinx.coroutines.JobSupport.continueCompleting(JobSupport.kt:935) 
        at kotlinx.coroutines.JobSupport.access$continueCompleting(JobSupport.kt:27) 
        at kotlinx.coroutines.JobSupport$ChildCompletion.invoke(JobSupport.kt:1155) 
        at kotlinx.coroutines.JobSupport.notifyCompletion(JobSupport.kt:1519) 
        at kotlinx.coroutines.JobSupport.completeStateFinalization(JobSupport.kt:323) 
        at kotlinx.coroutines.JobSupport.finalizeFinishingState(JobSupport.kt:240) 
        at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:906) 
        at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:863) 
        at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:828) 
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:100) 
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) 
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) 

Steps to Reproduce

Make a getObject request using Ktor client and pass invalid key of the object, it result in crash.

val result = runCatching {
    s3.getObject(GetObjectRequest {
                key = INVALID_KEY
                bucket = BUCKEY_NAME
            }) { response -> ... 

            }
}

Possible Solution

No response

Context

No response

AWS Kotlin SDK version used

0.12.0-beta

Platform (JVM/JS/Native)

Native

Operating System and version

Android 31

[kggilmer]

Thanks for the detailed bug report @sdhuka . I've repro'ed the issue, and Android platform is not required to reproduce. Complete program:

fun main() = runBlocking {
    val config = AwsClientConfig.fromEnvironment()

    S3Client(config) {
        httpClientEngine = KtorEngine()
    }.use { client ->
        client.getObject(GetObjectRequest {
            key = "adsf"
            bucket = "asdf"
        }) { }
    }
}

[ianbotsf]

The problem seems to be that receiving an error response interrupts the waiter mechanism in the Ktor engine. Specifically, in KtorEngine.execute, a Waiter is configured to signal upon completion and also to begin waiting after the body is ready to consume. The underlying implementation of Waiter is a rendezvous channel which fails to send (i.e., signal completion) if no consumer is actively receiving (i.e., waiting).

In the case of this S3 error, an exception is thrown before the waiting can begin meaning that the signal of completion will fail. This can be most simply overcome by replacing the waiter's implementation with a simple mutex.

PR coming soon...

[ianbotsf]

PR in smithy-kotlin#589.

[ianbotsf]

PR approved and pushed to smithy-kotlin/main. This fix will be in the next release of the SDK.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.