Fatal exception in coroutines machinery for DispatchedContinuation

[saibotma]

Describe the bug

App crashed with exception.

kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[Dispatchers.IO, Continuation at app.appella.api.persistence.jooq.KotlinDslContext$transaction$3.invokeSuspend(KotlinDslContext.kt)@cbf1194]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
    at kotlinx.coroutines.DispatchedTask.handleFatalException(DispatchedTask.kt:144)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:115)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
    at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
Caused by: java.util.concurrent.RejectedExecutionException: event executor terminated
    at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)
    at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351)
    at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
    at io.ktor.server.netty.NettyDispatcher.dispatch(CIO.kt:69)
    at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:320)
    at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith$default(DispatchedContinuation.kt:276)
    at kotlinx.coroutines.DispatchedCoroutine.afterResume(Builders.common.kt:258)
    at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
    ... 6 more

This was the log message that contained the stack trace:
Unhandled exception caught for CoroutineName(call-handler)
Seems to happen during Ktor call handling.

Versions:
kotlinx.coroutine Packages

• org.jetbrains.kotlinx:kotlinx-coroutines-test:1.7.1
• org.jetbrains.kotlinx:kotlinx-coroutines-slf4j:1.7.1

Ktor Server Packages

• io.ktor:ktor-server-core-jvm:2.3.7
• io.ktor:ktor-server-netty-jvm:2.3.7
• io.ktor:ktor-serialization-jackson:2.3.7
• io.ktor:ktor-server-status-pages:2.3.7
• io.ktor:ktor-server-call-id:2.3.7
• io.ktor:ktor-server-double-receive:2.3.7
• io.ktor:ktor-server-data-conversion:2.3.7
• io.ktor:ktor-server-cors:2.3.7
• io.ktor:ktor-server-content-negotiation:2.3.7
• io.ktor:ktor-server-locations:2.3.7
• io.ktor:ktor-server-auth:2.3.7
• io.ktor:ktor-server-html-builder:2.3.7
• io.ktor:ktor-server-metrics-micrometer:2.3.7
• io.ktor:ktor-server-sessions:2.3.7
• io.ktor:ktor-server-forwarded-header:2.3.7
• io.ktor:ktor-server-websockets:2.3.7
• io.ktor:ktor-server-rate-limit:2.3.7
• io.ktor:ktor-server-test-host:2.3.7

Kotlin Version: 1.9.22

Will try to upgrade all the packages and Kotlin and then report if the issue persists.

Provide a Reproducer
Have not been able to reproduce this. The project where it is happening can not be shared.

[dkhalanskyjb]

There is a PR that will make this a more pleasant error message: #4181
However, even with it, your code will still crash.

The problem seems to be that you are closing a dispatcher before all tasks that use it have run to completion.

I'm not sure which code uses io.netty.util.concurrent.SingleThreadEventExecutor (maybe it's code outside your control), but the recommended solution is to avoid single-threaded dispatchers and executors in favor of val singleThreadedDispatcher = Dispatchers.IO.limitedParallelism(1). This doesn't have the issue of needing to be closed, and so it can't reject tasks.

If it's impossible to change the dispatcher, then you will have to ensure that all tasks using this dispatcher are finished before it is closed.

[dkhalanskyjb]

I hope your question was answered! I'll close the issue, but please ping me if something is still unclear.