Repository navigation
READONLY init uses ForkJoinPool.commonPool(); blocked common pool causes replica-read timeouts #3890
SummaryIn Redis Cluster with if (key.connectionIntent == ConnectionIntent.READ) {
connection = connection.thenCompose(c -> {
RedisFuture<String> future = c.async().readOnly();
return future.thenApply(s -> c).whenCompleteAsync((s, throwable) -> {
if (throwable != null) {
c.close();
}
});
});
}
whenCompleteAsync without an executor schedules on ForkJoinPool.commonPool(). If the
common pool is saturated by other blocking work in the JVM, the chained stage never
completes, getConnectionAsync(...) never returns, and the caller blocks in
LettuceConnection.await() until command timeout — indistinguishable from a slow Redis
command (SLOWLOG shows nothing).
Reproduction
1. Saturate common pool workers with blocking tasks.
2. Issue the first read against a replica (fresh connection, replica-preferred).
3. Read hangs and times out after the configured timeout (5s); succeeds immediately once
workers are released.
Production correlation: bursts of RedisCommandTimeoutException across GET/HGET/EXISTS/
ZRANGE right after startup / rolling deploys, with thread dumps showing all common pool
workers blocked and ~58 threads stuck in LettuceConnection.await().
Environment
Lettuce 6.3.2.RELEASE; same code still present on current main.
Question
Is there a specific reason whenCompleteAsync was chosen here over plain whenComplete,
given the callback (c.close()) is non-blocking? As far as I can tell, running it
synchronously on the completing thread would remove the dependency on
ForkJoinPool.commonPool() entirely. Are there other spots where Lettuce implicitly relies
on the common pool that could hit the same problem? |
Answered by
a-TODO-rov
Aug 14, 2026
Replies: 1 comment
|
Hey @nei10u |
0 replies
Answer selected by
a-TODO-rov
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey @nei10u
c.close() is blocking and the reason it was offloaded to ForkJoinPool.commonPool() after issue #453 is because the callback would have run on Nettys eventloop otherwise.
However, later closeAsync() was introduced, which is indeed non-blocking. If we use it i think we can safely switch to whenComplete.
I think there are a few other occurrences of delegation to ForkJoinPool.commonPool() in RedisHandshake path and MultiDB.
You are welcome to open an issue if you like.