IllegalMonitorStateException in ThreadPoolExecutor with Redisson Integration

[cuongqluu]

When using BlockHound with a Spring Boot application, an IllegalMonitorStateException is encountered in a ThreadPoolExecutor thread.

Expected Behavior

The application should run without encountering IllegalMonitorStateException or other synchronization-related exceptions when BlockHound is enabled.

Actual Behavior

The application throws IllegalMonitorStateException with the following stack trace:

Exception in thread "redisson-4-5203"
    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1059)
    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
    at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:1583)

Steps to Reproduce

1. Setup Redis Server by Docker:

docker run -d -it -v %cd%/redis_data:/data -p 6379:6379 redis:7.2.4-alpine redis-server --requirepass "P@ssword123"

2. Add redisson dependency:

<dependency>
	<groupId>org.redisson</groupId>
	<artifactId>redisson-spring-boot-starter</artifactId>
	<version>3.27.2</version>
</dependency>

3. Add RedisConfig:

@Configuration
public class RedisConfig {

    @Bean(destroyMethod = "shutdown")
    @ConditionalOnMissingBean({RedissonClient.class})
    public RedissonClient redisson() {
        String ip = System.getProperty("redis.ip");
        String port = System.getProperty("redis.port");
        String password = System.getProperty("redis.password");

        Config config = new Config();
        String redisUrl = "redis://" + ip + ":" + port;
        config.useSingleServer().setAddress(redisUrl);

        // Set Redis server password if provided
        if (password != null && !password.isEmpty()) {
            config.useSingleServer().setPassword(password);
        }

        return Redisson.create(config);
    }

    @Bean
    public CacheManager cacheManager(RedissonClient redissonClient) {
        return new RedissonSpringCacheManager(redissonClient);
    }
}

4. Add VM Options:

-Dredis.ip=localhost
-Dredis.port=6379
-Dredis.password=P@ssword123

5. Start a Spring Boot application with BlockHound enabled.

public static void main(String[] args) throws Exception {
    BlockHound.install();
    SpringApplication.run(AccountApplication.class, args);
}

Environment

BlockHound Version: 1.0.8.RELEASE
Redisson Version: 3.27.2
Spring Boot Version: 3.2.1
Java Version: JDK 21
Operating System: Windows

Additional Information

• The application uses RedissonClient for Redis operations, which might be related to the issue.

[pderop]

thanks for reporting, will take a look.

[pderop]

I could reproduce using this sample project:

403-repro.tgz

now, it seems that there are some BlockingOperationError exception that are happening but they are not logged.
For example, I have hacked the BlockHound BlockingOperationError class like this:

    public BlockingOperationError(BlockingMethod method) {
        super(String.format("Blocking call! %s", method));
        this.method = method;
        StringWriter stringWriter = new StringWriter();
        PrintWriter printWriter = new PrintWriter(stringWriter);
        this.printStackTrace(printWriter);
        System.out.println("[" + Thread.currentThread().getName() + "] BlockingOperationError: " + stringWriter.toString());
    }

then I can see a tons of exceptions like this:

[redisson-3-1] BlockingOperationError: reactor.blockhound.BlockingOperationError: Blocking call! jdk.internal.misc.Unsafe#park
	at reactor.blockhound.BlockHound$Builder.lambda$new$0(BlockHound.java:260)
	at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:490)
	at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89)
	at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
	at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
	at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
	at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
	at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

there is a permitBlockingCalls method in FastThreadLocalThread which can be used to mark a FastThreadLocalThread so it can run blocking tasks, but I don't know what is exactly happening for the moment.

while trying to debug, I observed for example that RedisClient can schedule some FastThreadLocalRunnable tasks like in the following stacktrace:

java.lang.Exception: Stack trace
	at java.base/java.lang.Thread.dumpStack(Thread.java:2209)
	at io.netty.util.concurrent.FastThreadLocalRunnable.<init>(FastThreadLocalRunnable.java:24)
	at io.netty.util.concurrent.FastThreadLocalRunnable.wrap(FastThreadLocalRunnable.java:38)
	at io.netty.util.concurrent.DefaultThreadFactory.newThread(DefaultThreadFactory.java:105)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
	at org.redisson.client.RedisClient$1$1.run(RedisClient.java:288)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:31)
	at java.base/java.lang.Thread.run(Thread.java:1583)

so, it looks like some blocking codes are scheduled in some netty FastThreadLocalThreads, which are assumed to be non blocking.

honestly, I'm not sure to understand everything here, but I cannot see any issues in BlockHound itself for the moment.

[cuongqluu]

I think I can skip blocking inside by
BlockHound.builder().allowBlockingCallsInside(Vertx.class.getName(), "executeBlocking").install(),
but it seems to skip all blocking operations. Can you guide me how I can find the exactly class and methods to use allowBlockingCallsInside().

[pderop]

I tried it like the following, but it does not seem to resolve the problem:

		BlockHound.install(builder -> builder.allowBlockingCallsInside("io.vertx.core.Vertx", "executeBlocking"));

However, I tried this which works, but it's probably not a good idea to allow blocking calls from netty FastThreadLocalThread, maybe the following could be done with adding a flag in order to only allow blocking calls during the startup time ? The following allows all blocking calls from netty FastThreadLocalThread threads:

		BlockHound.install(builder -> builder.nonBlockingThreadPredicate(p ->
				thread -> thread instanceof FastThreadLocalThread ? false : p.test(thread)));

you can also do this, which allow blocking calls from any FastThreadLocalRunnable.run methods (but is it what should be done, I'm not sure):

		BlockHound.install(builder -> builder.allowBlockingCallsInside("io.netty.util.concurrent.FastThreadLocalRunnable", "run"));

you can check (Dis-)allowing blocking calls inside methodssection from docs for more examples.

[draguljce]

I've hit the same issue and managed to work around it like so:

BlockHound.install(builder -> {
            builder.nonBlockingThreadPredicate(p -> thread -> !thread.getName().contains("redisson") && p.test(thread));
        });

[cuongqluu]

It seems to the Redisson does not support Reactive completely, so maybe allow Redisson call blocking inside.
@pderop the builder.nonBlockingThreadPredicate(..) works for me.
But the builder.allowBlockingCallsInside(..) does not work.
@draguljce I tried your code and it just passed the redisson startup time. Because Redisson still call FastThreadLocalRunnable in runtime.

[pderop]

if you don't mind, I will close this issue, because for the moment I don't see any issues to fix in BlockHound itself.
but feel free to reopen, if it's necessary.