3.11.2 Caused by: io.netty.channel.StacklessClosedChannelException: null

[ljbonbon]

redisson version: 3.11.2
redisson config:
singleServerConfig:
address: "redis://127.0.0.1:6379"
password: null
clientName: null
database: 0
idleConnectionTimeout: 10000
pingTimeout: 1000
connectTimeout: 10000
timeout: 3000
retryAttempts: 3
retryInterval: 1500
reconnectionTimeout: 3000
failedAttempts: 3
subscriptionsPerConnection: 5
subscriptionConnectionMinimumIdleSize: 1
subscriptionConnectionPoolSize: 50
connectionMinimumIdleSize: 32
connectionPoolSize: 3000
dnsMonitoringInterval: 5000
threads: 80
nettyThreads: 60

when we used redisson to addlock,the following errors were encountered
2022-07-11 08:28:32,952 i-C38365BB ERROR [90831.c2c445d.275.9688] [rc] io.iec.edp.caf.commons.exception.handler.ExceptionLogger [https-jsse-nio-5200-exec-5] Unable to send command! Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:49:queue:0, freeConnectionsAmount=28, freeConnectionsCounter=value:2996:queue:0, freezed=false, freezeReason=null, client=[addr=redis://127.0.0.1:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@583039672 [redisClient=[addr=redis://127.0.0.1:6379], channel=[id: 0xaa884b89, L:/127.0.0.1:58094 ! R:127.0.0.1/127.0.0.1:6379], currentCommand=null], command: (EVAL), params: [if (redis.call('exists', KEYS[1]) == 0) then redis.call('hset', KEYS[1], ARGV[2], 1); redis.call('pe..., 1, 10000-barc-manager-0f3ac37a-0449-40eb-87e4-3e8576929c7c, 20000, 1d447097-835a-47f3-b4d7-d0d4b79a46ad:275] after 3 retry attempts
org.redisson.client.WriteRedisConnectionException: Unable to send command! Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:49:queue:0, freeConnectionsAmount=28, freeConnectionsCounter=value:2996:queue:0, freezed=false, freezeReason=null, client=[addr=redis://127.0.0.1:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@583039672 [redisClient=[addr=redis://127.0.0.1:6379], channel=[id: 0xaa884b89, L:/127.0.0.1:58094 ! R:127.0.0.1/127.0.0.1:6379], currentCommand=null], command: (EVAL), params: [if (redis.call('exists', KEYS[1]) == 0) then redis.call('hset', KEYS[1], ARGV[2], 1); redis.call('pe..., 1, 10000-barc-manager-0f3ac37a-0449-40eb-87e4-3e8576929c7c, 20000, 1d447097-835a-47f3-b4d7-d0d4b79a46ad:275] after 3 retry attempts
at org.redisson.command.CommandAsyncService.checkWriteFuture(CommandAsyncService.java:884)
at org.redisson.command.CommandAsyncService.access$000(CommandAsyncService.java:98)
at org.redisson.command.CommandAsyncService$7.operationComplete(CommandAsyncService.java:803)
at org.redisson.command.CommandAsyncService$7.operationComplete(CommandAsyncService.java:800)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:1017)
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:878)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.StacklessClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)

The network is ok. When I use redis-cli to set a value, it is ok.
But when I use redisson api,such as
RLock redLock = getClient(this.redissonConfig).getLock("10000-barc-manager-0f3ac37a-0449-40eb-87e4-3e8576929c7c");
redLock.lock(100, TimeUnit.SECONDS);
it doest not work and the following errors are encountered.

2022-07-11 08:28:32,952 i-C38365BB ERROR [90831.c2c445d.275.9688] [rc] io.iec.edp.caf.commons.exception.handler.ExceptionLogger [https-jsse-nio-5200-exec-5] Unable to send command! Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:49:queue:0, freeConnectionsAmount=28, freeConnectionsCounter=value:2996:queue:0, freezed=false, freezeReason=null, client=[addr=redis://127.0.0.1:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@583039672 [redisClient=[addr=redis://127.0.0.1:6379], channel=[id: 0xaa884b89, L:/127.0.0.1:58094 ! R:127.0.0.1/127.0.0.1:6379], currentCommand=null], command: (EVAL), params: [if (redis.call('exists', KEYS[1]) == 0) then redis.call('hset', KEYS[1], ARGV[2], 1); redis.call('pe..., 1, 10000-barc-manager-0f3ac37a-0449-40eb-87e4-3e8576929c7c, 20000, 1d447097-835a-47f3-b4d7-d0d4b79a46ad:275] after 3 retry attempts
org.redisson.client.WriteRedisConnectionException: Unable to send command! Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:49:queue:0, freeConnectionsAmount=28, freeConnectionsCounter=value:2996:queue:0, freezed=false, freezeReason=null, client=[addr=redis://127.0.0.1:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@583039672 [redisClient=[addr=redis://127.0.0.1:6379], channel=[id: 0xaa884b89, L:/127.0.0.1:58094 ! R:127.0.0.1/127.0.0.1:6379], currentCommand=null], command: (EVAL), params: [if (redis.call('exists', KEYS[1]) == 0) then redis.call('hset', KEYS[1], ARGV[2], 1); redis.call('pe..., 1, 10000-barc-manager-0f3ac37a-0449-40eb-87e4-3e8576929c7c, 20000, 1d447097-835a-47f3-b4d7-d0d4b79a46ad:275] after 3 retry attempts
at org.redisson.command.CommandAsyncService.checkWriteFuture(CommandAsyncService.java:884)
at org.redisson.command.CommandAsyncService.access$000(CommandAsyncService.java:98)
at org.redisson.command.CommandAsyncService$7.operationComplete(CommandAsyncService.java:803)
at org.redisson.command.CommandAsyncService$7.operationComplete(CommandAsyncService.java:800)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:1017)
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:878)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.StacklessClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)

[mrniko]

Try to set connectionPoolSize = 64

[ljbonbon]

Try to set connectionPoolSize = 64

when I set connectionPoolSize = 64, the following errors are sometimes encountered .

org.springframework.dao.InvalidDataAccessApiUsageException: Unable to send command! Try to increase 'nettyThreads' and/or connection pool size settings Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:48:queue:0, freeConnectionsAmount=31, freeConnectionsCounter=value:2999:queue:0, freezed=false, freezeReason=null, client=[addr=redis://127.0.0.1:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@101990895 [redisClient=[addr=redis://127.0.0.1:6379], channel=[id: 0xc39821d7, L:/127.0.0.1:56072 - R:127.0.0.1/127.0.0.1:6379], currentCommand=null], command: (HGETALL), params: [[99, 97, 102, 45, 115, 101, 115, 115, 105, 111, ...]] after 3 retry attempts; nested exception is org.redisson.client.RedisTimeoutException: Unable to send command! Try to increase 'nettyThreads' and/or connection pool size settings Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:48:queue:0, freeConnectionsAmount=31, freeConnectionsCounter=value:2999:queue:0, freezed=false, freezeReason=null, client=[addr=redis://127.0.0.1:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@101990895 [redisClient=[addr=redis://127.0.0.1:6379], channel=[id: 0xc39821d7, L:/127.0.0.1:56072 - R:127.0.0.1/127.0.0.1:6379], currentCommand=null], command: (HGETALL), params: [[99, 97, 102, 45, 115, 101, 115, 115, 105, 111, ...]] after 3 retry attempts
at org.redisson.spring.data.connection.RedissonExceptionConverter.convert(RedissonExceptionConverter.java:48)
at org.redisson.spring.data.connection.RedissonExceptionConverter.convert(RedissonExceptionConverter.java:35)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
at org.redisson.spring.data.connection.RedissonConnection.transform(RedissonConnection.java:234)
at org.redisson.spring.data.connection.RedissonConnection.syncFuture(RedissonConnection.java:229)
at org.redisson.spring.data.connection.RedissonConnection.sync(RedissonConnection.java:459)
at org.redisson.spring.data.connection.RedissonConnection.read(RedissonConnection.java:840)
at org.redisson.spring.data.connection.RedissonConnection.hGetAll(RedissonConnection.java:1537)
at org.springframework.data.redis.core.DefaultHashOperations.lambda$entries$13(DefaultHashOperations.java:245)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:228)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96)
at org.springframework.data.redis.core.DefaultHashOperations.entries(DefaultHashOperations.java:245)
at org.springframework.data.redis.core.DefaultBoundHashOperations.entries(DefaultBoundHashOperations.java:183)
at org.springframework.session.data.redis.RedisIndexedSessionRepository.getSession(RedisIndexedSessionRepository.java:440)
at org.springframework.session.data.redis.RedisIndexedSessionRepository.findById(RedisIndexedSessionRepository.java:412)
at org.springframework.session.data.redis.RedisIndexedSessionRepository.findById(RedisIndexedSessionRepository.java:249)
at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.getRequestedSession(SessionRepositoryFilter.java:351)
at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.getRequestedSessionId(SessionRepositoryFilter.java:333)
at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.commitSession(SessionRepositoryFilter.java:227)
at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.access$100(SessionRepositoryFilter.java:192)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:144)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1726)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.redisson.client.RedisTimeoutException: Unable to send command! Try to increase 'nettyThreads' and/or connection pool size settings Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:48:queue:0, freeConnectionsAmount=31, freeConnectionsCounter=value:2999:queue:0, freezed=false, freezeReason=null, client=[addr=redis://127.0.0.1:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@101990895 [redisClient=[addr=redis://127.0.0.1:6379], channel=[id: 0xc39821d7, L:/127.0.0.1:56072 - R:127.0.0.1/127.0.0.1:6379], currentCommand=null], command: (HGETALL), params: [[99, 97, 102, 45, 115, 101, 115, 115, 105, 111, ...]] after 3 retry attempts
at org.redisson.command.CommandAsyncService$6.run(CommandAsyncService.java:715)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:669)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:744)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:469)
... 1 common frames omitted

[mrniko]

then try connectionPoolSize = 128, 256 ...

[ljbonbon]

then try connectionPoolSize = 128, 256 ...

Can you tell me the relationship between the io.netty.channel.StacklessClosedChannelException: null and the connectionPoolSize.In my opinion, to solve these errors is to constantly adjust parameters.I don't quite understand the relationship between them.

[mrniko]

Your vendor might have a connections amount limitation

[ljbonbon]

Your vendor might have a connections amount limitation

I don't quite understand the meaning of your vendor might have a connections amount limitation.Can you explain why the following errors occur when freeconnectionsamount=31 and freeConnectionsCounter=value:2999.It seems that there is still an idle connection through freeconnectionsamount=31. Why do Redisson still prompt to try to increase'nettythreads'and/or connection pool size settings.

Caused by: org.redisson.client.RedisTimeoutException: Unable to send command! Try to increase 'nettyThreads' and/or connection pool size settings Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:48:queue:0, freeConnectionsAmount=31, freeConnectionsCounter=value:2999:queue:0, freezed=false, freezeReason=null, client=[addr=redis://127.0.0.1:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@101990895 [redisClient=[addr=redis://127.0.0.1:6379], channel=[id: 0xc39821d7, L:/127.0.0.1:56072 - R:127.0.0.1/127.0.0.1:6379], currentCommand=null], command: (HGETALL), params: [[99, 97, 102, 45, 115, 101, 115, 115, 105, 111, ...]] after 3 retry attempts
at org.redisson.command.CommandAsyncService$6.run(CommandAsyncService.java:715)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:669)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:744)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:469)
... 1 common frames omitted

[mrniko]

Sorry, unable to reproduce the issue. If it's occurs with HGETALL command only then try to increase retry interval.

[ljbonbon]

Sorry, unable to reproduce the issue. If it's occurs with HGETALL command only then try to increase retry interval.

I understand that it is difficult to reproduce this problem.Can you help me understand why unable to send command ！Try to increase'nettythreads'and/or connection pool size settings when freeconnectionsamount=31 and freeconnectionscounter=value:2999!

[mrniko]

unable to send command error may arise if data hasn't been sent due to broken connection or other network issues.