Request org.redisson.client.RedisTimeoutException during high load #1402
Comments
|
same situation |
|
Hello, can we get some advice on this please? |
|
Can network be a bottleneck here? |
|
Hi Nikita, Thanks for your response. Unfortunately it's not caused by the network. Any help would be greatly appreciated. |
|
Is there any other error preceded this error? Like it was in #1377 |
|
No, not on our side. It's just different command type with same error pattern: We are using org.redisson.codec.FstCodec for serialization (but I supposed it didn't matter?). |
|
+1 |
1 similar comment
|
+1 |
This error means that Redisson unable to get response from Redis in time |
|
Please see the FAQ for more advice. |
|
@jackygurui I think they already did it that as mentioned in description. |
|
I think there is a bit of misunderstanding about the types of timeouts. We need to clarify that |
|
+1 |
|
@hypercube1024 what is the CPU load in your case? |
|
+1 |
|
try reduce connectionMinimumIdleSize to small value ? especially on test environment, because the redission application will create at least (connectionMinimumIdleSize + 1) TCP connection at the same time when RedissonClient instance created. (I noticed this by wireshark) |
|
+1 |
|
+1 org.redisson.client.RedisTimeoutException: Unable to send command! |
|
@lying1234 use |
|
@anjia0532 I tried redis-benchmark on client server to connect to redis server, it runs fast. |
|
@lying1234 paste your redisson config plz. |
|
Here it is: |
|
u config is |
|
@lying1234 paste your redis server config.
|
|
We have two environments: AWS and on-prem. For AWS, it will be configuration endpoint; on-prem, we list all master/salve redis servers actually. From one of the server:
|
|
@lying1234 first , exec exec maybe u should use Replicated mode for AWS redis server , rather than Cluster mode. both of aws and on-premises are timeout? accidental or un-accidental? the max value length (get showlog ref SLOWLOG subcommand [argument])? the
redis server's io , cpu , mem when redisson throws timeout exception? is the redis startup log print |
|
@anjia0532 Thanks for checking. The timeout issue was happened mostly on on-prem, we don't have many users on AWS now. When the timeout issue happened, we didn't see a lot of load on Redis server, connected_clients was lower, and never over 132 on each redis server, which make me to think this is not server side issue. |
|
the same ，+1 |
|
@drluorose command:(HGETALL) ? exec |
|
hlen key = 4，not the key size or redis problem，is redisson network problem！I guess. |
|
change redisson to jedis. |
|
+1 compile("org.redisson:redisson:3.7.0")
|
|
@drluorose Your issue is related to the network environment, as you might have guessed |
|
@jinshubao Your problem is totally different to others here. The problem you have described is caused by a |
|
@jackygurui 问题解决了，连接池没有空闲连接导致的 |
|
@jinshubao |
|
@jinshubao 问题解决了，使用的jackson的序列化方式，entity的包名改了，反序列化出现错误，紧接着就出现了很多超时。把redis清空就好了。 |
|
We still have this RedisTimeoutException happened sporadically. It happened after we got decode error during deserialization for some reason. We already adjusted ConnectionPoolSize/retryAttempts/retryInterval, but looks it didn't help, and we didn't see a lot of connections from server side when the error thrown. It makes me think this may relate to redisson code since others have the similar issue. Also flush redis data is not an option for us. |
|
@lying1234 The timeout exception caused by decode error during deserialisation is inevitable, since the server response timer covers deserialisation process. These type errors are considered unrecoverable and they would cause results to be discarded and the waiting threads would never able to receive the results they are waiting for until the timer is up. Because those commands had already dispatched and executed, Redisson would not going to repeat them. Please consider the timeout exception happens under this circumstance as an indication of a programming error. |
Also please update to 3.6.5 or later. This release contains bug fix for your case. #1377 |
|
I understand timeout exception happened on the request/response with deserialisation failure, but just wonder why different requests from different uses will get the same RedisTimeoutException at that time? Is it because they share the same connection pool? Also, here is part of the error I got for decode error, not sure #1377 will handle this kind of error or not: |
|
The same issue with v3.7.5 |
|
@soockie Please open a new issue ticket and describe your case according to the template requirement. In most cases, your cause is different from those above. |
|
We are using client v3.7.5 and server is v4 . |
|
Could you share the stack trace? |
|
Here it is: i currently have this much. exception Redis server response timeout (200 ms) occured for command: (EVAL) with params: [local value = redis.call('hget', KEYS[1], ARGV[2]); if value == false then return nil; end; local t,..., 5, mya.contracts, redisson__timeout__set:{mya.contracts}, redisson__idle__set:{mya.contracts}, redisson__map_cache__last_access__set:{mya.contracts}, {mya.contracts}:redisson_options, 1539290794706, PooledUnsafeDirectByteBuf(ridx: 0, widx: 114, cap: 256)] channel: [id: 0xadcc4c88, L:/10.110.1.219:45992 - R:/10.110.151.90:6379],org.redisson.client.RedisTimeoutException: Redis server response timeout (200 ms) occured for command: (EVAL) with params: [local value = redis.call('hget', KEYS[1], ARGV[2]); if value == false then return nil; end; local t,..., 5, mya.contracts, redisson__timeout__set:{mya.contracts}, redisson__idle__set:{mya.contracts}, redisson__map_cache__last_access__set:{mya.contracts}, {mya.contracts}:redisson_options, 1539290794706, PooledUnsafeDirectByteBuf(ridx: 0, widx: 114, cap: 256)] channel: [id: 0xadcc4c88, L:/10.110.1.219:45992 - R:/10.110.151.90:6379] |
|
+1 We too face the same issue. We tried various options on client configurations, and reduced keys. Sometimes this is random more frequent, and sometimes the issue happens 1% of times. |
|
@satishcode Can you create a separate issue ticket for your problem, please? The RedisTimeoutException is Redisson reporting to you that it has encountered an issue that's related to the environment it is in, so what you are seeing might not be the same as others. |
|
I did some improvements in retry logic 1a08db7#diff-ed3e437a905b4c04c09be926b2d0c93dR886. So now command will be retried if response couldn't be get. Released in 2.15/3.10 version |
|
@jinshubao @jackygurui |
Hello,
I'm using redisson v3.6.5 with Redis 4. I have 3 masters and 3 slaves nodes for 5 tomcat clusters. However during high load, my team experienced a lot of org.redisson.client.RedisTimeoutException exceptions such as below:
20-Apr-2018 11:21:16.641 SEVERE [http-nio-18080-exec-1790] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [jsp] in context with path [] threw exception [org.redisson.client.RedisTimeoutException: Unable to send command! Node source: NodeSource [slot=null, addr=null, redisClient=null, redirect=null, entry=org.redisson.connection.MasterSlaveEntry@6a5e3f8], connection: [id: 0x8ccbc494, L:/184.170.243.238:38841 - R:/104.200.148.138:6380], command: (HGET), command params: [redisson_tomcat_session:7AB9E70459E2041FED447330E28EF802, PooledUnsafeDirectByteBuf(ridx: 0, widx: 12, cap: 256)] after 3 retry attempts] with root cause
org.redisson.client.RedisTimeoutException: Unable to send command! Node source: NodeSource [slot=null, addr=null, redisClient=null, redirect=null, entry=org.redisson.connection.MasterSlaveEntry@6a5e3f8], connection: [id: 0x8ccbc494, L:/184.170.243.238:38841 - R:/104.200.148.138:6380], command: (HGET), command params: [redisson_tomcat_session:7AB9E70459E2041FED447330E28EF802, PooledUnsafeDirectByteBuf(ridx: 0, widx: 12, cap: 256)] after 3 retry attempts
at org.redisson.command.CommandAsyncService$8.run(CommandAsyncService.java:544)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:668)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:743)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:471)
at java.lang.Thread.run(Thread.java:748)
org.redisson.client.RedisTimeoutException: Unable to send command! Node source: NodeSource [slot=null, addr=null, redisClient=null, redirect=null, entry=org.redisson.connection.MasterSlaveEntry@6a5e3f8], connection: [id: 0x354aeaed, L:/184.170.243.238:57085 - R:/104.200.148.138:6379], command: (HSET), command params: [redisson_tomcat_session:D5C32289A6DD5483FBEABA4A98C5B32A, PooledUnsafeDirectByteBuf(ridx: 0, widx: 12, cap: 256), PooledUnsafeDirectByteBuf(ridx: 0, widx: 178, cap: 256)] after 3 retry attempts
at org.redisson.command.CommandAsyncService$8.run(CommandAsyncService.java:544)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:668)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:743)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:471)
at java.lang.Thread.run(Thread.java:748)
We have increased the timeout param but there seemed to make no difference. Are the parameters mentioned in https://github.com/redisson/redisson/wiki/16.-FAQ#q-i-saw-a-redistimeoutexception-what-does-it-mean-what-shall-i-do-can-redisson-team-fix-it the only variables to tune?
Could you please point me to the direction, as I'm lost with these issues here?
Thank you.
Kevin
The text was updated successfully, but these errors were encountered: