java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out #1561

ant103 · 2017-08-16T07:17:06Z

Expected behavior

No "read time out" should occur.

Actual behavior

"Read time out" often occurs in a live environment.

Steps to reproduce:

I understand that this post might not be related to this board, but as I can’t find any solution to it, I am writing here.
Please understand the situation.

Currently, I am using jedis version 2.1.
I am using pool setting as follow, and I often get “read time out”.

maxActive 600
maxIdle 800
minIdle 10
maxWait 100000

When it occurred, I suspected overload on the expected load capacity.
However, looking at the server index, I found that it also occurred when there is no overload as well.

It mostly occurs when it is reading hash type data.

If there is anything I can check to solve this, please let me know.

===================================
=> java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
#011at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:47)
#011at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:37)
#011at org.springframework.data.redis.connection.jedis.JedisConverters.toDataAccessException(JedisConverters.java:117)
#011at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:169)
#011at org.springframework.data.redis.connection.jedis.JedisConnection.get(JedisConnection.java:1008)
#011at org.springframework.data.redis.connection.DefaultStringRedisConnection.get(DefaultStringRedisConnection.java:264)
#011at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:46)
#011at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:51)
#011at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:185)
#011at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153)
#011at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86)
#011at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:43)
#011at com.nx.npa.dao.impl.NPPKMapDaoImpl.getUserMasterKeyByNxSN(NPPKMapDaoImpl.java:481)

===================================

Redis / Jedis Configuration

Sentinel
Master - Slave

Jedis version:

2.1

Redis version:

2.8.21

Java version:

openjdk 1.7

Etc:

spring-data-redis 1.1.0
commons-pool 1.5.7

Any help is good.
Thank you.
thanks.

marcosnils · 2017-08-16T16:57:43Z

You're using an extremely old Jedis version which we don't support anymore :(

ant103 · 2017-08-16T17:08:22Z

If so, should I upgrade the version?
Is there a problem with changing from 2.1 to 2.7.X?
I am unsure if it is backwards compatible.
Ask for advice.

marcosnils · 2017-08-16T17:10:35Z

@ant103 I think there might be some binary compatibility issues. My suggestion would be to upgrade the dependency and then recompile your app to make sure.

ant103 · 2017-08-16T17:12:52Z

Thank you for your kind reply.
I will upgrade according to your advice.
Have a nice day!

bandikishores · 2017-12-22T18:21:41Z

@ant103 We're facing a similar issue in our production boxes. Was your issue resolved on upgrading to newer binaries?
My Service was a legacy service and is using using spring-data-redis 1.0.0, It was working fine all these years but from last couple of weeks we started running out of connections in the pool and all our requests are hung at

java.lang.Thread.State: WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <28c26a60> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
   at java.lang.Object.wait(Object.java:502)
   at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1104)
   at redis.clients.util.Pool.getResource(Pool.java:20)
   at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:148)
   at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)

Finally resulting in

org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; nested exception is redis.clie
nts.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
        at org.springframework.data.redis.connection.jedis.JedisUtils.convertJedisAccessException(JedisUtils.java:75) ~[iam-service-4.0.37.jar:
4.0.37]
        at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:113) ~[iam-service-
4.0.37.jar:4.0.37]
        at org.springframework.data.redis.connection.jedis.JedisConnection.get(JedisConnection.java:789) ~[iam-service-4.0.37.jar:4.0.37]
        at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:45) ~[iam-service-4.0.37.jar:4.0.37
]
        at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:49) ~[iam-s
ervice-4.0.37.jar:4.0.37]

@marcosnils I understand this is an old version, but could you please point me in the direction as to how I can debug this out? My last resort would be to upgrade the version.
In my scenario, the JedisConnector is initialized with default values (no args constructor). The application as such runs without any issues for a week or 2 and suddenly stops, as all my threads are hung while getting a connection from GenericObjectPool
So even reproducing this would be difficult (So far I had no luck reproducing this issue in local machines).

bandikishores · 2017-12-26T03:12:45Z

Found the issue.
In spring-data-redis 1.0.0 version there was a connection leak bug, which was causing this above issue.

Explanation:
When RedisTemplate invokes RedisConnectionUtils.getConnection(factory)which inturn invokes constructor JedisConnection(Jedis jedis, Pool<Jedis> pool, int dbIndex), then the library was trying to make a call to redis to change the databaseIndex.

// select the db
		if (dbIndex > 0) {
			select(dbIndex);
		}

Unfortunately, we're facing some network issues these days due to which this call resulted in a SocketTimeout. When an exception is thrown from this particular line, none of the above callers handle this exception and close the connection.
because of which this particular connection is never returned to the pool (Connection leak), resulting in the pool thinking this connection is being used. So overtime all our connections from the pool are getting locked and exhausted finally we end up with the above mentioned trace, where there are no connections available.

As @marcosnils suggested, its better to upgrade the version of spring-data-redis, if you're facing this connection leak.

In our scenario, upgrading spring-data-redis has problems of upgrading other spring dependency framework and aspectj versions which are not compatible with some of our code. We're planning on taking this change as part of a bigger release.

For now, we extended the RedisTemplate class overriding execute method to handle the closing of connections if getConnection fails.

PS : I verified in spring-data-redis 1.6.4 and this bug was fixed. (Checked only this version as I wanted redis sentinel feature)

mrafeeq · 2018-01-17T16:32:57Z

I am still having this issue in 1.5.9.RELEASE of spring-boot-starter-data-redis
am using redis cache in spring boot application v 1.5.9.RELEASE

xiongben-tongxue · 2018-06-04T07:31:41Z

You're using an extremely old Jedis version which we don't support

<dependency>
  		<groupId>org.springframework.boot</groupId>
  		<artifactId>spring-boot-starter-data-redis</artifactId>
  		<version>1.5.13.RELEASE</version>
  	</dependency>

rakeshbmurthy · 2018-08-18T10:35:51Z

Am using the latest version of Jedis 2.9.0 with spring-data-redis 1.8.14.RELEASE, facing the same issue

cechaodeng · 2018-10-26T06:52:48Z

using the latest version of Jedis 2.9.0 , facing the same issue

Andi-Lo · 2018-12-07T13:07:12Z

we got the same problem (as @cechaodeng ) with the version

snaketl · 2018-12-11T11:16:05Z

I've the same issue sometimes connecting to AWS Elasticache.
After some time (4 - 7 days), connections just start throwing timeout and after a few seconds everything returns to normal. Any clue?

jssika · 2019-02-10T21:42:17Z

we have been seeing these exceptions in live environment. Doesn't seem to happen in nonprod environments .
" java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out"

Redis / Jedis Configuration

spring-data-redis => 1.8.9.RELEASE

Jedis version: 2.9.0

Redis version: 3.2

Java version: 1.8

In the above thread it is mentioned that these exceptions could occur if you are using older version of spring-data-redis this could occur , but in our case we are using latest version and we are still seeing these exceptions
Any clue ? any body?

licoco · 2019-02-15T09:08:47Z

we got the same problem with the version 2.9.0 @marcosnils

Felicite93 · 2019-03-18T07:55:09Z

we got the same problem in live production after run 1-2hours:
nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out

spring-data-redis 1.8.6
jedis 2.9.0

any clue?

MrOhana · 2019-04-12T14:24:04Z

Hi guys I don't know if help you, but, im my case, I was trying connect in Redis using SSL in the port 6379 in my localhost. So if possible, verify your connection is using SSL method or not and change de connection parameter to the correct scenario.
In Spring the parameter to that is [spring.redis.ssl=true]

galsivan · 2019-05-03T20:34:37Z

Experiencing same issue. not using spring, but using version 2.9.0. funny thing is that i didn't get this when i was connecting to a redis instance running on a kubernetes cluster.
as soon as we switched to a physical machine - boom. i'll try this ssl thing. maybe it's part of the config of the redis servers that is different between the k8s cluster and the physical ubuntu box.

prayjourney · 2020-11-04T09:39:39Z

just use spring-session-data-redis,

org.springframework.boot
spring-boot-starter-data-redis

jjiema · 2021-01-21T16:55:33Z

Hi guys I don't know if help you, but, im my case, I was trying connect in Redis using SSL in the port 6379 in my localhost. So if possible, verify your connection is using SSL method or not and change de connection parameter to the correct scenario.
In Spring the parameter to that is [spring.redis.ssl=true]

@adrian-o is this for the application.properties?

LuisEC2016 · 2021-11-05T18:24:17Z

Experiencing same issue. not using spring, but using version 2.9.0. funny thing is that i didn't get this when i was connecting to a redis instance running on a kubernetes cluster. as soon as we switched to a physical machine - boom. i'll try this ssl thing. maybe it's part of the config of the redis servers that is different between the k8s cluster and the physical ubuntu box.

Excuse me, how do I solve the problem?

pankushrii · 2021-11-25T19:21:32Z

I am facing a similar kind of issue. daily at 12 AM for a few seconds I am getting a time-out. any clue how to fix this?

pankushrii · 2021-11-25T19:23:25Z

I've the same issue sometimes connecting to AWS Elasticache. After some time (4 - 7 days), connections just start throwing timeout and after a few seconds everything returns to normal. Any clue?

@snaketl how did you fix this?

snaketl · 2022-02-04T10:08:02Z

Actually never found a way to fix it, in case someone found I way, please share with us :)

prayjourney · 2022-02-04T10:08:25Z

我已收到您的来信，谢谢！祝您生活愉快！

rajatjupiter · 2022-12-13T16:50:23Z

How did we solve this one we are facing the same issue in production.
Libraries Involved:

JedisVersion: 3.6.0
spring-boot-starter-data-redis-2.3.5.RELEASE.
Redis: AWS Elastic Cache (Cluster Mode Off) (Node: cache.r6g.large)

we are getting intermittent timeouts with error.
error: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out

prayjourney · 2022-12-13T16:50:44Z

我已收到您的来信，谢谢！祝您生活愉快！

rajatjupiter · 2022-12-13T17:12:57Z

hey did you resolve this ?

BradGamez · 2022-12-21T03:00:50Z

Has anyone found a resolution to this issue. I am also currently experiencing this issue.

redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
    at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:208)
    at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:46)
    at redis.clients.jedis.Protocol.process(Protocol.java:126)
    at redis.clients.jedis.Protocol.read(Protocol.java:192)
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:316)
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:243)
    at redis.clients.jedis.Jedis.ping(Jedis.java:356)
    at redis.clients.jedis.JedisFactory.validateObject(JedisFactory.java:221)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:343)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223)
    at redis.clients.jedis.util.Pool.getResource(Pool.java:38)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:378)
    at com.moonsidemc.moonsidebungeecore.databasemanager.redis.RedisManager.getJedis(RedisManager.java:42)
    at com.moonsidemc.moonsidebungeecore.databasemanager.redis.RedisManager.getAllCacheAccountIDs(RedisManager.java:119)
    at com.moonsidemc.moonsidebungeecore.utilities.StaffChatMethods.sendMessage(StaffChatMethods.java:13)
    at com.moonsidemc.moonsidebungeecore.databasemanager.redis.utilities.RedisListener.onMessage(RedisListener.java:22)
    at redis.clients.jedis.JedisPubSub.process(JedisPubSub.java:156)
    at redis.clients.jedis.JedisPubSub.proceed(JedisPubSub.java:125)
    at redis.clients.jedis.Jedis.subscribe(Jedis.java:7915)
    at com.moonsidemc.moonsidebungeecore.databasemanager.redis.utilities.RedisConnection.run(RedisConnection.java:14)
    at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
    at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
    at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
    at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
    at java.base/java.net.Socket$SocketInputStream.read(Socket.java:976)
    at java.base/java.io.InputStream.read(InputStream.java:218)
    at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:202)
    ... 20 more
    ```

vedev9 · 2023-03-09T09:19:44Z

Has anyone found a resolution to this issue. I am also currently experiencing this issue.

redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
    at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:208)
    at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:46)
    at redis.clients.jedis.Protocol.process(Protocol.java:126)
    at redis.clients.jedis.Protocol.read(Protocol.java:192)
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:316)
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:243)
    at redis.clients.jedis.Jedis.ping(Jedis.java:356)
    at redis.clients.jedis.JedisFactory.validateObject(JedisFactory.java:221)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:343)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223)
    at redis.clients.jedis.util.Pool.getResource(Pool.java:38)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:378)
    at com.moonsidemc.moonsidebungeecore.databasemanager.redis.RedisManager.getJedis(RedisManager.java:42)
    at com.moonsidemc.moonsidebungeecore.databasemanager.redis.RedisManager.getAllCacheAccountIDs(RedisManager.java:119)
    at com.moonsidemc.moonsidebungeecore.utilities.StaffChatMethods.sendMessage(StaffChatMethods.java:13)
    at com.moonsidemc.moonsidebungeecore.databasemanager.redis.utilities.RedisListener.onMessage(RedisListener.java:22)
    at redis.clients.jedis.JedisPubSub.process(JedisPubSub.java:156)
    at redis.clients.jedis.JedisPubSub.proceed(JedisPubSub.java:125)
    at redis.clients.jedis.Jedis.subscribe(Jedis.java:7915)
    at com.moonsidemc.moonsidebungeecore.databasemanager.redis.utilities.RedisConnection.run(RedisConnection.java:14)
    at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
    at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
    at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
    at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
    at java.base/java.net.Socket$SocketInputStream.read(Socket.java:976)
    at java.base/java.io.InputStream.read(InputStream.java:218)
    at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:202)
    ... 20 more
    ```

Same here
Has anyone found the reason/solution?

thanks

itsKarad · 2023-05-19T05:46:18Z

Has anyone found a resolution? I have also tried setting the connection timeout.

prayjourney · 2023-05-19T05:46:40Z

我已收到您的来信，谢谢！祝您生活愉快！

juzunoglu · 2023-06-10T15:55:24Z

Any solutions?

prayjourney · 2023-06-10T15:55:45Z

我已收到您的来信，谢谢！祝您生活愉快！

itsKarad · 2023-06-11T11:10:07Z

What I found:
You can upgrade Jedis and Redis to use newer versions since this issue is reoccurring only in older versions.
What worked for me: Since I was using AWS Elasticache, I tried to upgrade the node_type of my cache from t2.micro to m4.xlarge, which resolved the issue for me.

More details

queliang96 · 2024-07-10T02:00:49Z

we have been seeing these exceptions in live environment. Doesn't seem to happen in nonprod environments . " java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out"

Redis / Jedis Configuration

spring-data-redis => 1.8.9.RELEASE

Jedis version: 2.9.0

Redis version: 3.2

Java version: 1.8

In the above thread it is mentioned that these exceptions could occur if you are using older version of spring-data-redis this could occur , but in our case we are using latest version and we are still seeing these exceptions Any clue ? any body?

How did you solve the problem in the end?

marcosnils closed this as completed Aug 16, 2017

Athou mentioned this issue May 27, 2023

API docs down/moved Athou/commafeed#1060

Closed

java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out #1561

java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out #1561

Comments

ant103 commented Aug 16, 2017

Expected behavior

Actual behavior

Steps to reproduce:

Redis / Jedis Configuration

Jedis version:

Redis version:

Java version:

Etc:

marcosnils commented Aug 16, 2017

ant103 commented Aug 16, 2017

marcosnils commented Aug 16, 2017

ant103 commented Aug 16, 2017

bandikishores commented Dec 22, 2017 • edited Loading

bandikishores commented Dec 26, 2017 • edited Loading

mrafeeq commented Jan 17, 2018

xiongben-tongxue commented Jun 4, 2018 • edited Loading

rakeshbmurthy commented Aug 18, 2018

cechaodeng commented Oct 26, 2018

Andi-Lo commented Dec 7, 2018

snaketl commented Dec 11, 2018

jssika commented Feb 10, 2019

Redis / Jedis Configuration

Jedis version: 2.9.0

Redis version: 3.2

Java version: 1.8

licoco commented Feb 15, 2019

Felicite93 commented Mar 18, 2019

MrOhana commented Apr 12, 2019

galsivan commented May 3, 2019

prayjourney commented Nov 4, 2020

jjiema commented Jan 21, 2021

LuisEC2016 commented Nov 5, 2021

pankushrii commented Nov 25, 2021

pankushrii commented Nov 25, 2021

snaketl commented Feb 4, 2022

prayjourney commented Feb 4, 2022 via email

rajatjupiter commented Dec 13, 2022

prayjourney commented Dec 13, 2022 via email

rajatjupiter commented Dec 13, 2022

BradGamez commented Dec 21, 2022 • edited Loading

vedev9 commented Mar 9, 2023 • edited Loading

itsKarad commented May 19, 2023

prayjourney commented May 19, 2023 via email

juzunoglu commented Jun 10, 2023

prayjourney commented Jun 10, 2023 via email

itsKarad commented Jun 11, 2023

queliang96 commented Jul 10, 2024

Redis / Jedis Configuration

Jedis version: 2.9.0

Redis version: 3.2

Java version: 1.8

bandikishores commented Dec 22, 2017 •

edited

Loading

bandikishores commented Dec 26, 2017 •

edited

Loading

xiongben-tongxue commented Jun 4, 2018 •

edited

Loading

BradGamez commented Dec 21, 2022 •

edited

Loading

vedev9 commented Mar 9, 2023 •

edited

Loading