JedisPool exhausted in Jedis 2.10.0 #1920

raytz · 2018-12-21T03:41:27Z

Jedis.java

@Override
public void close() {
    if (dataSource != null) {
        if (client.isBroken()) {
            this.dataSource.returnBrokenResource(this);
        } else {
            this.dataSource.
                    returnResource(this);
        }
        this.dataSource = null;  // <-- This line 
    } else {
        super.close();
    }
}

JedisSentinelPool.java

@Override
public Jedis getResource() {
    while (true) {
        Jedis jedis = super.getResource();
        jedis.setDataSource(this);  // <-- This line 

        // get a reference because it can change concurrently
        final HostAndPort master = currentHostMaster;
        final HostAndPort connection = new HostAndPort(jedis.getClient().getHost(), jedis.getClient()
                .getPort());

        if (master.equals(connection)) {
            // connected to the correct master
            return jedis;
        } else {
            returnBrokenResource(jedis);
        }
    }
}

In the case of concurrency

Thread A return an object but not run to "this.dataSource = null" yet
Thread B borrow an object and set dataSource to this;
And Then Thread A run this.dataSource = null;
Finally Thread A will never returnResource because dataSource is null

Jedis version:

2.10.0

The text was updated successfully, but these errors were encountered:

gkorland · 2018-12-23T16:22:54Z

See #1911

sazzad16 · 2018-12-24T18:10:40Z

@gkorland Isn't this issue actually related to #1918 ?

oliverbestmann · 2018-12-28T10:56:48Z

Sorry, looks like this is still broken in 2.10.1. We are still leaking connections.

kutzi · 2019-01-07T09:31:21Z

Is there any update on this? Should we open a new bug against 2.10.1?

sazzad16 · 2019-01-07T11:07:05Z

I think, it's because of 02f2cc5#diff-df2421269af8d142bf842cb0141f3a95R3639

@gkorland ping

pavel-hp · 2019-01-15T16:45:54Z

@gkorland
We moved from version 2.9.0 to 2.10.1 and we've faced with the same issue.
Leaking connection to Redis.

All our threads have this StackTrace (it waits infinite time)

java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006404ef2d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:587)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:440)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:50)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:235)
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:70)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:110)
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:77)
at redis.clients.jedis.BinaryJedisCluster.evalsha(BinaryJedisCluster.java:1403)

Fix #1920 (#1918) Fix bad backport of commit

kutzi · 2019-01-28T08:44:41Z

FYI: we're running now with 2.10.2 for several days without any connection leak.
Thanks for fixing the issue!

sazzad16 · 2019-01-28T12:41:41Z

@kutzi that's great!

Reset shardjedis dataSource before returned to pool Related to #1920 As it had fixed in Jedis.java, SharedJedis.java should be fixed too.

muhufuk · 2019-03-29T08:26:57Z

I have updated our jedis version from 2.9.1 (Spring boot 1.5.19) to 2.10.2 we have got following exception;

"pool-8-thread-12" #67 prio=5 os_prio=0 tid=0x00005603bac71800 nid=0x60 waiting on condition [0x00007f68ae799000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.springframework.data.redis.cache.RedisCache$AbstractRedisCacheCallback.waitForLock(RedisCache.java:617)
at org.springframework.data.redis.cache.RedisCache$AbstractRedisCacheCallback.doInRedis(RedisCache.java:577)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:207)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:157)
at org.springframework.data.redis.cache.RedisCache.doLookup(RedisCache.java:325)
at org.springframework.data.redis.cache.RedisCache.get(RedisCache.java:184)
at org.springframework.data.redis.cache.RedisCache.get(RedisCache.java:133)
at org.springframework.cache.interceptor.AbstractCacheInvoker.doGet(AbstractCacheInvoker.java:71)
at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:533)
at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:499)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:385)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:323)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)

Reset shardjedis dataSource before returned to pool Related to #1920 As it had fixed in Jedis.java, SharedJedis.java should be fixed too.

xuben · 2019-06-07T12:15:27Z

I use Jedis 2.9.3, it seems like the problem still remains in spite of #1947 . In our case, we set spring.redis.pool.max-active to 4 and ran into the same connection pool leak problem.

sun.misc.Unsafe.park(Native method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:587) org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:440) org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) redis.clients.util.Pool.getResource(Pool.java:49) redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) redis.clients.jedis.JedisPool.getResource(JedisPool.java:16) org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194) org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348) org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:129) org.springframework.data.redis.core.RedisConnectionUtils.bindConnection(RedisConnectionUtils.java:67) org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:192) org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169) org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:91) org.springframework.data.redis.core.DefaultValueOperations.increment(DefaultValueOperations.java:63)

NitishBangera · 2019-08-24T12:53:16Z

I am also facing the same issue with Jedis 2.9.3. The application runs fine without issues for around 3-4 hours and then all the connections show timed waiting in the thread stack and my application throws the "Could not get resource" exception.

"ForkJoinPool.commonPool-worker-1" #3305 daemon prio=5 os_prio=0 tid=0x00007fa458117800 nid=0x3f1357 runnable [0x00007fa5e8ac9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c059fe78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:638)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:427)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:346)
        at redis.clients.util.Pool.getResource(Pool.java:49)
        at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
        at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
        at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:271)
        at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:464)
        at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:132)
        at org.springframework.data.redis.core.RedisConnectionUtils.bindConnection(RedisConnectionUtils.java:70)
        at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:209)
        at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184)

Is this issue resolved in Jedis 2.10.2?

ghost · 2019-11-04T06:39:34Z

There is a little mistake in the top comment.

In the case of concurrency
Thread A return an object but not run to "this.dataSource = null" yet
Thread B borrow an object and set dataSource to this;
And Then Thread A run this.dataSource = null;
Finally Thread A will never returnResource because dataSource is null

It should be:
Finally Thread B will never returnResource because dataSource is null

jack199322zx · 2020-04-23T02:52:44Z

I encountered the same situation with jedis 2.9.1.All connections are exhausted, and all threads are blocked waiting for connections

"Restlight-Biz#99" #217 prio=5 os_prio=0 tid=0x00007fc458028800 nid=0x4f4ab waiting on condition [0x00007fc4e865d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006d2fea6a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:638) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:427) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:346) at redis.clients.util.Pool.getResource(Pool.java:49) at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:70) at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:113) at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:30) at redis.clients.jedis.JedisCluster.hget(JedisCluster.java:413)

loop · 2021-03-29T22:26:27Z

@sazzad16 @gkorland Why is this issue closed? This still seems to be happening and causing major issues. Is there a fix for this or some config we should be setting up?

sazzad16 · 2021-03-29T23:49:22Z

@loop

Which Jedis version are you using?
Which commons-pool2 version are you using?

loop · 2021-03-30T00:08:27Z

@sazzad16 I just ran mvn dependency:tree and got the following:

[INFO] +- redis.clients:jedis:jar:3.5.2:compile
[INFO] |  \- org.apache.commons:commons-pool2:jar:2.6.2:compile

I do not see any other commons-pool2 being used in my project

sazzad16 · 2021-03-30T00:12:56Z

@loop Can you try a commons-pool2 version which is neither of 2.5.x, 2.6.x, 2.7.x?

loop · 2021-03-30T00:14:26Z

@sazzad16 Will do. Do I need to set any additional config as well? Because I'm just using creating my cluster like this:

JedisCluster jedisCluster = new JedisCluster(redisClusterNodes);

sazzad16 · 2021-03-30T00:19:02Z

@loop There's no perfect answer. It really depends on your application.

gkorland added a commit that referenced this issue Dec 25, 2018

Fix #1920 (port #1918)

02f2cc5

gkorland closed this as completed Dec 25, 2018

sazzad16 reopened this Jan 7, 2019

gkorland added a commit that referenced this issue Jan 17, 2019

remove reset after connection was returned to pool

291d37b

Fix #1920 (#1918) Fix bad backport of commit

gkorland mentioned this issue Jan 17, 2019

Remove reset after connection was returned to pool #1935

Merged

sazzad16 pushed a commit that referenced this issue Jan 17, 2019

remove reset after connection was returned to pool (#1935)

bcc0265

Fix #1920 (#1918) Fix bad backport of commit

sazzad16 closed this as completed Jan 17, 2019

gkorland self-assigned this Jan 20, 2019

gkorland added this to the 2.10.2 milestone Jan 20, 2019

timothyzhw mentioned this issue Jan 23, 2019

Reset shardjedis datasource #1939

Merged

jruusu mentioned this issue Mar 14, 2019

Update jedis to 2.10.2 - fixes connection leak issue in JedisPool cb372/scalacache#243

Closed

sazzad16 pushed a commit that referenced this issue Mar 27, 2019

Reset shardjedis datasource (#1939)

79baab4

Reset shardjedis dataSource before returned to pool Related to #1920 As it had fixed in Jedis.java, SharedJedis.java should be fixed too.

marcosnils pushed a commit that referenced this issue May 19, 2019

Reset shardjedis datasource (#1939)

191e9b8

Reset shardjedis dataSource before returned to pool Related to #1920 As it had fixed in Jedis.java, SharedJedis.java should be fixed too.

marcosnils pushed a commit that referenced this issue May 19, 2019

Reset shardjedis datasource (#1939)

1673e9f

Reset shardjedis dataSource before returned to pool Related to #1920 As it had fixed in Jedis.java, SharedJedis.java should be fixed too.

ghost mentioned this issue Nov 6, 2019

报告jedis依赖bug alibaba/jetcache#377

Closed

freshgeek mentioned this issue Oct 28, 2020

[bug] 在3.2.3 中出现死锁,造成应用假死 alexxiyang/shiro-redis#137

Closed

sazzad16 mentioned this issue Apr 6, 2021

Jedis connection in block state #1974

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JedisPool exhausted in Jedis 2.10.0 #1920

JedisPool exhausted in Jedis 2.10.0 #1920

raytz commented Dec 21, 2018

gkorland commented Dec 23, 2018

sazzad16 commented Dec 24, 2018 •

edited

oliverbestmann commented Dec 28, 2018

kutzi commented Jan 7, 2019

sazzad16 commented Jan 7, 2019

pavel-hp commented Jan 15, 2019 •

edited

kutzi commented Jan 28, 2019

sazzad16 commented Jan 28, 2019

muhufuk commented Mar 29, 2019

xuben commented Jun 7, 2019

NitishBangera commented Aug 24, 2019 •

edited

ghost commented Nov 4, 2019

jack199322zx commented Apr 23, 2020

loop commented Mar 29, 2021 •

edited

sazzad16 commented Mar 29, 2021

loop commented Mar 30, 2021

sazzad16 commented Mar 30, 2021

loop commented Mar 30, 2021

sazzad16 commented Mar 30, 2021

JedisPool exhausted in Jedis 2.10.0 #1920

JedisPool exhausted in Jedis 2.10.0 #1920

Comments

raytz commented Dec 21, 2018

Jedis version:

gkorland commented Dec 23, 2018

sazzad16 commented Dec 24, 2018 • edited

oliverbestmann commented Dec 28, 2018

kutzi commented Jan 7, 2019

sazzad16 commented Jan 7, 2019

pavel-hp commented Jan 15, 2019 • edited

kutzi commented Jan 28, 2019

sazzad16 commented Jan 28, 2019

muhufuk commented Mar 29, 2019

xuben commented Jun 7, 2019

NitishBangera commented Aug 24, 2019 • edited

ghost commented Nov 4, 2019

jack199322zx commented Apr 23, 2020

loop commented Mar 29, 2021 • edited

sazzad16 commented Mar 29, 2021

loop commented Mar 30, 2021

sazzad16 commented Mar 30, 2021

loop commented Mar 30, 2021

sazzad16 commented Mar 30, 2021

sazzad16 commented Dec 24, 2018 •

edited

pavel-hp commented Jan 15, 2019 •

edited

NitishBangera commented Aug 24, 2019 •

edited

loop commented Mar 29, 2021 •

edited