JedisCluster leaks connections

[serge-groshev]

When redis cluster gets problems (e.g., some nodes crashed and are being restarted), we get some exceptions like this:

redis.clients.jedis.exceptions.JedisException: Could not return the resource to the pool
at redis.clients.util.Pool.returnBrokenResourceObject(Pool.java:100)
at redis.clients.jedis.JedisPool.returnBrokenResource(JedisPool.java:112)
at redis.clients.jedis.JedisPool.returnBrokenResource(JedisPool.java:12)
at redis.clients.jedis.Jedis.close(Jedis.java:3355)
at redis.clients.jedis.JedisClusterCommand.releaseConnection(JedisClusterCommand.java:160)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:154)
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:59)
at redis.clients.jedis.BinaryJedisCluster.get(BinaryJedisCluster.java:98)
at com.jnetx.redis.RedisClient.get(RedisClient.java:92)
at com.jnetx.perffw.RedisPerf$ReadWriteThread.doOperation(RedisPerf.java:78)
at com.jnetx.perffw.ThreadBase.run(ThreadBase.java:173)
Caused by: java.lang.IllegalStateException: Invalidated object not currently part of this pool
at org.apache.commons.pool2.impl.GenericObjectPool.invalidateObject(GenericObjectPool.java:643)
at redis.clients.util.Pool.returnBrokenResourceObject(Pool.java:98)
... 10 more

The reason why it appears is currently unclear to us.

Then we get some exceptions like this:

redis.clients.jedis.exceptions.JedisDataException: LOADING Redis is loading the dataset in memory
at redis.clients.jedis.Protocol.processError(Protocol.java:117)
at redis.clients.jedis.Protocol.process(Protocol.java:151)
at redis.clients.jedis.Protocol.read(Protocol.java:205)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:196)
at redis.clients.jedis.BinaryJedis.ping(BinaryJedis.java:105)
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:42)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:113)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:131)
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:59)
at redis.clients.jedis.BinaryJedisCluster.get(BinaryJedisCluster.java:98)
at com.jnetx.redis.RedisClient.get(RedisClient.java:83)
at com.jnetx.perffw.RedisPerf$ReadWriteThread.doOperation(RedisPerf.java:82)
at com.jnetx.perffw.ThreadBase.run(ThreadBase.java:173)

The connections leak here!

As I see in sources of jedis 2.8.0 and jedis master branch, when such exception occurs, method JedisSlotBasedConnectionHandler.getConnection() does not return the connection back to the pool since it catches JedisConnectionException, but not JedisDataException.

Also we get many exceptions like this:

redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections?
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:97)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:131)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:152)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:131)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:152)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:131)
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:59)
at redis.clients.jedis.BinaryJedisCluster.get(BinaryJedisCluster.java:98)
at com.jnetx.redis.RedisClient.get(RedisClient.java:92)
at com.jnetx.perffw.RedisPerf$ReadWriteThread.doOperation(RedisPerf.java:78)
at com.jnetx.perffw.ThreadBase.run(ThreadBase.java:173)

In this case, it seems that jedis returns connections to the pool, but I am not sure.

And some later, we see all the generator threads blocked with the following stack trace:

Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 186
AbstractQueuedSynchronizer$ConditionObject.await() line: 2043
LinkedBlockingDeque.takeFirst() line: 583
GenericObjectPool.borrowObject(long) line: 442
GenericObjectPool.borrowObject() line: 363
JedisPool(Pool).getResource() line: 48
JedisPool.getResource() line: 99
JedisSlotBasedConnectionHandler.getConnection() line: 36
JedisSlotBasedConnectionHandler.getConnectionFromSlot(int) line: 65
BinaryJedisCluster$3(JedisClusterCommand).runWithRetries(byte[], int, boolean, boolean) line: 115
BinaryJedisCluster$3(JedisClusterCommand).runBinary(byte[]) line: 59
RedisClient$MyCluster(BinaryJedisCluster).get(byte[]) line: 98
RedisClient.get(String) line: 83
RedisPerf$ReadWriteThread.doOperation() line: 82
RedisPerf$ReadWriteThread(ThreadBase).run() line: 173

---

JedisCluster instance is created using the following code:
Set nodes = new LinkedHashSet<>();
....... fill nodes .......
JedisCluster jedisCluster = new JedisCluster(nodes);

All data accesses are performed via calls to jedisCluster.get(byte[]) and jedisCluster.set(byte[], byte[]) in muliple threads.