JedisCluster works in 2.7.x but doesn't work in 2.9.x with JedisConnectionException and Connection refused. (Single node cluster)

[yanghuirong]

Expected behavior

JedisCluster in Jedis 2.9.x works as same as Jedis 2.7.x

Actual behavior

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
	at redis.clients.util.Pool.getResource(Pool.java:53) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:66) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:116) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisCluster.get(JedisCluster.java:124) ~[jedis-2.9.x.jar:?]
.........
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused (Connection refused)
	at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.x.jar:?]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836) ~[commons-pool2-2.2.jar:2.2]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434) ~[commons-pool2-2.2.jar:2.2]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.2.jar:2.2]
	at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.x.jar:?]
	... 45 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_144]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_144]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_144]
	at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.x.jar:?]
	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.x.jar:?]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836) ~[commons-pool2-2.2.jar:2.2]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434) ~[commons-pool2-2.2.jar:2.2]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.2.jar:2.2]
	at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.x.jar:?]
	... 45 more

Steps to reproduce:

Summary: I have the Jedis 2.7.x for several months. But i found I have the same issue as #1625 (JedisPool.getResource block when redis server restart) . So I switched from 2.7.x to 2.9.x. I am using the same way to create JedisCluster. But it failed to connect and get data from Jedis. I created a singleton JedisCluster instance and every time i just use JedisCluster.get() to get data. I don't close connections for cluster.

Please create a reproducible case of your problem. Make sure
that case repeats consistently and it's not random

1. I have Jedis 2.7.x . Code below worked:

    @Singleton
    JedisCluster jedisCluster(){
            Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
            jedisClusterNodes.add(new HostAndPort("10.xx.xx.xx", 6379));
            GenericObjectPoolConfig jedisPoolConfig = new GenericObjectPoolConfig();
            jedisPoolConfig.setMaxTotal(16);
            jedisPoolConfig.setMaxIdle(16);
            return new JedisCluster(jedisClusterNodes, 2000, 1000, 5, jedisPoolConfig);
   }
   //When using cluster
    jedisCluster.set("foo", "bar");
    String value = jedisCluster.get("foo");

2. I switched to Jedis 2.9.x, below code worked

           Jedis jedis = new Jedis("xx.xx.xxx.xx",6379);
           System.out.println(jedis.get("foo"));

3. I ssh to the Redis host and run "get foo" worked.

127.0.0.1:6379> get foo
"bar

4. Code below is the same as 1 but it doesn't work. I switched back to Jedis 2.7.x and it worked. I tried to switch Jedis version several times and only 2.7.x worked.

    @Singleton
    JedisCluster jedisCluster(){
            Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
            jedisClusterNodes.add(new HostAndPort("10.xx.xx.xx", 6379));
            GenericObjectPoolConfig jedisPoolConfig = new GenericObjectPoolConfig();
            jedisPoolConfig.setMaxTotal(16);
            jedisPoolConfig.setMaxIdle(16);
            return new JedisCluster(jedisClusterNodes, 2000, 1000, 5, jedisPoolConfig);
   }
   //When using cluster
    jedisCluster.set("foo", "bar");
    String value = jedisCluster.get("foo");

log from redis side

3088:M 21 Mar 21:42:24.187 - Accepted xxx.xxx.xx.xx:36064
3088:M 21 Mar 21:42:24.197 - Accepted xxx.xxx.xx.xx:36065
3088:M 21 Mar 21:42:24.212 - Reading from client: Connection reset by peer
3088:M 21 Mar 21:42:28.481 - DB 0: 12xxxxxx keys (0 volatile) in 16777216 slots HT.
3088:M 21 Mar 21:42:28.482 - 1 clients connected (0 slaves), 1852164712 bytes in use

Redis / Jedis Configuration

I am testing with one single host and use it as a cluster.

127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:1
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

Jedis version:

2.7.x works fine
2.9.x having problem

Redis version:

3.2.8

Java version:

I don't know. I am using JDK8

[marcosnils]

what does your cluster slots command return?

[yanghuirong]

@marcosnils
Yeah,

1) 1) (integer) 0
   2) (integer) 16383
   3) 1) ""
      2) (integer) 6379
      3) "c97xxxxxxxxxxxxxxxxxxxxxxxxxx"

[marcosnils]

I believe that's the reason. Jedis uses cluster slots from 2.8 onwards (more info here: #1211). So your client needs to be able to resolve / connect to the hostname that appears there.

[yanghuirong]

@marcosnils WOW!!!!! This solved my problem!!! Previously I didn't create cluster in my single host and I could directly use it as a cluster in 2.7.x. I justran the below command which solved my problem.

for i in {0..16383}; do bin/redis-cli -h <ipOfHost1> CLUSTER ADDSLOTS $i; done
cluster meet <ip> 6379

Feel free to close this issue.
Thank you very much for your help! I will create new issue with my other questions.