Cluster node is duplicated after cluster redeploy

[roboslone]

Hi! I'm running Redis 3.2.0 in cluster mode with 9 nodes in a cloud.

At some point I want to redeploy all the nodes, one by one. That means shutting down old node (by SIGTERM), setting up new directory for a new node, creating new nodes.conf with the same node id, but no other nodes and starting a new node.

Newly created nodes.conf looks like this:

hostname.mydomain-port :0 myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0

Since only one node is restarted at a time it should look like host going down and up in several seconds for other nodes. And it almost does. Except one thing.

Node, that has been redeployed last, is duplicated. Here are outputs of cluster nodes before and after redeploy:

# before
A1 ... myself,master ... connected 0-5460
B1 ... master ... connected 5461-10922
C1 ... master ... connected 10923-16383
A2 ... slave A1 ... connected
A3 ... slave A1 ... connected
B2 ... slave B1 ... connected
B3 ... slave B1 ... connected
C2 ... slave C1 ... connected
C3 ... slave C1 ... connected

# after
A1 ... myself,master ... connected 0-5460
B1 ... master ... connected 5461-10922
C1 ... master ... connected 10923-16383
A2 ... slave A1 ... connected
A3 ... slave A1 ... connected
B2 ... slave B1 ... connected
B3 ... slave B1 ... connected
C2 ... slave C1 ... connected
C3 ... slave C1 ... connected
C3 :0 slave,noaddr C1 ... disconnected

As you can see, there's extra C3 after redeploy, that has no address and is disconnected. Here's that node's logs before and after redeploy:

# old node
19:M 24 Aug 03:01:28.437 # Server started, Redis version 3.2.0
19:M 24 Aug 03:03:52.265 # configEpoch set to 9 via CLUSTER SET-CONFIG-EPOCH
19:M 24 Aug 03:03:52.584 # IP address for this node updated to <C3-addr>
19:M 24 Aug 03:03:57.589 # Cluster state changed: ok
19:S 24 Aug 03:09:18.035 # Connection with master lost.
19:S 24 Aug 03:09:18.447 # Error condition on socket for SYNC: Connection refused
19:S 24 Aug 03:09:19.447 # Error condition on socket for SYNC: Connection refused
19:S 24 Aug 03:09:20.452 # Error condition on socket for SYNC: Connection refused
19:S 24 Aug 03:09:21.455 # Error condition on socket for SYNC: Connection refused
19:S 24 Aug 03:09:22.454 # Error condition on socket for SYNC: Connection refused
19:S 24 Aug 03:09:23.457 # Error condition on socket for SYNC: Connection refused
19:S 24 Aug 03:09:24.462 # Error condition on socket for SYNC: Connection refused
19:S 24 Aug 03:09:25.459 # Error condition on socket for SYNC: Connection refused
19:signal-handler (1471997506) Received SIGTERM scheduling shutdown...
19:S 24 Aug 03:11:46.832 # User requested shutdown...
19:S 24 Aug 03:11:46.832 # Redis is now ready to exit, bye bye...

19:S 24 Aug 03:09:18.035 # Connection with master lost. - that's probably a master redeploy.

# new node
19:M 24 Aug 03:11:54.303 # Server started, Redis version 3.2.0
19:S 24 Aug 03:11:54.304 # Cluster state changed: ok

How could I fix that?

P.S. Cluster seems to work fine even with that extra node.

[roboslone]

Here are C1 logs, if it's any help:

# old node
19:M 24 Aug 03:00:12.289 # Server started, Redis version 3.2.0
19:M 24 Aug 03:03:52.216 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH
19:M 24 Aug 03:03:52.501 # IP address for this node updated to <C1-addr>
19:M 24 Aug 03:03:57.061 # Cluster state changed: ok
19:signal-handler (1471997357) Received SIGTERM scheduling shutdown...
19:M 24 Aug 03:09:18.024 # User requested shutdown...
19:M 24 Aug 03:09:18.024 # Redis is now ready to exit, bye bye...

# new node
19:M 24 Aug 03:09:25.466 # Server started, Redis version 3.2.0
19:M 24 Aug 03:09:27.470 # Cluster state changed: ok
19:M 24 Aug 03:10:35.456 # Connection with slave <C2-addr> lost.
19:M 24 Aug 03:11:46.839 # Connection with slave <C3-addr> lost.