Cluster: nodes have a different slots mapping after a reshard

[jcsc]

Randomly, after a reshard, nodes don't share the same slot mapping.

Here is an example. Before reshard, there are 4 nodes:

127.0.0.1:7000> CLUSTER NODES
decc1ead97b1f3cb5697932908ac7dd5cdbddcc9 127.0.0.1:7000 myself,master - 0 0 5 connected 0-4095
84d9ae4782d9ac54a7e42245d168408ae3192547 127.0.0.1:7004 master - 0 1450708872446 9 connected 8192-12287
96487b2c01d16479baa61839e5831c17551f9694 127.0.0.1:7002 master - 0 1450708872446 8 connected 4096-8191
a880d1c55a57bf7d3c85869c28aa9b55f7c87cc5 127.0.0.1:7006 master - 0 1450708872446 10 connected 12288-16383

One node is removed, and after reshard, the slots 5458-5461 are not correctly updated on one node:

127.0.0.1:7000> CLUSTER NODES
decc1ead97b1f3cb5697932908ac7dd5cdbddcc9 127.0.0.1:7000 myself,master - 0 0 11 connected 0-5458
84d9ae4782d9ac54a7e42245d168408ae3192547 127.0.0.1:7004 master - 0 1450708955427 13 connected 10923-16383
96487b2c01d16479baa61839e5831c17551f9694 127.0.0.1:7002 master - 0 1450708955427 12 connected 5459-10922

127.0.0.1:7002> CLUSTER NODES
decc1ead97b1f3cb5697932908ac7dd5cdbddcc9 127.0.0.1:7000 master - 0 1450708969194 11 connected 0-5461
84d9ae4782d9ac54a7e42245d168408ae3192547 127.0.0.1:7004 master - 0 1450708969194 13 connected 10923-16383
96487b2c01d16479baa61839e5831c17551f9694 127.0.0.1:7002 myself,master - 0 0 12 connected 5462-10922


127.0.0.1:7004> CLUSTER NODES
84d9ae4782d9ac54a7e42245d168408ae3192547 127.0.0.1:7004 myself,master - 0 0 13 connected 10923-16383
decc1ead97b1f3cb5697932908ac7dd5cdbddcc9 127.0.0.1:7000 master - 0 1450709010033 11 connected 0-5461
96487b2c01d16479baa61839e5831c17551f9694 127.0.0.1:7002 master - 0 1450709010033 12 connected 5462-10922

127.0.0.1:7000> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 5458
   3) 1) "127.0.0.1"
      2) (integer) 7000
2) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7004
3) 1) (integer) 5459
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7002

127.0.0.1:7002> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 5461
   3) 1) "127.0.0.1"
      2) (integer) 7000
2) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7004
3) 1) (integer) 5462
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7002

127.0.0.1:7000> INFO
# Server
redis_version:3.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:3accd3e0dcba738
redis_mode:cluster
os:Linux 3.10.0-229.1.2.36242.MSSr4.el7.centos.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.2
process_id:8856
run_id:bb4c35f70e29baa5616d56b93d26317484477669
tcp_port:7000
uptime_in_seconds:755
uptime_in_days:0
hz:10
lru_clock:7868681
config_file:/var/opt/envivio/share/xps/config/redisCluster/127.0.0.1_7000/node.cfg

# Clients
connected_clients:2
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:2439952
used_memory_human:2.33M
used_memory_rss:9707520
used_memory_peak:3226432
used_memory_peak_human:3.08M
used_memory_lua:36864
mem_fragmentation_ratio:3.98
mem_allocator:jemalloc-3.6.0

# Persistence
loading:0
rdb_changes_since_last_save:28494
rdb_bgsave_in_progress:0
rdb_last_save_time:1450708502
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_current_size:1206172
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

# Stats
total_connections_received:57
total_commands_processed:120689
instantaneous_ops_per_sec:24
total_net_input_bytes:8069373
total_net_output_bytes:1334030
instantaneous_input_kbps:0.62
instantaneous_output_kbps:0.35
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:39089
keyspace_misses:5110
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:8.63
used_cpu_user:25.93
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=6659,expires=0,avg_ttl=0

nodes_7000.conf

decc1ead97b1f3cb5697932908ac7dd5cdbddcc9 127.0.0.1:7000 myself,master - 0 0 11 connected 0-5458
84d9ae4782d9ac54a7e42245d168408ae3192547 127.0.0.1:7004 master - 0 1450708923614 13 connected 10923-16383
96487b2c01d16479baa61839e5831c17551f9694 127.0.0.1:7002 master - 0 1450708923615 12 connected 5459-10922
vars currentEpoch 13 lastVoteEpoch 0

nodes_7002.conf

decc1ead97b1f3cb5697932908ac7dd5cdbddcc9 127.0.0.1:7000 master - 0 1450708923628 11 connected 0-5461
84d9ae4782d9ac54a7e42245d168408ae3192547 127.0.0.1:7004 master - 0 1450708923628 13 connected 10923-16383
96487b2c01d16479baa61839e5831c17551f9694 127.0.0.1:7002 myself,master - 0 0 12 connected 5462-10922
vars currentEpoch 13 lastVoteEpoch 0

nodes_7004.conf

84d9ae4782d9ac54a7e42245d168408ae3192547 127.0.0.1:7004 myself,master - 0 0 13 connected 10923-16383
decc1ead97b1f3cb5697932908ac7dd5cdbddcc9 127.0.0.1:7000 master - 0 1450708923576 11 connected 0-5461
96487b2c01d16479baa61839e5831c17551f9694 127.0.0.1:7002 master - 0 1450708923576 12 connected 5462-10922
vars currentEpoch 13 lastVoteEpoch 0

[antirez]

Hello, if I understand this correctly in order to replicate this I should:

1. Setup a 4 nodes cluster.
2. Fill it with some data.
3. Reshard away all the slots of a given node to the other three nodes (how do you do this exactly, so that I can do things as similarly as possible?)
4. Finally remove the master entirely.

Am I right? Could you kindly provide the steps you use so I can try to reproduce this? Thanks.

[jcsc]

Yes you are rigth.

My application is written in nodejs. Slots are moved in series.
In our case, slots are moved according to the following table:

Slot 4096 127.0.0.1:7002 -> 127.0.0.1:7000
...
Slot 5458 127.0.0.1:7002 -> 127.0.0.1:7000
Slot 8192 127.0.0.1:7004 -> 127.0.0.1:7002
...
Slot 10922 127.0.0.1:7004 -> 127.0.0.1:7002
Slot 12288 127.0.0.1:7006 -> 127.0.0.1:7004
...
Slot 16383 127.0.0.1:7006 -> 127.0.0.1:7004

Here is the methods use to move each slot:

/**
 * Move a single slot from one node to another.
 * Async.
 * @param slot              {number} the slot to move
 * @param previousNode      {object} RedisNode object
 * @param newNode           {object} RedisNode object
 * @param callback          {function} function(err)
 */ 
RedisCluster.prototype.MoveOneSlot = function(slot, previousNode, newNode, callback)
{   
    var self = this;

    // 1. Set the destination node slot to importing state using CLUSTER SETSLOT <slot> IMPORTING <source-node-id>.
    newNode.cluster(["SETSLOT", slot, "IMPORTING",  newNode._id], function(err) {
        if(err) {
            return callback(err);
        } 
        // 2. Set the source node slot to migrating state using CLUSTER SETSLOT <slot> MIGRATING <destination-node-id>.
        previousNode.cluster(["SETSLOT", slot, "MIGRATING",  previousNode._id], function(err) {
            if(err) {
               return callback(err);
            } 
            // 3. Get keys from the source node with CLUSTER GETKEYSINSLOT command 
            //    and move them into the destination node using the MIGRATE command.
            self.MigrateKeys(slot, previousNode, newNode, function(err) {
                if(err) {
                    return callback(err);
                }                
                // 4. Use CLUSTER SETSLOT <slot> NODE <destination-node-id> to all nodes
                async.eachSeries(self._nodes,
                    function(node, eachCb) {
                        node.cluster(["SETSLOT", slot, "NODE",  newNode._id], eachCb);
                    },
                    function(err) {
                        return callback(err);
                    }
                );
            }); 
        });
    });
}



/**
 * Move keys of a slot from one node to another.
 * Async.
 * @param slot              {number} the slot where the keys are taken.
 * @param previousNode      {object} RedisNode object
 * @param newNode           {object} RedisNode object
 * @param callback          {function} function(err)
 */ 
RedisCluster.prototype.MigrateKeys = function(slot, previousNode, newNode, callback)
{    
    var self = this;

    var nbKeys = 0;
    async.doWhilst(
        function(doCallback) {
            // Get keys from the source node with CLUSTER GETKEYSINSLOT slot count
            previousNode.CLUSTER(["GETKEYSINSLOT", slot, 100], function(err, keys) {
                if(err) {
                    nbKeys = 0;
                    return doCallback(err);
                }
                nbKeys = keys.length;
                if(nbKeys == 0) {
                    return doCallback(err);
                }
                previousNode._redisClient.send_command("migrate", [newNode._ip, newNode._port, "", 0, 1500, "KEYS"].concat(keys), function(err) {
                    doCallback(err);
                });
            });
        },            
        // doCallback
        function(err) {
            return nbKeys > 0;
        },
        // doWhilst finished: no more key to move
        function(err) {
            callback(err);
        }
    );
}

[antirez]

Thanks for the info, it is possible that you try to move multiple slots at the same time? Btw I see you are not using the redis-trib utility to move slots, so it is possible that there is some kind of bug in the Node.js code (I don't understand it enough to spot subtle bugs, sorry). Did you tried if a similar thing happens with redis-trib? Using the new version if redis-trib you can empty a node just using the rebalance command assigning a given node the weight of zero.

[antirez]

Btw I'm checking better the whole report in order to spot if there is something obviously wrong in some way.

[antirez]

Basically it's not yet clear how it happens, but this is what happens:

1. If you check the output, the node with not updated information about the slot is the receiving node itself.
2. For some reason, this node, has no information about the fact it got the slots assigned (your script bug? Some race condition in Redis Custer by using commands to setup the slot in a certain order?). Or perhaps the node used to know about the config, but later this information is cleared by an UPDATE packet or alike.
3. The other nodes are informed of the change correctly.

So now there is no node claiming this set of slots in the cluster. When this happens the information can't propagate, since we propagate cluster slots info in two ways:

1. Slot owner ping another node: the info in the receiver node is updated.
2. Past slot owner claims to have slots, but other nodes know there is a node with greater configuration epoch owning them, so an UPDATE packet is set.

So this is a stalled condition. Now the issue is, how this stalled condition is reached? Can you try to replicate this issue in the most minimal way possible?

[jcsc]

Hi Salvatore,

I am currently working at rewritting a light version of my nodejs app in ruby using redis-trib.
I will try to reproduce the issue with it.

My feeling is that your hypothesis "the node used to know about the config, but later this information is cleared by an UPDATE packet or alike." is right.
I'm trying to make a full cluster resharding as fast as possible, and maybe I have some requests in parallel instead of series.

I will keep you informed.

[antirez]

Thanks! I'll investigate more in the meantime.

[jcsc]

I can't reproduce the issue with redis-trib, so I guess there is an error in my nodejs application.
Probably, the application should begin to move slots from a second node while the migration on a first node is not yet finished.
The cluster's resharding is almost 2 times faster in nodejs.

Here is the redis-trib patch is used for my tests.
It is add a command which reads a json file containing slot mapping

@@ -23,6 +23,7 @@

 require 'rubygems'
 require 'redis'
+require 'json'

 ClusterHashSlots = 16384
 MigrateDefaultTimeout = 60000
@@ -911,7 +912,108 @@
         load_cluster_info_from_node(argv[0])
         show_cluster_info
     end
+    
+    
+    def config_cluster_cmd(argv, opt)
+    
+        clutserAddr = argv[0]
+        configFile = argv[1]
+    
+        config = JSON.parse( IO.read(configFile))
+        load_cluster_info_from_node(clutserAddr)
+        first = @nodes[0].info
+    
+        slotsMap = []
+        nodesMap = {}
+        nodesInConfig = []
+        nodesToRemove = []
+        nodesToAdd = []
+    
+        # Build a map of slots: slots => node address
+        config.each{|n|
+            nodesInConfig.push(n['addr'])
+            nodesToAdd.push(n['addr'])
+            slotsCfg = n['slots']
+            slotsRanges = slotsCfg.split(",")
+            slotsRanges.each{|sr|
+                if sr.index("-")
+                    start,stop = sr.split("-")
+                    ((start.to_i)..(stop.to_i)).each{|s|
+                        if slotsMap[s]
+                            puts "Error: slot #{s} already set"
+                        else
+                            slotsMap[s] = n['addr']
+                        end    
+                        
+                    }
+                else 
+                    s = sr.to_i
+                    if slotsMap[s]
+                        puts "Error: slot #{s} already set"
+                    else
+                        slotsMap[s] = n['addr']
+                    end  
+                end
+            }    
+        }
+    
+        # Build lists of node to add and node to remove
+        @nodes.each{|n|
+            addr =  n.info[:host] + ":" + n.info[:port]
+            nodesMap[addr] = n
+            if ! nodesInConfig.include?(addr)
+                nodesToRemove.push(n)   
+            else
+                nodesToAdd.delete(addr);
+            end
+        }
+        
+        # Add new nodes
+        nodesToAdd.each{|n| 
+            new = ClusterNode.new(n)
+            new.connect(:abort => true)
+            new.assert_cluster
+            new.load_info
+            new.assert_empty
+            add_node(new)
+            new.r.cluster("meet", first[:host], first[:port])
+            nodesMap[n] = new
+        }
+        
+        # Set slots
+        for s in 0..slotsMap.length-1
+            if slotsMap[s] && nodesMap[slotsMap[s]]
+                targetNode = nodesMap[slotsMap[s]]
+                currentNode = get_slot_owner(s)
+                if !currentNode
+                    targetNode.r.cluster("addslots",s)
+                elsif currentNode != targetNode
+                    move_slot(currentNode, targetNode, s, :quiet=>true, :dots=>false);
+                end    
+            end
+        end

+        
+        
+        # Remove old nodes
+        nodesToRemove.each{|node| 
+            if node
+                # Send CLUSTER FORGET to all the nodes but the nodes to remove
+                @nodes.each{|n|
+                    next if nodesToRemove.include?(n) 
+                    if n.info[:replicate] && n.info[:replicate].downcase == id
+                        # Reconfigure the slave to replicate with some other node
+                        master = get_master_with_least_replicas
+                        n.r.cluster("replicate",master.info[:name])
+                    end
+                    n.r.cluster("forget", node.info[:name])
+                }
+            node.r.cluster("reset")
+            end
+        }      
+    end
+    
+    
     def rebalance_cluster_cmd(argv,opt)
         opt = {
             'pipeline' => MigrateDefaultPipeline,
@@ -1562,6 +1664,7 @@
     "set-timeout" => ["set_timeout_cluster_cmd", 3, "host:port milliseconds"],
     "call" =>    ["call_cluster_cmd", -3, "host:port command arg arg .. arg"],
     "import" =>  ["import_cluster_cmd", 2, "host:port"],
+    "config" => ["config_cluster_cmd", -1, "host:port configFile"],
     "help"    => ["help_cluster_cmd", 1, "(show this help)"]
 }

A script use to create json config file:

 #!/usr/bin/env ruby
# Create a json cluster config file
# Usage:
#    ./genConfig.rb <fileName> <list of ip:addr>
# Example:
#   ./genConfig.rb config.json 127.0.0.1:7000 127.0.0.1:7002 127.0.0.1:7004

require 'json'
ClusterHashSlots = 16384

if ARGV.length < 2
    puts ARGV
    puts "Usage: /genConfig.rb <fileName> <list of ip:addr>"
    exit 1
end

slotMap = []
fileName = ARGV.shift
nbNodes = ARGV.length

nbSlotsPerNode = ClusterHashSlots / nbNodes
extraSlots = ClusterHashSlots % nbNodes
slot = 0 

ARGV.each{|addr|
    c = {}
    c['addr'] = addr

    lastSlot = slot + nbSlotsPerNode - 1 
    if extraSlots != 0
        extraSlots -= 1
        lastSlot += 1
    end    
    range = slot.to_s + "-" + lastSlot.to_s
    c['slots'] = range
    slot = lastSlot+1
    slotMap.push(c);    
}

config = slotMap.to_json
File.write(fileName, config)

[jcsc]

After more analysis, I guess this case happens because a migrate command fails and my scripst doesn't handle correctly this error:

    Set the destination node slot to importing state => OK
    Set the source node slot to migrating state => OK
    Get keys from the source node => OK
        Migrate  10 keys loop => OK 
        Migrate  10 keys loop => KO ( This error should stop resharding but currently it doesn't)     
    Use CLUSTER SETSLOT <slot> NODE <destination-node-id> to all nodes    

So I will change my application, but I don't see why the migrate command fails sometines.
With different scripts (nodejs and redis-trib), migrate command fails with the following errors:
- ERR Target instance replied with error: CLUSTERDOWN The cluster is down
- ERR Target instance replied with error: READONLY You can't write against a read only slave.
- LOADING Redis is loading the dataset in memory (Redis::CommandError

[yurial]

Hi, i have the same problem on redis-cluster 4.0.8. My cluster configured as 19 shards (57 nodes total), network latency ~1,76ms.
Problem was not reproduced by redis-trib, cause redis-trib processed nodes one-by-one (migrate all slots from one node, then migrate all slots from next node).
To reproduce this bug, i'm write a script (on error, an exception will occur):
migration method: https://pastebin.com/QZLpNyv7
migration script: https://pastebin.com/htvGJU5i
log tail (at the begin of the log, messages are the same): https://pastebin.com/fJJmGR2T