Codis should validate server validity before setting migrate status for a slot

[therealbill]

Scenario to reproduce:

1. Set up cluster
2. add data
3. add new group and server master
   3a) on new server connect directly and do a "config slaveof # any non-clustered redis node will do
4. observe the node shows maxmemory settings and used memory
5. initiate an autorebalance

You might be able to replicate by adding a non-existent server.

What happens is the slots are marked as migrating but they can't migrate because the server they are moving to won't accept commands. There are two key factors here:

1. Codis marks the slot as "migrating" before it validates it can migrate keys to that target.
2. Once attempted and failed there is no useful error reporting, and no way to resolve it short of manually looping through the slots to find all keys destined for that target and setting them back to their original group, then retrying the entire process. Once this has happened that slot is marked as offline.

If Codis were to check that it can write (ie. connect and get role. if role !=master or no connection established) this would prevent this condition from occurring.

Codis should, as part of this, automatically mark as a slave any master it is told to assign to a slot if it pulls the role and finds it to be a slave and not a master. It would also be ideal to be able to tell codis-config to re-start or resume the migration process for a given slot.

[ngaut]

@therealbill Thanks.
Will do it later in a few days.

[therealbill]

@ngaut Any update available yet?