New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codis should validate server validity before setting migrate status for a slot #173

Closed
therealbill opened this Issue Mar 31, 2015 · 2 comments

Comments

Projects
None yet
3 participants
@therealbill

therealbill commented Mar 31, 2015

Scenario to reproduce:

  1. Set up cluster
  2. add data
  3. add new group and server master
    3a) on new server connect directly and do a "config slaveof # any non-clustered redis node will do
  4. observe the node shows maxmemory settings and used memory
  5. initiate an autorebalance

You might be able to replicate by adding a non-existent server.

What happens is the slots are marked as migrating but they can't migrate because the server they are moving to won't accept commands. There are two key factors here:

  1. Codis marks the slot as "migrating" before it validates it can migrate keys to that target.
  2. Once attempted and failed there is no useful error reporting, and no way to resolve it short of manually looping through the slots to find all keys destined for that target and setting them back to their original group, then retrying the entire process. Once this has happened that slot is marked as offline.

If Codis were to check that it can write (ie. connect and get role. if role !=master or no connection established) this would prevent this condition from occurring.

Codis should, as part of this, automatically mark as a slave any master it is told to assign to a slot if it pulls the role and finds it to be a slave and not a master. It would also be ideal to be able to tell codis-config to re-start or resume the migration process for a given slot.

@ngaut ngaut added the todo label Apr 1, 2015

@ngaut

This comment has been minimized.

Show comment
Hide comment
@ngaut

ngaut Apr 1, 2015

Contributor

@therealbill Thanks.
Will do it later in a few days.

Contributor

ngaut commented Apr 1, 2015

@therealbill Thanks.
Will do it later in a few days.

@therealbill

This comment has been minimized.

Show comment
Hide comment
@therealbill

therealbill Jun 18, 2015

@ngaut Any update available yet?

therealbill commented Jun 18, 2015

@ngaut Any update available yet?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment