Skip to content

Commit

Permalink
Updated redis auto failover readme
Browse files Browse the repository at this point in the history
  • Loading branch information
boosty committed Jul 12, 2010
1 parent f0c7b82 commit bcf14c5
Showing 1 changed file with 27 additions and 19 deletions.
46 changes: 27 additions & 19 deletions REDIS_AUTO_FAILOVER.rdoc
Expand Up @@ -35,52 +35,60 @@ On every worker server runs another daemon, the Redis Configuration Client

If the RCS finds another potential Redis Master, it sends out a message to see
if all known RCCs are still available (once again to eliminate the risk of a
partitioned network). If all RCCs have answered to that message, the RCS sends
out a message which tells the RCCs to invalidate the current master. This
happens by deleting the contents of a special configuration file which is used
partitioned network) and if they agree to the master switch.

If all RCCs have answered to that message, the RCS sends out a message which
tells the RCCs to invalidate the current master.

This happens by deleting the contents of a special file which is used
by the workers to store the current Redis master (the content of that file is
the hostname:port of the currently active Redis master). By doing that, it is
ensured that no operations are done to any Redis server anymore, because the
AMQP workers check this files mtime and reads its contents in case that the
ensured that no operations are done to the old Redis master server anymore, because the
AMQP workers check this file's mtime and reads its contents in case that the
file changed, before every Redis operation. When the file has been emptied, the
RCCs respond to the "invalidate" message of the RCS. When all RCCs have
responded, the RCS knows for sure that it is save to switch the Redis master
responded, the RCS knows for sure that it is safe to switch the Redis master
now. It sends a "reconfigure" message with the new Redis master hostname:port
to the RCCs, which then write that value into the redis master configuration
file.
to the RCCs, which then write that value into their redis master file.

Additionally, the RCS sends reconfigure messages with the current Redis master
periodically, to allow new RCCs to pick up the current master. Plus it turns
all other redis servers into slaves of the current master.

=== Prerequisites

* one redis-configuration-server process ("RCS", on one server), one redis-configuration-client process ("RCC") on every worker server
* the RCS knows about all possible RCCs using a configured list
* the RCS knows about all possible RCCs using a list of client ids
* the RCS and RCCs exchange messages via a "system queue"

=== Flow of actions

* on startup, an RCC can determine the current redis-master without the help of the RCS by checking that either only one redis is available (as master) or by checking that redis master and slave are available
* when the RCS finds the redis-master to be down, it will retry a couple of times before starting a reconfiguration round
* on startup, an RCC can consult its redis master file to determine the current master without the help of the RCS by checking that it's still a master (or wait for the periodic reconfigure message with the current master from the RCS)
* when the RCS finds the master to be down, it will retry a couple of times before starting a reconfiguration round
* the RCS sends all RCCs a "ping" message to check if every client is there and able to to answer
* the RCCs acknowledge via a "pong" message
* the RCCs acknowledge via a "pong" message if they can confirm the current master to be unavailable
* the RCS waits for *all* RCCs to reply via pong
* the RCS tells all RCCs to stop using the redis-master by sending an "invalidate" message
* the RCCs acknowledge via an "invalidated" message
* the RCS tells all RCCs to stop using the master by sending an "invalidate" message
* the RCCs acknowledge via an "invalidated" message if they can still confirm the current master to be unavailable
* the RCS waits for *all* RCCs to acknowledge the invalidation
* the RCS promotes the former redis-slave to become the new redis-master (by sending SLAVEOF no one)
* the RCS sends a "reconfigure" message containing the new redis-master to every RCC
* the RCCs write the new redis-master to their redis master file
* the RCS promotes the former slave to become the new master (by sending SLAVEOF no one)
* the RCS sends a "reconfigure" message containing the new master to every RCC
* the RCCs write the new master to their redis master file

=== Configuration

See Beetle::Configuration for setting redis configuration server and client options.

Please note:
Beetle::Configuration#redis_server must be a file path (not a redis host:port string) to use the redis failover.
The RCS and RCCs store the current redis master in that file, and the handlers read from it.
Beetle::Configuration#redis_server must be a file path (not a redis host:port string) to use the redis failover. The RCS and RCCs store the current redis master in that file, and the handlers read from it.

== How to use it

This example uses two worker servers, identified by rcc-1 and rcc-2.

Please note:
All command line options can also be given as a yaml configuration file via the --config-file option.

=== On one server

Start the Redis Configuration Server:
Expand Down

0 comments on commit bcf14c5

Please sign in to comment.