Autopilot Pattern Redis
Redis designed for automated operation using the Autopilot Pattern.
The initial work was sponsored by Faithlife.
A running cluster includes the following components:
- Redis: we're using Redis 3.2.
- Redis Sentinel: manage failover.
- ContainerPilot: included in our Redis containers to orchestrate bootstrap behavior and coordinate replication using keys and checks stored in Consul in the startup,
- Consul: is our service catalog that works with ContainerPilot and helps coordinate service discovery, replication, and failover
- Manta: the Joyent object store, for securely and durably storing our Redis backups.
manage.sh: a small bash script that ContainerPilot will call into to do the heavy lifting of bootstrapping Redis.
When a new Redis node is started.
onStart task is run as part of app start, not from ContainerPilot's
because the Consul agent must be running, and ContainerPilot doesn't start coprocesses until
onStart performs the following:
- Is this container configured as a replica? If yes:
- Wait for the master to become healthy in the service registry.
- If there is no healthy master, try to reconfigure as master and restart.
- Is this container configured as the master? If yes:
- Verify this node should still start as master.
- If this node shouldn't be master, reconfigure as a replica and restart.
- Write redis and sentinel configurations based on the master in the service registry or this node if there is no master.
- Restore the last backup if one exists.
health performs the following:
- Ping redis, verify the response.
- Verify the service configuration (master or replica) matches redis's role (master or slave). Sentinel may have performed a failover and changed this node's role. If the role is changed, the service registry needs to be updated so any containers started in the future are configured correctly. If the service configuration and role do not match, reconfigure to match the current role.
healthSentinel pings sentinel.
ContainerPilot calls the
backup handler via a recurring task. The backup handler will:
- Check the backup run TTL health check on the redis service.
- If the TTL has expired:
- Pass the check.
- Create a backup.
- Upload the backup to Manta.
Running the cluster
Starting a new cluster is easy once you have your
_env file set with the configuration details, just run
docker-compose up -d and in a few moments you'll have a running Redis master. Both the master and replicas are described as a single
docker-compose service. During startup, ContainerPilot will ask Consul if an existing master has been created. If not, the node will initialize as a new master and all future nodes will self-configure replication with the master via Sentinel.
docker-compose scale redis=3 to add replicas. The replicas will automatically configure themselves to to replicate from the master and will register themselves in Consul as replicas once they're ready. There should be at least 3 nodes to have a quorum in case of a node failure.
Pass these variables via an
_env file. The included
examples/triton/setup.sh can be used to test your Docker and Triton environment, and to encode the Manta SSH key in the
MANTA_URL: the full Manta endpoint URL. (ex.
MANTA_USER: the Manta account name.
MANTA_SUBUSER: the Manta subuser account name, if any.
MANTA_ROLE: the Manta role name, if any.
MANTA_KEY_ID: the MD5-format ssh key id for the Manta account/subuser (ex.
1a:b8:30:2e:57:ce:59:1d:16:f6:19:97:f2:60:2b:3d); the included
setup.shwill encode this automatically
MANTA_PRIVATE_KEY: the private ssh key for the Manta account/subuser; the included
setup.shwill encode this automatically
MANTA_BUCKET: the path on Manta where backups will be stored. (ex.
/myaccount/stor/manage); the bucket must already exist and be writeable by the
These variables are optional but you most likely want them:
LOG_LEVEL: will set the logging level of the
manage.shscript. Set to
DEBUGfor more logging.
CONSULis the hostname for the Consul instance(s) to join. Defaults to
Where to store data
This pattern automates the data management and makes container effectively stateless to the Docker daemon and schedulers. This is designed to maximize convenience and reliability by minimizing the external coordination needed to manage the database. The use of external volumes (
-v, etc.) is not recommended.
On Triton, there's no need to use data volumes because the performance hit you normally take with overlay file systems in Linux doesn't happen with ZFS.
Using an existing database
If you start your Redis container instance with a data directory that already contains a database (specifically, a appendonly.aof file), the pre-existing database won't be changed in any way.