Project status: alpha
The basic features have been completed, and while no breaking API changes are currently planned, the API can change in a backwards incompatible way before the project is declared stable.
Fundamental things to know about Redis Operator:
3is a minimum number of Redis instances. Having
3instances allows to always maintain a simple master-replica pair thus making it possible to replicate data even with Redis persistence turned off.
- Redis Operator is stateless. It means that it does not store any information about Redis instances internally. If an instance of the operator terminates in the middle of the failover process it will reconnect to Redis instances and reconfigure them if it is still required.
- Redis Operator is not a distributed system. It leverages a simple leader election protocol. You can run multiple instances of Redis Operator. Detailed description of leader election can be found here.
- One Redis Operator deployment is designed to rule multiple Redis replication setups. However you should bear in mind that current implementation is limited to reconfiguring one Redis replication at a time.
- Redis Operator does not provide continuous monitoring, notification and service discovery. Those are provided by Kubernetes itself.
- Redis clients don't need Sentinel support. Appropriate
rolelabels are added to each pod and end users are encouraged to use services to connect to master or replica nodes.
- Redis 5.0 is the minimum supported version.
Deploying the Redis operator
Create a namespace for the operator:
kubectl create namespace redis-operator
Create all the necessary resources and deploy the operator:
kubectl apply -Rf deploy
Verify that the operator is running:
$ kubectl -n redis-operator get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE redis-operator 1 1 1 1 5m
Redis can be deployed by creating a
Redis Custom Resource(CR).
Create a Redis CR that deploys a 3 node Redis replication in high availablilty mode:
kubectl create -f example/k8s_v1alpha1_redis_cr.yaml
Wait until the
redis-example-...pods for the Redis replication are up and check the status of 'redis'. It will show the name for the Pod of the current master instance and the total number of replicas in the setup:
$ kubectl get redis example NAME MASTER REPLICAS DESIRED AGE example redis-example-0 3 3 24d
Scale the deployment:
$ kubectl scale redis example --replicas 4 redis.k8s.amaiz.com/example scaled $ kubectl get redis example NAME MASTER REPLICAS DESIRED AGE example redis-example-0 4 4 24d
Redis Operator creates the following resources owned by the corresponding
RedisCR. Please note that the name of
examplein this case) is used as an infix or suffix for the names of the generated resources:
redis-example(in case the password is set up)
redis-example- covers all instances
redis-example-headless- covers all instances, headless
redis-example-master- service for access to the master instance
All configuration of Redis is done via editing the
Redis resourse file. Full annotated example can be found in the
examples directory of the repo.
Uninstalling Redis operator
Delete the operators and CRDs. Kubernetes will garbage collect all operator-managed resources:
kubectl delete namespace redis-operator kubectl delete crd redis.k8s.amaiz.com
Design and goals
The main goal of the project is to create a Kubernetes native service for managing highly available Redis deployments. Sentinel is great for simple Redis replication but it does not fit into Kubernetes model for a number of reasons:
- yet another distributed system to maintain. In order to support automatic failover and high availability of Redis one should figure out some way to support automatic failover and high availability of Sentinel.
- feature overlap. Monitoring(periodic instance health check), notifications(events) and service discovery are something Kubernetes already provides out of the box and can be leveraged by implementing your own controller and Custom Resources.
- Sentinel allows to resist without human intervention to certain kind of failures. The goal of the operator is to create a Redis deployment that would resist without human intervention to most kind of failures.
Another imporatant goal of this project is to resist failures even with persistence turned off. In some scenarios persisting data on disk is not permitted and all the data should reside only in-memory no matter what. And at the same time losing this data is undesirable.
Automatic failover algorithm details
Redis Operator is not a distributed system. Instead it leverages the Kuberenetes API to perform a leader election upon startup. Current implementation of the leader election algorithm precludes the possibility of 2 instances mistakenly running as leaders (split brain).
Redis Operator watches for changes to the
Redis resource as well as the resources managed by the operator and owned by
StatefulSets. Should anything happen to any of the above the operator will check the state of the Kubernetes resources along with the state of Redis replication and reconfigure them if needed.
All the managed resources are created or updated in the first place. The resources already present are always compared to the resources generated by the operator and updated if they differ.
Once all the resources are in sync the list of Redis instances is compiled from the list of
Pods owned by the corresponding
Pods with all containers running and ready are taken into account.
Minimum failover size is
2 represents a simple master-replica pair essential for running replication. If the number of instances is less than the minimum failover size no reconfiguration will be performed. With this in mind it is absolutely normal to lose all instances but one at the same time. Even with persistence turned off the data will be preserved and replicated across all
Pods that come in place of the terminated ones.
Redis Operator is stateless. It means that the state of replication is determined every time the list of Redis instances is analyzed.
The state of replication is determined based on the
INFO REPLICATION output of every instance from the list.
A healthy replication is the state when there is a single master and all other instances are connected to it. In this case the operator will do nothing.
Master is an instance with at least one connected replica. If there is no masters found then there is one of two cases met:
- the master is lost. Then there's at least one replica and one of the replicas should be promoted to master
- all instances are masters. This is considered to be the initial state thus any instance can be chosen as a master
In case the master has been lost the list of candidate instances are sorted according to their replica priority and replication offset. Instances with replica priority equal to
0 are filtered out prior to sorting.
A replica with the lowest priority and/or higher replication offset is promoted to master.
With the master in place all other instances that do not report themselves as the master's replicas are reconfigured appropriately. All replicas in question are reconfigured simultaneously.
Once the reconfiguration has been finished all
Pods are labeled appropriately with
role=replica labels. Current master's Pod name and the total quantity of connected instances are written to the status field of the
Redis resource. The
ConfigMap is updated with the master's IP address.
- add more testing