CassKop - Cassandra Kubernetes operator
The CassKop Cassandra Kubernetes operator makes it easy to run Apache Cassandra on Kubernetes. Apache Cassandra is a popular, free, open-source, distributed wide column store, NoSQL database management system. The operator allows to easily create and manage racks and data centers aware Cassandra clusters.
The Cassandra operator is based on the CoreOS operator-sdk tools and APIs.
NOTE: This is an alpha-status project. We do regular tests on the code and functionality, but we can not assure a production-ready stability at this time. Our goal is to make it run in production as quickly as possible.
CassKop creates/configures/manages Cassandra clusters atop Kubernetes and is by default space-scoped which means that :
- CassKop is able to manage X Cassandra clusters in one Kubernetes namespace.
- You need X instances of CassKop to manage Y Cassandra clusters in X different namespaces (1 instance of CassKop per namespace).
This adds security between namespaces with a better isolation, and less work for each operator.
TL;DR - CassKop presentation
We have some slides for a CassKop demo
At this time of the project, the goal is to deploy a Cassandra cluster in 1 Kubernetes datacenter, but this will change in next versions to deal with Kubernetes in multi-datacenters.
The following features are supported by CassKop:
- Deployment of a C* cluster (rack or AZ aware)
- Scaling up the cluster (with cleanup)
- Scaling down the cluster (with decommission prior to Kubernetes scale down)
- Pods operations (removenode, upgradesstable, cleanup, rebuild..)
- Adding a Cassandra DC
- Removing a Cassandra DC
- Setting and modifying configuration files
- Setting and modifying configuration parameters
- Update of the Cassandra docker image
- Rolling update of a Cassandra cluster
- Update of Cassandra version (including upgradesstable in case of major upgrade)
- Update of JVM
- Update of configuration
- Rolling restart of a Cassandra rack
- Stopping a Kubernetes node for maintenance
- Process a remove node (and create new Cassandra node on another Kubernetes node)
- Process a replace address (of the old Cassandra node on another Kubernetes node)
- Manage operations on pods through CassKop plugin (cleanup, rebuild, upgradesstable, removenode..)
- Monitoring (using Instaclustr Prometheus exporter to Prometheus/Grafana)
- Performing live backup of Cassandra datas (using Instaclustr sidecar)
- Performing live restore of datas (using Instaclustr sidecar)
- Performing live Cassandra repairs through the use of Cassandra reaper
- Pause/Restart operations through CassKoP plugin.
CassKop doesn't use nodetool but invokes operations through authenticated JMX/Jolokia call
Operator SDK is part of the operator framework provided by RedHat & CoreOS. The goal is to provide high-level abstractions that simplifies creating Kubernetes operators.
The quick start guide walks through the process of building the Cassandra operator using the SDK CLI, setting up the RBAC, deploying the operator and creating a Cassandra cluster.
You can find this in the Developer section
Users should only need Kubectl & helm cli
- kubectl version v1.13.3+.
- Helm version v2.12.2+.
- Access to a Kubernetes v1.13.3+ cluster.
- Cassandra needs fast local storage (we have tested with local storage provisioner, GKE ssd storage, and Rancher local-path-provisioner)
Install CassKop kubectl plugin
You can install the plugin by copying the file into your PATH.
We uses CircleCI as our CI tool to build and test the operator.
To accelerate build phases we have created a custom build-image used by the CircleCI pipeline:
You can find more info in the developer Section
The Cassandra operator image is automatically built and stored on Docker Hub
Associated Cassandra image
The operator works with specific Docker Cassandra image which is build in the repository: https://github.com/Orange-OpenSource/cassandra-image
This table shows compatibility between CassKop and associated Cassandra image
docker image: orangeopensource/cassandra-image:3.11.4-8u212-0.3.2-release or 3.11.4-8u212-0.3.2-release-cqlsh if you need cqlsh
Deploy the Cassandra operator in the cluster
First, we need to create a Kubernetes
namespace in order to host our operator & cluster
kubectl create namespace cassandra
Deploy the Cassandra Operator and it's CRD with Helm
To ease the use of the Cassandra operator, a Helm chart has been created
We are looking where to store our helm in the future
Add the Helm incubator repo if you do not already have it:
helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
Helm is available in the official helm/charts/incubator:
helm install --name casskop incubator/cassandra-operator
you can also add the CassKop repository from Github
helm repo add casskop https://Orange-OpenSource.github.io/cassandra-k8s-operator/helm
$ helm install --name casskop casskop/cassandra-operator NAME: casskop LAST DEPLOYED: Thu May 23 15:34:27 2019 NAMESPACE: cassandra-demo STATUS: DEPLOYED RESOURCES: ==> v1/ServiceAccount NAME SECRETS AGE cassandra-k8s-operator 1 0s ==> v1beta1/Role NAME AGE cassandra-k8s-operator 0s ==> v1/RoleBinding NAME AGE cassandra-k8s-operator 0s ==> v1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE casskop-cassandra-k8s-operator 1 1 1 0 0s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE casskop-cassandra-k8s-operator-78786b9bf-cjggg 0/1 ContainerCreating 0 0s
You can find more information in the Cassandra operator Helm readme
If you have problem you can see troubleshooting section
This creates a Kubernetes Deployment for the operator, with RBAC settings.
Once deployed, you may find the Pods created by the Charts. If you deploy a release
casskop, then your pod will have a name similar to :
$ kubectl get pod NAME READY STATUS RESTARTS AGE casskop-cassandra-operator-78786b9bf-cjggg 1/1 Running 0 1h
You can view the CassKop logs using
$ kubectl logs -f cassandra-cassandra-operator-78786b9bf-cjggg
The charts also deploy the cassandracluster CRD we can check that it is deployed:
$ kubectl get crd NAME AGE ... cassandraclusters.db.orange.com 1h ...
Deploy a Cassandra cluster
From local yaml spec
Once the operator is deployed inside a Kubernetes cluster, a new API will be accessible, so you'll be able to create, update and delete cassandraclusters.
In order to deploy a new cassandra cluster a specification has to be created:
For example :
kubectl apply -f samples/cassandracluster.yaml
see pods coming into life :
kubectl get pods -w
You can watch the status updates in real time on your CassandraCluster object :
watch 'kubectl describe cassandracluster cassandra-demo | tail -20'
Cassandra cluster status
You can find mode information on the
CassandraCluster.status in this section
Make operation on the cluster
You can do a lot of operations on your Cassandra cluster.
Cassandra operator recovery
If the Cassandra operator restarts, it can recover its previous state thanks to the CRD objects
CassandraClusters which stored directly in Kubernetes, description and state of the Cassandra cluster.
Uninstaling the Charts
If you want to delete the operator from your Kubernetes cluster, the operator deployment should be deleted.
$ helm delete casskop
The command removes all the Kubernetes components associated with the chart and deletes the helm release.
The CRD created by the chart are not removed by default and should be manually cleaned up (if required)
Manually delete the CRD:
kubectl delete crd cassandraclusters.dfy.orange.com
If you delete the CRD then !!!!!!WAAAARRRRNNIIIIINNG!!!!!!
It will delete ALL Clusters that has been created using this CRD!!!
Please never delete a CRD without very very good care
CassKop is build using operator SDK:
We can quickly setup monitoring for our deployed Cassandra nodes using Prometheus operator.
You can deploy the CoreOs Prometheus operator on your cluster: You can find example helm value.yaml to configure the Prometheus operator:
$ kubectl create namespace monitoring $ helm install --namespace monitoring --name prometheus stable/prometheus-operator
Add ServiceMonitor for Cassandra
Then you have to define a ServiceMonitor object to monitor cluster deployed by your cassandra operator (one time), update this to specify which namespace to monitor.
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kube-prometheus-cassandra-k8s-jmx labels: k8s-apps: cassandra-k8s-jmx prometheus: kube-prometheus component: cassandra release: prometheus spec: jobLabel: kube-prometheus-cassandra-k8s-jmx selector: matchLabels: k8s-app: exporter-cassandra-jmx namespaceSelector: matchNames: - cassandra - cassandra-demo - default endpoints: - port: promjmx interval: 15s
Your namespace need to be listed in the
Add Grafana dashboard for Cassandra
You can import this dashboard to retrieve metrics about your Cassandra cluster.
See CONTRIBUTING for details on submitting patches and the contribution workflow.
CassKop is under Apache 2.0 license. See the LICENSE file for details.