mck8s, short for multi-cluster Kubernetes, allows you to automate the deployment of multi-cluster applications on multiple Kubernetes clusters by offering enhanced configuration possibilities. The main aim of mck8s is maximizing resource utilization and supporting elasitcity across multiple Kubenetes clusters by providing multiple placement policies, as well as bursting, cloud resource provisioning, autoscaling and de-provisioning capabilities. mck8s builds upon other open-source software such as Kubernetes, Kubernetes Federation, kopf, serf, Cilium, Cluster API, and Prometheus.
This code is in active development and not stable, and thus not production ready.
All contributions are welcome :)
The figure below shows the architecture of mck8s.
- A Kubernetes cluster to act as the mck8s
management cluster
(namedcluster0
). Since mck8s components need access to the Kubernetes control plane, managed Kubernetes offerings such as GKE are not supported at the moment. - Few Kubernetes clusters to be managed by the management cluster and on which workloads run. We call these
workload clusters
(namedcluster1
,cluster2
,cluster3
, ...). We assume that you have administrative access to all these clusters and the Kuberneteskubeconfig
files of all clusters are available. - If traffic routing between clusters is desired, it is recommended to deploy Cilium Cluster Mesh on the workload clusters with distinct Pod CIDRs.
- If proximity-aware placement is desired, serf should be deployed on at least one node of each workload cluster.
- If cloud provisioning and autoscaling ia desired, cloud credentials are required. For now, we support OpenStack clusters.
- Clone this repository to your computer.
- Copy the
kubeconfig
file of themanagement cluster
ascluster0
in~/.kube/
directory of your computer . - Copy the
kubeconfig
files from theworkload clusters
to~/.kube/
directory of your computer and rename these files as per their cluster names such ascluster1
,cluster2
, orcluster3
, ..... - Make sure that the
cluster
,context
, anduser
names is these files are distinct from each other. - Before running
prepare.sh
, which assumes that there are fiveworkload clusters
,cluster1
tocluster5
, edit thevalues.yaml
file as per the number of yourworkload clusters
and enter the IP addresses of the master nodes of your clusters. - Run the
prepare.sh
script, which sets up Kubernetes Federation, Prometheus Operator, and Cilium. This script assumes that there are fiveworkload clusters
,cluster1
tocluster5
. If you have a different number ofworkload clusters
, please adjust the script accordingly. - Copy the
~/.kube/config
file to the~/.kube/
directory of themaster node
of themanagement cluster
.
-
Switch to the
cluster0
context to be able to deploy theCRDs
on the management cluster.kubectl config use-context cluster0
-
Create the
RBAC
requiredkubectl apply -f manifests/crds/01_rbac_mck8s.yaml
-
Deploy the
CRDs
kubectl apply -f manifests/crds/
-
Buid the image of the
Multi Cluster Scheduler
cd multi-cluster-scheduler && docker build -t REPO_NAME/IMAGE_NAME .
-
Build the image of the
Multi Cluster HPA
cd multi-cluster-horizontal-pod-autoscaler && docker build -t REPO_NAME/IMAGE_NAME .
-
Build the image of the
Cloud Provisioner and Cluster Autoscaler
cd cloud-cluster-provisioner-autoscaler && docker build -t REPO_NAME/IMAGE_NAME .
-
Build the image of the
Deployment Rescheduler
cd multi-cluster-rescheduler && docker build -t REPO_NAME/IMAGE_NAME .
NOTE: The pods of these controllers are going to be scheduled on the master node of the management cluster
. In the manifest files below, Replace the MASTER_NODE_HOST_NAME
, REPO_NAME/IMAGE_NAME
, and /PATH/TO/HOME/DIRECTORY/
accordingly before running the following.
-
Deploy the
Multi Cluster Scheduler
Deploymentkubectl apply -f manifests/controllers/01_deployment_multi_cluster_scheduler.yaml
-
Deploy the
Multi Cluster HPA
Deploymentkubectl apply -f manifests/controllers/02_deployment_multi_cluster_hpa.yaml
-
Deploy the
Cloud Provisioner and Cluster Autoscaler
Deploymentkubectl apply -f manifests/controllers/03_deployment_cloud_provisioner_cluster_autoscaler.yaml
Demo: Deployment of mck8s CRDs, controllers, multicluster scheduler, and example multicluster deployment
Let's see a demo of how to deploy the multiclusterscheduler, a sample multiclusterdeployment and a multiclusterservice.
In this demo, we show how to access a back-end application deployed on cluster3 via a front-end application deployed on cluster2.
[1] Mulugeta Tamiru, Guillaume Pierre, Johan Tordsson, Erik Elmroth. mck8s: An orchestration platform for geo-distributed multi-cluster environments. ICCCN 2021 - 30th International Conference on Computer Communications and Networks, Jul 2021, Athens, Greece. pp.1-12. (pdf)
[1] I gave a virtual talk titled "McK8s: Container Orchestration in Kubernetes Multi-Clusters" at KubeCon + CloudNativeCon NA 2021 https://www.youtube.com/watch?v=U1iHBZhEWUA&t=824s