This project provides a custom operator for implementing a strong multi-tenant environment in Kubernetes. Capsule is not intended to be yet another PaaS, instead, it has been designed as a lightweight tool with a minimalist approach leveraging only the standard features of upstream Kubernetes.
Kubernetes introduced the namespace resource to create logical partitions of the cluster. A Kubernetes namespace creates a sort of isolated slice in the cluster: Network and Security Policies, Resource Quota, Limit Ranges, and RBAC can be used to enforce isolation among different namespaces. Namespace isolation shines when Kubernetes is used to isolate the different environments or the different types of applications. Also, it works well to isolate applications serving different users when implementing the SaaS delivery model.
However, implementing advanced multi-tenancy scenarios, for example, a private or public Container-as-a-Service platform, it becomes soon complicated because of the flat structure of Kubernetes namespaces. In such scenarios, different groups of users get assigned a pool of namespaces with a limited amount of resources (e.g.: nodes, vCPU, RAM, ephemeral and persistent storage). When users need more namespaces or move resources from one namespace to another, they always need the intervention of the cluster admin because each namespace still works as an isolated environment. To work around this, and not being overwhelmed by continuous users' requests, cluster admins often choose to create multiple smaller clusters and assign a dedicated cluster to each organization or group of users leading to the well know and painful phenomena of the clusters sprawl.
Capsule takes a different approach. It aggregates multiple namespaces assigned to an organization or group of users in a lightweight abstraction called Tenant. Within each tenant, users are free to create their namespaces and share all the assigned resources between the namespaces of the tenant. The Network and Security Policies, Resource Quota, Limit Ranges, RBAC, and other constraints defined at the tenant level are automatically inherited by all the namespaces in the tenant leaving the tenant's users to freely allocate resources without any intervention of the cluster administrator.
Please, refer to the corresponding section for a more detailed list of use cases that Capsule can address.
Ensure you have kubectl and kustomize
installed in your PATH. Also, make sure you have access to a Kubernetes cluster as an administrator.
Clone this repository and move to the repo folder:
make deploy
# /home/prometherion/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
# cd config/manager && /usr/local/bin/kustomize edit set image controller=quay.io/clastix/capsule:latest
# /usr/local/bin/kustomize build config/default | kubectl apply -f -
# namespace/capsule-system created
# customresourcedefinition.apiextensions.k8s.io/tenants.capsule.clastix.io created
# clusterrole.rbac.authorization.k8s.io/capsule-namespace:deleter created
# clusterrole.rbac.authorization.k8s.io/capsule-namespace:provisioner created
# clusterrole.rbac.authorization.k8s.io/capsule-proxy-role created
# clusterrole.rbac.authorization.k8s.io/capsule-metrics-reader created
# clusterrolebinding.rbac.authorization.k8s.io/capsule-manager-rolebinding created
# clusterrolebinding.rbac.authorization.k8s.io/capsule-namespace:provisioner created
# clusterrolebinding.rbac.authorization.k8s.io/capsule-proxy-rolebinding created
# secret/capsule-ca created
# secret/capsule-tls created
# service/capsule-controller-manager-metrics-service created
# service/capsule-webhook-service created
# deployment.apps/capsule-controller-manager created
# mutatingwebhookconfiguration.admissionregistration.k8s.io/capsule-mutating-webhook-configuration created
# validatingwebhookconfiguration.admissionregistration.k8s.io/capsule-validating-webhook-configuration created
Log verbosity of the Capsule controller can be increased by passing the --zap-log-level option with a value from 1 to 10 or the basic keywords although it is suggested to use the --zap-devel flag to get also stack traces.
Capsule implements Kubernetes multi-tenancy capabilities using a minimum set of standard Admission Controllers enabled on the Kubernetes APIs server: --enable-admission-plugins=PodNodeSelector,LimitRanger,ResourceQuota,MutatingAdmissionWebhook,ValidatingAdmissionWebhook. In addition to these default controllers, Capsule implements its own set of Admission Controllers through the Dynamic Admission Controller, providing callbacks to add further validation or resource patching.
All these requests must be served via HTTPS and a CA must be provided to ensure that the API Server is communicating with the right client. Capsule upon installation is setting its custom Certificate Authority as a client certificate as well, updating all the required resources to minimize the operational tasks.
Each tenant comes with a delegated user acting as the tenant admin. In the Capsule jargon, this user is called the Tenant Owner. Other users can operate inside a tenant with different levels of permissions and authorizations assigned directly by the Tenant owner.
Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the capsule.clastix.io group.
Assignment to a group depends on the authentication strategy in your cluster. For example, users authenticated through a X.509 certificate must have capsule.clastix.io as Organization: -subj "/CN=${USER}/O=capsule.clastix.io"
Users authenticated through an OIDC token must have
...
"users_groups": [
"/capsule.clastix.io",
"other_group"
]in their token.
The hack/create-user.sh can help you set up a dummy kubeconfig for the alice user acting as owner of a tenant called oil
./hack/create-user.sh alice oil
creating certs in TMPDIR /tmp/tmp.4CLgpuime3
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
........................+++++
e is 65537 (0x010001)
certificatesigningrequest.certificates.k8s.io/alice-oil created
certificatesigningrequest.certificates.k8s.io/alice-oil approved
kubeconfig file is: alice-oil.kubeconfig
to use it as alice export KUBECONFIG=alice-oil.kubeconfigUse the scaffold Tenant and simply apply as Cluster Admin.
kubectl apply -f config/samples/capsule_v1alpha1_tenant.yaml
tenant.capsule.clastix.io/oil created
The related Tenant owner alice can create Namespaces according to their assigned quota: happy Kubernetes cluster administration!
Similar to deploy, you can get rid of Capsule using the remove target.
make remove
# /home/prometherion/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
# /usr/local/bin/kustomize build config/default | kubectl delete -f -
# namespace "capsule-system" deleted
# customresourcedefinition.apiextensions.k8s.io "tenants.capsule.clastix.io" deleted
# clusterrole.rbac.authorization.k8s.io "capsule-namespace:deleter" deleted
# clusterrole.rbac.authorization.k8s.io "capsule-namespace:provisioner" deleted
# clusterrole.rbac.authorization.k8s.io "capsule-proxy-role" deleted
# clusterrole.rbac.authorization.k8s.io "capsule-metrics-reader" deleted
# clusterrolebinding.rbac.authorization.k8s.io "capsule-manager-rolebinding" deleted
# clusterrolebinding.rbac.authorization.k8s.io "capsule-namespace:provisioner" deleted
# clusterrolebinding.rbac.authorization.k8s.io "capsule-proxy-rolebinding" deleted
# secret "capsule-ca" deleted
# secret "capsule-tls" deleted
# service "capsule-controller-manager-metrics-service" deleted
# service "capsule-webhook-service" deleted
# deployment.apps "capsule-controller-manager" deleted
# mutatingwebhookconfiguration.admissionregistration.k8s.io "capsule-mutating-webhook-configuration" deleted
# validatingwebhookconfiguration.admissionregistration.k8s.io "capsule-validating-webhook-configuration" deleted
Any contribution is welcome! Please refer to the corresponding section.
Capsule is still in an alpha stage, so don't use it in production!
tbd
tbd
tbd