[Platforms] Publish Kubernetes multitenancy whitepaper

[AlexsJones]

There is an ever-growing demand for guidance and resources to classify how to support multi-tenant Kubernetes clusters.
In the spirit of the recently published Operator Whitepaper there is an opportunity to collaborate and aggregate industry examples to provide a rich resource for those looking to understand the history, context, and direction of multi-tenant Kubernetes architecture.

Examples:

• Cloud providers running tenant clusters within a single parent cluster.
• Development teams wishing to partition their resources and teams.
• Security groups looking to ensure isolation between workload tenancy.

Currently, there are several approaches teams are following:

• Running individual clusters per tenant
• Running tenants in different namespaces
• Running virtual cluster in cluster (like cluster)

Please comment below :

• if you are delivering multi-tenant applications
• provide infrastructure for multi-tenancy on Kubernetes
• are involved in OSS projects related to the topic

[raesene]

I think this would be a very interesting project. One aspect that's definitely worth considering is comparison between soft and hard multi-tenancy, as the security requirements of hard multi-tenancy can be tricky to achieve with stock Kubernetes.

[randomvariable]

I think @sedefsavas and I could potentially be interested as we built our own multitenancy implementation for Kubernetes Cluster API here and have submitted kubecon talk about it too.

[adrianludwin]

See also kubernetes/website#31479

[AlexsJones]

Thanks, @adrianludwin this feels like something we could apply a combined effort on.
Whilst we certainly target Kubernetes as the substrate, the App Delivery TAG angle is going to be heavily predicated in how this pattern is used E2E and its enablement for the delivery of tenant workloads ( So the scope might vary to some degree and be slightly broader ).

That said, I would be keen to know what work has been started if any and how we can pool our resources 🚀

I do notice a few projects mentioned in the related issue and I would love to provide some additional examples to make sure we have a fair and representative view of a) the landscape of OSS and CNCF projects b) the commercial vendors and their approaches + challenges.

[mkorbi]

I have discussed this recently with @AloisReitbauer and would be happy to give my/our five cents.

We work with various customers together to deliver multi-tenant clusters, and give guidance to the dev teams on how to tackle that.

[richburroughs]

Running virtual cluster in cluster (like cluster)

@AlexsJones did you mean vcluster there in the parens? :)

Besides vcluster our commerical product Loft is focused on multi-tenancy too.

[joshgav]

We may want to differentiate between provider-side multitenancy, like how Azure or AWS support multiple organizations; and user-side multitenancy, between groups in a single organization.

[AlexsJones]

After the App Delivery TAG call the consensus is to move this into the Cooperative delivery WG and take this forward.
It will be put on the docket for the next meeting and we can carry this forward.

[roberthstrand]

I'm really excited about this, as it's a topic I would love to get into the details of. A lot of room for collaboration here. 👏

[yogeek]

Really interested about this !
I can share our current implementation and how we are currently trying to improve it :

We ("platform" team) provides several kubernetes cluster for different projects inside our organization :

• one K8S cluster per environment (dev, staging, production region 1, production region 2...)
• in each K8S cluster, multiple tenants (projects) can deploy resources

So we do what you called "user-side" multitenancy

When a project wants to be hosted in our platform, they have to do Pull Request in our self-hosted gitlab 'onboarding' repository to

• add their desired namespaces into a YAML file like :

project1:
- project1_ns1
- project1_ns2
project2:
- project2_ns1
- project2_ns2
...

• optionally add an argocd applicationset custom resource (usually with a 'gitlab generator' for argocd appset controller to autodiscover their git 'deployment' repositories and generate corresponding argocd applications)
• optionally add additional RBAC for specific needs

This PR is reviewed and if merged, the Jenkins pipeline initializes the tenant with an 'admin' kubeconfig to create all the resources :

• creates desired namespaces with a label 'owner=project1' and a label 'netpol-gen=true' (will be used by a kyverno clusterpolicy to generate network policies for these namespaces)
• generates kubeconfigs for this project (1 for the project's admin/sre/devops roles that will allow 'admin-like' permission in the project's namespaces and 1 for devs that will have less permissions)
• generates RBAC for the project to limit their actions to their namespaces (we use RBACManager project here)
• creates a k8s dockerregistry secret with harbor credentials to pull only project's images from harbor
  (the imagePullSecret of the namespace is set to point to this specific secret)
• adds potential additional RBAC for specific needs
• launch some tests to check the isolation and kubeconfig scope
• applies the potential argocd applicationset resources to automatically generate all project's argocd applications

This is currently done with a shell script... (using templated YAML for resources like namespaces, RBAC...)

But we are currently study a refactoring of the repository to use :

• kyverno to handle RBACManager resources generation (as we already doing today for network policies)
• kyverno to handle imagePullSecret secret generation and the NS default serviceaccount corresponding modification
• argocd to apply namespace manifests

And some other ideas to test :

• use hierarchical namespaces to give namespaces control back to the projects : we would only handle the 'parent' namespaces creation for each project and leave them the freedom to create all the sub-namespaces they want (all the netpols, secrets, rbac would be propagated from the parent NS to the child NS)
• use a homemade 'Project' CRD for the project to describe their needs in a declarative way and kyverno to generate all needed resources in reaction to a 'Project' CR creation and write a kyverno policy to detect any new 'Project' custom resource and generates all necessary resources in reaction (instead of writing a full controller to do this, we think kyverno can do the job)
  (I found it was a bit like what Capsule proposes with its 'Tenant' CRD, but it would be more specific to our needs. By the way, Kapsule may have been a nice solution for us I think if it existed at the time of we set up our onboarding process...)

Voilà ! I hope it is clear enough and it can be of any use 😅

[devdattakulkarni]

I am interested in this topic and can share the insights that we have gained through our customers. We have been focusing on supporting the multi-instance multi-tenancy use-case (soft multi-tenancy). Lot of our customers want to deliver k8s-native applications to their end users in a service form. Multi-instance multi-tenancy provides a natural approach towards this. We have built the KubePlus Operator that enables such a service-based delivery of Kubernetes applications. Will be happy to share the challenges that arise in this form of multi-tenancy and how we are addressing them in KubePlus.