Multi-Tenancy Strategies

[stefanprodan]

Personas

Platform Admin

• Has cluster admin access to the fleet of clusters
• Has maintainer access to the fleet Git repository
• Manages cluster wide resources (CRDs, controllers, cluster roles, etc)
• Onboards the tenant’s main GitRepository and Kustomization
• Manages tenants by assigning namespaces and role binding to the tenant apps

Tenant

• Has admin access to the namespaces assigned to them by the platform admin
• Has maintainer access to the tenant Git repository and apps repositories
• Manages app deployments with GitRepositories and Kustomizations
• Manages app releases with HelmRepositories and HelmReleases

Current solution

See https://github.com/fluxcd/flux2-multi-tenancy

Proposal

A tenant should be able to add sources with GitRepository, HelmRepository and Bucket objects in their namespaces and define how those sources are reconciled on the cluster with Kustomization and HelmRelease objects.

Platform admins should be able to make use of Kubernetes RBAC and restrict the reconciliation scope of a tenant's Kustomizations and HelmReleases.

Defining tenants

Platform admins can define tenants by creating namespaces, RBAC and role bindings in the Git repository bootstrapped with flux.

Tenant with full access to a namespace:

---
apiVersion: v1
kind: Namespace
metadata:
    name: web-team
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: gotk-reconciler
  namespace: web-team
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: gotk:web-team:reconciler

When a Kustomization or HelmRelease is being reconciled in the web-team namespace, it will run under the gotk:web-team:reconciler user that's restricted to the web-team namespace only.

A cluster admin can extend the tenant access to be able to create custom resource definitions or any other cluster wide object:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: crd-admin
rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
    name: web-team-crd-reconciler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: crd-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: gotk:web-team:reconciler

Configuring multi-tenancy

Cluster admins can enable multi-tenancy isolation when installing or upgrading flux.

When a Kustomization or HelmRelease is being reconciled in a namespace other than the namespace where flux is installed, the reconciliation will run under the gotk:<NAMESPACE>:reconciler user. If the user doesn't exists or if the objects being reconciled don't belong to that namespace, the reconcilers will not be able to apply the tenant's sources on the cluster.

The cluster admins can provision tenants with flux create tenant, the command generates a namespace and role binding definition that grants the reconcilers full access to that namespace and that namespace only.

Cluster admin workflow:

# boostrap staging cluster with multi-tenancy enabled
flux bootstrap github \
--owner=org \
--repository=fleet-infra \
--path=staging \
--watch-all-namespaces=true \
--multi-tenancy=true

# clone the repo locally
git clone github.com/org/fleet-infra
cd fleet-infra

# create the dev-team tenant with access to two namespaces
flux create tenant dev-team \
--with-namespace=app1 \
--with-namespace=app2 \
--export > ./staging/dev-team-namespaces.yaml

# add the dev-team-infra repository 
flux create source git dev-team \
--url=https://github.com/org/dev-team-infra \
--branch=main \
--export > ./staging/dev-team-repository.yaml

# reconcile the staging dir of the dev-team-infra repository
flux create kustomization dev-team \
--source=dev-team \
--path="./staging" \
--export > ./staging/dev-team.yaml

# push dev-team tenant
git add -A && git commit -m "Add dev-team to staging" && git push

Dev team workflow:

# clone the repo locally
git clone github.com/org/dev-team-infra
cd dev-team-infra
mkdir staging

# add app1 Git repository
flux -n app1 create source git app1 \
--url=https://github.com/org/app1 \
--branch=staging \
--export > ./staging/app1-repository.yaml

# reconcile app1 on the staging cluster
flux -n app1 create kustomization app1 \
--source=app1 \
--path="./deploy" \
--export >  ./staging/app1.yaml

# add app2 Helm repository
flux -n app2 create source helm app2 \
--url=https://org.github.io/app2-charts \
--export > ./staging/app2-repository.yaml

# reconcile app2 on the staging cluster
flux -n app2 create helmrelease app2 \
--source=HelmRepository/app2 \
--chart=app2 \
--values="./values-staging.yaml" \
--export > ./staging/app2.yaml

# push apps deployments for the staging cluster
git add -A && git commit -m "Add apps to staging" && git push

[squaremo]

After having this explained to me at length (thanks Stefan!), I think there's a simpler design possible.

To recap, the problem to be solved is how to set things safely up so a tenant can create their own GitRepository and Kustomization objects (I refer to these together as "syncs", in the following), without being able to step outside the bounds given by the administrator. This means there's a chain of syncs:

fleet repo --> tenant repo --> app repos

i.e., the fleet config delegates app syncing to the tenant, by syncing a tenant repo containing definitions of GitRepository and Kustomization objects for the app repos.

Since you don't want to give tenant's any access to the system namespace, you can't let them create syncs there. You could create the delegating sync in the tenant namespace; however this means it's accessible to the tenant. Ideally you would want the delegating sync to live inside the system namespace, while anything defined within the tenant repo to be confined to the tenant namespace.

The solution given in the original post is that impersonation of service accounts can be switched off for tenants -- and when it's switched off, a user outside of the control of the tenant is impersonated instead. This has some downsides:

• the behaviour of impersonation is inconsistent (sometimes it's respected, sometimes it's not)
• you have to run the controller in a different mode, and you can't change your mind without risking breaking things

The multi-tenancy mode isn't strictly required -- if the tenant is restricted to creating syncs in their namespace(s), they can't impersonate a service account outside it. To achieve this, you have to have a service account (in the system namespace) for the delegating sync, that has the restricted permissions. Then the problem becomes this much simpler one: service accounts are costly at startup time, because crypto, so you don't really want to have to create one per tenant.

A design which doesn't introduce another controller mode is: let syncs specify a user to impersonate. Then the delegating syncs can refer to a user with restricted permissions for the tenant namespace, rather than having to have a service account created for them. App syncs can try to impersonate a service account or user, or leave it to the default -- they won't be able to access outside the namespace they are in.

[stefanprodan]

I'm ok with removing the namespace field from the service account ref to avoid privilege escalation 👍

[jonathan-innis]

Coming back to the discussion around the default behavior, what's the reasoning behind wanting multi-tenancy enabled by default? As Stefan has mentioned, it seems like quite a bit of onboarding work for a user that does not want this kind of multi-tenancy support to have to create a user/service account for each namespace in which they are reconciling.

[squaremo]

what's the reasoning behind wanting multi-tenancy enabled by default

It's not really having multitenancy by default, it's closing a security loophole by default. The loophole is that you can act as a cluster-admin simply by not mentioning a service account in your Kustomization object.

[stefanprodan]

There is no security loophole if you don't have tenants and your main branch is protected. It works like Flux1, you will open a PR to make changes and when it's merged it gets reconciled on the cluster under cluster-admin.

[squaremo]

I came to think of that as a loophole in Flux v1 -- to be safe it either has to be essentially single tenant, or you have to run it (multiply) with much a more constrained service account. And it is very easy to fall somewhere in the middle of those.

Whether or not you think of this as a security loophole, I hope you would at least agree that it's safer in general to default to giving people very few (or no) permissions, and add what is needed. As has been pointed out, this comes with a heavy penalty in terms of having to think carefully and do extra work, which certainly weighs against it.

In any case, the difference between the two models is drastic enough that I think people should at least be asked to choose one explicitly. Many people will not need to consider permissions given to untrusted users, because there are no untrusted users. It should be easy to run an installation which supports that and doesn't get in the way -- but it also should be obvious that is what you are choosing to do. Similarly, it should be easy and obvious to run an installation which assumes no trust.

With that in mind, I suggest making this a mandatory choice in flux install (and flux bootstrap). Something like

flux install --trusted-tenants|--untrusted-tenants ...

(--single-tenant|--multi-tenant doesn't quite work IMO, but it may be clearer to some).

This makes the question of which way the controller flag runs somewhat moot, since neither alternative is the default at the point of access (that is, in the install command).

[stefanprodan]

Service account impersonation has been implemented for HelmReleases: fluxcd/helm-controller#157

[jonathan-innis]

Another point that was brought up to me by a colleague of mine was the potential for DOSing resources that aren't present in your namespace by setting up a webhook receiver referencing resources in a namespace you don't have permissions to access, triggering continuous reconciliation on resources you don't own. I'm not sure if the docs are consistent on this point as it relates to the current state of the source code, but as it stands in the docs, this also seems to be a multi-tenancy hole.

[stefanprodan]

Good point, receivers should be subject to the multi-tenancy policy here: https://github.com/fluxcd/flux2-multi-tenancy/blob/main/infrastructure/kyverno-policies/flux-multi-tenancy.yaml

[stealthybox]

Thanks for your comment on this @jonathan-innis.
I've updated #582 to include notification-controller impersonation, so that this can be denied across namespaces by default with the option to open it up via RBAC.

[stealthybox]

I've written up #582 which proposes changes inspired by these strategies.
A good number of the concerns here are addressed in some way.
I've added several new ones as well.

We could probably add an entire section based on @stefanprodan's cluster owner and tenant workflows using the flux cli.
After looking at this thread again, I think that's lacking detail in the proposal.

Please give it a look when you have a moment!
Feedback is super desired.

[stealthybox]

Just noting:
This is not a competing proposal to what @stefanprodan posted above.
We've been collaborating on this approach since the beginning, so it's an iteration on what is here.