Cluster Onboarding

This repository contains an example implementation of OpenShift project onboarding via GitOps. It's implemented with kustomize, but could be probably also done via helm (see here).

Table of contents

• General scheme
• Nomenclature
• Use cases considered
• Requirements considered
• Tools used
• Tenants organization
• Sharing a common base of configurations
• Additional Folders
• OpenShift Gitops

General scheme

The purpose of platform onboarding is to provide an automated way of creating required OpenShift resources like

• namespaces
• quotas
• network policies

for developer teams and application operations teams.

Yes we know this is not DevOps, but corresponds to things we see in real life. But what exactly is DevOps anyways?

We defined two repositories:

• Onboarding-base
• Onboarding

Onboarding-base is used to provide manifests reused for all tenants (see Nomenclature).

We also define different T-Shirt sizes for namespaces in the base respository.

Onboarding contains manifest for the actual onboarding of tenants.

Nomenclature

• Tenant: A team requiring OpenShift resources in one or more clusters.
• Cluster: An OpenShift cluster managed by a platform team

Use cases considered

1. As developer I would like to get namespaces in a DEV cluster
2. As application ops I require a namespaces in the DEV cluster
3. As application ops I require namespaces in the TEST cluster
4. As application ops I require namespaces in the PROD cluster
5. As application ops I require different settings per cluster
6. As a platform team I would like to support different resource quotas for namespaces depending on clusters
7. As a platform team I would like to leverage a common base for all Kubernetes manifests.

Requirements considered

1. A tenant has one to many namespaces
2. Not all tenant namespaces are in all clusters
3. Namespaces might have different settings per cluster e.g.
   • CI/CD namespace on cluster dev is allowed to create 100 pods
   • CI/CD namespace on cluster prod is allowed to create 10 pods
4. Resource requirements might be different between clusters

Tools used

We required kustomize for rendering manifests to onboard tenants. Helm is also possible and there are examples available, see helper-proj-onboarding and the customers folder in openshift-cluster-bootstrap

Kustomize allows use to deploy any Kubernetes manifest in any folder and also patching of existing manifest if there's the need.

Tenants organization

We leverage the following folder structure for tenants:

tenants/
├── dinosaurs
│   ├── clusters
│   │   ├── dev
│   │   ├── prod
│   │   └── test
│   └── namespaces
│       ├── dinosaurs-accounting
│       │   ├── base
│       │   ├── large
│       │   └── medium
│       └── dinosaurs-cicd
└── goldfish
    ├── clusters
    │   └── prod
    └── namespaces
        ├── goldfish-bookstore
        └── goldfish-cicd

Every new tenant requiring onboarding is represented by a folder in the tenants directory.

Within tenants we have a clusters/ folder for every cluster the tenant should be able to use. In the corresponding cluster folder (e.g. dinosaurs/clusters/prod) we include namespaces that should be deployed to the prod cluster. Those namespaces are defined in dinosaurs/namespaces/.

The following diagram depicts connections between clusters, namespaces and the onboarding-base repository:

Sharing a common base of configurations

Namespaces in <tenants>/<tenant name>/namespaces share a common base. The base is in a separate GIT repository onboarding-base.

Technically onboarding-base could be stored in the same repository as onboarding but we have reasons to use a dedicated repository:

• Making changes to base explicit: changes to the common base for all namespace is potentially a dangerous operation. It could effect all namespaces on all cluster.

  We think that having this common base in a dedicated repository strengthens the awareness for this risk.

• Allow usage of dedicated tags / releases: We use tags to reference a (hopefully ) frozen state of the onboarding-base repository. If onboarding-base changes upstream the changes will not effect downstream namespaces. See for example here.

  Using a newer base for a namespace requires an explicit change.

onboarding-base is a way to share common configurations, relevant for all namespaces in all clusters. For example it defines the following t-shirt shape sizes for projects:

• small
• medium
• large

Because of using kustomize, there are ways to overwrite base settings. For example in the CI/CD namespace for the goldfish project we use the medium overlay from onboarding-base, but overwrite the memory limit to 7Gi.

Additional Folders

• onboarding: Contains ArgoCD ApplicationSet's for creating ArgoCD Applications from the tenants folder.
• argocd: Additional ArgoCD configuration like repositories and clusters.
• docs: Diagrams and other documentation artifacts

OpenShift Gitops

OpenShift GitOps leverages ArgoCD to sync Kubernetes manifests from a GIT repository to a cluster.

An ArgoCD Application defines which repository to sync into which cluster.

ArgoCD uses projects to organize applications in groups.

Specifically we use ApplicationSets to generate required Applications. See onboarding/onboarding-applicationset.yaml for more information.