Allow pre-bootstrap customisation of Flux components

[stefanprodan]

This PRs changes the bootstrap logic to allow customising the Flux manifests before or after bootstrap.

Overview

This PR enables users to set annotations (IAM role bindings, safe-to-evict, etc), node affinity and tolerations, change the number of replicas, the resources requests and limits, the container group id and user id, etc before the controllers are deployed on the cluster. Besides patching the controllers, users can patch the sync manifests too, this would allow disabling pruning, enabling SOPS and other customisations not cover by bootstrap cmd args.

Customize Flux manifests

You can customize the Flux components before or after running bootstrap.

Assuming you want to customize the Flux controllers before they get deployed on the cluster,
first you'll need to create a Git repository and clone it locally.

Create the file structure required by bootstrap with:

mkdir -p clusters/my-cluster/flux-system
touch clusters/my-cluster/flux-system/gotk-components.yaml \
    clusters/my-cluster/flux-system/gotk-patches.yaml \
    clusters/my-cluster/flux-system/gotk-sync.yaml \
    clusters/my-cluster/flux-system/kustomization.yaml

Assuming you want to add custom annotations and labels to the Flux controllers,
edit clusters/my-cluster/gotk-patches.yaml and set the metadata for source-controller and kustomize-controller pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: source-controller
  namespace: flux-system
spec:
  template:
    metadata:
      annotations:
        custom: annotation
      labels:
        custom: label
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustomize-controller
  namespace: flux-system
spec:
  template:
    metadata:
      annotations:
        custom: annotation
      labels:
        custom: label

Edit clusters/my-cluster/kustomization.yaml and set the resources and patches:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patchesStrategicMerge:
  - gotk-patches.yaml

Push the changes to main branch:

git add -A && git commit -m "add flux customisations" && git push

Now run the bootstrap for clusters/my-cluster:

flux bootstrap git \
  --url=ssh://git@<host>/<org>/<repository> \
  --branch=main \
  --path=clusters/my-cluster

When the controllers are deployed for the first time on your cluster, they will contain all
the customizations from gotk-patches.yaml.

You can make changes to the patches after bootstrap and Flux will apply them in-cluster on its own.

Issues

Fix: #1166
Fix: #1180
Refs: #829 fluxcd/terraform-provider-flux#120 fluxcd/terraform-provider-flux#130

[hiddeco]

Looks good to me 🍎