home-infra-k8s-flux

This is my Kubernetes Flux repository. It contains everything installed in my kubernetes cluster and keeps the cluster up to date with this repo.

Apps:

See apps/production.

Usage

Handy CLI Commands working with Flux

See Flux Troubleshooting Cheatsheet.

# usually most informative:
flux get kustomizations --watch

# very informative too, but very detailed:
flux logs -f

# force reconciliation to source:
flux reconcile kustomization flux-system --with-source

# force reconciliation of (docker) image repository:
flux -n scott-willeke-com-prod reconcile image repository repo-scott-willeke-com
flux -n ramblefeed-prod reconcile image repository repo-ramblefeed-app

# Show all Flux objects that are not ready !
flux get all -A --status-selector ready=false

# watch flux events:
flux events -w

# Show flux warning events
kubectl get events -n flux-system --field-selector type=Warning

flux get kustomizations --watch

###############
#
# To fix something manually while flux won't constantly replace them do this:
flux suspend kustomization apps
# then make changes
# then resume:
flux resume kustomization apps
#
###############

flux reconcile kustomization apps

# I find it helpful to get logs directly from the kustomization controller:
kubectl -n flux-system logs -f deployment/kustomize-controller


# Automated Image Updates:
# check the image repository (per https://fluxcd.io/flux/guides/image-update/)
flux get image repository -n tayle-prod repo-tayle-app

# list images flux is tracking:
flux get images all --all-namespaces

# list the image policies:
flux get images -A policy

# list all image repositories:
kubectl get -A imagerepository

# list the tags found in an image repository:
kubectl get -n tayle-prod -o=yaml imagerepository/repo-tayle-worker

# reconcile an image repository:
flux -n gpupoet-prod reconcile image repository repo-gpupoet-app
flux -n tinkerbell-prod reconcile image repository repo-tinkerbell-app
flux -n tayle-prod reconcile image repository repo-tayle-app
flux -n tayle-prod reconcile image repository repo-tayle-worker

# a handy way to do a dry run on the kustomize (this prints a lot of warnings when it works but returns non-zero as long as there are no errors):
kubectl kustomize apps/production | kubectl apply --dry-run='server' -f -

Cluster Layout

├── apps
│   ├── base
│   ├── production
│   └── staging
├── infrastructure
│   ├── base
│   ├── production
│   └── staging
└── clusters
    ├── production
    └── staging

per https://fluxcd.io/flux/guides/repository-structure/ example at https://github.com/fluxcd/flux2-kustomize-helm-example

Secrets

Using sops + age.

Encrypting

TLDR: put .env files in a dirctory and then run /scripts/encrypt-env-files.sh <dir> on the dir containing the .env file and it will save .env*.encrypted files that you can reference in kustomization files like:

secretGenerator:
  # db
  - name: db-creds
    envs:
      - .env.secret.db.encrypted

Per https://fluxcd.io/flux/guides/mozilla-sops/#encrypting-secrets-using-age

Decrypting

The flux+kustomize knows how to decrypt SOPS secrets via secret generator. So we just have to have a sops-age secret in the flux-system namespace in the cluster.

See /infrastructure/configs/create-sops-age-decryption-secret.sh

Per https://fluxcd.io/flux/guides/mozilla-sops/#encrypting-secrets-using-age

Image Pull Secrets

Image Pull Secrets (to Pull an Image from a Private Registry) using .dockerconfigjson secrets are kinda just like json secrets. Run:

./scripts/create-image-pull-secret-ghcr.sh

Per https://fluxcd.io/flux/components/kustomize/kustomizations/#kustomize-secretgenerator

Image Updates & Image Scanning

Image scanning for one app setup at apps/production/tayle/image-scanning per https://fluxcd.io/flux/guides/image-update/

OCI Labels Required for GHCR Webhook Events

Each app's Docker build workflow must include OCI labels (org.opencontainers.image.source and org.opencontainers.image.revision) on the container image. These labels link the GHCR package to its GitHub repo, which is required for GitHub to send package webhook events. Without them, Flux image automation won't receive notifications when new images are published. See the labels: parameter in each app's docker/build-push-action step.

Image Updates from Github Web Hooks for Continuous Deployment

A flux webhook receiver is set up in /infrastructure/base/configs/image-scanning-webhook-receiver. It has configured which ImageRepositories to refresh. More can be added.

Add a webhook to github like:

Get the ReceiverURL by running kubectl -n flux-system get receiver it will print it out as its status.

On GitHub, navigate to your repository and click on the “Add webhook” button under “Settings/Webhooks”. Fill the form with: Payload URL: compose the address using the receiver LB and the generated URL http:/// Secret: use the token string

With the above settings, when you push a commit to the repository, the following happens:

GitHub sends the Git push event to the receiver address Notification controller validates the authenticity of the payload using HMAC Source controller is notified about the changes Source controller pulls the changes into the cluster and updates the GitRepository revision Kustomize controller is notified about the revision change Kustomize controller reconciles all the Kustomizations that reference the GitRepository object

per https://fluxcd.io/flux/guides/webhook-receivers/

Restoring a Cluster from Flux Repo

You can use the flux bootstrap at anytime to re-provision the cluster with everything in flux. I've done this on a clean k3s a few times now and it works flawlessly:

  flux bootstrap github \
  --token-auth \
  --owner=activescott \
  --repository=home-infra-k8s-flux \
  --branch=main \
  --path=clusters/nas1 \
  --personal \
  --components-extra=image-reflector-controller,image-automation-controller                                                                                                                                                                                                                                                                                                                  

Since I use SOPS to encrypt secrets and keep them in git, need to provision that root key

./scripts/create-sops-age-decryption-secret.sh

Force reconcile:

flux reconcile kustomization infra-configs

...that should should show some output "waiting for Kustomization reconciliation..."

Then you can monitor with:

flux logs -f

# or flux get kustomization --watch

Should see output lines like Namespace/activescott-redirect created for each flux resource as it is created.

YAML+Kustomize

I prefer plain "kubectl yaml" and Kustomize over helm. Helm is great for packaging up an app into an opaque package and provide it to others, but IMHO not for managing a cluster directly. When consuming apps, I prefer consuming yaml if provided, but don't mind consuming Helm.

Posterity / Done

• Bootstrap See script for this in scripts dir. it was updated.

flux bootstrap github \
--token-auth \
--owner=activescott \
--repository=home-infra-k8s-flux \
--branch=main \
--path=clusters/nas1 \
--personal