Kubernetes clusters using the GitOps tool Flux.
The Git repository is the driving the state of the Kubernetes clusters.
The awesome Flux SOPS integration is used to encrypt secrets with age.
The Git repository contains the following directories:
📁
├─📁 apps # apps available for intallation
├─📁 cluster-apps # kustomization and overlays for app installations per cluster
│ ├─📁 staging
│ └─📁 production
├─📁 charts # helm chart repos
├─📁 configs # configs per cluster
└─📁 base
├─📁 flux-system # flux & gitops operator
├─📁 staging # flux configuration per cluster
└─📁 production # flux configuration per clusterThe following apps are installed on the clusters.
| Software | Purpose |
|---|---|
| Flux2 | GitOps Tool managing the cluster |
| Longhorn | Persistent Block Storage Provisioner |
| NGINX Ingress Controller | Cluster Ingress controller |
| MetalLB | Bare metal LoadBalancer |
| Cert-Manager | Letsencrypt certificates with Cloudflare DNS |
| ExternalDNS | Configure Cloudflare DNS Servers |
| kube-vip | Virtual IP Load-Balancer for Control Plane High Availability |
| Kube-Prometheus Stack | Prometheus & Exporters to monitor the cluster |
| Grafana | Monitoring & Logging Dashboard |
| Alertmanager | Monitoring Alerts |
| Grafana Loki | Log aggregation system |
| System Upgrade Controller | Automated k3s upgrades |
| Weave GitOps | Powerful WebUI extension to Flux for deployment insights |
| MinIO | Amazon S3 compatible high Performance Object Storage |
| Authelia | SSO & 2FA authentication server for Cluster Web Apps |
| Paperless-ngx | Document management system |
| Tandoor-Receipes | Receipe Manager and Meal Planner |
| Rancher | Kubernetes Management Dashboard |
| Homer | Static dashboard for the cluster applications |
| Pod-Gateway | Route traffic through a VPN gateway |
| Descheduler | Evicts pods to optimize scheduling |
| Goldilocks | Utility to help identifying good resource requests and limits |
| X.509 Certificate Exporter | A Prometheus exporter to monitor x509 certificates |
| SMB CSI Driver | CSI Driver for SMB |
| kured | Kubernetes Reboot Daemon (only used for Monitoring) |
| Kubernetes Metrics Server | Source of container resource metrics for Kubernetes |
Renovate Bot makes sure the components are never outdated.
It creates PullRequests when Helm charts or Docker images have newer versions available and even keeps Flux and k3s up-to-date.
Big shout out to k8s@home and everyone from awesome-home-kubernetes for the inspiration ❤️
📍 Installation Notes
tl;dr
kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -
sops -d ./base/flux-system/init/flux-sops-age-secret.sops.yaml | kubectl apply -f -
sops -d ./base/flux-system/init/flux-secret.sops.yaml | kubectl apply -f -
kubectl apply --kustomize=./base/flux-system
kubectl apply --kustomize=./base/staging
- Pre-create the
flux-systemnamespace
kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -- Add the Flux age key in-order for Flux to decrypt SOPS secrets
sops -d ./base/flux-system/init/flux-sops-age-secret.sops.yaml | kubectl apply -f -- (Optional) Add the Flux SSH key in-order for Flux to pull private git repositories
sops -d ./base/flux-system/init/flux-secret.sops.yaml | kubectl apply -f -- Install Flux
kubectl apply --kustomize=./base/flux-system- Configure Flux
kubectl apply --kustomize=./base/staging🚧 PostgreSQL Major Version Upgrade
I now use a tianon/postgres-upgrade init-container for PostgreSQL Major Upgrades.
Always take backups, dataloss is possible. Old data gets removed and replaced by output of pg_upgrade.
## Init Container for Major PostgreSQL Upgrades, not needed permanently
initContainers:
- name: pg-upgrade
image: tianon/postgres-upgrade:15-to-16
securityContext:
runAsUser: 0
runAsGroup: 0
volumeMounts:
- name: data
mountPath: /bitnami/postgresql
env:
- name: "PG_OLD"
value: "15"
- name: "PG_NEW"
value: "16"
command:
- /bin/bash
- -c
- |
if [[ $(< /bitnami/postgresql/data/PG_VERSION) -eq $PG_NEW ]]; then echo "PostgreSQL is already up2date"; exit 0; fi
if [[ "$PGBINOLD" != "/usr/lib/postgresql/$PG_OLD/bin" ]]; then echo "Wrong postgres-upgrade image"; exit -1; fi
if [[ "$PGBINNEW" != "/usr/lib/postgresql/$PG_NEW/bin" ]]; then echo "Wrong postgres-upgrade image"; exit -1; fi
echo "Upgrading PostgreSQL from $PG_OLD to $PG_NEW"
cp -r /bitnami/postgresql/data /var/lib/postgresql/$PG_OLD
usermod -u 1001 postgres
groupmod -g 1001 postgres
chown -R postgres:postgres /var/lib/postgresql
su postgres -c 'PGDATA="$PGDATANEW" eval "initdb $POSTGRES_INITDB_ARGS"'
cp -p $PG_NEW/data/postgresql.conf $PG_NEW/data/pg_hba.conf $PG_OLD/data/
chmod 700 $PG_OLD/data
gosu postgres pg_upgrade
gosu postgres pg_ctl -D /var/lib/postgresql/$PG_NEW/data -l logfile start
gosu postgres /usr/lib/postgresql/$PG_NEW/bin/vacuumdb --all --analyze-in-stages
gosu postgres pg_ctl -D /var/lib/postgresql/$PG_NEW/data -l logfile stop
rm /var/lib/postgresql/$PG_NEW/data/pg_hba.conf /var/lib/postgresql/$PG_NEW/data/postgresql.conf
rm -rf /bitnami/postgresql/data
mv /var/lib/postgresql/$PG_NEW/data /bitnami/postgresql/🚧 Previous Manual Upgrade Approach
Based on bitnami/charts#1798 (comment)
- Preparation
export NAMESPACE=selfhosted
export APPLICATION_DEPLOYMENT=paperless-ngx
export POSTGRES_DEPLOYMENT=${APPLICATION_DEPLOYMENT}-postgresql
export POSTGRES_MAJOR_VERSION=12
export POSTGRES_PVC_SIZE=4Gi
export POSTGRES_DB=${APPLICATION_DEPLOYMENT}
export POSTGRES_USERNAME=${APPLICATION_DEPLOYMENT}
export POSTGRES_PASSWORD=yourSecretPasswordNote: POSTGRES_MAJOR_VERSION is the helm version, not postgresql version.
- Scale down application that uses the database
kubectl scale deployment ${APPLICATION_DEPLOYMENT} -n ${NAMESPACE} --replicas 0- Deploy new major version of the database
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install ${POSTGRES_DEPLOYMENT}-upgrade bitnami/postgresql --version ${POSTGRES_MAJOR_VERSION} -n ${NAMESPACE} --wait \
--set auth.username=${POSTGRES_USERNAME} \
--set auth.password=${POSTGRES_PASSWORD} \
--set auth.database=${POSTGRES_DB} \
--set primary.persistence.size=${POSTGRES_PVC_SIZE}- Migrate data to new postgresql deployment
kubectl exec -it ${POSTGRES_DEPLOYMENT}-upgrade-0 -n ${NAMESPACE} -- bash -c "export PGPASSWORD=${POSTGRES_PASSWORD}; time pg_dump -h ${POSTGRES_DEPLOYMENT} -U ${POSTGRES_USERNAME} | psql -U ${POSTGRES_USERNAME}"From here on you have multiple possibilities, e.g. just use your app with the new db deployment.
I personally prefer to backup the volume of the new DB, uninstall both database deployments & delete their PVCs.
After that I restore the backup of the PVC with the name of the old database deployment & upgrade my Helmrelease version.
helm uninstall ${POSTGRES_DEPLOYMENT}-upgrade -n ${NAMESPACE}
kubectl scale sts ${POSTGRES_DEPLOYMENT} -n ${NAMESPACE} --replicas 0The volume deletion and restore is done in Longhorn UI. Afterwards helm upgrade for the postgresql deployments can be done.
- Scale up application that uses the database
kubectl scale deployment ${APPLICATION_DEPLOYMENT} -n ${NAMESPACE} --replicas 1