⛵ Build Amazon EKS using Terraform, GitHub Actions and GitOps

... managed by Flux/ArgoCD and serviced with Renovate ...

• GitHub repository: https://github.com/ruzickap/k8s-tf-eks-gitops
• Web Pages: https://ruzickap.github.io/k8s-tf-eks-gitops

---

📖  Overview

---

✨  Cluster setup

---

🎨  Cluster components

• cilium: For cluster networking.
• ingress-nginx: Provides ingress cluster services.
• SOPS: Encrypts secrets which is safe to store - even to a public repository.
• external-dns: Creates DNS entries in Cloud Provider's DNS service.
• cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.

---

📂  Repository structure

flux tree kustomization flux-system --compact

Output:

Kustomization/flux-system/flux-system
├── Kustomization/flux-system/cluster-apps
│   ├── HelmRelease/metrics-server/metrics-server
│   ├── HelmRelease/polaris/polaris
│   ├── Kustomization/flux-system/cert-manager
│   │   └── HelmRelease/cert-manager/cert-manager
│   ├── Kustomization/flux-system/cert-manager-certificate
│   ├── Kustomization/flux-system/cert-manager-clusterissuer
│   ├── Kustomization/flux-system/cert-manager-crds
│   ├── Kustomization/flux-system/cluster-autoscaler
│   │   └── HelmRelease/cluster-autoscaler/cluster-autoscaler
│   ├── Kustomization/flux-system/crossplane
│   │   └── HelmRelease/crossplane-system/crossplane
│   ├── Kustomization/flux-system/crossplane-provider
│   ├── Kustomization/flux-system/crossplane-providerconfig
│   ├── Kustomization/flux-system/dex
│   │   └── HelmRelease/dex/dex
│   ├── Kustomization/flux-system/external-dns
│   │   └── HelmRelease/external-dns/external-dns
│   ├── Kustomization/flux-system/ingress-nginx
│   │   └── HelmRelease/ingress-nginx/ingress-nginx
│   ├── Kustomization/flux-system/kube-prometheus-stack
│   │   └── HelmRelease/kube-prometheus-stack/kube-prometheus-stack
│   ├── Kustomization/flux-system/kubernetes-dashboard
│   │   └── HelmRelease/kubernetes-dashboard/kubernetes-dashboard
│   ├── Kustomization/flux-system/oauth2-proxy
│   │   └── HelmRelease/oauth2-proxy/oauth2-proxy
│   └── Kustomization/flux-system/podinfo
│       └── HelmRelease/podinfo/podinfo
├── Kustomization/flux-system/cluster-apps-secrets
├── Kustomization/flux-system/sources
│   ├── HelmRepository/flux-system/autoscaler
│   ├── HelmRepository/flux-system/bitnami
│   ├── HelmRepository/flux-system/crossplane
│   ├── HelmRepository/flux-system/dex
│   ├── HelmRepository/flux-system/fairwinds-stable
│   ├── HelmRepository/flux-system/ingress-nginx
│   ├── HelmRepository/flux-system/jetstack
│   ├── HelmRepository/flux-system/kubernetes-dashboard
│   ├── HelmRepository/flux-system/metrics-server
│   ├── HelmRepository/flux-system/oauth2-proxy
│   ├── HelmRepository/flux-system/podinfo
│   └── HelmRepository/flux-system/prometheus-community
└── GitRepository/flux-system/flux-system

---

🤖  Automate all the things

• GitHub Actions for checking code formatting
• Renovate Renovate GitHub action keeps my application charts and container images up-to-date

---

🕸️  Secrets

There are several secrets:

• cluster-apps-vars-terraform-secret - used for providing Terraform variables to Flux/Kustomizations: eks.tf
• cluster-apps-secrets - secrets specific to cluster: cluster-apps-secrets.yaml
• cluster-apps-group-secrets - secrets specific to cluster group: cluster-apps-secrets.yaml

---

🤷‍♂️  Notes

• Describe the directory structure
• Check emails form policy-reporter
• Put all HelmRepository objects to flux-system instead of "namespaces" to be able to share them
• Check snapshots (cnpg/velero) + KMS keys (if they are being deleted)
• kubernetes-dashboard - auto login not working

---

🤝  Thanks

A lot of inspiration for my cluster came from the people that have shared their clusters over at k8s-at-home and many other "GitHub" repositories...

Requirements for K8s cluster management

Not complete...

• Run change on only 3% of the clusters, then another 3%, ...
• Changes applied to group of clusters
• Easily add / remove application from specific cluster / cluster group
• One source (file) for "variables/secrets"
• Move cluster from one cluster to another... ?
• Different TF code for different clusters / cluster groups