Please don't judge the commit history and some janky components too harshly, I'm just doing this for fun in my spare time with the majority of these services not exposed to the Internet.
This is a repository for my Kubernetes cluster. (Shamelessly stolen README) I have all of my ArgoCD applications here to deploy the infrastructure and services I use at home.
My K8S cluster is deployed with the Ansible script in the Ansible folder. This sets up a K3S flavoured K8S cluster.
Once the k8s cluster is up, the Ansible script will bootstrap ArgoCD and then my app of apps, which will deploy all applications to the cluster.
After that the terraform process needs to be run to onboard necessary secrets and create the needed cloud infrastructure.
- argo-cd: Continous deployment tool used to deploy manifests/charts from git to my homelab
- authentik: SSO for the services that support LDAP/SAML/OIDC.
- cert-manager: Creates SSL certificates for services in my Kubernetes cluster.
- cloudnative-pg: Operator for Postgresql DB management.
- external-dns: Automatically manages DNS records from my cluster in a cloud DNS provider.
- external-secrets: Used to store my secrets in a reversible way so DR is trivial.
- ingress-nginx: Ingress controller to expose HTTP traffic to pods over DNS.
- kyverno: Policy management for cluster (ie. Setup all pods with timezone Australia/Sydney)
- loki: Centralized log storage
- metallb: Load balancer provisioning service for bare metal LBs
- prometheus/alert manager/ grafana: Observability and alerting
- rook: Distributed block storage for peristent storage.
- volsync and snapscheduler: Backup and recovery of persistent volume claims.
This Git repository contains the following directories.
📁 .taskfiles # Script files for common operations I perform
📁 ansible # Ansible playbook to deploy K3S binaries onto nodes
📁 kubernetes # Contains all k8s manifests deployed using Argo CD
└─📁 helm
└─📁 ${namespace} # Folder name used by Argo to determine application's namespace
└─📁 ${application} # folder name used to name the Helm Application created by Argo
└─📁 manifests # optional : manifests related to application (i.e. Secrets, ClusterPolicy, etc)
📁 terraform # Create cloud infrastructure and k8s manifests in a repeatable wayWhile most of my infrastructure and workloads are selfhosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about two things. (1) Dealing with chicken/egg scenarios and (2) services I critically need whether my cluster is online or not.
| Service | Use | Cost |
|---|---|---|
| G Suite | ~$15/mo | |
| Backblaze B2 | Backups | ~$5/mo |
| GCP | Secrets management | ~$2/mo |
| Cloudflare | Domain, DNS and proxy management | ~$30/yr |
| GitHub | Hosting this repository and continuous integration/deployments | Free |
| Zen Duty | Push notifications alerting when health issue occurs | Free |
| Total: ~$24.5/mo |
Over WAN, I have port forwarded ports 80 and 443 to the load balancer IP of my ingress-nginx controller that's running in my Kubernetes cluster.
CoreDNS is deployed in my k8s cluster. All DNS queries for k8s.tylercash.dev domains are forwarded to k8s_gateway that is running in my cluster. With this setup k8s_gateway has direct access to my clusters ingresses and services and serves DNS for them in my internal network.
The CoreDNS deployment has a volume with a hosts file present on it. This hostfile is updated frequently by a CronJob run in cluster. CoreDNS is configured to drop any entries that are matched in this file, which results in ads failing to load.
external-dns is deployed in my cluster and configure to sync DNS records to Cloudflare. The only ingresses external-dns looks at to gather DNS records to put in Cloudflare are ones that have an annotation of external-dns.alpha.kubernetes.io/target
My home IP can change at any given time and in order to keep my WAN IP address up to date on Cloudflare. pfSense has a native Dynamic DNS tool which I use to update my IP in cloudflare
| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose |
|---|---|---|---|---|---|---|
| DIY machine (i5 11400) | 2 | 256GB SSD | 1x 1TB SSD, 1x 4TB HDD | 32GB | Ubuntu 22.04 | K8S Node |
| HP ProDesk 600 G2 (i5 6500) | 1 | 256GB SSD | 1x 1TB SSD, 4x 1TB HDD | 32GB | Ubuntu 22.04 | K8S Node |
| Soft Router (Celeron N5105) | 1 | 256GB SSD | N/A | 8GB | pfSense | Router |
Thanks to all the people who donate their time to the Kubernetes @Home Discord community. A lot of inspiration for my cluster comes from the people that have shared their clusters using the k8s-at-home GitHub topic. Be sure to check out the Kubernetes @Home search for ideas on how to deploy applications or get ideas on what you can deploy.