Build a Kubernetes (k3s) cluster with RPis and utilize GitOps for managing cluster state. I would like to give a shout-out to k8s-gitops, the big brother of this repo, created by @billimek.
Note: A lot of files in this project have @CHANGEME comments, these are things that are specific to my set up that you may need to change.
- 3x RPi4 (recommended 4GB RAM) and at least 32GB SD Cards
- 3x USB 3.1 256GB Flash drives (used for k3s local storage on each node, any size is fine)
- A NFS server for storing persistent data
Note: I use the fish shell for a lot of my commands. Some will work in Bash but others will not, see here for more information.
.
├── ./deployments # Flux will only scan and deploy from this directory
├── ./setup # Setup of the cluster
│ └── ./ansible # Ansible playbook to run after the RPis have been flashed
├── ./hack # Helper scripts
└── ./docs # Documentation
| IP | Function |
|---|---|
| 192.168.1.1 | Router (USG) |
| 192.168.1.15 | Dedicated RPi running PiHole |
| 192.168.1.170 | NFS Server |
| 192.168.42.1/24 | k3s cluster CIDR, VLAN 42 |
| 192.168.42.23 | k3s master (pik3s00) |
| 192.168.42.24 | k3s worker (pik3s01) |
| 192.168.42.25 | k3s worker (pik3s02) |
See ubuntu.md
Ansible is a great automation tool and here I am using it to provision the RPis.
See ansible.md and review the files in the setup/ansible folder.
k3sup is a neat tool provided by @alexellis that helps get your k3s cluster up and running quick.
For manual deployment see k3sup.md, and for an automated script see bootstrap-cluster.sh
Helm is a package manager for Kubernetes.
For manual deployment see helm-flux.md, and for an automated script see bootstrap-cluster.sh
Flux is the GitOps tool I've chosen to have this Git Repository manage my clusters state.
For manual deployment see helm-flux.md, and for an automated script see bootstrap-cluster.sh
Sealed Secrets are a "one-way" encrypted Secret that can be created by anyone, but can only be decrypted by the controller running in the target cluster. The Sealed Secret is safe to share publicly, upload to git repositories, give to the NSA, etc. Once the Sealed Secret is safely uploaded to the target Kubernetes cluster, the sealed secrets controller will decrypt it and recover the original Secret.
See sealed-secrets.md and review the files in the setup folder.
I opted to let a HAProxy server which lives outside of my cluster handle SSL termination and domain routing since I have multiple domains pointing to my Homes IP address. Since every domain doesn't exactly point to my k3s cluster I found this the easy way to have any number of domains I own point to different servers in my house. See assets/_k3s.png for my network topology. This setup is documented in docs/haproxy-cloudflare.md if anyone is interested.
The path I have chosen above doesn't exactly fit everyones use-case so if you would like SSL terminated at NginX I would take a look at this repo and see how it's made possible with cert-manager.
MinIO is pioneering high performance object storage, think of this as self-hosted AWS S3.
Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.
For manual deployment see minio-velero.md
The projects in the table below I have compiled multi-arch ARM (arm, arm64) Docker images for.
| Project | Repository | DockerHub | Issue requesting ARM images |
|---|---|---|---|
| Flux | onedr0p/flux | onedr0p/flux | fluxcd/flux#1761 |
| Helm Operator | onedr0p/helm-operator | onedr0p/helm-operator | fluxcd/helm-operator#147 |
| Velero | onedr0p/velero | onedr0p/velero | vmware-tanzu/velero#720 |
| Velero Plugin for AWS | onedr0p/velero-plugin-for-aws | onedr0p/velero-plugin-for-aws | vmware-tanzu/velero-plugin-for-aws#18 |
| Kured | onedr0p/kured | onedr0p/kured | weaveworks/kured#23 |
| Sealed Secrets | onedr0p/sealed-secrets | onedr0p/sealed-secrets | bitnami-labs/sealed-secrets#173 |
- Samsung 128GB EVO Plus Class 10 Micro SDXC
- AUKEY Quick Charge 3.0 6-Port USB Wall Charger
- AUKEY USB C Cable Short
- Cablecc Mini Size 5Gbps Super Speed USB 3.0 to Micro SD SDXC TF Card Reader Adapter
- Samsung MUF-256AB/AM FIT Plus 256GB - 300MB/s USB 3.1 Flash Drive
- Argon NEO Raspberry Pi 4 Case and/or iUniker Raspberry Pi 4 Cluster Case,