Kubernetes on Raspberry Pis

Build a Kubernetes (K3s) cluster with Raspberry Pis, using Anisble to bootstrap the machines, and utilizing ArgoCD for managing cluster state.

Do not expect the git history for this repository to be clean and tidy.

Prerequisites:

• This was done using a cluster of 4 x RPi 4 4GB
• All Pi's are connected via a local ethernet switch on a 10.0.0.0/24 LAN
• The master node connects to the outside world on WiFi, and provides NAT for the rest of the cluster.

Directory layout description

.
│   # Ansible playbooks to bootstrap the cluster
├── ./ansible
│   # Docker builds for ARM devices
├── ./docker
│   # ArgoCD will deploy from this directory
├── ./gitops
│   # Scripts and config for setting up the Raspberry Pis
└── ./scripts
    │  # Cloud-init config for Raspberry Pi Imager
    └─ ./cloud-init

Network topology

IP	Function	MAC Address
192.168.115.1	Router
192.168.115.193	Master wifi interface
10.0.0.0/24	k3s cluster CIDR
10.0.0.1	k3s master (master)	dc:a6:32:67:76:f1
10.0.0.2	k3s worker (node-1)	dc:a6:32:67:77:3e
10.0.0.3	k3s worker (node-2)	dc:a6:32:67:76:b8
10.0.0.4	k3s worker (node-3)	dc:a6:32:67:77:06

Hardware list

• 4 x Raspberry Pi 4 Model B 4GB
• 4 x Raspberry Pi PoE Hat
• 4 x SanDisk Ultra microSDHC Memory Card
• 4 x 15cm Flat Cat 6 cables
• 4 x Low profile heatsinks
• 4 x WD Green 240GB 2.5" SSD
• 4 x SSD to USB 3.0 Cable
• Uctronics Cluster Enclosure V3.0
• MicroSD Extender Set for Uctronics Cluster Cases
• NETGEAR 5-Port Gigabit Ethernet PoE Switch

Bootstrapping the cluster

1. Ensure that Raspberry Pi Imager is installed along with kubectl and kubecm
2. Run make flash to flash the SSD drive for each RPI using RPI Imager. It will flash with Ubuntu Server 24.04.3 LTS (64-bit) and copy the network-config & user-data relevant to the node to the SSD.
3. Edit ~/.ssh/config on the local machine to include the following (see REBELinBLUE/dotfiles)

Host master
    Hostname cluster.lan # Or the IP address
    User ubuntu

Host node-1
    Hostname 10.0.0.2
    ForwardAgent yes
    User ubuntu
    ProxyJump master

Host node-2
    Hostname 10.0.0.3
    ForwardAgent yes
    User ubuntu
    ProxyJump master

Host node-3
    Hostname 10.0.0.4
    ForwardAgent yes
    User ubuntu
    ProxyJump master

4. Plug in the master RPI and let it boot
5. Run make master to set up the master
6. Plug in the 3 remaining nodes and let them boot
7. Run make workers to set up the worker nodes
8. Run make cluster to set up K3s, this will copy and merge the kubeconfig locally and set the context to pi.
9. Once the cluster has been started, ArgoCD will been installed and this with then install the content of the cluster.
10. Run make argo to get the ArgoCD initial admin password, then access the ArgoCD UI at http://localhost:8090 with username admin and the retrieved password.
11. Run make secrets to update sealed secrets so the remaining applications can start. The secrets are sourced from a 1password environment.