- KVM
- OpenTofu (the
terraform
fork)
- See the
scripts/
folder for various utility scripts. - You probably want to change
user
andgroup
tolibvirt-qemu
andkvm
respectively in/etc/libvirt/qemu.conf
to mitigate permission issues on storage pools.
By default we'll deploy a cluster on three nodes, they will have both the control-plane and worker roles.
If you're setting a lower resource values on each node then you might need to set: --ignore-preflight-errors=mem,numcpu
during kubeadm init
.
- Change the
k8s.auto.tfvars
to fit your needs! - Run
tofu init
- Run
tofu plan
- Run
tofu apply
- SSH to all nodes using the private and public key pair you referenced when deploying the cluster. You can find the IP addresses of the cluster nodes by running:
sudo virsh net-dhcp-leases k8s_net
- Proceed with the bootstrapping the Kubernetes cluster using e.g.
kubeadm
.
Flannel CNI:
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
Calico CNI:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml
Cilium CNI: https://docs.cilium.io/en/stable/installation/k8s-install-kubeadm/#deploy-cilium
Add control-plane nodes to the cluster:
echo "$(kubeadm token create --print-join-command) --control-plane --certificate-key $(kubeadm init phase upload-certs --upload-certs --skip-headers --skip-log-headers 2>/dev/null | tail -n 1)"
Add worker nodes to the cluster:
- Generate the join command on a control-plane node:
kubeadm token create --print-join-command
- Use the generate join command and run that on the worker node.
On the first control-plane node:
- Upgrade
kubeadm
:
export NEXT_VERSION="1.26.2"
apt-mark unhold kubeadm
apt-get update
apt-get install -y kubeadm=${NEXT_VERSION}-00
apt-mark hold kubeadm
- Check the upgrade plan:
kubeadm upgrade plan
- Apply the upgrade plan:
kubeadm upgrade apply ${NEXT_VERSION}
- Drain the node:
kubectl drain <node-to-drain> --ignore-daemonsets
- Upgrade
kubectl
andkubelet
:
apt-mark unhold kubelet kubectl
apt-get update
apt-get install -y kubelet=${NEXT_VERSION}-00 kubectl=${NEXT_VERSION}-00
apt-mark hold kubelet kubectl
- Restart the services:
sudo systemctl daemon-reload
sudo systemctl restart kubelet
- Uncordon the node to allow scheduling again:
kubectl uncordon <node-to-uncordon>
- Repeat on the rest of the control-plane nodes!
Run the clean-up utility script: scripts/clean_up.sh