- Kubernetes - k3s v1.27.9+k3s1
- Rancher - V2.8
Proxmox Hypervisor -> Linux LXC Containers -> K3S -> Nginx Ingress Controller -> Rancher
- Linux Host or WSL
- kubectl
- helm
- vim
This is installed on Bare Metal, just download the .iso here and make a bootable usb stick with RUFUS. Download rufus here
These can be set up in proxmox. Choose your linux flavor.
- Uncheck Privileged container
- no not start on create
- ssh into proxmox
- add the following to
/etc/pve/lxc/<conatinerId>.conf
lxc.apparmor.profile: unconfined lxc.cgroup.devices.allow: a lxc.cap.drop: lxc.mount.auto: "proc:rw sys:rw"
- start the containers
- run
pct push <container id> /boot/config-$(uname -r) /boot/config-$(uname -r)
for each containerId
This will need to be done in each container
you can get a shell in the container with lxc-attach <conatinerId>
- create this file
nano /usr/local/bin/conf-kmsg.sh
- Add the following
#!/bin/sh -e if [ ! -e /dev/kmsg ]; then ln -s /dev/console /dev/kmsg fi mount --make-rshared /
- Create this file
nano /etc/systemd/system/conf-kmsg.service
- add the following
[Unit] Description=Make sure /dev/kmsg exists [Service] Type=simple RemainAfterExit=yes ExecStart=/usr/local/bin/conf-kmsg.sh TimeoutStartSec=0 [Install] WantedBy=default.target
- run the following commands to start the service
chmod +x /usr/local/bin/conf-kmsg.sh systemctl daemon-reload systemctl enable --now conf-kmsg
Install the following packages:
- curl
- nfs-common
Followed most of the guide from here
Install k3s
curl -fsL https://get.k3s.io | INSTALL_K3S_VERSION=v1.27.9+k3s1 sh -s - --disable traefik --node-name control.k8s
Get kube config and save it locally under .kube/config (for linux) make sure to change the server to the ip address you gave your control.k8s
cat /etc/rancher/k3s/k3s.yaml
Get the Control Token (you will need this in the next step to set up the worker nodes)
cat /var/lib/rancher/k3s/server/node-token
save the config as a template
- shut down the machine.
- right click on it and click clone
- right click and click template
- save the token and kube config in the notes
Before you run the following command, shutdown the machine, clone, create a template and add the command below to the notes with the right token so that you can spin up more worker nodes easily. Install k3s
curl -fsL https://get.k3s.io | INSTALL_K3S_VERSION=v1.27.9+k3s1 K3S_URL=https://<Ip_Of_Control_Node>:6443 K3S_TOKEN=<Token_From_Control_Node> sh -s - --node-name worker-1.k8s
Do this for any additional nodes you need, just make sure to change the name of the nodes at the end of the command.
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
kubectl create namespace ingress
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress --set controller.publishService.enabled=true
In order to sign you website there is a few setup things you need to do with Cloudflare and your K8s cluster.
- Have a domain Registered and using DNS on Cloudflare.
- Set an A record pointing to your home IP address, make sure to set is as proxied
- Go to SSL/TLS on the left pane,
- In SSL/TLS->Overview select "Full (strict)"
- In SSL/TLS->Origin Server you will need to generate an origin Certificate to load as a k8s secret. (default options are fine, but change if you want more security)
- Use PEM format and copy the Origin Certificate to a tls.pem file.
- Copy the Private key to tls.key file.
- Download the Cloudflare Root CA Cert to the same directory as the above 2 files.
wget https://developers.cloudflare.com/ssl/static/origin_ca_rsa_root.pem
- Concatenate the tls.pem and origin_ca_rsa_root.pem
cat tls.pem origin_ca_rsa_root.pem > combined.crt
- Create the secret in the appropriate namespace
kubectl create secret tls my-tls-secret --cert=combined.crt --key=tls.key -n <namespace>
You will need a secret in each namespace for the ingress to pick up.
Add rancher latest repo
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
Create Namespace
kubectl create namespace cattle-system
Install Rancher using Certs from file option.
helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.my.org --set bootstrapPassword=admin --set ingress.tls.source=secret
Add Cloudflare Cert for your subdomain rancher.<your_domain>.com
kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=combined.crt --key=tls.key
You will also need to add an ingress class to rancher, or it will not work properly. Thanks to wrkode on GitHub for this fix
kubectl edit ing -n cattle-system rancher
add the following in annotations
metadata:
annotations:
#exsiting annotations
kubernetes.io/ingress.class: nginx
#additional annotations
This seems like the easier way, so I will present it first.
- Install the Turnkey Linux File Server
- Set the IP address
- Create a ZFS pool in Proxmox to hold all your files
- open a shell on the proxmox host and go to /etc/pve/lxc
- edit the config file for the nfs server and add the following:
mp0: /{{name of the zfs pool you made}},mp=/store
- now start the lxc and set up the nfs exports.