Looking for help: Set up K3s agent nodes on Raspberry Pi

[springCozyRock]

I was deploying K3s agents on my Raspberry Pi, but it is blocked here for a long time. Is it normal? I really need you guys help.

pi@knode3:~ $ curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_SKIP_SELINUX_RPM=true K3S_URL=https://192.168.70.5:6443 K3S_TOKEN=K1072dcd39a8cb7364b9659a651b097289f2c87ccbeece16af285db24a1d84b006e::server:fc39046c4461ac73725e6cc5badd608c sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.29.3+k3s1 as release
[INFO]  Downloading hash rancher-mirror.rancher.cn/k3s/v1.29.3-k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary rancher-mirror.rancher.cn/k3s/v1.29.3-k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, command exists in PATH at /usr/bin/crictl
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

[brandond]

Check the service logs via journalctl to see what it is doing: journalctl -u k3s-agent

[springCozyRock]

Thanks for your reply! Firstly, I found my multipass instance was not in the subnet of my Raspberry Pi. Then I fixed it. But something weird happened again. The error messages changed as follows. My raspberry pi is in 192.168.31.73/24

gent-containerd-multiple\x2dlowerdir\x2dcheck1669435036-merged.mount has successfully entered the 'dead' state.
Apr 24 13:45:54 knode3 k3s[2901]: time="2024-04-24T13:45:54+08:00" level=info msg="Module overlay was already loaded"
Apr 24 13:45:54 knode3 k3s[2901]: time="2024-04-24T13:45:54+08:00" level=info msg="Module nf_conntrack was already loaded"
Apr 24 13:45:54 knode3 k3s[2901]: time="2024-04-24T13:45:54+08:00" level=info msg="Module br_netfilter was already loaded"
Apr 24 13:45:54 knode3 k3s[2901]: time="2024-04-24T13:45:54+08:00" level=info msg="Module iptable_nat was already loaded"
Apr 24 13:45:54 knode3 k3s[2901]: time="2024-04-24T13:45:54+08:00" level=info msg="Module iptable_filter was already loaded"
Apr 24 13:45:54 knode3 k3s[2901]: time="2024-04-24T13:45:54+08:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Apr 24 13:45:54 knode3 k3s[2901]: time="2024-04-24T13:45:54+08:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/r>
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="containerd is now running"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Getting list of apiserver endpoints from server"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Updated load balancer k3s-agent-load-balancer default server address -> 192.168.70.8:6443"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 192.168.70.8:6443"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Removing server from load balancer k3s-agent-load-balancer: 192.168.31.13:6443"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Updated load balancer k3s-agent-load-balancer server addresses -> [192.168.70.8:6443] [default: 192.168.70.8:6443]"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Connecting to proxy" url="wss://192.168.70.8:6443/v1-k3s/connect"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhoo>
Apr 24 13:46:05 knode3 k3s[2901]: W0424 13:46:05.853319    2901 reflector.go:539] k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSe>
Apr 24 13:46:05 knode3 k3s[2901]: I0424 13:46:05.853800    2901 trace.go:236] Trace[853769364]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229 (24-Apr-2024 13:45:55.850) (total time: 10002ms):
Apr 24 13:46:05 knode3 k3s[2901]: Trace[853769364]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeou>
Apr 24 13:46:05 knode3 k3s[2901]: Trace[853769364]: [10.002684353s] [10.002684353s] END
Apr 24 13:46:05 knode3 k3s[2901]: E0424 13:46:05.853909    2901 reflector.go:147] k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/names>
Apr 24 13:46:15 knode3 k3s[2901]: time="2024-04-24T13:46:15+08:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Cl>
Apr 24 13:46:16 knode3 k3s[2901]: W0424 13:46:16.876034    2901 reflector.go:539] k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSe>
Apr 24 13:46:16 knode3 k3s[2901]: I0424 13:46:16.876314    2901 trace.go:236] Trace[1842518669]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229 (24-Apr-2024 13:46:06.872) (total time: 10003ms):
Apr 24 13:46:16 knode3 k3s[2901]: Trace[1842518669]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeo>
Apr 24 13:46:16 knode3 k3s[2901]: Trace[1842518669]: [10.00334577s] [10.00334577s] END
Apr 24 13:46:16 knode3 k3s[2901]: E0424 13:46:16.876417    2901 reflector.go:147] k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/names>
Apr 24 13:46:28 knode3 k3s[2901]: W0424 13:46:28.590415    2901 reflector.go:539] k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSe>
Apr 24 13:46:28 knode3 k3s[2901]: I0424 13:46:28.590713    2901 trace.go:236] Trace[885925905]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229 (24-Apr-2024 13:46:18.587) (total time: 10003ms):
Apr 24 13:46:28 knode3 k3s[2901]: Trace[885925905]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeou>
Apr 24 13:46:28 knode3 k3s[2901]: Trace[885925905]: [10.003504009s] [10.003504009s] END
Apr 24 13:46:28 knode3 k3s[2901]: E0424 13:46:28.590860    2901 reflector.go:147] k8s.io/client-go@v1.29.3-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/names>
~

I am quite confused now. And I really really need your help. @brandond
My k3s agent (Raspberry Pi) download scripts now is curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_SKIP_SELINUX_RPM=true K3S_URL=https://192.168.31.13:6443 K3S_TOKEN=K10250dd4680c00d297cfac5354873d503caf99de96eaa59751d97a68420cfa8629::server:9baac9e9d0bf62faf0609494cb60f5ad sh -

[brandond]

Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Updated load balancer k3s-agent-load-balancer default server address -> 192.168.70.8:6443"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 192.168.70.8:6443"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Removing server from load balancer k3s-agent-load-balancer: 192.168.31.13:6443"
Apr 24 13:45:55 knode3 k3s[2901]: time="2024-04-24T13:45:55+08:00" level=info msg="Updated load balancer k3s-agent-load-balancer server addresses -> [192.168.70.8:6443] [default: 192.168.70.8:6443]"

Your server node has multiple interfaces, and has chosen to use 192.168.70.8, but it appears that you wanted it to use 192.168.31.13. Start the server with --node-ip=192.168.31.13.

[brandond]

These also look like DHCP addresses - make sure that you have static IPs or at least DHCP reservations so that the IPs dont change randomly when you reboot your nodes.

[springCozyRock]

Yes. Thanks for your reply. It is because my instance is in two subnet. So sudo ifconfig ens3 down and sudo systemctl restart k3s. And everything works well then. Thanks again!