K3s-agent hangs indefinitely on install

[danjac]

Environmental Info:
K3s Version:

k3s version v1.31.5+k3s1 (56ec5dd)
go version go1.22.10

Node(s) CPU architecture, OS, and Version:

Ubuntu 24.0

Cluster Configuration:
1 server, 1 agent

Describe the bug:

When installing an agent on a new VM, the process hangs indefinitely at systemd: Starting k3s-agent

Steps To Reproduce:

root@k3s-agent-1:~# curl -sfL https://get.k3s.io | K3S_TOKEN=$TOKEN K3S_URL=https://10.0.0.7 sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.31.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.31.5+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.31.5+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

Expected behavior:

Agent is successfully up and running

Actual behavior:

Hangs indefinitely

Additional context / logs:

Server and worker node are running in the same private network. There are no connectivity issues. Server node is installed with no problems.

journalctl -u k3s-agent has a number of these errors:

Feb 17 22:05:14 k3s-agent-1 systemd[1]: Starting k3s-agent.service - Lightweight Kubernetes...
Feb 17 22:05:14 k3s-agent-1 sh[1914]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Feb 17 22:05:14 k3s-agent-1 k3s[1921]: time="2025-02-17T22:05:14Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Feb 17 22:05:14 k3s-agent-1 k3s[1921]: time="2025-02-17T22:05:14Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/da3ffc1d30a49a23>
Feb 17 22:05:15 k3s-agent-1 k3s[1921]: time="2025-02-17T22:05:15Z" level=info msg="Starting k3s agent v1.31.5+k3s1 (56ec5dd4)"
Feb 17 22:05:15 k3s-agent-1 k3s[1921]: time="2025-02-17T22:05:15Z" level=info msg="Updated load balancer k3s-agent-load-balancer default server:>
Feb 17 22:05:15 k3s-agent-1 k3s[1921]: time="2025-02-17T22:05:15Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 >
Feb 17 22:05:15 k3s-agent-1 k3s[1921]: time="2025-02-17T22:05:15Z" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacert": read tcp 127.0.0.1:59650->127.0.0.1:6444: read: connection reset by peer" # multiple errors like this

[danjac]

Further info: finding #6789 I checked to see if certs were available from the master node:

curl -vks https://10.0.0.7:6443/cacerts

This successfully fetched certs with no issues. Both nodes are in the same private network, no firewall issues.

[brandond]

You've redacted or at least truncated the address of the server node that the agent load-balancer is trying to use. Make sure that the server's node IP is correct and reachable from the agent. If it has multiple interfaces, you may need to use --node-ip to select the correct one that's reachable from the agent.

[danjac]

From journalctl:

Updated load balancer k3s-agent-load-balancer default server: 10.0.0.8:443

This might explain this error:

Feb 17 22:05:15 k3s-agent-1 k3s[1921]: time="2025-02-17T22:05:15Z" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacert": read tcp 127.0.0.1:59650->127.0.0.1:6444: read: connection reset by peer"

10.0.0.8 is the IP the agent itself is running on (i.e. localhost).

The server IP is 10.0.0.7 - as per the docs this is how I installed the agent:

curl -sfL https://get.k3s.io | K3S_TOKEN=$TOKEN K3S_URL=https://10.0.0.7 sh -

[danjac]

Sorry - stupidity on my side - a misconfigured Ansible inventory file :-(.

With the correct master IP address, it worked fine.

[brandond]

shouldn't the --server argument be present if K3S_URL environment variable is provided?

No, we don't convert env vars to their equivalent flags. If you set it via an env var it will remain set via env var, in the .env file.

Updated load balancer k3s-agent-load-balancer default server: 10.0.0.8:443

That's not right. The agent load-balancer would not be connecting to itself, nor would it use that port.

What do you see in kubectl get node -o wide on the server? Are you SURE you have the IPs set properly, and have not overridden the advertise-address or anything odd like that?

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.