rke2-agent failed to start on MicroOS #9311

vnwnv · 2025-11-26T15:53:01Z

vnwnv
Nov 26, 2025

I'm running RKE2 cluster on MicroOS.

After MicroOS auto-updated in 2025-11-26, which updated RKE2 to v1.34.2+rke2r1. After reboot, the agent node can not start and give out some errors.

Nov 26 15:48:13 wk0-alpha-starbase.lan sh[30438]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=warning msg="Unknown flag --embedded-registry found in config.yaml, skipping\n"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=warning msg="Unknown flag --tls-san found in config.yaml, skipping\n"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=warning msg="Unknown flag --disable found in config.yaml, skipping\n"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=warning msg="Unknown flag --cni found in config.yaml, skipping\n"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=warning msg="not running in CIS mode"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Applying Pod Security Admission Configuration"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Starting rke2 agent 1.34.2+rke2r1 (v1.34.2+rke2r1)"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Updated load balancer rke2-agent-load-balancer default server: 172.16.21.10:9345"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Adding server to load balancer rke2-agent-load-balancer: 172.16.21.100:9345"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Adding server to load balancer rke2-agent-load-balancer: 172.16.21.182:9345"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Adding server to load balancer rke2-agent-load-balancer: 172.16.21.165:9345"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Updated load balancer rke2-agent-load-balancer server addresses -> [172.16.21.100:9345 172.16.21.182:9345 172.16.21.165:9345] [default: 172.16.21.10:9345]"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Running load balancer rke2-agent-load-balancer 127.0.0.1:6444 -> [172.16.21.100:9345 172.16.21.182:9345 172.16.21.165:9345] [default: 172.16.21.10:9345]"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Server 172.16.21.100:9345@UNCHECKED->RECOVERING from successful dial"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Updated load balancer rke2-api-server-agent-load-balancer default server: 172.16.21.10:6443"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Adding server to load balancer rke2-api-server-agent-load-balancer: 172.16.21.165:6443"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Adding server to load balancer rke2-api-server-agent-load-balancer: 172.16.21.100:6443"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Adding server to load balancer rke2-api-server-agent-load-balancer: 172.16.21.182:6443"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Updated load balancer rke2-api-server-agent-load-balancer server addresses -> [172.16.21.165:6443 172.16.21.100:6443 172.16.21.182:6443] [default: 172.16.21.10:6443]"
Nov 26 15:48:13 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:13Z" level=info msg="Running load balancer rke2-api-server-agent-load-balancer 127.0.0.1:6443 -> [172.16.21.165:6443 172.16.21.100:6443 172.16.21.182:6443] [default: 172.16.21.10:6443]"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Module overlay was already loaded"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Module nf_conntrack was already loaded"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Module br_netfilter was already loaded"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Module iptable_nat was already loaded"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Module iptable_filter was already loaded"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=warning msg="Failed to load kernel module nft-expr-counter with modprobe"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Removed kube-proxy static pod manifest"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=error msg="Not starting distributed registry mirror: no registries configured for distributed mirroring"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Creating rke2-cert-monitor event broadcaster"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Getting list of apiserver endpoints from server"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Got apiserver addresses from supervisor: [172.16.21.100:6443 172.16.21.165:6443 172.16.21.182:6443]"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Connecting to proxy" url="wss://172.16.21.182:9345/v1-rke2/connect"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Connecting to proxy" url="wss://172.16.21.100:9345/v1-rke2/connect"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Connecting to proxy" url="wss://172.16.21.165:9345/v1-rke2/connect"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Connected to proxy" url="wss://172.16.21.182:9345/v1-rke2/connect"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Remotedialer connected to proxy" url="wss://172.16.21.182:9345/v1-rke2/connect"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Connected to proxy" url="wss://172.16.21.165:9345/v1-rke2/connect"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Connected to proxy" url="wss://172.16.21.100:9345/v1-rke2/connect"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Remotedialer connected to proxy" url="wss://172.16.21.165:9345/v1-rke2/connect"
Nov 26 15:48:14 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:14Z" level=info msg="Remotedialer connected to proxy" url="wss://172.16.21.100:9345/v1-rke2/connect"
Nov 26 15:48:15 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:15Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=wk0-alpha-starbase.lan --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Nov 26 15:48:15 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:15Z" level=info msg="Server 172.16.21.100:9345@RECOVERING->ACTIVE from successful health check"
Nov 26 15:48:15 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:15Z" level=info msg="Server 172.16.21.182:9345@UNCHECKED->RECOVERING from successful health check"
Nov 26 15:48:15 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:15Z" level=info msg="Server 172.16.21.165:9345@UNCHECKED->RECOVERING from successful health check"
Nov 26 15:48:15 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:15Z" level=info msg="Server 172.16.21.165:6443@UNCHECKED->RECOVERING from successful health check"
Nov 26 15:48:15 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:15Z" level=info msg="Server 172.16.21.100:6443@UNCHECKED->RECOVERING from successful health check"
Nov 26 15:48:15 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:15Z" level=info msg="Server 172.16.21.182:6443@UNCHECKED->RECOVERING from successful health check"
Nov 26 15:48:16 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:16Z" level=info msg="Server 172.16.21.182:9345@RECOVERING->PREFERRED from successful health check"
Nov 26 15:48:16 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:16Z" level=info msg="Server 172.16.21.165:9345@RECOVERING->PREFERRED from successful health check"
Nov 26 15:48:16 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:16Z" level=info msg="Server 172.16.21.165:6443@RECOVERING->PREFERRED from successful health check"
Nov 26 15:48:16 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:16Z" level=info msg="Server 172.16.21.100:6443@RECOVERING->PREFERRED from successful health check"
Nov 26 15:48:16 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:16Z" level=info msg="Server 172.16.21.182:6443@RECOVERING->PREFERRED from successful health check"
Nov 26 15:48:34 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:34Z" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
Nov 26 15:48:54 wk0-alpha-starbase.lan rke2[30444]: time="2025-11-26T15:48:54Z" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""

When I rollback to v1.34.1+rke2r1, the agent node run normally. I think the main reason is /run/k3s/containerd/containerd.sock But I have no idea to find out what happened.

Answered by brandond

Dec 3, 2025

I installed RKE2 from openSUSE's repository

That is not supported by this project. Please open an issue with OpenSUSE; they build and package things without any involvement from this project.

there are no 1.34.2 folders

Then something is very wrong with your installation. Enabling debug may give more of a clue, but it shouldn't be getting as far as it is without creating a v1.34.2 folder and extracting things to it. This may be something that opensuse has broken in their packaging?

View full answer

brandond · 2025-11-26T21:18:28Z

brandond
Nov 26, 2025
Maintainer

There are a lot of odd things going on here.

You have a bunch of server-only options in the config file on this node. Are you trying to run this node as a server, or agent?

You've also enabled the embedded registry but have no mirrors listed in registries.yaml.

It looks like containerd isn't being started for some reason. Are you using a custom container-runtime-endpoint, or are you trying to use the containerd that comes with rke2? What does your complete config.yaml look like on this node?

5 replies

vnwnv Nov 27, 2025
Author

I'm using this node as an agent, above logs came from rke2-agent.service.

The registry config was misconfiged

I'm NOT using custom container runtime endpoint. I'm using the default containerd, which came with rke2.

I disabled the embed registry on this node and my /etc/rancher/rke2/config.yaml is

selinux: true
embedded-registry: false
tls-san:
  - "172.16.21.10"
server: "https://172.16.21.10:9345"
token: "my-token"
disable:
  - "rke2-ingress-nginx"
cni: canal

But I'm still facing the same problem.

and this is /var/lib/rancher/rke2/agent/etc/containerd/config.toml and I do not have any config.toml.tmpl in the same directory

# File generated by rke2. DO NOT EDIT. Use config.toml.tmpl instead.
version = 3
root = "/var/lib/rancher/rke2/agent/containerd"
state = "/run/k3s/containerd"

[grpc]
  address = "/run/k3s/containerd/containerd.sock"

[plugins.'io.containerd.internal.v1.opt']
  path = "/var/lib/rancher/rke2/agent/containerd"

[plugins.'io.containerd.grpc.v1.cri']
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"

[plugins.'io.containerd.cri.v1.runtime']
  enable_selinux = true
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  device_ownership_from_security_context = false

[plugins.'io.containerd.cri.v1.images']
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
  use_local_image_pull = true

[plugins.'io.containerd.cri.v1.images'.pinned_images]
  sandbox = "index.docker.io/rancher/mirrored-pause:3.6"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
  runtime_type = "io.containerd.runhcs.v1"

[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = "/var/lib/rancher/rke2/agent/etc/containerd/certs.d"

brandond Dec 1, 2025
Maintainer

It looks like it's not even trying to start containerd for some reason. What do you get from find /var/lib/rancher/rke2/data/v1.34.2-*?

You might try removing that directory, and restarting the service. If it fails again, please grab complete logs, with debug: true enabled if possible.

vnwnv Dec 3, 2025
Author

I installed RKE2 from openSUSE's repository, and there are no 1.34.2 folders.

ls -l /var/lib/rancher/rke2/data
total 0
drwxr-xr-x. 1 root root 18 Apr 28  2025 v1.32.3-rke2r1-543f84e7e830
drwxr-xr-x. 1 root root 18 May  7  2025 v1.32.4-rke2r1-2a30dd9c8583
drwxr-xr-x. 1 root root 18 May 14  2025 v1.33.0-rke2r1-c069fc368e5c
drwxr-xr-x. 1 root root 18 Jun  2  2025 v1.33.1-rke2r1-c1c56f262526
drwxr-xr-x. 1 root root 18 Jul  4 00:31 v1.33.2-rke2r1-d50c19db40de
drwxr-xr-x. 1 root root 18 Aug  3 01:21 v1.33.3-rke2r1-73cc3544ad1b
drwxr-xr-x. 1 root root 18 Aug 30 00:52 v1.33.4-rke2r1-daf85d39fe51
drwxr-xr-x. 1 root root 18 Sep 22 00:56 v1.34.1-rke2r1-8522b4227a96

here are the logs

Dec 03 03:37:01 wk0-alpha-starbase.lan systemd[1]: rke2-agent.service: Scheduled restart job, restart counter is at 42.
Dec 03 03:37:01 wk0-alpha-starbase.lan systemd[1]: Starting Rancher Kubernetes Engine v2 (agent)...
Dec 03 03:37:01 wk0-alpha-starbase.lan sh[206772]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=warning msg="Unknown flag --embedded-registry found in config.yaml, skipping\n"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=warning msg="Unknown flag --tls-san found in config.yaml, skipping\n"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=warning msg="Unknown flag --disable found in config.yaml, skipping\n"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=warning msg="Unknown flag --cni found in config.yaml, skipping\n"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=warning msg="not running in CIS mode"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Applying Pod Security Admission Configuration"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Starting rke2 agent 1.34.2+rke2r1 (v1.34.2+rke2r1)"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Updated load balancer rke2-agent-load-balancer default server: 172.16.21.10:9345"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Adding server to load balancer rke2-agent-load-balancer: 172.16.21.182:9345"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Adding server to load balancer rke2-agent-load-balancer: 172.16.21.165:9345"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Adding server to load balancer rke2-agent-load-balancer: 172.16.21.100:9345"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Updated load balancer rke2-agent-load-balancer server addresses -> [172.16.21.182:9345 172.16.21.165:9345 172.16.21.100:9345] [default: 172.16.21.10:9345]"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Running load balancer rke2-agent-load-balancer 127.0.0.1:6444 -> [172.16.21.182:9345 172.16.21.165:9345 172.16.21.100:9345] [default: 172.16.21.10:9345]"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Supervisor proxy using supervisor=https://127.0.0.1:6444 apiserver=https://127.0.0.1:6444 lb=true"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Server 172.16.21.182:9345@UNCHECKED->RECOVERING from successful dial"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Updated load balancer rke2-api-server-agent-load-balancer default server: 172.16.21.10:6443"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Adding server to load balancer rke2-api-server-agent-load-balancer: 172.16.21.182:6443"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Adding server to load balancer rke2-api-server-agent-load-balancer: 172.16.21.100:6443"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Adding server to load balancer rke2-api-server-agent-load-balancer: 172.16.21.165:6443"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Updated load balancer rke2-api-server-agent-load-balancer server addresses -> [172.16.21.182:6443 172.16.21.100:6443 172.16.21.165:6443] [default: 172.16.21.10:6443]"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Running load balancer rke2-api-server-agent-load-balancer 127.0.0.1:6443 -> [172.16.21.182:6443 172.16.21.100:6443 172.16.21.165:6443] [default: 172.16.21.10:6443]"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Supervisor proxy apiserver port changed; apiserver=https://127.0.0.1:6443 lb=true"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Module overlay was already loaded"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Module nf_conntrack was already loaded"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Module br_netfilter was already loaded"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Module iptable_nat was already loaded"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Module iptable_filter was already loaded"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=warning msg="Failed to load kernel module nft-expr-counter with modprobe"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="getConntrackMax: using conntrack-min"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Removed kube-proxy static pod manifest"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=error msg="Not starting distributed registry mirror: no registries configured for distributed mirroring"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: W1203 03:37:02.934241  206778 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "/run/k3s/containerd/containerd.sock", ServerName: "%2Frun%2Fk3s%2Fcontainerd%2Fcontainerd.sock", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for crun container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for crun in $PATH: exec: \"crun\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for nvidia container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for nvidia in $PATH: exec: \"nvidia-container-runtime\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for nvidia-experimental container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for nvidia-experimental in $PATH: exec: \"nvidia-container-runtime-experimental\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for nvidia-cdi container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for nvidia-cdi in $PATH: exec: \"nvidia-container-runtime.cdi\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for wasmtime container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for wasmtime in $PATH: exec: \"containerd-shim-wasmtime-v1\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for lunatic container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for lunatic in $PATH: exec: \"containerd-shim-lunatic-v1\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for slight container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for slight in $PATH: exec: \"containerd-shim-slight-v1\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for spin container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for spin in $PATH: exec: \"containerd-shim-spin-v2\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for wws container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for wws in $PATH: exec: \"containerd-shim-wws-v1\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for wasmedge container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for wasmedge in $PATH: exec: \"containerd-shim-wasmedge-v1\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Searching for wasmer container runtime"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Error searching for wasmer in $PATH: exec: \"containerd-shim-wasmer-v1\": executable file not found in $PATH"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=debug msg="Starting rke2-cert-monitor with monitoring period 30m0s"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Creating rke2-cert-monitor event broadcaster"
Dec 03 03:37:02 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:02Z" level=info msg="Getting list of apiserver endpoints from server"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Got apiserver addresses from supervisor: [172.16.21.100:6443 172.16.21.165:6443 172.16.21.182:6443]"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=debug msg="Set health check for load balancer rke2-agent-load-balancer: 172.16.21.100:9345"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=debug msg="Set health check for load balancer rke2-api-server-agent-load-balancer: 172.16.21.100:6443"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=debug msg="Set health check for load balancer rke2-agent-load-balancer: 172.16.21.165:9345"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=debug msg="Set health check for load balancer rke2-api-server-agent-load-balancer: 172.16.21.165:6443"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=debug msg="Set health check for load balancer rke2-agent-load-balancer: 172.16.21.182:9345"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=debug msg="Set health check for load balancer rke2-api-server-agent-load-balancer: 172.16.21.182:6443"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Connecting to proxy" url="wss://172.16.21.100:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Connecting to proxy" url="wss://172.16.21.182:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Connecting to proxy" url="wss://172.16.21.165:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Connected to proxy" url="wss://172.16.21.182:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Remotedialer connected to proxy" url="wss://172.16.21.182:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Connected to proxy" url="wss://172.16.21.165:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Remotedialer connected to proxy" url="wss://172.16.21.165:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Connected to proxy" url="wss://172.16.21.100:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Remotedialer connected to proxy" url="wss://172.16.21.100:9345/v1-rke2/connect"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Server 172.16.21.182:9345@RECOVERING->ACTIVE from successful health check"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Server 172.16.21.165:9345@UNCHECKED->RECOVERING from successful health check"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Server 172.16.21.100:9345@UNCHECKED->RECOVERING from successful health check"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=wk0-alpha-starbase.lan --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Server 172.16.21.182:6443@UNCHECKED->RECOVERING from successful health check"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Server 172.16.21.100:6443@UNCHECKED->RECOVERING from successful health check"
Dec 03 03:37:03 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:03Z" level=info msg="Server 172.16.21.165:6443@UNCHECKED->RECOVERING from successful health check"
Dec 03 03:37:04 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:04Z" level=info msg="Server 172.16.21.165:9345@RECOVERING->PREFERRED from successful health check"
Dec 03 03:37:04 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:04Z" level=info msg="Server 172.16.21.100:9345@RECOVERING->PREFERRED from successful health check"
Dec 03 03:37:04 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:04Z" level=info msg="Server 172.16.21.182:6443@RECOVERING->PREFERRED from successful health check"
Dec 03 03:37:04 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:04Z" level=info msg="Server 172.16.21.100:6443@RECOVERING->PREFERRED from successful health check"
Dec 03 03:37:04 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:04Z" level=info msg="Server 172.16.21.165:6443@RECOVERING->PREFERRED from successful health check"
Dec 03 03:37:22 wk0-alpha-starbase.lan rke2[206778]: W1203 03:37:22.934783  206778 logging.go:55] [core] [Channel #3 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "/run/k3s/containerd/containerd.sock", ServerName: "%2Frun%2Fk3s%2Fcontainerd%2Fcontainerd.sock", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
Dec 03 03:37:22 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:22Z" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
Dec 03 03:37:42 wk0-alpha-starbase.lan rke2[206778]: W1203 03:37:42.935001  206778 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "/run/k3s/containerd/containerd.sock", ServerName: "%2Frun%2Fk3s%2Fcontainerd%2Fcontainerd.sock", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
Dec 03 03:37:42 wk0-alpha-starbase.lan rke2[206778]: time="2025-12-03T03:37:42Z" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
lines 907-1000/1000 (END)

brandond Dec 3, 2025
Maintainer

I installed RKE2 from openSUSE's repository

That is not supported by this project. Please open an issue with OpenSUSE; they build and package things without any involvement from this project.

there are no 1.34.2 folders

Then something is very wrong with your installation. Enabling debug may give more of a clue, but it shouldn't be getting as far as it is without creating a v1.34.2 folder and extracting things to it. This may be something that opensuse has broken in their packaging?

Answer selected by dereknola

vnwnv Dec 3, 2025
Author

Thank you for your reply.

I'll Open a new issue with openSUSE.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rke2-agent failed to start on MicroOS #9311

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

rke2-agent failed to start on MicroOS #9311

Uh oh!

vnwnv Nov 26, 2025

Replies: 1 comment · 5 replies

Uh oh!

brandond Nov 26, 2025 Maintainer

Uh oh!

Uh oh!

vnwnv Nov 27, 2025 Author

Uh oh!

Uh oh!

brandond Dec 1, 2025 Maintainer

Uh oh!

vnwnv Dec 3, 2025 Author

Uh oh!

Uh oh!

brandond Dec 3, 2025 Maintainer

Uh oh!

vnwnv Dec 3, 2025 Author

vnwnv
Nov 26, 2025

Replies: 1 comment 5 replies

brandond
Nov 26, 2025
Maintainer

vnwnv Nov 27, 2025
Author

brandond Dec 1, 2025
Maintainer

vnwnv Dec 3, 2025
Author

brandond Dec 3, 2025
Maintainer

vnwnv Dec 3, 2025
Author