Cilium Fails After Power Loss – RKE2 Unable to Start Even After Reinstallation

[imoyakin]

I encountered an issue where Cilium fails to start after a power outage. When I reboot the RKE2 service, it refuses to start. Even after uninstalling and reinstalling RKE2, the service still does not start and throws errors related to network connectivity.And the only solution is to reinstall the host machine.
My rke2 instance is based on this article, but the difference is that I built it through a single node.
In a recent failure, I performed a snapshot recovery of rke2 as usual, but the log was as follows:

server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-kuber-master-1-174595320
4
WARN[0000] not running in CIS mode                      
INFO[0000] Applying Pod Security Admission Configuration 
INFO[0000] Static pod cleanup in progress               
INFO[0000] Logging temporary containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log 
INFO[0000] Running temporary containerd /var/lib/rancher/rke2/bin/containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd 
INFO[0010] Static pod cleanup completed successfully    
INFO[0010] Starting rke2 v1.31.7+rke2r1 (7b18bda1c5ec1e110cec206f9163f6aba3a2154d) 
INFO[0010] Managed etcd cluster bootstrap already complete and initialized 
INFO[0010] Pre-restore etcd database moved to /var/lib/rancher/rke2/server/db/etcd-old-1746258327 
{"level":"info","ts":"2025-05-03T15:45:27.153316+0800","logger":"rke2-etcd-client","caller":"snapshot/v3_snapshot.go:265","msg":"restoring snapshot","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-kuber-master-1-1745953204","wal-dir":"/var/lib/rancher/rke2/server/db/etcd/member/wal","data-dir":"/var/lib/rancher/rke2/server/db/etcd","snap-dir":"/var/lib/rancher/rke2/server/db/etcd/member/snap","initial-memory-map-size":0}
INFO[0010] certificate CN=rke2,O=rke2 signed by CN=rke2-server-ca@1743745733: notBefore=2025-04-04 05:48:53 +0000 UTC notAfter=2026-05-03 07:45:27 +0000 UTC 
WARN[0010] dynamiclistener [::]:9345: no cached certificate available for preload - deferring certificate load until storage initialization or first client request 
INFO[0010] Active TLS secret / (ver=) (count 15): map[listener.cattle.io/cn-10.1.61.69:10.1.61.69 listener.cattle.io/cn-10.110.0.1:10.110.0.1 listener.cattle.io/cn-100.127.6.114:100.127.6.114 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-127.0.100.0:127.0.100.0 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-explosion.zip:explosion.zip listener.cattle.io/cn-ice.local:ice.local listener.cattle.io/cn-ice.localdomain:ice.localdomain listener.cattle.io/cn-kuber-master-1:kuber-master-1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.ice.localdomain:kubernetes.default.svc.ice.localdomain listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=7527BE2119E4F9E7C1EC8E53FF7FCD3889E2D708] 
{"level":"info","ts":"2025-05-03T15:45:27.346926+0800","logger":"rke2-etcd-client","caller":"membership/store.go:141","msg":"Trimming membership information from the backend..."}
{"level":"debug","ts":"2025-05-03T15:45:27.346955+0800","logger":"rke2-etcd-client","caller":"membership/store.go:147","msg":"Removed member from the backend","member":"c92b227fe494bb25"}
{"level":"info","ts":"2025-05-03T15:45:27.362778+0800","logger":"rke2-etcd-client","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"83792c4c21f5c319","local-member-id":"0","added-peer-id":"8c9d90c0bac52fb9","added-peer-peer-urls":["https://10.1.61.69:2380"],"added-peer-is-learner":false}
{"level":"info","ts":"2025-05-03T15:45:27.404041+0800","logger":"rke2-etcd-client","caller":"snapshot/v3_snapshot.go:293","msg":"restored snapshot","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-kuber-master-1-1745953204","wal-dir":"/var/lib/rancher/rke2/server/db/etcd/member/wal","data-dir":"/var/lib/rancher/rke2/server/db/etcd","snap-dir":"/var/lib/rancher/rke2/server/db/etcd/member/snap","initial-memory-map-size":0}
INFO[0010] Starting etcd for new cluster, cluster-reset=true 
INFO[0010] Failed to test data store connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused" 
INFO[0010] Server node token is available at /var/lib/rancher/rke2/server/token 
INFO[0010] Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: connection refused" 
INFO[0010] To join server node to cluster: rke2 server -s https://10.1.61.69:9345 -t ${SERVER_NODE_TOKEN} 
INFO[0010] Agent node token is available at /var/lib/rancher/rke2/server/agent-token 
INFO[0010] To join agent node to cluster: rke2 agent -s https://10.1.61.69:9345 -t ${AGENT_NODE_TOKEN} 
INFO[0010] Wrote kubeconfig /etc/rancher/rke2/rke2.yaml 
INFO[0010] Run: rke2 kubectl                            
INFO[0010] Updated load balancer rke2-agent-load-balancer default server: 127.0.0.1:9345 
INFO[0010] Running load balancer rke2-agent-load-balancer 127.0.0.1:6444 -> [] [default: 127.0.0.1:9345] 
WARN[0010] Failed to get apiserver address from etcd: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused" 
INFO[0010] Running load balancer rke2-api-server-agent-load-balancer 127.0.0.1:6443 -> [] [default: ] 
INFO[0010] Password verified locally for node kuber-master-1 
INFO[0010] certificate CN=kuber-master-1 signed by CN=rke2-server-ca@1743745733: notBefore=2025-04-04 05:48:53 +0000 UTC notAfter=2026-05-03 07:45:28 +0000 UTC 
INFO[0011] certificate CN=system:node:kuber-master-1,O=system:nodes signed by CN=rke2-client-ca@1743745733: notBefore=2025-04-04 05:48:53 +0000 UTC notAfter=2026-05-03 07:45:28 +0000 UTC 
INFO[0011] certificate CN=system:kube-proxy signed by CN=rke2-client-ca@1743745733: notBefore=2025-04-04 05:48:53 +0000 UTC notAfter=2026-05-03 07:45:28 +0000 UTC 
INFO[0011] certificate CN=system:rke2-controller signed by CN=rke2-client-ca@1743745733: notBefore=2025-04-04 05:48:53 +0000 UTC notAfter=2026-05-03 07:45:28 +0000 UTC 
INFO[0011] Module overlay was already loaded            
INFO[0011] Module nf_conntrack was already loaded       
INFO[0011] Module br_netfilter was already loaded       
INFO[0011] Module iptable_nat was already loaded        
INFO[0011] Module iptable_filter was already loaded     
WARN[0011] Failed to load kernel module nft-expr-counter with modprobe 
INFO[0011] Runtime image index.docker.io/rancher/rke2-runtime:v1.31.7-rke2r1 bin and charts directories already exist; skipping extract 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values 
INFO[0011] No cluster configuration value changes necessary for manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-validation-webhook.yaml 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-traefik.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller-crd.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-traefik-crd.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-flannel.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-runtimeclasses.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values 
INFO[0011] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller.yaml to set cluster configuration values 
INFO[0011] Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log 
INFO[0011] Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml 
INFO[0012] containerd is now running                    
INFO[0012] Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt 
INFO[0012] Image index.docker.io/rancher/rke2-cloud-provider:v1.31.2-0.20241016053446-0955fa330f90-build20241016 has already been pulled 
INFO[0012] Imported docker.io/rancher/rke2-cloud-provider:v1.31.2-0.20241016053446-0955fa330f90-build20241016 
INFO[0012] Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 7.309125ms 
INFO[0012] Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt 
INFO[0012] Image index.docker.io/rancher/hardened-etcd:v3.5.19-k3s1-build20250306 has already been pulled 
INFO[0012] Imported docker.io/rancher/hardened-etcd:v3.5.19-k3s1-build20250306 
INFO[0012] Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 6.756554ms 
INFO[0012] Pulling images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt 
INFO[0012] Image index.docker.io/rancher/hardened-kubernetes:v1.31.7-rke2r1-build20250312 has already been pulled 
INFO[0012] Imported docker.io/rancher/hardened-kubernetes:v1.31.7-rke2r1-build20250312 
INFO[0012] Imported images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt in 6.775818ms 
INFO[0012] Pulling images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt 
INFO[0012] Image index.docker.io/rancher/hardened-kubernetes:v1.31.7-rke2r1-build20250312 has already been pulled 
INFO[0012] Imported docker.io/rancher/hardened-kubernetes:v1.31.7-rke2r1-build20250312 
INFO[0012] Imported images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt in 702.02µs 
INFO[0012] Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt 
INFO[0012] Image index.docker.io/rancher/hardened-kubernetes:v1.31.7-rke2r1-build20250312 has already been pulled 
INFO[0012] Imported docker.io/rancher/hardened-kubernetes:v1.31.7-rke2r1-build20250312 
INFO[0012] Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 418.741µs 
INFO[0012] Pulling images from /var/lib/rancher/rke2/agent/images/runtime-image.txt 
INFO[0012] Image index.docker.io/rancher/rke2-runtime:v1.31.7-rke2r1 has already been pulled 
INFO[0012] Imported docker.io/rancher/rke2-runtime:v1.31.7-rke2r1 
INFO[0012] Imported images from /var/lib/rancher/rke2/agent/images/runtime-image.txt in 6.624506ms 
INFO[0012] Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cluster-dns=10.110.0.10 --cluster-domain=ice.localdomain --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=kuber-master-1 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --kubelet-cgroups=/rke2 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --max-pods=1000 --node-ip=10.1.61.69 --node-labels= --node-status-update-frequency=4s --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key 
ERRO[0013] Kubelet exited: exit status 1                
ERRO[0013] Sending HTTP/1.1 503 response to 127.0.0.1:41140: runtime core not ready 
INFO[0015] Failed to test data store connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused" 
WARN[0015] Failed to get apiserver address from etcd: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused" 
WARN[0017] failed to list etcd members: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused" 
ERRO[0018] Kubelet exited: exit status 1                
INFO[0020] Failed to test data store connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
 ---- error message loop ----

I need some hints, maybe it has something to do with ebpf?

[joestringer]

It looks like etcd is not starting up correctly, and the connectivity is for a local host IP. Are you sure etcd is healthy in such a scenario?

I would suggest inspecting system logs for key components like etcd, cilium-agent, any k8s or rke components. If you can access the Cilium agent container then you could also inspect traffic with hubble observe.