[Release-1.25] - Cannot start control-plane-only server

[brandond]

Backport fix for Cannot start control-plane-only server

• Cannot start control-plane-only server #8637

[VestigeJ]

FWIW @brandond I did not hit this on v1.25.14+k3s1

$ kubectl get nodes

NAME               STATUS   ROLES                  AGE   VERSION
ip-1-1-1-109       Ready    etcd                   22s   v1.25.14+k3s1
ip-1-1-1-111       Ready    control-plane,master   25s   v1.25.14+k3s1

$ kgp -A

NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-8b9777675-dvz9c                   1/1     Running     0          88s
kube-system   helm-install-traefik-8kckf                0/1     Completed   0          88s
kube-system   helm-install-traefik-crd-zvcsg            0/1     Completed   0          88s
kube-system   local-path-provisioner-69dff9496c-cq68f   1/1     Running     0          88s
kube-system   metrics-server-854c559bd-blxrm            1/1     Running     0          88s
kube-system   svclb-traefik-b2e12014-lssc2              2/2     Running     0          73s
kube-system   svclb-traefik-b2e12014-qtqr7              2/2     Running     0          73s
kube-system   traefik-66fd46ccd-nhsb6                   1/1     Running     0          73s

[brandond]

@VestigeJ this would only occur on recent commit builds. It was broken by some reorganization in the etcd snapshot refactor, which has not yet shipped.

[VestigeJ]

Running latest commit instead of last release on v1.25 LGTM

##Environment Details
COMMIT=ec31704c1ab3d59f2adc40be806d75533fb83abe

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

$ k3s -v

k3s version v1.25.14+k3s-ec31704c (ec31704c)
go version go1.20.8

$ kg no,po -A

NAME                    STATUS   ROLES                       AGE   VERSION
node/ip-1-1-1-184       Ready    control-plane,master        13s   v1.25.14+k3s-ec31704c
node/ip-1-1-7-42        Ready    control-plane,etcd,master   70s   v1.25.14+k3s-ec31704c

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-8b9777675-rtj7v                   1/1     Running     0          55s
kube-system   pod/helm-install-traefik-crd-ggl74            0/1     Completed   0          55s
kube-system   pod/helm-install-traefik-vz88t                0/1     Completed   1          55s
kube-system   pod/local-path-provisioner-69dff9496c-vq8r2   1/1     Running     0          55s
kube-system   pod/metrics-server-854c559bd-8xl58            1/1     Running     0          55s
kube-system   pod/svclb-traefik-301195df-pv7vx              2/2     Running     0          38s
kube-system   pod/svclb-traefik-301195df-tt5cx              2/2     Running     0          12s
kube-system   pod/traefik-7c48d7b8b4-h5jc6                  1/1     Running     0          38s

Config.yaml:

node-external-ip: on initial node to join
cluster-init: true

Details

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_K3S_COMMIT=ec31704c1ab3d59f2adc40be806d75533fb83abe INSTALL_K3S_SKIP_ENABLE=true ./install-k3s.sh
$ sudo /usr/local/bin/k3s server --server https://1.1.7.42:6443 --cluster-init --disable-etcd --token YOUR_TOKEN_HERE
$ get_report //generate this template 

Results:

LGTM