[Release-1.29] - leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur

[brandond]

Backport fix for leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur

• leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur #10046

[VestigeJ]

##Environment Details
Reproduced here - #10046 (comment)

Validated using COMMIT=da2625d1a9993902a2fe443d39b72df073419199

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

kgn
NAME               STATUS   ROLES                       AGE     VERSION
ip-1-1-1-72        Ready    control-plane,etcd,master   4m50s   v1.29.5+k3s-da2625d1
ip-1-1-1-32        Ready    <none>                      5m21s   v1.29.5+k3s-da2625d1
ip-1-1-1-22        Ready    control-plane,etcd,master   5m54s   v1.29.5+k3s-da2625d1
ip-1-1-1-52        Ready    control-plane,etcd,master   5m5s    v1.29.5+k3s-da2625d1

Config.yaml:

node-external-ip: 1.1.1.22
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
cluster-init: true
embedded-registry: true

Steps

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ COMMIT=da2625d1a9993902a2fe443d39b72df073419199
$ sudo INSTALL_K3S_COMMIT=$COMMIT INSTALL_K3S_EXEC=server ./install-k3s.sh
$ kg cm -n kube-system
$ kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=test=ing
$ kg cm -n kube-system
$ kg leases k3s -n kube-system
$ kg leases k3s-etcd -n kube-system
$ k edit lease k3s-etcd -n kube-system
$ kg leases k3s-etcd -n kube-system
$ kg cm -n kube-system

Results:

$ kg cm -n kube-system

NAME                                                   DATA   AGE
chart-content-traefik                                  0      2m49s
chart-content-traefik-crd                              0      2m49s
cluster-dns                                            2      2m54s
coredns                                                2      2m52s
extension-apiserver-authentication                     6      2m57s
k3s-etcd-snapshot-extra-metadata                       1      42s
k3s-etcd-snapshots                                     1      8s
kube-apiserver-legacy-service-account-token-tracking   1      2m57s
kube-root-ca.crt                                       1      2m41s
local-path-config                                      4      2m52s

$ k edit lease k3s-etcd -n kube-system

lease.coordination.k8s.io/k3s-etcd edited

$ kg lease k3s-etcd -n kube-system

NAME       HOLDER             AGE
k3s-etcd   ip-1-1-1-22   4m12s

$ kg lease k3s -n kube-system

NAME   HOLDER             AGE
k3s    ip-1-1-1-72   4m17s

$ kg cm -n kube-system

NAME                                                   DATA   AGE
chart-content-traefik                                  0      4m31s
chart-content-traefik-crd                              0      4m31s
cluster-dns                                            2      4m36s
coredns                                                2      4m34s
extension-apiserver-authentication                     6      4m39s
k3s-etcd-snapshot-extra-metadata                       1      2m24s
k3s-etcd-snapshots                                     9      110s
kube-apiserver-legacy-service-account-token-tracking   1      4m39s
kube-root-ca.crt                                       1      4m23s
local-path-config                                      4      4m34s