[etcd] Member list

[rmammadli]

Environmental Info:
RKE2 Version: rke2 version v1.24.4+rke2r1

Node(s) CPU architecture, OS, and Version: Ubuntu 20.04
5.4.0-132-generic #148-Ubuntu SMP Mon Oct 17 16:02:06 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: 3 server, 3 agents

Describe the bug:

Etcd member list is not updated after removing cp node ( with etcd role) from the cluster.
> for etcdpod in $(kubectl -n kube-system get pod -l component=etcd --no-headers -o custom-columns=NAME:.metadata.name); do kubectl -n kube-system exec $etcdpod -- sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl member list"; done

Output:

2dd3d49d0a1b31a6, started, workload-cluster-test-01-cp01-v20221128-ec93204a9p8k8-378a6338, https://10.10.74.144:2380, https://10.10.74.144:2379, false
8f391ee98aee99ba, started, workload-cluster-test-01-cp02-v20221128-47eeed63kfpfb-8e51049e, https://10.10.74.139:2380, https://10.10.74.139:2379, false
e8270da1d6838cd8, started, workload-cluster-test-01-cp03-v20221128-7e3edf71kvfln-b359de0d, https://10.10.74.140:2380, https://10.10.74.140:2379, false
2dd3d49d0a1b31a6, started, workload-cluster-test-01-cp01-v20221128-ec93204a9p8k8-378a6338, https://10.10.74.144:2380, https://10.10.74.144:2379, false
8f391ee98aee99ba, started, workload-cluster-test-01-cp02-v20221128-47eeed63kfpfb-8e51049e, https://10.10.74.139:2380, https://10.10.74.139:2379, false
e8270da1d6838cd8, started, workload-cluster-test-01-cp03-v20221128-7e3edf71kvfln-b359de0d, https://10.10.74.140:2380, https://10.10.74.140:2379, false

where 10.10.74.144 is the ip address of cp-node removed.

Steps To Reproduce:

Remove cp node with etcd role from the cluster.
Execute cmd above to get the etcd member list.

• Installed RKE2:

Expected behavior:

Etcd node is removed from the etcd member list.

Actual behavior:

Node is not removed from etcd member list.

Additional context / logs:

Although checking of endpoint status, shows only available etcd nodes.
> for etcdpod in $(kubectl -n kube-system get pod -l component=etcd --no-headers -o custom-columns=NAME:.metadata.name); do kubectl -n kube-system exec $etcdpod -- sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status"; done

Output:
https://127.0.0.1:2379, 8f391ee98aee99ba, 3.5.4, 41 MB, true, false, 45, 80567408, 80567408
https://127.0.0.1:2379, e8270da1d6838cd8, 3.5.4, 25 MB, false, false, 45, 80567411, 80567411

This issue creates another problem during adding new control plane node to the existing cluster.
Due to the fact that removed etcd node is still in the member list, new node is unable to join the etcd cluster using / checking / connecting to this ip address from the member list.

[rmammadli]

Can not reproduce this issue anymore...
Probably it happened due to some inconsistency in etcd cluster during our heavy testing ( delete / add cp node(s) ).
I will just reopen the issue, if we encounter the same problem again.