Retry cluster join on "too many learners" error

[brandond]

Proposed Changes

Retry cluster join on "too many learners" error

Types of Changes

bugfix

Verification

See linked issue

Testing

Linked Issues

• k3s server exits instead of retrying when receiving "too many learners" error from etcd join #7000

User-Facing Change

K3s now retries the cluster join operation when receiving a "too many learners" error from etcd. This most frequently occurred when attempting to add multiple servers at the same time.

Further Comments

With this change, when adding multiple servers at the same time, you should now see this repeat:

INFO[0011] Adding member k3s-server-3-88058c08=https://172.17.0.6:2380 to etcd cluster [k3s-server-1-c18966c7=https://172.17.0.4:2380 k3s-server-2-8b00056d=https://172.17.0.5:2380]
{"level":"warn","ts":"2023-04-25T23:12:35.335Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b03340/172.17.0.4:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: too many learner members in cluster"}
INFO[0011] Waiting for other members to finish joining etcd cluster

instead of K3s exiting with a fatal error:

INFO[0001] Adding member k3s-server-3-300fe049=https://172.17.0.6:2380 to etcd cluster [k3s-server-2-47159b14=https://172.17.0.5:2380 k3s-server-1-75ea6829=https://172.17.0.4:2380]
{"level":"warn","ts":"2023-04-25T21:05:26.878Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00056f880/172.17.0.4:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: too many learner members in cluster"}
FATA[0001] ETCD join failed: etcdserver: too many learner members in cluster