Simultaneously started K3s servers may race to create CA certificates when using external SQL

[brandond]

Opening this to fix the issues described at:

• SURE-6082
• Nodes aren't joining a new cluster when using an external DB #3226 (comment)
• Error using kubeconfig when connecting to a multi-master K3s cluster #6525

We could probably handle this better on the K3s side by having the nodes do some sort of "check/lock/check" step by pre-creating the bootstrap key prior to creating all the CA certs on disk. This would ensure that they don't both create the CA certs, and then blow up when trying to write them into the datastore when the second node finds out that the first node created the bootstrap key while it was preparing to do so.

[ShylajaDevadiga]

##Environment Details
VERSION=v1.26.3+k3s1
COMMIT=

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and version:

Linux 5.4.0-1009-aws x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 20.04 LTS"

Cluster Configuration:
Multi node cluster

Config.yaml:

write-kubeconfig-mode: "0644"
tls-san:
  - <public ip>
token: <TOKEN>
datastore-endpoint: <mydb>

NAME               STATUS   ROLES                  AGE     VERSION
ip-<REDACTED>58     Ready    control-plane,master   4m41s   v1.26.3+k3s1
ip-<REDACTED>56    Ready    <none>                 6m39s   v1.26.3+k3s1
ip-<REDACTED>74    Ready    control-plane,master   6m40s   v1.26.3+k3s1
ip-<REDACTED>1     Ready    control-plane,master   6m38s   v1.26.3+k3s1
ip-<REDACTED>115   Ready    control-plane,master   6m32s   v1.26.3+k3s1
ip-<REDACTED>9    Ready    control-plane,master   5m53s   v1.26.3+k3s1 

Results from reproducing the issue

$ journalctl -u k3s|grep ECDSA  |grep -i Unable
Apr 11 23:30:01 ip-<REDACTED> k3s[15490]: time="2023-04-11T23:30:01Z" level=warning msg="unable to verify existing certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"k3s-server-ca@1681255795\") - signing operation may change certificate issuer"
ubuntu@ip-172-31-3-28:~$

[ShylajaDevadiga]

Validated on k3s v1.26.4-rc1+k3s1 with above config using mysql as external db on a multi node cluster

kubectl get nodes
NAME               STATUS   ROLES                  AGE    VERSION
ip-<REDACTED>-21    Ready    <none>                 9m6s   v1.26.4-rc1+k3s1
ip-<REDACTED>-214    Ready    control-plane,master   12m    v1.26.4-rc1+k3s1
ip-<REDACTED>-30    Ready    control-plane,master   11m    v1.26.4-rc1+k3s1
ip-<REDACTED>-94     Ready    control-plane,master   11m    v1.26.4-rc1+k3s1
ip-<REDACTED>-27     Ready    control-plane,master   11m    v1.26.4-rc1+k3s1
ip-<REDACTED>-55    Ready    control-plane,master   11m    v1.26.4-rc1+k3s1
ip-<REDACTED>-239    Ready    control-plane,master   11m    v1.26.4-rc1+k3s1

$ journalctl -u k3s|grep ECDSA  |grep -i Unable
$