Add ability to reconcile bootstrap data between datastore and disk

[briandowns]

Proposed Changes

Adds the ability for k3s to detect when there are differences in bootstrap data in either the datastore or on disk and to update the whichever one is older.

This also adds the ability to migrate bootstrap data from the "older" format to the "newer" format. This situation will happen when clusters are upgraded and the new version started. The new bootstrap data contains a timestamp field which the old format didn't have. That older data is read, determined to be old, migrated to be in the new format and saved back to the datastore.

Types of Changes

Verification

1. Start a k3s server
2. Remove /var/lib/rancher/k3s/server/tls
3. Restart k3s
4. Verify certificates are rewritten and kubectl can successfully run

or

1. Start a k3s server
2. Remove a file from the /var/lib/rancher/k3s/server/tls directory
3. Restart k3s
4. Verify that the file is replaced and the cluster is functional

See comment below for additional verification scenarios.

Linked Issues

#3015

Further Comments

[brandond]

Can we add additional tests?

Custom certs:

1. Pre-create custom CA certs in /var/lib/rancher/k3s/server/tls using openssl
2. Start K3s on server A connected to datastore 1
3. Start K3s on server B connected to datastore 1
4. Verify certificates are written on server B, and match those from server A

Changing datastore:

1. Start K3s server A connected to external datastore 1
2. Start K3s server B connected to external datastore 2
3. Stop K3s on server B, change datastore address to point at datastore 1, start K3s
4. Verify certificates are written on server B, match those from server A, and kubectl can successfully run

Certificate renewal:

1. Start K3s server A connected to external datastore 1
2. Start K3s server B connected to external datastore 1
3. Stop K3s on both servers, and set time forward to 10 years in the future (after CA certs expire)
4. Start K3s on server A
5. Verify that CA and other cluster certs are updated on server A (are no longer expired), and kubectl can successfully run
6. Start K3s on server B
7. Verify certificates are written on server B, match those from server A (same expiration and CA hash), and kubectl can successfully run.

[codecov-commenter]

Codecov Report

Merging #3398 (57fda9a) into master (a1e3615) will decrease coverage by 0.15%.
The diff coverage is 10.57%.

@@            Coverage Diff             @@
##           master    #3398      +/-   ##
==========================================
- Coverage   11.61%   11.45%   -0.16%     
==========================================
  Files         136      136              
  Lines        8828     8983     +155     
==========================================
+ Hits         1025     1029       +4     
- Misses       7579     7730     +151     
  Partials      224      224              

Flag	Coverage Δ
inttests	0.67% <0.00%> (-0.02%)	⬇️
unittests	11.10% <10.57%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/cli/server/server.go	0.00% <0.00%> (ø)
pkg/clientaccess/kubeconfig.go	0.00% <ø> (ø)
pkg/cluster/bootstrap.go	0.00% <0.00%> (ø)
pkg/cluster/cluster.go	0.00% <0.00%> (ø)
pkg/cluster/storage.go	0.00% <0.00%> (ø)
pkg/daemons/control/deps/deps.go	44.53% <ø> (ø)
pkg/daemons/control/server.go	0.00% <0.00%> (ø)
pkg/etcd/etcd.go	16.58% <0.00%> (ø)
pkg/bootstrap/bootstrap.go	20.51% <16.66%> (-3.02%)	⬇️
pkg/clientaccess/token.go	89.83% <88.23%> (ø)
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1e3615...57fda9a. Read the comment docs.

[xiaods]

Dockerfile.dapper826934624 should be remove, this is generate file by dapper

[briandowns]

Dockerfile.dapper826934624 should be remove, this is generate file by dapper

Correct. Last commit was a push to get the data up due to hardware failure. PR is t ready for review and will be marked WIP shortly.

[brandond]

This doesn't seem to do the right thing when using an external SQL datastore; it overwrites the files on disk from the bootstrap data without printing any message about why it is doing so.

# start K3s and let it run until the system stabilizes
k3s server --debug --datastore-endpoint 'mysql://root:password@tcp(db.dev-backend.k3s.khaus:3306)/k3s' --token token

# generate a CSR to renew the server-ca cert
openssl x509 -x509toreq -in /var/lib/rancher/k3s/server/tls/server-ca.crt -out server-ca.csr -signkey /var/lib/rancher/k3s/server/tls/server-ca.key

# renew server-ca cert with 10000 days of validity
openssl x509 -req -days 10000 -in server-ca.csr -out /var/lib/rancher/k3s/server/tls/server-ca.crt -signkey /var/lib/rancher/k3s/server/tls/server-ca.key

# verify file timestamps and expiration - timestamp on the .crt file should be newer
ls -la /var/lib/rancher/k3s/server/tls/server-ca*
openssl x509 -noout -text -in /var/lib/rancher/k3s/server/tls/server-ca.crt  | grep After

# start K3s again; let it run until the system stabilizes
k3s server --debug --datastore-endpoint 'mysql://root:password@tcp(db.dev-backend.k3s.khaus:3306)/k3s' --token token

# note that the file timestamps are now the same, and the expiration has reverted to the original value (10 years from initial creation)
ls -la /var/lib/rancher/k3s/server/tls/server-ca*
openssl x509 -noout -text -in /var/lib/rancher/k3s/server/tls/server-ca.crt  | grep After

[brandond]

The same behavior can be seen when using a sqlite datastore (no --datastore-endpoint or --cluster-init)

[briandowns]

Changes made and pushed.

[brandond]

This LGTM, but I would like @rancher-max to make sure we're covering all the weird edge cases before we merge.