Support for RKE2 secrets encryption keys rotation

[snasovich]

Implement support to rotate secrets encryption keys. This will allow the UI equivalent to the following RKE1 UI to be implemented:

Rotate encryption keys:

This requires rancher/rke2#748 implemented on RKE2 side.

[sowmyav27]

Test cases to validate:

• Enable default encryption for Secrets on RKE2 cluster. Deploy secrets on the cluster. And perform Rotate Encryption Keys action on the cluster. Verify the key is rotated. and the secrets are encrypted with the new key.
• Enable custom encryption for Secrets on RKE2 cluster. Deploy secrets on the cluster. And perform Rotate Encryption Keys action on the cluster. Verify the key is rotated. and the secrets are encrypted with the new key.
• Enable default encryption for Secrets on RKE2 cluster. Deploy about 5k secrets. Verify the secrets are encrypted. Perform Rotate Encryption Keys action on the cluster. Verify the key is rotated. Verify the secrets are encrypted using the new key. And the rewriting of secrets happen smoothly.
• A few others listed in the internal confluence test plan for this feature

[jakefhyde]

How to rotate encryption keys

Although there will be a UI button to start rotating encryption keys before the v2.6.4 release, currently the encryption keys can be rotated by adding the following:

rotateEncryptionKeys:
  generation: 1

under the rkeConfig section of a cluster object's yaml.

Additional test cases

• single node rke2/k3s
• 3 node control plane rke2/k3s
• creating the cluster with rotateEncryptionKeys.generation in the spec (cluster should not rotate while provisioning).
• >5 node control plane

[jakefhyde]

Blocked waiting for rke2 and k3s kdm june releases since encryption key rotation is feature gated in KDM.

[jakefhyde]

Root cause

Encryption key rotation did not initially work for rke2/k3s, and plans for etcd only nodes would be in sync and not updated during subsequent reconciliations, which includes prepare, rotate, and reencrypt.

What was fixed, or what changes have occurred

The encryption key rotation feature for rke2/k3s has gone through a few iterations in order to reach a working state for all cluster configurations. A fix has been made to ensure that these system-agent plans are run precisely when they need to.

Areas or cases that should be tested

I've personally tested rke2 & k3s v1.22.11, compiled from source with the required fixes merged in, as well as some testing afterwards. I tested the following cluster configurations using Digital Ocean provisioned nodes.

configuration	status
1 node all roles	✔️
1 etcd, 1 cp, 1 worker	✔️
3 nodes all roles	✔️
3 etcd, 2 cp, 3 worker	✔️
3 etcd, 3 cp, 3 worker	✔️
3 etcd, 5 cp, 3 worker	✔️

I personally recommend testing each kubernetes version, both rke2 & k3s, under a variable amount of load (new cluster, >1000 secrets, >1000 secrets and currently adding more, etc.). I also recommend messing with the cluster in various ways, such as deleting a machine.

I've tested the standard user case, and the standard user can rotate encryption keys no problem.

What areas could experience regressions?

N/A

Are the repro steps accurate/minimal?

Set CATTLE_SYSTEM_CHART_DEFAULT_BRANCH=dev-v2.6 when running rancher to use the dev-v2.6 KDM branch.
Rotate Encryption Keys is now an option in the UI.

[thaneunsoo]

Test Environment:

Rancher version: v2.6-head 2c21373
Rancher cluster type: HA
Docker version: 20.10

---

Testing:

Tested all scenarios listed here

Result
All scenarios have passed except for the following scenario:

• Rotate Encryption Key for a RKE2 cluster which has encryption enabled and 10,000 secrets
  Issues found are:
• 38277 - Unable to rotate encryption keys after rotating once
• 38283 - Unable to rotate encryption keys for large number of secrets

Closing this ticket as testing is complete and issues will be tracked in the tickets listed above.