Rotate certificates for RKE2 provisioned clusters #4485

snasovich · 2021-11-02T02:07:48Z

Detailed Description
The UI should provide an option to rotate certificates on RKE2 provisioned cluster, including an option to rotate certificates for an individual component. The UX should probably match what we have for RKE1 with context menu item and pop-up dialog asking to specify the settings:

Context

This is needed for RKE2 provisioning parity with RKE1

Additional Details
rancher/rancher#35343

catherineluse · 2022-02-08T04:15:14Z

@snasovich I believe no UI changes will be needed if the backend includes rotateCertificates in the action links within the cluster data.

To determine what actions are available, the UI looks at the actions in the cluster data at <rancher-server-url>/v3/clusters/<cluster-id>. Currently the actions for RKE2 clusters include these actions:

"actions":{
      "apply":"https://192.168.0.9:8005/v1/management.cattle.io.clusters/c-m-b4qzzdqg?action=apply",
      "enableMonitoring":"https://192.168.0.9:8005/v3/clusters/c-m-b4qzzdqg?action=enableMonitoring",
      "generateKubeconfig":"https://192.168.0.9:8005/v3/clusters/c-m-b4qzzdqg?action=generateKubeconfig",
      "importYaml":"https://192.168.0.9:8005/v3/clusters/c-m-b4qzzdqg?action=importYaml"
   }

The actions for RKE1 clusters include the additional rotateCertificates action:

"rotateCertificates":"https://192.168.0.9:8005/v3/clusters/c-hsv9l?action=rotateCertificates",

To confirm that it would work if rotateCertificates were included in the action links, I used mock data to pretend it was included in the action links for an RKE2 cluster. In that scenario, the action is available from the context menu, the modal appears properly, and when you submit a request from the modal, it sends the appropriate network request.

snasovich · 2022-02-08T16:18:49Z

@catherineluse , based on the implementation PR rancher/rancher#36347 it doesn't look like that's how it's supported on backend as the changes seem to be limited to v1 API.
@paynejacob , could you please confirm? Also, does it make sense / feasible to add rotateCertificates option for v3 API?

paynejacob · 2022-02-08T18:31:04Z

@catherineluse this is adding support for v2 clusters. v2 provisioning clusters are not supported in the v3 api. Steve does not support actions so the rotation will need to be triggered with a patch on the provisioning cluster through the v1 api kubectl patch -n fleet-default --patch '{"spec":{"rkeConfig":{"rotateCertificates":{"caCertificates":true,"generation":1}}}}' --type=merge cluster/cert-rotate

catherineluse · 2022-02-15T00:26:23Z

@paynejacob I noticed that when the UI does the cert rotation for RKE1 clusters, we set caCertificates to false, whereas in your example command it is true. Is it correct that for RKE1 it should be false and for RKE2 it should be true?

paynejacob · 2022-02-15T16:26:55Z

@catherineluse I forgot to update the ticket the caCertificates field is going away entirely once this pr is merged. The only fields will be the service list and the generation. If no services are selected you just need to increment the generation to kick of rotation.

rancher-max · 2022-03-09T22:25:57Z

Validating on v2.6-head commit `572ab58` -- Reopening due to last point below

Field is available and looks correct:

After clicking on it, valid options:

Selecting individual services correctly selects and rotates that service
Selecting to rotate all fails and only does etcd. Here is what gets passed in the request payload. Snippet posted first of the error spot, followed by the entire payload. According to my understanding, the services array should be empty in order to rotate all certs.

      "rotateCertificates": {
        "generation": 4,
        "services": [
          "etcd"
        ]
      },

{
  "id": "fleet-default/maxcert1",
  "type": "provisioning.cattle.io.cluster",
  "links": {
    "remove": "https://173.230.149.125/v1/provisioning.cattle.io.clusters/fleet-default/maxcert1",
    "self": "https://173.230.149.125/v1/provisioning.cattle.io.clusters/fleet-default/maxcert1",
    "update": "https://173.230.149.125/v1/provisioning.cattle.io.clusters/fleet-default/maxcert1",
    "view": "https://173.230.149.125/apis/provisioning.cattle.io/v1/namespaces/fleet-default/clusters/maxcert1"
  },
  "apiVersion": "provisioning.cattle.io/v1",
  "kind": "Cluster",
  "metadata": {
    "annotations": {
      "field.cattle.io/creatorId": "user-qhm2g"
    },
    "creationTimestamp": "2022-03-09T18:19:40Z",
    "fields": [
      "maxcert1",
      "true",
      "maxcert1-kubeconfig"
    ],
    "finalizers": [
      "wrangler.cattle.io/provisioning-cluster-remove",
      "wrangler.cattle.io/rke-cluster-remove"
    ],
    "generation": 6,
    "managedFields": [
      {
        "apiVersion": "provisioning.cattle.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:status": {
            ".": {},
            "f:clientSecretName": {},
            "f:clusterName": {},
            "f:conditions": {},
            "f:observedGeneration": {},
            "f:ready": {}
          }
        },
        "manager": "rancher",
        "operation": "Update",
        "subresource": "status",
        "time": "2022-03-09T18:25:58Z"
      },
      {
        "apiVersion": "provisioning.cattle.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:metadata": {
            "f:finalizers": {
              ".": {},
              "v:\"wrangler.cattle.io/provisioning-cluster-remove\"": {},
              "v:\"wrangler.cattle.io/rke-cluster-remove\"": {}
            }
          },
          "f:spec": {
            ".": {},
            "f:cloudCredentialSecretName": {},
            "f:kubernetesVersion": {},
            "f:localClusterAuthEndpoint": {},
            "f:rkeConfig": {
              ".": {},
              "f:chartValues": {
                ".": {},
                "f:rke2-calico": {}
              },
              "f:etcd": {
                ".": {},
                "f:snapshotRetention": {},
                "f:snapshotScheduleCron": {}
              },
              "f:machineGlobalConfig": {
                ".": {},
                "f:cni": {},
                "f:disable-kube-proxy": {},
                "f:etcd-expose-metrics": {},
                "f:profile": {}
              },
              "f:machinePools": {},
              "f:machineSelectorConfig": {},
              "f:registries": {},
              "f:rotateCertificates": {
                ".": {},
                "f:generation": {},
                "f:services": {}
              },
              "f:upgradeStrategy": {
                ".": {},
                "f:controlPlaneConcurrency": {},
                "f:controlPlaneDrainOptions": {
                  ".": {},
                  "f:timeout": {}
                },
                "f:workerConcurrency": {},
                "f:workerDrainOptions": {
                  ".": {},
                  "f:timeout": {}
                }
              }
            }
          }
        },
        "manager": "rancher",
        "operation": "Update",
        "time": "2022-03-09T18:40:46Z"
      }
    ],
    "name": "maxcert1",
    "namespace": "fleet-default",
    "relationships": [
      {
        "toId": "fleet-default/maxcert1",
        "toType": "cluster.x-k8s.io.cluster",
        "rel": "owner",
        "state": "provisioned"
      },
      {
        "toId": "fleet-default/maxcert1-managed-system-agent",
        "toType": "fleet.cattle.io.bundle",
        "rel": "owner",
        "state": "active",
        "message": "Resource is Ready"
      },
      {
        "toId": "fleet-default/r-cluster-maxcert1-view-p-dbhxs-creator-project-owner-esc-3b72d",
        "toType": "rbac.authorization.k8s.io.rolebinding",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "c-m-lcshvl5k/c-m-lcshvl5k-fleet-default-owner",
        "toType": "management.cattle.io.clusterroletemplatebinding",
        "rel": "applies",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/r-cluster-maxcert1-view-p-jrtns-creator-project-owner-esc-67214",
        "toType": "rbac.authorization.k8s.io.rolebinding",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/r-cluster-maxcert1-view",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/crt-maxcert1-cluster-admin",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/crt-maxcert1-nodes-manage",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/crt-maxcert1-nodes-view",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/nc-maxcert1-allroles-2kwqb",
        "toType": "rke-machine-config.cattle.io.amazonec2config",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/maxcert1-kubeconfig",
        "toType": "secret",
        "rel": "owner",
        "state": "active",
        "message": "Resource is always ready"
      },
      {
        "toId": "fleet-default/maxcert1",
        "toType": "fleet.cattle.io.cluster",
        "rel": "applies",
        "state": "active",
        "message": "Resource is Ready"
      },
      {
        "toId": "fleet-default/r-cluster-maxcert1-view-p-m9r8c-creator-project-owner-esc-dc976",
        "toType": "rbac.authorization.k8s.io.rolebinding",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/crt-maxcert1-cluster-member",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "c-m-lcshvl5k",
        "toType": "management.cattle.io.cluster",
        "rel": "applies",
        "state": "active",
        "message": "Resource is Ready"
      },
      {
        "toId": "fleet-default/crt-maxcert1-cluster-owner",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
      },
      {
        "toId": "fleet-default/maxcert1-managed-system-upgrade-controller",
        "toType": "management.cattle.io.managedchart",
        "rel": "owner",
        "state": "active",
        "message": "Resource is Ready"
      }
    ],
    "resourceVersion": "85859",
    "state": {
      "error": false,
      "message": "Resource is Ready",
      "name": "active",
      "transitioning": false
    },
    "uid": "dc308447-eb80-41b2-a1b2-35459994a5b1"
  },
  "spec": {
    "cloudCredentialSecretName": "cattle-global-data:cc-t6xtp",
    "kubernetesVersion": "v1.21.10+rke2r1",
    "localClusterAuthEndpoint": {},
    "rkeConfig": {
      "chartValues": {
        "rke2-calico": {}
      },
      "etcd": {
        "snapshotRetention": 5,
        "snapshotScheduleCron": "0 */5 * * *"
      },
      "machineGlobalConfig": {
        "cni": "calico",
        "disable-kube-proxy": false,
        "etcd-expose-metrics": false,
        "profile": null
      },
      "machinePools": [
        {
          "controlPlaneRole": true,
          "etcdRole": true,
          "machineConfigRef": {
            "kind": "Amazonec2Config",
            "name": "nc-maxcert1-allroles-2kwqb"
          },
          "name": "allroles",
          "quantity": 3,
          "unhealthyNodeTimeout": "0s",
          "workerRole": true
        }
      ],
      "machineSelectorConfig": [
        {
          "config": {
            "protect-kernel-defaults": false
          }
        }
      ],
      "registries": {},
      "rotateCertificates": {
        "generation": 4,
        "services": [
          "etcd"
        ]
      },
      "upgradeStrategy": {
        "controlPlaneConcurrency": "10%",
        "controlPlaneDrainOptions": {
          "timeout": 0
        },
        "workerConcurrency": "10%",
        "workerDrainOptions": {
          "timeout": 0
        }
      }
    }
  },
  "status": {
    "clientSecretName": "maxcert1-kubeconfig",
    "clusterName": "c-m-lcshvl5k",
    "conditions": [
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:25:58Z",
        "status": "False",
        "transitioning": false,
        "type": "Reconciling"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:40Z",
        "status": "False",
        "transitioning": false,
        "type": "Stalled"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T22:10:39Z",
        "status": "True",
        "transitioning": false,
        "type": "Created"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T22:16:18Z",
        "status": "True",
        "transitioning": false,
        "type": "RKECluster"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:40Z",
        "status": "True",
        "transitioning": false,
        "type": "BackingNamespaceCreated"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:40Z",
        "status": "True",
        "transitioning": false,
        "type": "DefaultProjectCreated"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:40Z",
        "status": "True",
        "transitioning": false,
        "type": "SystemProjectCreated"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:41Z",
        "status": "True",
        "transitioning": false,
        "type": "InitialRolesPopulated"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T22:16:18Z",
        "status": "True",
        "transitioning": false,
        "type": "Provisioned"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:43Z",
        "status": "True",
        "transitioning": false,
        "type": "CreatorMadeOwner"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:25:44Z",
        "status": "True",
        "transitioning": false,
        "type": "Connected"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:25:26Z",
        "status": "True",
        "transitioning": false,
        "type": "Pending"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:26:12Z",
        "status": "True",
        "transitioning": false,
        "type": "Waiting"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:53Z",
        "status": "True",
        "transitioning": false,
        "type": "NoDiskPressure"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:53Z",
        "status": "True",
        "transitioning": false,
        "type": "NoMemoryPressure"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:19:58Z",
        "status": "True",
        "transitioning": false,
        "type": "SecretsMigrated"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:25:29Z",
        "status": "True",
        "transitioning": false,
        "type": "GlobalAdminsSynced"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:25:57Z",
        "status": "True",
        "transitioning": false,
        "type": "Ready"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:25:58Z",
        "status": "True",
        "transitioning": false,
        "type": "SystemAccountCreated"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:26:02Z",
        "status": "True",
        "transitioning": false,
        "type": "AgentDeployed"
      },
      {
        "error": false,
        "lastUpdateTime": "2022-03-09T18:26:07Z",
        "status": "True",
        "transitioning": false,
        "type": "Updated"
      }
    ],
    "observedGeneration": 6,
    "ready": true
  }
}

rancher-max · 2022-03-10T19:25:32Z

There's another thing that should be changed as well. The individual service list should be:

admin
api-server
controller-manager
scheduler
rke2-controller
rke2-server
cloud-controller
etcd
auth-proxy
kubelet
kube-proxy

If the desire is to only have the ones currently listed, then at least the following need to be changed:

kube-apiserver           -->  api-server
kube-scheduler           -->  scheduler
kube-controller-manager  -->  controller-manager

catherineluse · 2022-03-10T19:43:19Z

Oops, I didn't notice the service names are different in RKE2. I will change it to the full list.

rancher-max · 2022-03-15T19:12:01Z

Validated using rke2 `v1.21.10+rke2r2` on rancher `v2.6-head` commitid `99974f0` with the help of @bguzman-3pillar

The list has the expected services now:

Selecting to rotate all correctly rotates all certificates (passes an empty array to the API)

jtravee · 2022-03-16T22:24:05Z

Confirmed with @catherineluse and @gaktive to add release note label.

snasovich added status/waiting-backend area/rke2 labels Nov 2, 2021

snasovich added this to the v2.6.4 milestone Nov 2, 2021

gaktive added the area/clusterprovisioningv2 label Dec 3, 2021

This was referenced Dec 3, 2021

Cluster Provisioning v2: Day2 RKE2 Cluster Ops - On Demand Certificate Rotation #3274

Closed

EPIC: Cluster Provisioning v2 GA #3346

Closed

gaktive added [zube]: To Triage status/release-blocker labels Dec 13, 2021

Sahota1225 mentioned this issue Jan 5, 2022

[Group] RKE2 Provisioning parity work for RKE2 Provisioning GA rancher/rancher#36044

Closed

24 tasks

gaktive added [zube]: Backend Blocked and removed [zube]: To Triage labels Jan 11, 2022

Sahota1225 added the team/area2 Hostbusters label Jan 26, 2022

gaktive added [zube]: Backlog and removed status/waiting-backend [zube]: Backend Blocked labels Feb 1, 2022

gaktive assigned catherineluse Feb 4, 2022

catherineluse added [zube]: Working and removed [zube]: Next Up labels Feb 4, 2022

This was referenced Feb 8, 2022

Can rotate encryption keys for RKE2 clusters #4540

Closed

Rotate certs action is available for RKE2 clusters #5123

Merged

catherineluse added [zube]: Review and removed [zube]: Working labels Feb 15, 2022

catherineluse removed the [zube]: Review label Feb 17, 2022

catherineluse added the [zube]: To Test label Feb 17, 2022

sowmyav27 assigned rancher-max Feb 24, 2022

rancher-max added [zube]: QA Working and removed [zube]: To Test labels Mar 9, 2022

rancher-max added [zube]: Reopened and removed [zube]: QA Working labels Mar 9, 2022

catherineluse mentioned this issue Mar 11, 2022

RKE2 cert rotation applies to correct services #5362

Merged

catherineluse added [zube]: Review and removed [zube]: Reopened labels Mar 11, 2022

catherineluse mentioned this issue Mar 12, 2022

Remove "Rotate Encryption Keys" functionality from UI (if already introduced) #5379

Closed

rancher-max closed this as completed Mar 15, 2022

zube bot reopened this Mar 15, 2022

zube bot closed this as completed Mar 15, 2022

zube bot added [zube]: Done and removed [zube]: To Test labels Mar 15, 2022

jtravee added the release-note label Mar 16, 2022

zube bot removed the [zube]: Done label Jun 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rotate certificates for RKE2 provisioned clusters #4485

Rotate certificates for RKE2 provisioned clusters #4485

snasovich commented Nov 2, 2021

catherineluse commented Feb 8, 2022 •

edited

Loading

snasovich commented Feb 8, 2022

paynejacob commented Feb 8, 2022

catherineluse commented Feb 15, 2022 •

edited

Loading

paynejacob commented Feb 15, 2022

rancher-max commented Mar 9, 2022

rancher-max commented Mar 10, 2022

catherineluse commented Mar 10, 2022

rancher-max commented Mar 15, 2022

jtravee commented Mar 16, 2022

Rotate certificates for RKE2 provisioned clusters #4485

Rotate certificates for RKE2 provisioned clusters #4485

Comments

snasovich commented Nov 2, 2021

catherineluse commented Feb 8, 2022 • edited Loading

snasovich commented Feb 8, 2022

paynejacob commented Feb 8, 2022

catherineluse commented Feb 15, 2022 • edited Loading

paynejacob commented Feb 15, 2022

rancher-max commented Mar 9, 2022

Validating on v2.6-head commit 572ab58 -- Reopening due to last point below

rancher-max commented Mar 10, 2022

catherineluse commented Mar 10, 2022

rancher-max commented Mar 15, 2022

Validated using rke2 v1.21.10+rke2r2 on rancher v2.6-head commitid 99974f0 with the help of @bguzman-3pillar

jtravee commented Mar 16, 2022

catherineluse commented Feb 8, 2022 •

edited

Loading

catherineluse commented Feb 15, 2022 •

edited

Loading

Validating on v2.6-head commit `572ab58` -- Reopening due to last point below

Validated using rke2 `v1.21.10+rke2r2` on rancher `v2.6-head` commitid `99974f0` with the help of @bguzman-3pillar