✨ Automatically renew control plane machine certificates before expiration through machine repave #6983

ykakarap · 2022-07-26T16:42:09Z

What this PR does / why we need it:

This PR achieves certificate rotation on control plane machine by repaving the machines.

It is achieved by doing the following:

Add an annotation (machine.cluster.x-k8s.io/certificates-expiry-date) on KubeadmBootstrapConfig objects that captures the certificates expiry date (1 year from the creation time)
Update the machine status with certificate expiry date by either reading that annotation on the machine object or the bootstrap config object.
Add a field to KCP called kcp.spec.rolloutBefore.certificatesExpiryDays that can be used to trigger a rollout if the control plane machine's certificates will expire within the specified days.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #6529

internal/controllers/machine/machine_controller_phases.go

bootstrap/kubeadm/internal/controllers/kubeadmconfig_controller.go

internal/controllers/machine/machine_controller_phases.go

api/v1beta1/machine_types.go

internal/controllers/machine/machine_controller_phases.go

internal/controllers/machinehealthcheck/machinehealthcheck_targets.go

internal/controllers/machine/machine_controller_phases.go

internal/controllers/machinehealthcheck/machinehealthcheck_targets.go

sbueringer

Very nice.

I think mostly nits from my side

Very nice feature and not a lot of changes necessary to make it happen.

Should we change the PR description to Part of. Given that the PR implements the feature but doesn't include other stuff like e.g. the column and the documentation?

api/v1beta1/machine_types.go

api/v1beta1/machinehealthcheck_types.go

bootstrap/kubeadm/internal/controllers/kubeadmconfig_controller.go

api/v1beta1/machine_types.go

internal/controllers/machine/machine_controller_phases_test.go

internal/controllers/machinehealthcheck/machinehealthcheck_targets.go

sbueringer

did a high-level review on the delta between the time of my last review and now.

Should we flag the PR as WIP again for now?

controlplane/kubeadm/api/v1beta1/kubeadm_control_plane_types.go

controlplane/kubeadm/internal/control_plane.go

util/collections/machine_filters.go

util/collections/machine_filters_test.go

ykakarap · 2022-09-02T20:55:04Z

/retitle ✨ [WIP] Automatically renew control plane machine certificates before expiration through machine repave

sbueringer · 2022-09-06T13:27:42Z

Reviewed the delta, looks good so far. Once we finalized the API and the PR is otherwise ready I would do some in-depth manual testing and another detailed review

api/v1alpha4/conversion.go

api/v1beta1/machine_types.go

controlplane/kubeadm/api/v1alpha4/conversion.go

controlplane/kubeadm/api/v1beta1/kubeadm_control_plane_types.go

internal/controllers/machine/machine_controller_phases.go

util/collections/machine_filters_test.go

sbueringer · 2022-09-12T14:40:56Z

This looks pretty good!

Only nits, except (potentially) #6983 (comment)

fabriziopandini

only nits

api/v1alpha4/conversion.go

api/v1beta1/machine_types.go

api/v1alpha4/conversion.go

bootstrap/kubeadm/internal/controllers/kubeadmconfig_controller.go

controlplane/kubeadm/api/v1beta1/kubeadm_control_plane_webhook.go

controlplane/kubeadm/internal/control_plane.go

util/collections/machine_filters.go

sbueringer

2 nits otherwise delta looks perfect

controlplane/kubeadm/internal/control_plane.go

internal/controllers/machine/machine_controller_phases.go

sbueringer · 2022-09-14T08:48:50Z

Great work!

lgtm pending squash

ykakarap · 2022-09-14T15:52:25Z

Squashed.

sbueringer · 2022-09-14T15:53:30Z

Thx!

/lgtm

internal/controllers/machine/machine_controller_phases.go

sbueringer · 2022-09-16T06:50:19Z

lgtm pending squash

A new annotation is added to KubeadmConfig that captures the certificate expiry information. This information is propagated to machines. This new information can be used by KCP to perform a rollout if the machine certificates are about to expire. Note: The expiry time captured in KubeadmConfig is an approximate time of when the certificates will expire but it is guaranteed to be before the actual certificate expiry.

ykakarap · 2022-09-16T23:40:38Z

Squashed.

fabriziopandini · 2022-09-19T08:55:08Z

great job! this is a long-awaited feature for many CAPI users
/lgtm

sbueringer · 2022-09-19T09:22:51Z

Great work!

/approve

Would be good to surface this feature somewhere in our book

k8s-ci-robot · 2022-09-19T09:23:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fabriziopandini · 2022-09-19T10:45:24Z

Would be good to surface this feature somewhere in our book

@ykakarap do you mind opening an issue for this?

ykakarap · 2022-09-20T02:48:07Z

Would be good to surface this feature somewhere in our book

@ykakarap do you mind opening an issue for this?

Issue: #7247

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 26, 2022

k8s-ci-robot requested review from enxebre and killianmuldoon July 26, 2022 16:42

ykakarap changed the title ~~✨ [WIP][DO_NOT_REVIEW] MHC to mark unhealthy when certificates are about to expire for CP machines~~ ✨ [WIP][DO_NOT_REVIEW] MHC to mark CP machines as unhealthy when certificates are about to expire Jul 26, 2022

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 2, 2022

ykakarap commented Aug 3, 2022

View reviewed changes

internal/controllers/machine/machine_controller_phases.go Outdated Show resolved Hide resolved

ykakarap changed the title ~~✨ [WIP][DO_NOT_REVIEW] MHC to mark CP machines as unhealthy when certificates are about to expire~~ ✨ MHC to mark CP machines as unhealthy when certificates are about to expire Aug 3, 2022

fabriziopandini reviewed Aug 4, 2022

View reviewed changes

internal/controllers/machine/machine_controller_phases.go Outdated Show resolved Hide resolved

internal/controllers/machinehealthcheck/machinehealthcheck_targets.go Outdated Show resolved Hide resolved

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 5, 2022

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 18, 2022

sbueringer reviewed Aug 18, 2022

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 18, 2022

ykakarap force-pushed the certificate-rotation branch from d1f39be to 3c3fa0c Compare September 1, 2022 00:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 1, 2022

ykakarap force-pushed the certificate-rotation branch from 3c3fa0c to d6f18c4 Compare September 2, 2022 02:23

sbueringer reviewed Sep 2, 2022

View reviewed changes

k8s-ci-robot changed the title ~~✨ MHC to mark CP machines as unhealthy when certificates are about to expire~~ ✨ [WIP] Automatically renew control plane machine certificates before expiration through machine repave Sep 2, 2022

ykakarap changed the title ~~✨ [WIP] Automatically renew control plane machine certificates before expiration through machine repave~~ ✨ Automatically renew control plane machine certificates before expiration through machine repave Sep 10, 2022

sbueringer reviewed Sep 12, 2022

View reviewed changes

fabriziopandini reviewed Sep 12, 2022

View reviewed changes

sbueringer reviewed Sep 13, 2022

View reviewed changes

controlplane/kubeadm/internal/control_plane.go Outdated Show resolved Hide resolved

internal/controllers/machine/machine_controller_phases.go Outdated Show resolved Hide resolved

This was referenced Sep 14, 2022

Introduce a NeedsRollout filter in util/collections package #7215

Closed

How to handle deleting machine.cluster.x-k8s.io/certificates-expiry annotation on the kubeadmconfig #7216

Closed

ykakarap force-pushed the certificate-rotation branch from 2d49b40 to 0a93b4f Compare September 14, 2022 15:52

k8s-ci-robot assigned sbueringer Sep 14, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 14, 2022

fabriziopandini reviewed Sep 15, 2022

View reviewed changes

internal/controllers/machine/machine_controller_phases.go Outdated Show resolved Hide resolved

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 15, 2022

ykakarap force-pushed the certificate-rotation branch from 954be1f to bac8e27 Compare September 16, 2022 23:40

k8s-ci-robot assigned fabriziopandini Sep 19, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 19, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 19, 2022

k8s-ci-robot merged commit fd11302 into kubernetes-sigs:main Sep 19, 2022

k8s-ci-robot added this to the v1.3 milestone Sep 19, 2022

ykakarap mentioned this pull request Sep 21, 2022

📖 document automatic certificate renewal support in KCP #7268

Merged

sbueringer mentioned this pull request Sep 27, 2022

Tasks for Cluster API v1.2.3 #7292

Closed

5 tasks

ykakarap mentioned this pull request Oct 4, 2022

KCP should introspect machine certificate expiry information from node to enable automatic certificate renewal #7342

Closed

furkatgofurov7 mentioned this pull request Dec 12, 2022

Automatic CA rotation in CAPI #7721

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Automatically renew control plane machine certificates before expiration through machine repave #6983

✨ Automatically renew control plane machine certificates before expiration through machine repave #6983

ykakarap commented Jul 26, 2022 •

edited

sbueringer left a comment •

edited

sbueringer left a comment

ykakarap commented Sep 2, 2022

sbueringer commented Sep 6, 2022

sbueringer commented Sep 12, 2022

fabriziopandini left a comment

sbueringer left a comment

sbueringer commented Sep 14, 2022

ykakarap commented Sep 14, 2022

sbueringer commented Sep 14, 2022

sbueringer commented Sep 16, 2022

ykakarap commented Sep 16, 2022

fabriziopandini commented Sep 19, 2022

sbueringer commented Sep 19, 2022 •

edited

k8s-ci-robot commented Sep 19, 2022

fabriziopandini commented Sep 19, 2022

ykakarap commented Sep 20, 2022

✨ Automatically renew control plane machine certificates before expiration through machine repave #6983

✨ Automatically renew control plane machine certificates before expiration through machine repave #6983

Conversation

ykakarap commented Jul 26, 2022 • edited

sbueringer left a comment • edited

Choose a reason for hiding this comment

sbueringer left a comment

Choose a reason for hiding this comment

ykakarap commented Sep 2, 2022

sbueringer commented Sep 6, 2022

sbueringer commented Sep 12, 2022

fabriziopandini left a comment

Choose a reason for hiding this comment

sbueringer left a comment

Choose a reason for hiding this comment

sbueringer commented Sep 14, 2022

ykakarap commented Sep 14, 2022

sbueringer commented Sep 14, 2022

sbueringer commented Sep 16, 2022

ykakarap commented Sep 16, 2022

fabriziopandini commented Sep 19, 2022

sbueringer commented Sep 19, 2022 • edited

k8s-ci-robot commented Sep 19, 2022

fabriziopandini commented Sep 19, 2022

ykakarap commented Sep 20, 2022

ykakarap commented Jul 26, 2022 •

edited

sbueringer left a comment •

edited

sbueringer commented Sep 19, 2022 •

edited