Makes extensions health checks use a Lease resource

[plkokanov]

How to categorize this PR?

/area quality
/kind enhancement

What this PR does / why we need it:

• Adds a heartbeat extension controller that will maintain a heart beat lease which can be used during extension health checks.
• Extension controllers health check actuator no longer updates the extensions' status.conditions[].lastUpdateTime on each reconciliation.
• When the shoot care controller performs health checks for extension resources it will first check the last time when the extension controller's heart beat lease was renewed. This information will be used to determine whether the conditions in the extension resource's status have expired or not.

Which issue(s) this PR fixes:
Fixes #4798

Special notes for your reviewer:

Release note:

When `gardenlet` checks the conditions of extension resources as part of the shoot health check, it checks if the `gardener-extensions-heartbeat` `Lease` maintained by the extension controllers has been renewed within the `ShootCare` controller's `staleExtensionHealthChecks.thresholds[]` settings and sets the corresponding `Shoot` condition to `Unknown` if that is not the case. If the `Lease` is not found, the `status.conditions[].LastUpdateTime` of the extension resource is checked as well for backwards compatibility.

Health checks performed by the `healthcheck` library no longer update the extensions resources' `status.conditions[].LastUpdateTime` on each reconciliation. Instead, a new heartbeat controller was added to the extensions library that will renew a dedicated `Lease` resource named `gardener-extensions-heartbeat` every 30 seconds by default. Extension controllers have to enable this controller as the `gardener-extensions-heartbeat` `Lease` will be used when `gardenlet` checks whether the extension resources' conditions are stale or not. `gardenlet` expects to find this `Lease` inside the namespace where the extension controller is installed by the corresponding `ControllerInstallation`.

[gardener-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[rfranzke]

I feel that having a dedicated Lease resource makes more sense. This is also what we do for the health checks of gardenlets (they report to a Lease object in the gardener-seed-system-lease namespace in the garden cluster which is a different Lease object they use for their leader election).

[plkokanov]

I feel that having a dedicated Lease resource makes more sense. This is also what we do for the health checks of gardenlets (they report to a Lease object in the gardener-seed-system-lease namespace in the garden cluster which is a different Lease object they use for their leader election).

Yep sure, just wondering how would the new Lease resource be created/managed. For gardenlet - there's a controller that reconciles Seed objects and takes care of creating/updating the Lease.

[rfranzke]

WDYM? Can't extensions controller also register such controller?

[plkokanov]

I was mostly concerned about what resource would be reconciled - the Seed can't be used since it is in the garden cluster. One option would be to directly reconcile Leases, but then something has to create the lease (maybe it can be deployed as part of the installation chart). Another option would be to reconcile the extension controller's Namespace and a third could be to have a mapping between extension CustomResources and the Lease resource.

[rfranzke]

I'm confused, maybe I miss something or don't have the full picture.
From a high-level, my understanding is that an approach could be that extension controllers regularly report a heartbeat in a dedicated Lease resource (via a dedicated controller) just like we do it in the gardenlet. The controllerinstallation-care controller of gardenlet could maintain this information in a condition in the respective ControllerInstallation. Then the gardenlet can consider this information to become aware whether extensions are still alive (and perform their health checks) or not.

[plkokanov]

@rfranzke gave the idea to use something similar to https://github.com/gardener/gardener/blob/master/pkg/resourcemanager/controller/garbagecollector/add.go#L75-L81 to initiate the reconciliations for the new controller that would be responsible for the new heartbeat Lease object.
It seems like a good idea, so that's how I will proceed with the PR.

[plkokanov]

The controllerinstallation-care controller of gardenlet could maintain this information in a condition in the respective ControllerInstallation. Then the gardenlet can consider this information to become aware whether extensions are still alive (and perform their health checks) or not.

I've skipped this part and instead went did what I had the impression was discussed in the issue - gardenlet will directly retrieve the Lease from the namespace where the extension controller is installed during the health checks. This is so that it can check its RenewTime directly and whether it is within the ShootCare controller's staleExtensionHealthChecks.thresholds[]

[rfranzke]

/assign

[ialidzhikov]

/assign

[rfranzke]

Nice PR, well structured!

[rfranzke]

Joint review together with @ialidzhikov

[plkokanov]

Thanks for the review. I've addressed the comments and also added proper default and validation for the --heartbeat-renew-interval-seconds

[rfranzke]

Nice @plkokanov, let's continue now :)

/lgtm
/approve

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gardener-prow]

@plkokanov: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gardener-check-vulnerabilities	bc58ef1	link	false	/test pull-gardener-check-vulnerabilities

Full PR test history. Your PR dashboard. Command help for this repository.
Please help us cut down on flakes by linking this test failure to an open flake report or filing a new flake report if you can't find an existing one. Also see our testing guideline for how to avoid and hunt flakes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.