Dependency health checking (e.g. Gardener)

[ebensom]

Description

Implement periodic health checking of Gardener cluster API dependency by periodically querying of the version or health non-resource endpoint via gardener kubeclient in a separate goroutine and keep the latest check result up-to-date. Expose the current (up-to-date) healthcheck result on the Prometheus metrics endpoint via series like:

{app}_{subsys}_gardener_health{url="..", status="healthy"} 1
{app}_{subsys}_gardener_health{url="..", status="error"} 0
{app}_{subsys}_gardener_health{url="..", status="unknown"} 0

Reasons

Ability to cross-correlate infrastructure-manager errors with Gardener API (dependency) errors.

Attachments

[tobiscr]

@ebensom : can we please quickly sync about it? We have open questions

[tobiscr]

@ebensom : I will setup a call for it to clarify the purpose. We want avoid to increase load on Gardener caused by redundant health-checks from us + additional monitoring etc.

[tobiscr]

Istio offers HTTP request metrics but those metrics are only available if traffic is used via plain HTTP but for Gardener HTTP connections are not possible as it enforces HTTPS communication.

Option to implement a check via Prometheus client in KCP would be possible, but this won't reflect whether the KIM is really able to talk to Gardener (it's not reflecting the truth).

[tobiscr]

@ebensom : we will only implement it if you send to each of us a "Thank you " award ;)