☂️-issue for gardener component metrics/alerts/dashboards

[timebertt]

How to categorize this issue?

/area monitoring ops-productivity
/kind enhancement
/priority normal

What would you like to be added:
Currently the /metrics endpoint of our components (gcm, g-scheduler, gardenlet) only expose some very rudimentary metrics: {garden_cm,gardener_scheduler,gardenlet}_worker_amount, basic go runtime metrics and prometheus http handler metrics. AFAIK, we don't display any of those metrics in any dashboard.

We should augment the set of component metrics, that are collected and exposed to improve observability of the gardener core components. The client-go and controller-runtime libraries contain some helpers for collecting controller related metrics, that can/should be leveraged for that (see client-go and c-r).

Helpful client-side metrics would be:

• client-side API request count per resource/url
• client-side API request latency/duration (histogram)
• client-side rate limit/throttle latency/duration (histogram)
• API request response count per code
• client certificate expiry/rotation age
• metrics around lists/watches (e.g. short_watches_total, watch_duration, lists_total)
• controller workqueue stats (depth, dirty, inflight, duration, ...)

Further server-side metrics, that would also be helpful:

• webhook inflight requests, response code distribution (e.g. in gcm, seed-admission-controller webhooks)
• Gardener API server metrics (same as exposed by kube-apiserver, check if that's already enabled)

More advanced metrics, that could be displayed in dashboards or used in alerts, may include (but not so important):

• number of goroutines (for detecting goroutine leaks), number of system threads
• go garbage collection stats

We could then add dedicated dashboards for all gardener system components displaying:

• the mentioned component-aware metrics
• kubernetes metrics (e.g amount of container/pod restarts, number of replicas, leader election lease durations, ...) (similar to control plane component dashboards)
• resource metrics (e.g. cpu, memory, network usage) (similar to control plane component dashboards)

Once we gained some experience with those metrics, we could build more alerts based on those metrics regarding our system components, for example:

• high amount of short watches
• high percentage of request failures
• high request latency
• high amount of throttled requests / high throttling latency
• only few weeks/days left until the client certificate expires

Why is this needed:
While operating fast growing gardener systems, it is important to be able to have deeper insights for its system components.
In the past few months we have observed some issues regarding scalability/stability/availability which are very hard to debug, reproduce and fix (e.g. #2689, #2747).
Having component-aware metrics, alerts and dashboards will help in running and operating large-scale gardener installations, while maintaining quality of service and keeping operational effort low.

/cc @wyb1 @istvanballok

[timebertt]

We had an internal discussion with @wyb1 and @istvanballok. Thanks again for the productive session!

To summarize, we formalized the following action items / plan for this topic:

• add new prometheus instance without whitelisting/filtering (with low retention time) [@wyb1, @istvanballok]
  • this will help us to explore over time
    • which metrics are indeed helpful -> might be added to whitelists for prometheus with longer retention time
    • which metrics are not helpful/waste -> might be blacklisted accordingly
  • should be configurable, can be enabled/disabled in some way (probably gardenlet config)
• collect/expose the desired new metrics in gardener's components in the go code (starting with those listed above) [@wyb1, @istvanballok, @gardener/gardener-maintainers]
• collect experience during debugging sessions / with new augmented set of metrics [all, operators] and based on these experiences:
  • add new and relevant metrics to existing whitelists [@gardener/gardener-maintainers, operators]
  • define helpful alerts [@gardener/gardener-maintainers, operators]
  • design helpful dashboards [@wyb1, @istvanballok]

[rfranzke]

/assign @danielfoehrKn
as he showed some interest in this these topics and might start looking into it

[danielfoehrKn]

After initially looking into it, I have to say that I would need a considerable amount of time and focus for this topic ( this is one of the areas I am least familiar in ) that I do not have at the moment.
Lets discuss this in a bigger round when @timebertt is back.

/unassign

[timebertt]

/touch

[rfranzke]

/assign @timuthy @rfranzke

[rfranzke]

/unassign @timuthy @rfranzke for now in favor of #1723 and #1724 and #1725

[timebertt]

/touch

[rfranzke]

cc @shafeeqes

[rfranzke]

@shafeeqes @timebertt Can you check whether this issue is still up-to-date and if not change the description accordingly? :) Thank you!

[shafeeqes]

I think

controller workqueue stats (depth, dirty, inflight, duration, ...)

is completed with #7180

[timebertt]

I checked off a few of the tasks that are already done with the controller runtime refactoring.
I expect that alerts won't be done, as alerts are kind of orphaned and unmaintained in this repository.
The remaining open tasks for additional metrics and dashboards could partially still be interesting to add.