Implementing Monitoring and Observability

[ruanxin]

Description

To ensure operational readiness for new reconciler product, a comprehensive monitoring and observability solution is essential to have. We want to support operational aspects like observability and tracing by design and want to incorporate it early on.

AC

• Design and establish KCP Operator Grafana dashboard
  • Reporting on the overall health of KCP Operators (Lifecycle-Manager and Module-Manager)
  • Design and implement key business and systems metrics (both from kubebuilder and custom)
  • Contact and collaborate with SRE for establishing meaningful alerting rule if needed
• Make sure Lifecycle-Manager, Module-Manager log can be collected and presented in KCP Kibana
  example: https://kibana-sf-2eaed983-6984-422e-b6a3-2dbf1c1775cc.cls-03.cloud.logs.services.sap.hana.ondemand.com/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15d,to:now))&_a=(columns:!(_source),filters:!(),index:'14462540-9c20-11eb-bfd5-3d4e267b3508',interval:auto,query:(language:kuery,query:klm),sort:!())

Developer Notes

• Metrics Including but not limited to:
  • Successful Processing Duration 95th percentile for each module reconciliation
    • The duration time of one module state change from error or processing to ready.
    • Design this metrics as Histograms
  • Operator memory, cpu usage (in % and in peak %)
  • Operator worker queue related metrics (from kubebuilder)
  • Operator reconcilation error, success count (from kubebuilder)
• Dashboard data file should persistent and configured in a grafana dashboard ConfigMap, current control-plane repo reference
• Contact with Huskies to clarify who take responsibility and how to do the integration of Jaeger, Kiali.
  • Jaeger is not in productive ready state
  • Jaeger will be replaced with OpenTelemetry
  • Operator Listener/Watcher is not a typical use case for tracing
  • Conclusion: Don't integrate tracing tool

Related PR

kyma-project/module-manager#113

[tobiscr]

For testing, we can use KCP DEV and deploy the reconciler ecosystem manually until the automated pipelines are ready.

[tobiscr]

Further details how to confiugre the observability stack can be provided by @a-thaler .

[tobiscr]

Depends on #43 to get finally implemented.

[tobiscr]

Waiting for SRE alignemtn on requirements. Moving to blocked now ;)