Expose metrics in hcms and asset store

[derberg]

Description

• expose metrics on /metrics endpoint (so servicemonitor can point to it) in all controllers and services
• for controllers have standard metrics from kubebuilder + the extra one we need to measure SLO and get alerts
• for service have standard metrics - if we defined them in Kyma already
• nice an awesome dashboard, separate for asset store, and separate for hcms
• document what metrics do we expose (reference docs), and maybe something like this for tracing would make sense (templates are stretched) and just mention we enabled tracing

Stretch:

• have tracing enabled so I can see time spent on asset between fetching and uploading to minio, to measure which step is a bottleneck in performance. Make sure sampling is set to 100 in performance cluster (check if possible)
• have sidecars where needed for tracing if needed

Reasons

Have metrics for controllers and services to be able to see how solution behaves and if the load handling matches the defined SLO

[derberg]

AssetStore

• asset store controller example scenarios

  • 1/10/30 asset with markdown files (this includes metadata extraction and communication with webhook service that is doing it) with a given size (e.g. 8MB) and defined complexity (e.g. filter out something) is handled below 0.5sec or more if more assets -> this is the only complex scenario we want, do not now try to find as many scenarios as possible, that one is with markdown, one with asset store, one with both
  • 1/10/30 assets with openapi of a given size (e.g. 1MB) is a single-mode without filtering is handled below 0.2sec

• upload service

  • how many files in what size I can upload in a given time.

• metadata service

  • test how many markdown files metadata service is able to handle in a given time. Hit API directly bypassing AssetStore

Stretch:

hCMS

• cms controller
  • I(@derberg) have no idea how should we approach perf testing of k8s controller that creates other k8s resources. Is K8s testing performance of deployment to see how many pods it is able to create?
• asyncapi service
  • how many asyncapi files the service is able to validate in a given time. Hit API directly bypassing AssetStore
  • how many asyncapi files the service is able to convert in a given time. Hit API directly bypassing AssetStore

Important:

• Above scenario can bo modified and flow can be reversed so, instead of checking 1 asset, we should check the maximum and provide KPI that says AssetStore handles 1000k assets in 1minute.
• during performance tests, take a look at a current consumption of the controllers and services and Minio and check if we can lower default requests and limits. If we can lower, then lower please ;)

[hudymi]

I have prepared draft dashboards for Asset Store and hCMS:
dashboards.zip

[aerfio]

In followup, see if those metrics could be imported from istio, and not from our custom implemented metrics