SPIKE: Investigate kserve monitoring and metrics

[VedantMahabaleshwarkar]

As a part of the kserve integration into ODH, we need to look into how monitoring and metrics work in kserve and consider how they will fit into the monitoring story for ODH

• [P0] Configure KServe runtime in monitoring kserve#31
• [P0] Configure KServe controller for monitoring kserve#32
• [P0] Verify which metrics CAIKit/TGIS produce caikit#3
• Create a kserve auxillary controller to manage extra resources for LLM serving odh-model-controller#58

[VedantMahabaleshwarkar]

update 7/11

• According to kserve documentation, Queue proxy is needed for exporting metrics

  • The required annotations for queue proxy metrics aggregation result in a crashlooping pod
  • https://kserve.github.io/website/master/modelserving/observability/prometheus_metrics/#knativequeue-proxy-metrics

• Knative-serving has a HTTP requests dashboard

  • todo: deploy this dashboard and investigate the metrics being shown and their queries

• TODO: investigate istio metrics

  • https://cloud.redhat.com/blog/monitoring-services-like-an-sre-in-openshift-servicemesh

Open questions:

• Is TGIS/CAIKIT going to produce a latency metric?
  • Looks like kserve has latency dashboards for each serving runtime - https://kserve.github.io/website/master/modelserving/observability/prometheus_metrics/#metrics-for-lgbserver-paddleserver-pmmlserver-sklearnserver-xgbserver-custom-transformerpredictor
  • If not, each component in the stack has some options for latency metrics
    • knative-serving seems to have a request-latencies metric. TODO: is this data the actual model latency?

[VedantMahabaleshwarkar]

Update :

• deployed kserve stack according to what will be used in watsonx : https://github.com/Xaenalt/caikit-serving/blob/main/docs/demo/kserve/Kserve.md#kserve-with-caikit--tgis-runtime
• Applying the servicemonitors according to the instructions here https://knative.dev/docs/serving/observability/metrics/collecting-metrics/#setting-up-prometheus do not work. The resulting targets in OCP Observe are Down due to connection refused or connection reset by peer

[VedantMahabaleshwarkar]

First pass of manual instructions for metrics is done.
This spike needs some extra work according to new requirements to be able to see network metrics

[VedantMahabaleshwarkar]

opendatahub-io/odh-model-controller#65
opendatahub-io/caikit-tgis-serving#46
opendatahub-io/odh-manifests#912