Refactor kserve metrics solution to work with prometheus annotations

[spolti]

chore: Make sure that the vLLM metrics are getting correctly exposed when the
prometheus annotations are set.
Fixes [RHOAIENG-6264] - Ensure Metrics work in vLLM

Signed-off-by: Spolti fspolti@redhat.com

Description

This change removes the hardcode monitoring Service, ServiceMonitor and PeerAuthentication resources in favor of a centralized configuration using the PodMonitor and Istio's ServiceMonitor resources to fit our needs and make them available in the aggregated metrics endpoint: http://:15020/stats/prometheus

How Has This Been Tested?

To test it follow the following steps (Can be tested with OVMS ServingRuntime as well):

• Have a cluster with GPU nodes available (only if testing with vLLM)
• Deploy the following ServingRuntime (If testing with vLLM): Available in the jira
• Deploy the following ISVC (If testing with vLLM): Available in the jira
• Use the Following DSC:

spec:
  components:
    codeflare:
      managementState: Removed
    kserve:
      devFlags:
        manifests:
          - contextDir: config
            sourcePath: ''
            uri: 'https://github.com/spolti/odh-model-controller/tarball/RHOAIENG-6264-test'
      managementState: Managed
      serving:
        ingressGateway:
          certificate:
            type: SelfSigned
        managementState: Managed
        name: knative-serving
    trustyai:
      managementState: Removed
    ray:
      managementState: Removed
    kueue:
      managementState: Removed
    workbenches:
      managementState: Removed
    dashboard:
      managementState: Removed
    modelmeshserving:
      managementState: Removed
    datasciencepipelines:
      managementState: Removed

When everything is deployed, check the metrics in the Prometheus console, it should show the vllm prefixed metrics:

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[VedantMahabaleshwarkar]

I think we need to add logic such that existing resources related to the old solution are deleted.

[VedantMahabaleshwarkar]

I'd also recommend changing the commit/PR title to reflect the generic solution rather than being vLLM specific.
Maybe something like refactor kserve metrics solution to work with prometheus annotations

[spolti]

refactor kserve metrics solution to work with prometheus annotations

Sure, updated.

[VedantMahabaleshwarkar]

Code changes look ok to me (see minor comment about function name GetDSCIName. Testing on a cluster now.

[VedantMahabaleshwarkar]

lgtm

[israel-hdez]

The code looks good.
I haven't been able to try the changes. I'll use the upcoming ODH release to do any needed testings there. It is not idea, but it is for what I have time.

/lgtm
Great work @spolti & @VedantMahabaleshwarkar .

[israel-hdez]

For later improvement, I inspected the pod and it also gets the serving.kserve.io/inferenceservice label.
So, maybe we can change the selector similarly to what you did in the webhook fix.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: israel-hdez, spolti, VedantMahabaleshwarkar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [VedantMahabaleshwarkar,israel-hdez,spolti]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment