Add EPP operational metrics

[liu-cong]

What would you like to be added:

I would like to add the following list of metrics:

[ ] Overall latency for EPP to receive a request and make a decision on the target pod. Why? EPP provides benefits with the cost of added latency. This is a generally useful metric to quantify the cost, as well as detect any regressions in EPP.
[ ] Overall latency for EPP to process the response. Why? Same as above.
[ ] Latency/error count when EPP scrapes model server metrics Why? The freshness of metrics is vital to the effectiveness of the EPP algorithm. These metrics help detect issues with metric scraping due to regressions of EPP/model server or transient failures.
[ ] Age of caches (the model server metrics, etc.) Why? This is the observed cache freshness. It can be a useful metric to alert on if the cache age exceeds certain limit.
[ ] Overall latency of each component (flow controller, scheduler) . Why? These component level metrics provide latency breakdown of the overall EPP latency, and help with identify bottlenecks, source of regressions, etc.
[ ] Future: Prefix cache, queuing metrics (latency, error, size, and any other custom metrics) Why? These are important operational metrics to monitor. For instance, if the queue size is hitting the limit, it's a signal to perhaps increase the size limit.

Why is this needed:

[JeffLuoo]

Can you please add "Why is this needed:" for each proposed metrics?

[kfswain]

/assign liu-cong

To address the above comment

[liu-cong]

@JeffLuoo Added "whys", please take a look.

[liu-cong]

/unassign

[JeffLuoo]

Can you clarify on:

1. What is the difference between

Overall latency of each component (flow controller, scheduler) https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/581. 

and

Overall latency for EPP to receive a request and make a decision on the target pod

#581 mentions about the scheduling latency but the first metric is also about the scheduling latency.

2. For

Overall latency for EPP to process the response.

Do you mean how long it takes for EPP to process the response header and then the response body?

[liu-cong]

Sorry for the confusion.

For 1, #581 captures the overall latency for the scheduler component, it's a sub task of this issue.

For 2, the latency between EPP receives the response from the backend to EPP returns the response back to the LB. The intent behind this is to understand the overall latency cost added by EPP in the request/response paths.