Does KServe allow custom prometheus metrics in predict step?

[sharonliu-gfg]

As we would like to collect application level metrics (e.g. histogram of model confidence values), is there any way we can have custom metrics in the predict step for a custom model?

We have tried using prometheus metrics within the @serve.deployment portion, but it cannot be picked up by the metrics handler/queue-proxy and cannot be registered to a global registry due to cloudpickle used in ray serve.