ClusterStorageContainer Feature with ModelMesh

[lampajr]

Is your feature request related to a problem? If so, please describe.

As part of model registry initiative[1] we started exploring[2] the new ClusterStorageContainer capability introduced since KServe 0.11[3].

I was able to make it working for plain Kserve deployments but I am struggling making it working using ModelMesh deployments as it seems custom URIs are not supported; I keep getting failed to fetch CR from kubebuilder cache for predictor fraud: the InferenceService test-model-registry-initializer/fraud has an unsupported storageUri scheme model-registry while trying to provide a custom URI like model-registry://....

After some inspection in the source code I think that the URI parsing pass through this method

modelmesh-serving/pkg/predictor_source/inferenceservice_registry.go

Line 131 in 6c86da9

func processInferenceServiceStorage(inferenceService *v1beta1.InferenceService, nname types.NamespacedName) (

and I couldn't find any way to workaround that, therefore I started thinking that the ClusterStorageContainer feature is not supported with ModelMesh deployments.

Is this true?

Is there any fix or trick, that I might have missed, to make it working?

[1] kubeflow/kubeflow#7396
[2] kserve/kserve#3343
[3] https://kserve.github.io/website/master/modelserving/storage/storagecontainers/

Describe your proposed solution

If the feature is not supported with ModelMesh yet, I would suggest to extend that feature here too such that the ClusterStorageContainer capability works for both deployment modes.

Describe alternatives you have considered

n/a

Additional context

Custom model registry storage container example: https://github.com/lampajr/model-registry-storage-initializer