Custom Model Repository

[jgallardorama-itx]

Hello!

We would like to extend the model repository to enable fetch models from other sources (database, services, etc...).
Have you thought how to enable this customation or extension?
For example, if ModelRepository was a interface, and an injection mechanishm like with a model is provided, it could enable this feature. What is your roadmap about this feature?

Thank you in advance.

Javi

[adriangonz]

Hey @jgallardorama-itx,

Thanks for opening this one.

In general, we prefer to implement the download of external artefacts through upstream orchestration engines, like Seldon Core or KServe. Main reason is that, in production setups, you generally don't want the inference server to spend time performing the download, and is best to delegate that on other infra pieces.

Having said that, I'd be keen to allow the user to inject other implementations of the ModelRepository class. I could see this being configured on the settings.json file, which could read a Python import path (e.g. mlserver.repository.ModelRepository) and use that as the repository class. If these repository implementations inferface with external sources, they would probably go on a diff package though.

[jgallardorama-itx]

Hi team!

I opened a PR about this issue #849

Please, could you reopen this issue, sorry for the inconveniences

[adriangonz]

Just saw that PR - sure thing, will reopen right ahead 👍 Thanks a lot for your work on the PR!

[adriangonz]

Fixed by #849