Lazily load models and on insufficient resources for a load, look to unload idle models

[kimjuny]

Feature Request

Describe the problem the feature is intended to solve

I have thousands of models to be served, but quite a big part of these models are not frequently requested, actually only a few of them are. Loading all of these models into memory at the same time is quite resource-consuming & time-consuming.

Describe the solution

I wonder if there's an option to lazy load models and only caching those most frequently requested models in memory?

Describe alternatives you've considered

None yet.

Additional context

Actually I'm not sure if this is a feature request or we are already having this feature.

[rmothukuru]

Can you please check this link about Lazy Loading (Warm Starting) and let us know if it helps.

[kimjuny]

Can you please check this link about Lazy Loading (Warm Starting) and let us know if it helps.

I didn't set any "warm up", I'm using tf.keras API.

Saving the model by tf.keras.experimental.export_saved_model(model, saved_model_path) API(2.0.0-beta-1).

I think the link above is something that reduces the first request latency and requires more memory on init(model loading stage), kind of opposite feature that I need?

[rmothukuru]

@kimjuny ,
Sorry for the misunderstanding. Can you please elaborate your question so that we can provide better support.

[kimjuny]

@rmothukuru I have lots of models to be served(tens of thousands of SavedModel), I've configured them in Serving Config. When I'm starting the tensorflow/serve, it consumes too much time & memory to startup. So I wonder if there is a startup option that disables loading all the models specified in Serving Config in the beginning but loading each specific models later when it is requested.

[peddybeats]

Hey @kimjuny the use case definitely makes sense. There's nothing on the roadmap currently addressing this need but I'll leave this open to see if other folks develop the same need.
Can you elaborate on the use case? For example, why do you have O(10K) models that can/should not be combined? And how would you know you need to load any specific one? Are they models trained for specific users and you won't know you need one of them loaded until that user sends an inference request? This kind of information will help us understand for example what the model load time SLO (<second or is minutes fine) would need to be in order for such a feature to be useful to you.

[kimjuny]

@unclepeddy Thanks for your reply. I'm currently trying to work out a market prediction project. In my case I have to train models individually(for every stock) for better performance, with different versions of them.
I'm only running the inference procedure when the market is closed, so actually I have plenty of time, which means a few seconds of latency(of each model's loading stage) is totally acceptable for me.
So here are two features (I assumed) that could really help me:

1. lazy-load option
When -e LAZY_LOAD=true is set when starting tf/serve, the server should start immediately without loading any of the models defined in Serving Config. Each model (that specified in the Serving Config) would only be loaded into memory when a specific inference request is sent.

2. memory management e.g. FIFO、LRU...
When -e LAZY_LOAD=true is set, tf/serve automatically manage memory by an option(something like -e FIFO_TOTAL=500, which means only load 500 models at most by "first in first out" principle)

Both of these feature would be a huge enhancement for my case, first one would really boost up the starting stage, and second one would save a lot of memory when the server is running.