Idle model management

[dtrawins]

Problem statement

Users and applications need to experiment and use various models depending of the situation and application

AIPC cannot host all needed models at the same time because of VRAM limitations

Alternatives

Llama-swap with configuration file which defines groups of related models and idle time.

Disadvantages on llama-swap

Complicated deployment of additional component and configuration file

Extra latency on the processing with additional hop

Delayed model startup - waiting for ready status checked regularly

Design concept

We need the option to group the model if they are needed together – for example – 3 models for RAG, or LLM with audio etc

Selected models can be considered permanent and excluded from idle model cleanup – for example needed in all use cases or loading for too long.

Groups should be started and unloaded together

Only one group can be loaded at a time (except permanent group)

Starting the group should be done on demand – request sent to any model in the group

Unloading should be done after idle period or when another group is to be started

Unloading can’t be initialized then active requests are running or then loading is in progress

Special group name “permanent” would ensure this group is not unloaded

Deployment

Idle model management relevant only when started from config.json with multiple models

Additional config parameter and CLI– idle_time_unload

Each model should have new optional parameter – group_name in config.json. By default the group_name is the same like model name

By default, each model/pipeline would be independent of each without models grouping