Model Concurrency Matrix

[rhyst]

Is your feature request related to a problem? Please describe.
I would like more control over which models can load concurrently. At the moment we can use "Max Active Backends" but that would allow 3 large models to load which may not fit.

Describe the solution you'd like

Literally the matrix configuration from llama-swap: mostlygeek/llama-swap#643
This would allow specifying more complicated rules like "allow my zed prediciton model to run alongside anything but don't allow my two 120b models to run alongside eachother"

Describe alternatives you've considered
Max Active Backends - but has problem as mentioned above.

Additional context

[mudler]

I'm assuming you are talking about standalone mode of LocalAI (not distributed, where there you can specify placement with node labels).

I gave it a thought, and I think the same feature can be achieved in a much simpler way without requiring a full DSL. For instance, by just specifying concurrency_groups (like Github actions, so to speak).

This would allow to specify "do not run these models in the same group at the same time". So in the YAML config you would just write:

name: heavy_model_1
concurrency_groups: ["heavy_model"]

and, another heavy model to sit with the same concurrency group (mutually exclusive)

name: heavy_model_2
concurrency_groups: ["heavy_model"]

[rhyst]

Yes, standalone. Your suggestion sounds good to me, it would solve my problem.