Model loading und unloading policy

[fl0rianr]

Feature Description

Introduce a more explicit resident model policy instead of relying mainly on a single max_loaded_models value.

The policy should support:

• a global limit for all loaded models,
• per-hardware limits for CPU, GPU, and NPU,
• and optionally different handling for utility models such as embedders or rerankers.

Example goals:

• keep 1 primary LLM loaded,
• allow 1 small utility model to stay resident,
• limit NPU residency more strictly than GPU or CPU,
• avoid unnecessary eviction of useful small models.

The eviction logic should also be more policy-aware than plain LRU. It should consider things like:

• hardware type,
• current resource pressure,
• whether the model is active,
• whether the model is a small utility model,
• and estimated reload cost.

In the Web UI, this should be presented as a resident model policy rather than a single ambiguous number. For example:

• Total loaded models
• CPU loaded models
• GPU loaded models
• NPU loaded models
• Keep utility models warm
• Eviction preference

This would make the behavior easier to predict and easier to explain.

Use Case / Motivation

Motivation

The current UX is difficult to understand because a single numeric setting does not reflect the real scheduling problem.

Users usually do not want “N arbitrary loaded models”. They want something more specific, such as:

• one main model,
• plus one small embedder,
• with different limits depending on whether the model is on CPU, GPU, or NPU.

This becomes especially confusing when the configured number is increased above 1, but the actual behavior still depends on backend limits, hardware constraints, or runtime placement decisions.

There is also an important UX problem when multiple models are kept loaded without distinguishing between small and large models:

• keeping multiple small models resident may be fine,
• but keeping multiple large models resident can unexpectedly cause memory pressure,
• which may lead to swapping, severe slowdown, or load/runtime errors.
• In extreme cases, loading multiple large models can lead to OOM conditions and system instability.

From the user perspective this feels inconsistent:

• sometimes increasing the number works,
• sometimes it silently causes eviction,
• sometimes it causes poor performance,
• and sometimes it fails with an error.

A more explicit policy would make the system safer and more predictable:

• small helper models could remain loaded when reasonable,
• large models could be limited more conservatively,
• and hardware-specific constraints could be reflected directly in both logic and UI.

Platform Relevance

All platforms

Additional Context

No response

[jeremyfowers]

Hi @fl0rianr I'm having trouble understanding the suggestion.

There are clear written rules for eviction in the server spec. Can you reformat the issue into a table of the current rules and the exact new rules you want?

[fl0rianr]

Thanks @jeremyfowers — here is the concrete breakdown.

My request is not to change the existing rules by default.
I am proposing to keep the current rules as default behavior and add optional extra policy layers for mixed CPU/GPU/NPU setups. The current spec says that max_loaded_models is a shared slot limit across model types, each type has its own LRU, NPU has explicit exclusivity rules, and CPU/GPU have no inherent residency limit beyond available RAM. As far as i got it correct.

Area	Current rules	Proposed new rules
Base limit	max_loaded_models is one shared slot limit used by all model types; default is 1, -1 means unlimited. Each type has its own independent LRU cache.	Keep this unchanged as default. Add an optional global residency cap, e.g. max_loaded_models_total.
Per-type behavior	LLM, Embedding, Reranking, Audio, and Image are separate model types, each with its own LRU, but all using the same slot count.	Keep this unchanged as default. Optionally add role-aware handling for small utility models such as embedders/rerankers.
Hardware constraints	NPU has backend-specific exclusivity rules. CPU/GPU have no inherent limit beyond available RAM.	Add optional per-hardware residency caps, e.g. CPU/GPU/NPU limits, so users can express policy directly.
Eviction	When a slot is full, Lemonade evicts the least recently used model of that same type, then loads the new model. Busy models cannot be evicted until they finish. If loading fails (except file-not-found), all models are evicted and load is retried.	Keep LRU as default. Optionally add a more policy-aware mode, for example preferring to evict idle GPU models or lower-priority utility models first.
Memory / UX outcome	On CPU/GPU, multiple models may coexist as long as memory allows. This means that increasing the count can work fine for multiple small models, but can also unexpectedly cause swapping, slowdowns, load failures, or OOM/system instability when the extra resident models are large. This is not expressed by the current single-number UI.	Make this explicit in both policy and UI: allow conservative defaults for large models, while still allowing small helper models to stay resident when reasonable.

In one sentence: keep the current rules, but add optional global/per-hardware limits and optional utility-model-aware eviction so residency is more predictable and safer on heterogeneous systems.

[jeremyfowers]

Ok I understand the technical change better now. Can you also help me understand the justification? Like why not just set max_loaded_model=-1 (unlimited) if you want a lot of models? Why not build an app layer on top of lemonade that has the specific eviction policies you want, managed via explicit load and unload calls?

Can you describe specific scenarios where you have model set {X} loaded, you want to load a new model Y, and the current eviction policy is unsuitable?

[fl0rianr]

Good question, I'll try with a comprehensive answer. The main justification is that max_loaded_models = -1 is only a count policy, not a safety policy.

According to the current server spec, CPU/GPU have no inherent residency limit beyond available RAM, while NPU has backend-specific exclusivity rules; eviction is LRU within the same model type when the configured slot limit is reached.

That means -1 does not answer the important question:

“Is it actually safe to keep this additional model resident on this hardware right now?”

That gap is also exactly why the current memory-calculation work on #1608 matters as well: Lemonade does not yet have a reliable runtime memory model that can predict when a new placement will cause memory pressure, OOM, or instability.

Why not just use -1?

Because -1 works fine only when the additional resident models are small enough.
If the extra models are large, Lemonade may still try to load them because the residency limit is count-based, not memory-aware. On CPU/GPU that can lead to swapping, severe slowdown, load/runtime errors, or in extreme cases OOM/system instability. The spec itself says CPU/GPU are effectively bounded only by available memory.

My concrete case is:

• I set the value to 2 or -1 because I want to allow concurrent NPU usage in the cases where that is valid.
• Later, when nothing is currently on the NPU, I load another large model.
• Lemonade may still attempt the load without a memory-aware admission check.
• The result is not a useful residency policy; it is just “try to load and hope”, which can push the system into swap or failure instead of rejecting or evicting more intelligently.

Why not solve it purely in an app layer?

An app layer can help with custom workflows, but it cannot fully solve this cleanly because the server is the component that actually knows:

• which models are resident,
• which backend/device they are using,
• which models are busy and therefore not evictable,
• and what the real placement/eviction outcome will be.
• not every wants or can handle this custom workflow

So an app layer can only approximate policy by issuing explicit load/unload calls. The server still needs a safe and predictable admission + eviction policy underneath. Otherwise every app has to reimplement the same scheduling heuristics externally, and it still cannot prevent the server from trying an unsafe load when the count policy allows it. That is also why the docs frame multi-model support as a server-side capability with server-managed eviction.

Concrete scenarios where the current policy is unsuitable

Scenario 1: “Unlimited” is too blunt

• Loaded set {X}: one large model already resident
• Want to load Y: another large model
• Current policy: max_loaded_models = -1, so no count-based eviction happens
• Problem: Lemonade may try to load Y even though the safe outcome should be deny load or warn / require eviction first

Why this is unsuitable:
The current policy treats “one tiny embedder” and “one second huge model” as the same kind of increment: +1 loaded model.

Scenario 2: Keep two small helpers, not two huge models

• Loaded set {embedding, reranker}
• Want to load Y: a large primary LLM
• Current policy: count-based slots or same-type LRU
• Desired behavior: keep the small helper models if they are cheap and useful, and make the decision based on memory pressure, not just count

Why this is unsuitable:
A user often wants “1 big model + 2 small utility models”, not “3 arbitrary models”.

Scenario 3: Mixed hardware should not share one mental model

• Loaded set {GPU LLM, NPU model}
• Want to load Y: another model whose placement may stress a different device
• Current policy: slot count plus backend-specific rules
• Desired behavior: a residency policy that can say “NPU is stricter, GPU may allow one helper model, CPU is RAM-bound”

Why this is unsuitable:
A single count does not express heterogeneous hardware constraints well.

Scenario 4: LRU is not the same as “best model to evict”

• Loaded set {small embedder, small reranker, large idle model}
• Want to load Y: active primary model
• Current policy: evict by same-type LRU once slots are hit
• Desired behavior: prefer evicting the least useful / most expensive-to-keep resident model, not just the least recently used one

Why this is unsuitable:
Users care about utility and safety, not just recency.

Bottom line

The justification is not “I want more models loaded.”
It is:

• max_loaded_models is a count limit
• but the real problem is a memory- and hardware-aware residency decision
• and without that, raising the count can produce unexpectedly bad outcomes when the additional models are large rather than small.

[jeremyfowers]

I set the value to 2 or -1 because I want to allow concurrent NPU usage in the cases where that is valid.

The NPU never allows more than one model of each type, this is moot.

I feel like I'm talking to a bot here :(

If there isn't a concise, concrete problem that needs to be solved then I'm going to close the issue.

[fl0rianr]

I'm sorry that we are on a misunderstanding here (I was not asking for more than one model at the same time on the NPU).

Let’s narrow the case down to memory pressure and UX convenience / functionality.

Memory pressure

Use case: a gfx1151 system is bought -> nice, I can load all sorts of models. With default max_loaded_models = 1, every model is exchanged.

Next exploration step: NPU is nice, saves energy, cannot run every model, but allows simultaneous runtime with little GPU impact - so max_loaded_models = 2.

Now two models can be loaded and potentially active at the same time. Once you try to exchange your frontier 120B-grade OSS with another one to test if it is better, you suddenly have a problem where before everything was working fine.

If not at that point, then later with an optional CPU model at the same time, for example for speech, or if the user sets max_loaded_models = -1. Sure, more resident models means more memory pressure.

This enhancement would not fully fix memory pressure, but it would reduce surprises and the likelihood of this happening. The full fix I see in unison with solving issue #1608.

UX convenience

Basically it is more convenient for the user to set it up in the way he actually wants (default still works fine, so no tuning overhead).

Maybe I want an embedder loaded while I have an LLM for chatting or coding, but specifically not two large LLMs on the GPU.

Maybe in the future we want a sub-agent tag, or a draft-model tag for speculative decoding, but again specifically not two large proper LLMs at the same time on the GPU.

Additionally would be nice if a user can add a keep warm attribute to it's most favorite every day model.

Functionality-based UX

There is seldom one fixed model, every week a new one comes around. But functionality stays more fixed.

Maybe you have a home assistant and you always want your audio model to be ready and respond to commands ASAP, so keep this warm, even if you tested a few other models since then and otherwise it would have been unloaded (unless max_loaded_models = -1 is active).

---

My point is only:

• avoid extreme failure cases once the residency limit is relaxed,
• reduce surprises,
• and allow users to express “keep these small useful models warm, but be conservative with additional large models”.

I do believe this would be useful for Lemonade.

[VladimirVLF]

I feel like I'm talking to a bot here :(

I have that feeling since ~February or March, wherever you go in Github and outside it too. Content to be written, not read. Brain overflow.

[jeremyfowers]

Would the problem be solved if there was a recipe_option field to pin a specific model, such that once its loaded it cant be automatically evicted? We would just have to require that max_loaded_models was larger than the number of pinned models so that load requests wouldnt hang forever.

[fl0rianr]

Thank you for that idea, simple and good! That way the user can ensure a specific long-term task model stays active.

But it does not solve the issue regarding memory budget (if we have more than one active model allowed - default is 1; currently the NPU accepts more than that in default mode as well, e.g. LLM + Embedding).

With PR #1744, a model that doesn't fit can't be loaded (better than the current load into bad OOM behavior - at least on my gfx1151 Linux system).
But that also means the user's load request might not be fulfilled (an error is reported).

I would say that is fine for a pinned model, e.g.:
ERROR: Loading failed due to pinned model/models (list of model names) blocking resources.

Side note: a warning for pinning large models relative to raw memory budget would make sense here as well.

Here is my idea how to fix this:

• log the memory consumption each model adds during loading (rough estimate of its current consumption)
• log this info separately for host RAM and dGPU VRAM (since you don't want to unload CPU models for a dGPU-only model)
• for backends that expose runtime memory info, e.g. future vLLM path, query that directly where possible
• now evict models until enough resources are available (estimation), according to the eviction policy
• if the model still does not fit, evict more and try again
• include waiting for busy models as well
• but never evict pinned models, that's the user's territory

Policy:
A minimal: try to unload the least amount of models (larger models will lose here)
B conservative: unload older/idle models first
C recency-aware: closest to the current LRU logic
Mixed: e.g. maybe first B, then C

I hope this makes the idea clearer while still keeping the current simple single-number model limit for the user.

[bong-water-water-bong]

Investigation: per-hardware max_loaded_models

Since this issue was filed, several related features landed: model pinning, VRAM-pressure eviction with per-model idle timeouts, and auto-tune memory checking (PR #1744).

The remaining gap was that max_loaded_models was a single integer applied uniformly. On heterogeneous systems like Strix Halo, users could not express "1 GPU model, 1 NPU model, 2 CPU models."

The attached PR extends max_loaded_models to accept a JSON object:

{ "max_loaded_models": { "gpu": 1, "npu": 1, "cpu": 2, "total": 3 } }

Per-device caps use LRU within the device (respecting pin status). Omitted keys default to unlimited. Integer format is fully backward compatible.

This directly addresses Scenario 3: "Mixed hardware should not share one mental model."