Eval bug: Router mode `--models-max` not enforced under concurrent requests (TOCTOU race in unload_lru)

[BenRacicot]

Name and Version

Name and Version:

• b8189
• macOS 15.5, Apple Silicon (M4 Max, 64GB unified memory)
• Operating systems: macOS
• GGML backends: Metal

This took a lot to find and repro, I do hope it's very helpful and thanks for looking into it.

Problem description & steps to reproduce:

Two related issues with router mode memory management that make --models-max unreliable on
memory-constrained systems.

Issue 1: --models-max race condition with concurrent requests

With --models-max 2 and 5 GGUF models in the directory, sending concurrent requests for different models loads 3+ models simultaneously, ignoring the limit entirely. After all requests complete, the loaded count never settles back to 2.

Root cause

In server-models.cpp, the load() method calls unload_lru() outside the mutex, then acquires the lock afterward:

void server_models::load(const std::string &name) {
    // ...
    unload_lru();                          // ← checks count WITHOUT lock
    std::lock_guard<std::mutex> lk(mutex); // ← lock acquired AFTER check
    // proceeds to load without re-checking capacity
}

unload_lru() checks count_active >= models_max without synchronization. When multiple requests arrive concurrently, they all observe count_active < models_max and all proceed to load, exceeding the limit.

This is a classic TOCTOU (time-of-check-time-of-use) race.

Steps to reproduce

# 1. Place 5+ GGUF models in a directory
# 2. Start router with limit of 2
llama-server --models-dir /path/to/models --models-max 2 --port 8086 -ngl 99

# 3. Send 5 concurrent requests for different models
for model in ModelA ModelB ModelC ModelD ModelE; do
  curl -s http://127.0.0.1:8086/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d
"{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":5}" & done wait

# 4. Check loaded models — expect 2, observe 3+
curl -s http://127.0.0.1:8086/v1/models | jq '.data[] | select(.status.value == "loaded") | .id'

Observed behavior

Time after requests	Child processes	Peak RSS	Models loaded
5s	3	15.5 GB	3
15s	4	18.2 GB	3
60s	4	12.5 GB	3 (never drops to 2)

Sequential requests work correctly — the limit is enforced when requests arrive one at a time.

Expected behavior

The router should serialize or gate model loads so that at most --models-max models are loaded simultaneously. Requests for unloaded models should queue when at capacity, not bypass the limit.

Suggested fix

Move unload_lru() inside the critical section, or re-check capacity after acquiring the lock:

void server_models::load(const std::string &name) {
  if (!has_model(name)) {
    throw std::runtime_error("model name=" + name + " is not found");
  }
  std::lock_guard<std::mutex> lk(mutex); // ← lock FIRST
  unload_lru();                          // ← then check/evict under lock
  // proceed to load
}

---

Issue 2: -cram budget duplicated per child process, not shared

The -cram value is passed to each child process independently. With --models-max 2 -cram 15000, the intended total cache budget is 15 GB, but the actual allocation is 2 × 15 GB = 30 GB.

Visible in the /v1/models response — each model entry's status.args shows the full --cache-ram
value:

{
  "id": "SomeModel-Q4_K_M",
  "status": {
    "value": "loaded",
    "args": ["...", "--cache-ram", "15000", "..."]
  }
}

This compounds with Issue 1: when 3+ models load instead of 2, each with the full cache budget, peak memory far exceeds what was intended.

Expected: -cram should either be divided across --models-max children automatically, or this behavior should be clearly documented so callers can pass cram / models_max themselves.

---

Combined impact

On a 64 GB Apple Silicon Mac with 5 models and --models-max 2 -cram 15000:

• Expected peak: ~6 GB (2 models + shared 15 GB cache budget)
• Actual peak: 18+ GB (3+ models loaded, each with 15 GB cache)

This causes severe memory pressure and makes router mode unreliable for its intended purpose of limiting resource usage on unified-memory systems.

Operating systems

Mac

GGML backends

Metal

Hardware

M2 Max / 32GB

Models

Any combination can be used to see the issue stated. I was working with Qwen 3 models during repro.

Problem description & steps to reproduce

Steps to reproduce

# 1. Place 5+ GGUF models in a directory
# 2. Start router with limit of 2
llama-server --models-dir /path/to/models --models-max 2 --port 8086 -ngl 99

# 3. Send 5 concurrent requests for different models
for model in ModelA ModelB ModelC ModelD ModelE; do
  curl -s http://127.0.0.1:8086/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d
"{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":5}" & done wait

# 4. Check loaded models — expect 2, observe 3+
curl -s http://127.0.0.1:8086/v1/models | jq '.data[] | select(.status.value == "loaded") | .id'

First Bad Commit

No response

Relevant log output

I don't have logs but ram usage is the key issue. If you require logs please let me know.

[giveen]

I am having the same issues with server not releasing after setting --models-max=1

[giveen]

Claude Sonnet 4.6 confirms the bug for Issue #1 and suggests

I reviewed the current code in server-models.cpp and can confirm the race condition diagnosis is correct.

The load() function calls unload_lru() before acquiring the main mutex. Since the HTTP server uses a thread pool, concurrent requests for different unloaded models can all observe the same stale capacity count at the same time, all decide eviction is not needed (or all evict the same model), and all proceed to spawn new child processes — exceeding --models-max entirely. With --models-max 1 this means the old model is not evicted before the new one starts loading, which is exactly the OOM-on-GPU-RAM symptom.

However, the proposed fix (acquire the lock first, then call unload_lru()) would deadlock on the current code. Both unload_lru() and unload() internally acquire the same non-recursive mutex, so calling either of them while the caller already holds that lock will block forever.

For the fix to work correctly, the capacity check, LRU eviction, and reservation of the new load slot would need to happen as a single atomic operation — meaning the wait-for-eviction step would need to release the lock during the wait (using a condition variable) and recheck capacity before committing to the new load, without calling any function that tries to re-acquire the same mutex while it is already held.

The race diagnosis in the issue is sound. The fix shape needs a locking refactor before it is safe to apply.

I will not submit any code for this.

[BenRacicot]

I would have done a PR but I'm sure a team member would want to dig into this with their entire team.
--models-max exists so its a broken feature.

If someone like @ngxson asks me to take a shot I will. Thanks ngxson for merging #20763

[HannesStrohkopp]

+1 on this.

Can confirm it: In router mode, the server correctly switches between models in idle state.
However, when model B is requested concurrently to the generation for model A, the model B request is not queued but model B is concurrently loaded, violating models-max = 1.

This is actually very annoying in production, curently preventing me from offering multiple models at all. Help would therefore be greatly appreciated! :)