Upstream: ggml-org#28698
With router --models-max 1, a queued request can evict a model immediately after it loads but before the request that loaded it is proxied.
Environment: Linux aarch64, NVIDIA GB10 / CUDA 13, current upstream df03399b885831b2a1603b3abb0d8c156808e363; Qwen2.5 1.5B and 0.5B Q4_K_M.
llama-server --models-preset models.ini --models-max 1
From a cold router, send /tokenize for model A, then 100 ms later send /tokenize for model B. Both presets use local GGUF files.
Observed: model A loads, tick() immediately logs evicting idle LRU name=model-a for a queued request, and model A's original request returns HTTP 500 proxy error: Could not establish connection; model B succeeds.
The loaded child is briefly considered idle before proxy_request() accounts for the waiting request. Upstream identifies f2dc72b8667ad71408c7f567c4adfc4bef2dc9d1 as first bad.
Upstream: ggml-org#28698
With router
--models-max 1, a queued request can evict a model immediately after it loads but before the request that loaded it is proxied.Environment: Linux aarch64, NVIDIA GB10 / CUDA 13, current upstream
df03399b885831b2a1603b3abb0d8c156808e363; Qwen2.5 1.5B and 0.5B Q4_K_M.From a cold router, send
/tokenizefor model A, then 100 ms later send/tokenizefor model B. Both presets use local GGUF files.Observed: model A loads,
tick()immediately logsevicting idle LRU name=model-a for a queued request, and model A's original request returns HTTP 500proxy error: Could not establish connection; model B succeeds.The loaded child is briefly considered idle before
proxy_request()accounts for the waiting request. Upstream identifiesf2dc72b8667ad71408c7f567c4adfc4bef2dc9d1as first bad.