Feature request: "autoload" model in router mode

[elfarolab]

Name and Version

Build version 7389

built on Jetson AGX Orin dev kit

Tegra release:

R36 (release), REVISION: 4.7, GCID: 42132812, BOARD: generic, EABI: aarch64, DATE: Thu Sep 18 22:54:44 UTC 2025

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

/opt/llama.cpp/bin/llama-server --models-dir /opt/llama-models --models-autoload --models-max 0 --models-preset /opt/llama.cpp/etc/models.ini --host 127.0.0.1 --port 8087 --threads-http 4 --parallel 2 --no-webui --flash-attn on

Problem description & steps to reproduce

I am trying to run llama-server with router feature enabled but the models are not autoloaded by default.
Using the integrated webui, if selecting the model to load, the model indeed get loaded, it is working fine.
I am not using the integrated webui, instead, I am developing a custom client.

Despite using configuration options of server:
--models-dir /opt/llama-models
--models-autoload
--models-max 0
--models-preset /opt/llama.cpp/etc/models.ini

models are not autoloaded.
Documentation says they should be autoloaded by default.

My models.ini is like this:

"""
version = 1

[Qwen3VL-8B-Instruct-Q8_0]
c = 65536
threads = 8
jinja = true
n-gpu-layers = 99
flash-attn = on
temp = 0.7
top-p = 0.8
top-k = 20
min-p = 0.0
presence-penalty = 1.5
batch-size = 2048
ubatch-size = 2048

[bartowski_nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF_nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q6_K]
temp = 0.7
top-p = 0.8
top-k = 20
min-p = 0.0
presence-penalty = 1.5
batch-size = 2048
ubatch-size = 2048
"""

In server.cpp at LOC 189 there is:
"""
//
// Start the server
//

std::function<void()> clean_up;

if (is_router_server) {
    LOG_INF("%s: starting router server, no model will be loaded in this process\n", __func__);

"""

it looks like the behaviour of not automatically loading the available models is hardcoded into the source.
So, do I am missing anything or maybe I've got it totally wrong?
Is this option: --models-autoload supposed to auload (by default) the models?

Thank you so much to everybody, great work!

First Bad Commit

No response

Relevant log output

Dec 14 17:02:54 xyz llama-server[2487]: Full command:
Dec 14 17:02:54 xyz llama-server[2487]: /opt/llama.cpp/bin/llama-server --models-dir /opt/llama-models --models-autoload --models-max 0 --models-preset /opt/llama.cpp/etc/models.ini --host 127.0.0.1 --port 8087 --threads-http 4 --parallel 2 --no-webui --flash-attn on
Dec 14 17:02:54 xyz llama-server[2487]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Dec 14 17:02:54 xyz llama-server[2487]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: yes
Dec 14 17:02:54 xyz llama-server[2487]: ggml_cuda_init: found 1 CUDA devices:
Dec 14 17:02:54 xyz llama-server[2487]:   Device 0: Orin, compute capability 8.7, VMM: yes
Dec 14 17:02:54 xyz llama-server[2487]: build: 7389 (c00ff929d) with GNU 11.4.0 for Linux aarch64
Dec 14 17:02:54 xyz llama-server[2487]: system info: n_threads = 12, n_threads_batch = 12, total_threads = 12
Dec 14 17:02:54 xyz llama-server[2487]: system_info: n_threads = 12 (n_threads_batch = 12) / 12 | CUDA : ARCHS = 870 | FORCE_CUBLAS = 1 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
Dec 14 17:02:54 xyz llama-server[2487]: init: using 4 threads for HTTP server
Dec 14 17:02:54 xyz llama-server[2487]: Web UI is disabled
Dec 14 17:02:54 xyz llama-server[2487]: srv  server_prese: Loaded 3 presets from /opt/llama.cpp/etc/models.ini
Dec 14 17:02:54 xyz llama-server[2487]: srv   load_models: Available models (2) (*: custom preset)
Dec 14 17:02:54 xyz llama-server[2487]: srv   load_models:   * Qwen3VL-8B-Instruct-Q8_0
Dec 14 17:02:54 xyz llama-server[2487]: srv   load_models:   * bartowski_nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF_nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q6_K
Dec 14 17:02:54 xyz llama-server[2487]: main: starting router server, no model will be loaded in this process
Dec 14 17:02:54 xyz llama-server[2487]: start: binding port with default address family
Dec 14 17:02:54 xyz llama-server[2487]: main: router server is listening on http://127.0.0.1:8087
Dec 14 17:02:54 xyz llama-server[2487]: main: NOTE: router mode is experimental
Dec 14 17:02:54 xyz llama-server[2487]: main:       it is not recommended to use this mode in untrusted environments

[elfarolab]

curl -v http://localhost:8087/models | jq

• Trying 127.0.0.1:8087...
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 8087 (#0)

GET /models HTTP/1.1
Host: localhost:8087
User-Agent: curl/7.81.0
Accept: /

• Mark bundle as not supporting multiuse
  < HTTP/1.1 200 OK
  < Keep-Alive: timeout=5, max=100
  < Content-Type: application/json; charset=utf-8
  < Server: llama.cpp
  < Content-Length: 2065
  < Access-Control-Allow-Origin:
  <
  { [2065 bytes data]
  100 2065 100 2065 0 0 2085k 0 --:--:-- --:--:-- --:--:-- 2016k
• Connection #0 to host localhost left intact
  {
  "data": [
  {
  "id": "Qwen3VL-8B-Instruct-Q8_0",
  "object": "model",
  "owned_by": "llamacpp",
  "created": 1765729854,
  "in_cache": false,
  "path": "/opt/llama-models/Qwen3VL-8B-Instruct-Q8_0/Qwen3VL-8B-Instruct-Q8_0.gguf",
  "status": {
  "value": "unloaded",
  "args": [
  "/opt/llama.cpp/bin/llama-server",
  "--host",
  "127.0.0.1",
  "--jinja",
  "--min-p",
  "0.0",
  "--port",
  "0",
  "--presence-penalty",
  "1.5",
  "--temp",
  "0.7",
  "--threads-http",
  "4",
  "--top-k",
  "20",
  "--top-p",
  "0.8",
  "--alias",
  "Qwen3VL-8B-Instruct-Q8_0",
  "--batch-size",
  "2048",
  "--ctx-size",
  "65536",
  "--flash-attn",
  "on",
  "--model",
  "/opt/llama-models/Qwen3VL-8B-Instruct-Q8_0/Qwen3VL-8B-Instruct-Q8_0.gguf",
  "--mmproj",
  "/opt/llama-models/Qwen3VL-8B-Instruct-Q8_0/mmproj-Qwen3VL-8B-Instruct-Q8_0.gguf",
  "--n-gpu-layers",
  "99",
  "--threads",
  "8",
  "--ubatch-size",
  "2048"
  ],
  "preset": "[Qwen3VL-8B-Instruct-Q8_0]\njinja = true\nmin-p = 0.0\noffline = false\npresence-penalty = 1.5\ntemp = 0.7\ntop-k = 20\ntop-p = 0.8\nbatch-size = 2048\nctx-size = 65536\nflash-attn = on\nn-gpu-layers = 99\nthreads = 8\nubatch-size = 2048\n\n"
  }
  },
  {
  "id": "bartowski_nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF_nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q6_K",
  "object": "model",
  "owned_by": "llamacpp",
  "created": 1765729854,
  "in_cache": false,
  "path": "/opt/llama-models/bartowski_nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF_nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q6_K.gguf",
  "status": {
  "value": "unloaded",
  "args": [
  "/opt/llama.cpp/bin/llama-server",
  "--host",
  "127.0.0.1",
  "--min-p",
  "0.0",
  "--port",
  "0",
  "--presence-penalty",
  "1.5",
  "--temp",
  "0.7",
  "--threads-http",
  "4",
  "--top-k",
  "20",
  "--top-p",
  "0.8",
  "--alias",
  "bartowski_nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF_nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q6_K",
  "--batch-size",
  "2048",
  "--flash-attn",
  "on",
  "--model",
  "/opt/llama-models/bartowski_nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF_nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q6_K.gguf",
  "--ubatch-size",
  "2048"
  ],
  "preset": "[bartowski_nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF_nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q6_K]\nmin-p = 0.0\npresence-penalty = 1.5\ntemp = 0.7\ntop-k = 20\ntop-p = 0.8\nbatch-size = 2048\nubatch-size = 2048\n\n"
  }
  }
  ],
  "object": "list"
  }

[ServeurpersoCom]

@elfarolab The behavior you're seeing is actually correct. "--models-autoload" doesn't mean "load all models at startup": it means "automatically load a model when a request for it arrives".

The router starts with all models in "unloaded" state. When you send a request to a specific model (via /v1/chat/completions for example), the router will automatically spawn the child process and load that model.

Try making a completion request to one of your models and you'll see it switch from "unloaded" to "loaded" in the /models endpoint.

This is by design: the router keeps models unloaded until they're actually needed, then manages loading/unloading based on --models-max (which is unlimited in your case with 0).

[elfarolab]

Hello, I appreciate your help.

It would be useful having an option to autoload all defined models form .ini file at startup.
I am developing a system with LLM+ASR+TTS and a custom client.
The autoloading of models would save time during user interactions.
New router feature is better of having 3 instances running of llama-server on different ports.
It would be a cleaner architecture.

I could also help with that, interested in a pull request with teh new option for llama-server?

[ServeurpersoCom]

@elfarolab I think that's a good idea. An option to preload all models defined in the .ini file at startup would be useful for larger infrastructures or low-latency use cases like yours! Feel free to open a PR.
cc @ngxson

[elfarolab]

@ServeurpersoCom
@ngxson

I'm preparing a PR for this feature and would appreciate your feedback.

Currently, "autoload" is used throughout the code, logic, and configs to mean making a model available for loading on-demand when requested.

In my opinion, this could cause confusion with preloading at startup. I'd like to introduce a new .ini parameter: preload-model = true to explicitly indicate that the model should be loaded immediately on start, without waiting for a trigger.

Changing the existing "autoload" term would be difficult due to its deep integration.

Do you agree with using "preload-model" for this boolean? Any alternative suggestions or thoughts?

Thanks!

[ServeurpersoCom]

On llama-swap there were different terms used :

hooks:
  on_startup:
    preload:
      - Dense-Llama-3_3-Nemotron-Super-49B-v1_5

preload-model look good, alternative : startup-model ? autostart-model ?

Being able to launch one or more models when the router starts up is not a bad idea.

[elfarolab]

maybe just "autostart = true" ?
It seems to me more intuitive. So we differentiate between starting and loading.
Thank you for the hint!

[ServeurpersoCom]

Not a problem because there is an abstraction layer for bool :
true = on = 1
false = off = 0

bool common_arg_utils::is_truthy(const std::string & value) {
    return value == "on" || value == "enabled" || value == "1";
}

bool common_arg_utils::is_falsey(const std::string & value) {
    return value == "off" || value == "disabled" || value == "0";
}

bool common_arg_utils::is_autoy(const std::string & value) {
    return value == "auto" || value == "-1";
}

[laurentperez]

Hi @elfarolab . Interested in seeing your PR if you're planning one.

Curious on how you'd autostart whenever --models-max=1. Which one would start first ?

[elfarolab]

Hello Laurent, hello everyone!

I'm currently working on it. I've already parsed the relevant source code files to implement the changes. This feature is essential for a system I'm building. I truly appreciate your interest. I closed the post because I prefer to avoid unnecessary noise, and I'll submit a pull request once I have something ready.

[elfarolab]

You raise a valid and insightful point. If 'autostart = true' is set for certain models, it would implies the user intends for those models to run continuously. In that case, enforcing '--models-max=1' could create a conflict, as it would limit the number of models regardless of the user's intent.

It makes sense to treat 'autostart = true' as a higher-priority signal meaning '--models-max=1' should either be ignored or overridden in such cases. This ensures the system respects the user’s explicit configuration while maintaining logical consistency.

Perhaps a better approach would be to:

Check for 'autostart = true' first.
If any model has this flag, ignore '--models-max' or dynamically adjust it to accommodate at least those models.
Alternatively, log a warning if '--models-max=1' is set but conflicting autostart models are detected.

Any better idea? thx

Hi @elfarolab . Interested in seeing your PR if you're planning one.

Curious on how you'd autostart whenever --models-max=1. Which one would start first ?