Misc. bug: Multi-model router does not support slot save/restore operations

[eleqtrizit]

Name and Version

Environment

• llama.cpp version: version: 7653 (193ee38)
• Operating System: Ubuntu 24 Linux
• Build: Standard build with CUDA support

Description

When running llama-server with --models-preset (multi-model router mode), the slot save/restore endpoints return 400 Bad Request with error message "Invalid action", even though the server is started with --slot-save-path.

Expected Behavior

The slot save/restore operations should work with multi-model router mode, just as they do with single-model mode.

Actual Behavior

The router's proxy_post handler (in tools/server/server-models.cpp:828) intercepts the request and attempts to proxy it to the individual model server, but the slot-specific endpoints are not handled correctly by the proxy logic.

Root Cause

In tools/server/server.cpp:155, when --models-preset is used:

routes.post_slots = models_routes->proxy_post;

The proxy_post handler expects standard request patterns but doesn't properly handle the slot save/restore operations which require:

• Path parameter :id_slot
• Query parameter ?action=save|restore|erase
• Special handling that differs from standard model inference requests

Workaround

Connect directly to the individual model server ports instead of the router port. The individual model servers support slot save/restore correctly.

Suggested Fix

The multi-model router should either:

1. Pass through slot operations directly to the target model server without modification
2. Implement proper slot save/restore handling in the router itself
3. Document that slot operations are not supported in multi-model mode

Additional Context

The slot save/restore functionality works correctly when running llama-server in single-model mode (without --models-preset). The issue only affects the multi-model router mode introduced for managing multiple models from a single endpoint.

Operating systems

No response

Which llama.cpp modules do you know to be affected?

No response

Command line

llama-server --models-preset models.ini --host 0.0.0.0 --port 8081 --models-max 1 --slot-save-path /tmp/kvcache --swa-full -np 2

Problem description & steps to reproduce

Steps to Reproduce

1. Start llama-server with multi-model router:

llama-server --models-preset /path/to/models.ini --host 0.0.0.0 --port 8081 --models-max 1 --slot-save-path /tmp/kvcache --swa-full -np 2

2. Attempt to save a slot:

❯ curl -X POST 'http://127.0.0.1:8081/slots/0?action=save' \
  -H 'Content-Type: application/json' \
  -d '{"filename": "test_save", "model": "ggml-org/gpt-oss-20b-GGUF"}'
{"error":{"code":400,"message":"Invalid action","type":"invalid_request_error"}}%   

3. Connect directly to llama-server behind the proxy:

❯ curl -X POST 'http://127.0.0.1:42515/slots/0?action=save' \
  -H 'Content-Type: application/json' \
  -d '{"filename": "test_save", "model": "ggml-org/gpt-oss-20b-GGUF"}'
{"id_slot":0,"filename":"test_save","n_saved":0,"n_written":36,"timings":{"save_ms":0.399}}%  

First Bad Commit

No response

Relevant log output

No response

[riven8192]

Regardless of the issue with the router, and its handling of query-parameters, I think there is a broader problem with the API.

This is how to we currently feed in data:

• id_slot: path parameter
• action: query parameter
• filename: json object attribute
• model: json object attribute (requirement added recently)

I'd much rather have a consistent way to pass parameters, for example, switching purely to a JSON payload.

POST /slots/save

{
   "id_slot": <int>,
   "filename": <string>,
   "model": <string>,
}

or even:

POST /slots

{
   "id_slot": <int>,
   "action": <string>,
   "filename": <string>,
   "model": <string>,
}

This cleans up the API, and 'accidentally' works around the issue with the router -- which could be fixed, but an API improvement would feel even better.

As a workaround for this specific bug, I patched tools/server/server-context.cpp in a local build, with:

std::string action = req.get_param("action");

++ // Router mode currently loses query params when proxying.
++ // Fallback: allow {"action": "..."} in request JSON.
++ if (action.empty() && !req.body.empty()) {
++     try {
++         const json request_data = json::parse(req.body);
++         if (request_data.contains("action")) {
++             action = request_data.at("action").get<std::string>();
++         }
++     } catch (...) {
++         // ignore parse errors; will fail below with "Invalid action"
++     }
++ }

if (action == "save") {
    return handle_slots_save(req, id_slot);
} else if (action == "restore") {
    return handle_slots_restore(req, id_slot);
} else if (action == "erase") {
    return handle_slots_erase(req, id_slot);
} else {
    res->error(format_error_response("Invalid action", ERROR_TYPE_INVALID_REQUEST));
    return res;
}

Naturally, I had to tweak the call-site to put the action in the json-payload.

[ServeurpersoCom]

This patch fixes the root cause at the proxy level by reconstructing the full path with query parameters before forwarding to the child server, so any current or future endpoint relying on query params through the router will work without endpoint-specific workarounds
Feel free to give it a try :)

[eleqtrizit]

Thanks ServeurpersoCom!