Towards Reconfigurable llama-server Runtimes

[wadealexc]

llama.cpp's server runtime maintains two different kinds of state:

Main model state

• weights
• tokenizer
• architecture

Auxiliary/Inference state

• kv cache
• mmproj
• speculative decoder
• (+ configuration for each of these)

Today, these are initialized and destroyed together. Loading a model means loading not just its weights, but also an mmproj, a speculative decoder, and a kv cache.

Problem: Inference Configuration on Model Load

Model and inference parameters can both be configured via CLI arguments, but this only happens at load time. Changing any configuration requires fully killing the process and restarting: removing model weights and inference state, then reinitializing all of it. For example, adding an mmproj to a model after it's been loaded:

graph LR
    User["User"]

    subgraph ServerA["llama-server"]
        A1["Read model from disk"]
        A2["Init kv cache"]

        A1 --> A2
    end
    
    User -- "1: load model" --> ServerA
    
    subgraph ServerB["llama-server"]
        B1["Read model from disk"]
        B2["Init kv cache"]
        B3["Init mmproj"]
        

        B1 --> B2
        B2 --> B3
    end
    
    User -- "2: kill process" --> ServerA
    User -- "3: load model + mmproj" --> ServerB

Loading

This pattern treats inference configuration as a decision to be made infrequently, as it has a significant time penalty attached. However, runtime requirements often change over the lifetime of a model.

As an example, here are some scenarios that might cause an inference backend to reconfigure the models it's serving:

• first image arrives in a conversation
• second user starts session
  • speculative decoding no longer useful due to inability to parallelize
• server has more available memory
  • long-context session goes inactive
  • sidecar/task model no longer needed
• server becomes memory constrained
  • sidecar/task model needed
  • conversation exceeds a certain length
  • speculative decoding is no longer worth the memory
  • kv cache quantization becomes worthwhile

Configure-on-load pushes configuration decisions to account for the "worst case" runtime requirements of all of these scenarios, making the average case worse.

Proposal: Runtime Configuration

In general, runtime inference configuration means reconstructing/reconfiguring a model's runtime context while its weights stay put (or without fully restarting). This can take many forms, but a few options I've been thinking of are:

• Load/unload mmproj
• Load/unload speculative decoder
• Shrink/grow context window
• Reconfigure kv cache precision
• Move models/layers between devices

While working on llama-server, I noticed that most of the ideas I had in this vein shared the same underlying limitation: model weight and mmproj/mtp/config lifetimes are tightly coupled. This coupling is the primary architectural limitation preventing runtime configuration from being implemented, and addressing it would require refactoring llama-server somewhat.

Additionally, I imagine exposing these runtime operations via HTTP endpoints, as that feels like the natural way to interact with them given how the built-in router mode (and other llama.cpp management libraries) work.

Example Use Case: Serving Multiple Models

By exposing these and similar methods for granular inference management, llama-server significantly enhances the model-serving usecases it already supports. It especially becomes easier to serve small task models alongside a single larger model, as memory requirements are more flexible.

For example, consider a chat application where a user talks to a single larger model, which is set up alongside a smaller model used for various agentic tasks.

For ease of visualization, let's assume we're loading both onto a GPU with 24 GB of capacity. There's a cost associated with each loaded component, deducted from our capacity:

• main model: 10 GB
• task model: 5 GB
• 10k context: 1 GB
• mmproj: 1 GB
• mtp: 1 GB

Both are initialized at startup with some basic capabilities, and as the user chats, the main model's context expands into available, unallocated space:

---
title: "llama-server (GPU: 20/24 GB)"
---
graph LR
    subgraph Main["Main (GPU: 15 GB)"]
        A1["Main model<br>+ mtp<br>+ mmproj"]
        A2["30k context"]
        
        A1 --- A2
    end
    
    subgraph Task["Task (GPU: 5 GB)"]
        B1["Task model"]
        B2["Empty context"]
        
        B1 --- B2
    end
    
    Main ~~~ Task

Loading

A request comes in that requires the task model to summarize results from a web search. This requires the task model's context window to expand to handle 50k context. 4 GB are already available, and the additional 1 GB is freed by unloading the main model's mmproj, which has not been used in the current session:

---
title: "llama-server (GPU: 24/24 GB)"
---
graph LR
    subgraph Main["Main (GPU: 14 GB)"]
        A1["Main model<br>+ mtp<br><span style='color:red'>- mmproj</span>"]
        A2["30k context"]
        
        A1 --- A2
    end
    
    subgraph Task["Task (GPU: 10 GB)"]
        B1["Task model"]
        B2["50k context"]
        
        B1 --- B2
    end
    
    Main ~~~ Task

Loading

As the task model completes its task, the bulk web page results can be cleared, and its entire context can be disposed. A summary of the results are sent to the main model, and its mmproj is restored:

---
title: "llama-server (GPU: 22/24 GB)"
---
graph LR
    subgraph Main["Main (GPU: 17 GB)"]
        A1["Main model<br>+ mtp<br><span style='color:green'>+ mmproj</span>"]
        A2["50k context"]
        
        A1 --- A2
    end
    
    subgraph Task["Task (GPU: 5 GB)"]
        B1["Task model"]
        B2["Empty context"]
        
        B1 --- B2
    end
    
    Main ~~~ Task

Loading

[EJainDev]

+1

I think this would significantly enhance the abilities of llama.cpp as well as enabling multi-modal deployments in VRAM constrained environments as models can switch devices using RAM as necessary. Although --fit does this, fine-grained control would allow the front end to control which model gets VRAM.