Strip parameters out of requests to preserve default server values

[mostlygeek]

When swapping models chat clients may not have the recommended params for temp, top_p, etc for a specific model. Many inference back ends allow defaults values to be set but not protected from being overridden.

llama-swap can do a bit more here by stripping out parameters in the JSON request to v1/chat/completions. If the parameters exist in the request they are deleted.

For example in llama-server there are many params that could be deleted from a request.

What the design would look like is:

macros:
    DRY_params: "dry_multipler, dry_base, dry_allowed_length, dry_penalty_max_n"
models:
    mymodel: 
        cmd: ... 
        filters: 
            strip_params: "temperature, top_k, min_p, ${DRY_params}"

In this design:

• a filters map is introduced. Defaults to an empty map
• the strip_params is a comma delimited string. This makes it easy to substitute in macros. Defaults to "" (empty string)

[bjodah]

Hi @mostlygeek, I like your idea of offering customization to model parameters in llama-swaps's config. May I suggest to make it even more powerful? Consider e.g. Qwen3: they recommend different parameters depending on /think vs. /no_think. I would like to be able to use either set of parameters without reloading the model. To that end, we cannot rely on a static set of parameters that the server was a launched with, instead we would need weak/strong overrides of parameters, and have more than one alias mapping to the same entry in llama-swap.

I hacked together my own middleware proxy by heavily patching an open-source python proxy to sit in front of llama-swap for this purpose, so I can specify e.g. {"model": "Qwen/Qwen3-30B-A3B@no-think"} or {"model": "Qwen/Qwen3-30B-A3B@think"} and it will adapt parameters without restarting the underlying server process. I opted for a user-defined special character, in this case @. If you're curious my config looks like this:

• https://github.com/bjodah/cot_proxy/blob/1b9ea5af06cea73e0575ec47f2dd26ee0d56e1e1/examples/cot_proxy.yaml

My approach also injects /no_think or /think, but I can also envision that e.g. a user defined regex looking for those strings by matching e.g. a regex to the message content could be used to trigger (weak or strong) parameter overrides..

I guess you have some ideas of your own, I just thought I should mention something that "works for me". Anyways, I deeply appreciate your continued work on llama-swap.

[mostlygeek]

This is pretty cool. I think for heavily custom rewriting of requests like this your approach is currently the best way. Have a proxy that sits in front of llama-swap.

The thinking behind the new filters is to support some light weight changes that many people would benefit from and upstream inference servers don’t support.

[bjodah]

Yeah, I can definitely see that this is a slippery slope. Just wanted to mention it in case it would be useful for inspiration.