Dynamically change sampling parameters when switching between thinking and non-thinking modes?

[dagbdagb]

Seeing how Qwen recommends different sampling parameters for thinking and non-thinking, I wonder what the approach is here.

The UI llama.cpp UI permits disabling thinking (for models and templates supporting that). But I only see how to set the sampling parameters at startup.

These are the suggestede parameters for Qwen 3.8 27B:

- Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

- Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Also, i still wonder about SPLIT_MODE_TENSOR and backend sampling. Is it fundamentally impossible to do, or simply not done yet?

[MLuc24]

The startup flags are only defaults — sampling is per-request, so nothing stops you sending different values for the two modes.

/completion takes them in the body (tools/server/README.md):

temperature: Adjust the randomness of the generated text. Default: 0.8
top_p: Limit the next token selection to a subset of tokens with a cumulative probability above a threshold P. Default: 0.95
presence_penalty: Repeat alpha presence penalty. Default: 0.0, which is disabled.

and the OpenAI-compatible endpoints accept the same, so any client that lets you save two presets gives you exactly what Qwen recommends without restarting anything.

What doesn't exist is a runtime switch on the server side. POST /props is documented as taking no options at all:

### POST `/props`: Change server global properties.
To use this endpoint with POST method, you need to start server with `--props`
*Options:*
- None yet

so you can't flip the server's defaults between requests either.

And there's no link between the thinking toggle and sampling — the toggle changes the template, nothing reaches the sampler. Whatever the UI has configured stays put when you flip it.

For the built-in UI, settings can at least be seeded rather than clicked in each time:

--ui-config, --webui-config JSON        JSON that provides default UI settings (overrides UI defaults)
--ui-config-file, --webui-config-file PATH

That gets you one profile on startup, not two on demand. If you switch modes often, a client with two presets pointed at the same server is less friction than editing settings each time — one server, two saved parameter sets, pick per conversation.

[dagbdagb]

In other words..... the llama.cpp web UI could hold two (or more...) sets of sampling parameters and be made to switch between them depending on the state of the thinking/non-thinking state? And so could any other client speaking the OpenAI API?

Is my interpretation correct?

Spinning on this.... :
Is a mechanism where the model itself deduces the likely optimal set of sampling parameters based on query, context and knowledge about itself theoretically feasible?

[MLuc24]

Correct on both counts. Nothing about the two-profile idea is blocked by the server — every sampler value is per-request, so a client holding two sets and picking one based on its own thinking toggle is just sending different JSON. Any OpenAI-API client can do it today; the built-in UI could, it simply doesn't yet.

On the second question — a model deducing its own sampling parameters — the static half already exists, and it's in llama.cpp.

GGUF has dedicated sampling metadata (include/llama.h):

enum llama_model_meta_key {
    LLAMA_MODEL_META_KEY_SAMPLING_SEQUENCE,
    LLAMA_MODEL_META_KEY_SAMPLING_TOP_K,
    LLAMA_MODEL_META_KEY_SAMPLING_TOP_P,
    LLAMA_MODEL_META_KEY_SAMPLING_MIN_P,
    LLAMA_MODEL_META_KEY_SAMPLING_XTC_PROBABILITY,
    LLAMA_MODEL_META_KEY_SAMPLING_XTC_THRESHOLD,
    LLAMA_MODEL_META_KEY_SAMPLING_TEMP,
    LLAMA_MODEL_META_KEY_SAMPLING_PENALTY_LAST_N,
    LLAMA_MODEL_META_KEY_SAMPLING_PENALTY_REPEAT,
    LLAMA_MODEL_META_KEY_SAMPLING_MIROSTAT,
    LLAMA_MODEL_META_KEY_SAMPLING_MIROSTAT_TAU,
    LLAMA_MODEL_META_KEY_SAMPLING_MIROSTAT_ETA,
};

with a public accessor, llama_model_meta_key_str, and the keys resolving to general.sampling.* in src/llama-model.cpp. So a quantizer can bake "this is how I should be sampled" into the file, and a client can read it back rather than hardcoding a table of per-model recommendations. That's the mechanism your Qwen numbers would ideally travel in.

What it can't express is your case: one value per key for the whole model, with no notion of a mode. A thinking/non-thinking split would need either two namespaces or a per-template override, and neither exists.

The dynamic version — the model deciding per query — runs into an ordering problem rather than an engineering one. Sampling happens after the forward pass, on the logits; the model has no channel to the sampler that is generating it. It could say which parameters it wants, but saying anything already requires sampling, so the first tokens come out under whatever settings you guessed. You'd end up with a cheap classifier pass to pick a profile, then a second request to answer under it — which is a real design, just not the model configuring itself so much as something else configuring it on the model's advice.

[dagbdagb]

Thank you for sharing your insight. Much appreciated. I absolutely love bouncing these kinds of ideas on people who are able and willing to answer. Apologies if I am abusing your kindness.

That 'cheap classifier pass' could be the model itself (or, technically, a completely different model) reasoning about the question (and context, and itself). And then it could autogenerate a request with the deduced sampling params to reason about the answer.

Reasoning about a question certainly needs sampling parameters, but is fairly narrow task: "Is the question about code, trivia, prose, live facts (weather, stock market)....? Is it a follow-up, do we have context, etc. Are there implied premises?" And so on.

A fixed set of sampling parameters for reasoning about the question is not inherently bad. I think?

[MLuc24]

Not abusing anything — this is a better use of a discussion thread than most.

Fixed parameters for the classifier pass aren't just acceptable, they're the right choice. Classification wants the mode of the distribution, not a sample from it: you're asking "which of these buckets", and creativity there is pure downside. Near-greedy — temperature at or near 0, top_k 1 — is what you want, and it makes that pass reproducible, which matters when you're debugging why the second request got the parameters it did.

The part worth knowing is that llama.cpp can make that pass a lot more reliable than "ask nicely and parse". The server takes a grammar per request:

grammar: Set grammar for grammar-based sampling. Default: no grammar
json_schema: Set a JSON schema for grammar-based sampling (e.g. {"items": {"type": "string"}, ...}). Default: no JSON schema.

Constrain the classifier to a schema whose only field is an enum of your profile names and the output can't be anything else — no prose, no hedging, no parse step. At that point the sampling question for pass one mostly dissolves: a grammar that admits three tokens doesn't leave temperature much room to matter.

Two practical notes for the two-pass shape:

cache_prompt defaults to true, so the second request reuses the KV cache for the shared prefix rather than reprocessing the question — the classifier costs you a short generation, not a second prompt ingest.

And you can skip generation entirely if the buckets are simple enough. n_probs returns the top-N token probabilities, so a one-token classifier plus a peek at the distribution tells you both the choice and how confident it was — useful for falling back to a default profile when the model is genuinely unsure rather than committing to a coin flip.

The honest limit is still that pass one is guessing from the question alone. "Is this about code" is answerable up front; "will this answer need to be creative" often isn't knowable until you're partway into writing it.