Translate reasoning-effort field for clients that can't vary the model ID (Open WebUI, OpenCode)

[asf0]

Clients like Open WebUI and OpenCode expose a reasoning-effort selector (e.g. reasoning_effort: "low"/"medium"/"high") as a plain request body field, sent alongside a fixed model value. There's currently no way to translate that into llama.cpp's native chat_template_kwargs.enable_thinking / thinking_budget_tokens fields.

The existing workaround (#584) requires the client to vary the model string itself per preset (e.g. model:low, model:med) via setParamsByID. That works for clients that let the user pick a model-ID variant, but Open WebUI/OpenCode don't — they keep model fixed and vary a separate effort field instead.

Request: a filter that maps a configurable input field (default reasoning_effort) to a set of presets, each preset setting chat_template_kwargs.enable_thinking / thinking_budget_tokens. Explicit client-supplied native fields should always win over the preset values. Since the model ID is never touched, switching effort levels shouldn't require unloading/reloading the model.

I have a PR up implementing this: #934.

[mostlygeek]

This thread has good info on why there is no universal way to make reasoning_effort a universal feature; the model has to be trained for it. Also, just setting reasoning tokens can degrade model performance.

The lcpp team is working on something; perhaps when that lands whatever llama-swap does will be obsolete.

[mostlygeek]

I looked through your PR and maybe we can go about this in a more generic way.

A filter that sets params by key and value would accomplish the same thing. Like setParamsById, it can set params by matching a key with a value.

I'm on my phone so excuse the formatting:

setParamsByMatch: 
  - key: reasoning_effort
    match: none
    set:
      a: b
  - key: reasoning_effort
    match: high
    set:
      chat_template_kwargs: ...

Something like that would be quite powerful for request time controls.

[asf0]

I rewrote the PR to use setParamsByMatch, as you suggested. The old reasoning filter is now gone.

I also moved the reasoning presets into a macro and imported them into each model:

macros:
  reasoning_rules:
    - key: reasoning_effort
      match: none
      set:
        chat_template_kwargs:
          enable_thinking: false

    - key: reasoning_effort
      match: high
      set:
        chat_template_kwargs:
          enable_thinking: true
          thinking_budget_tokens: 32768

models:
  "qwen3.6-27b":
    filters:
      setParamsByMatch: ${reasoning_rules}

For this to work, macros had to support lists, which they could not do before. Otherwise, I would need to duplicate the same list across all my models.

There is one remaining design decision regarding chat_template_kwargs. Currently, when a rule sets chat_template_kwargs, I merge it with the object already provided by the client. This preserves any additional keys sent by the client.

The alternative would be to replace the entire object, matching the behavior of setParams. However, that would discard any other keys already present in chat_template_kwargs. I am happy to switch to replacement behavior if you prefer consistency with setParams.

[tesurijp]

Do we really need to extend macros?
I think YAML's built-in merge key (<<: ) should be sufficient for what we want to achieve.

reasoning_rules: &_reasoning_rules
  filters:
    setParamsByMatch:
      - key: reasoning_effort
        match: none
        set:
          chat_template_kwargs:
            enable_thinking: false
      - key: reasoning_effort
        match: high
        set:
          chat_template_kwargs:
            enable_thinking: true
            thinking_budget_tokens: 32768

models:
  "qwen3.6-27b":
    <<: *_reasoning_rules

  "gemma-4-E2B":
    <<: *_reasoning_rules

Another concern with extending macros is that it could allow reusable fragments to be placed in parts of the configuration that are not representable or validatable by the Schema.

[asf0]

You're right that YAML anchors already work, and there are tests for them. I'm not trying to argue that structured macros are the only way to reuse these rules. I just think they solve a few cases that anchors do not.

The main one is reuse across files. Anchors only exist inside the YAML document where they are defined. With -config-dir, each file is parsed separately before everything is merged, so a model in one file cannot reference an anchor from another file. Global macros are merged before expansion, which means shared rules can live in one file and be used by models in other files.

They also allow model-specific overrides. A global macro can provide the default rules, and a model can redefine that macro locally, for example to use a smaller thinking budget. A model can select a different alias, but anchors do not provide the same global-default and model-local-override behavior under one name.

There is also some value in keeping reuse consistent. We already have global and model-level macros for scalar values. Supporting lists and maps through the same mechanism seems simpler than using one reuse system for scalars and YAML anchors for structured values.

The <<: approach also has the usual shallow merge behavior. If the model defines its own filters: mapping, it replaces the anchored mapping instead of merging inside it. Anchoring the setParamsByMatch list directly avoids that specific issue, but it still only works within one document and depends on declaration order.

I agree with the schema concern. JSON Schema cannot validate the contents hidden behind ${reasoning_rules}. I updated the schema so setParamsByMatch can be either the normal array or a whole-value macro reference, which removes the editor error for this case. We would need to add the same schema alternative if we want editor support for whole-value macros in other typed fields.

Runtime validation is unchanged. The macro has to replace the whole value, then the expanded result is decoded into []MatchRule, and every rule still goes through MatchRule.Validate(). Invalid rules will still fail when the config is loaded.

So I agree anchors cover the simple same-file case. I still think the small change is useful for cross-file configs, model-specific overrides, and keeping the reuse mechanism consistent.

[mostlygeek]

I appreciate the discussions on this thread. I started designing out a llama-swap custom DSL for filters that could replace all the YAML defined filters . Looking at the larger problem of modifying a request before it hits the LLM server it is clear that YAML is greatly lacking.

No promises on when it'd be ready but the goal is to be endgame for filtering. If it's not endgame, at least it'll provide a big step towards allowing users to customize and compose the behaviour they want.