Misc. bug: Llama Cpp - Model - Chat Template interactions = Mess (Devstral 2)

[ju6ge]

So I have been successfully running llama-cpp together with a quantized version of Devstral-Small-2 and things are going quite alright. Though I did have to make some adjustments to get here:

1. The default chat template from mistral seems to enforce a user -> assistant -> user -> assistant pattern, this is sub optimal because my main interest in this setup is using opencode. But that fails fast, so I have adjusted the jinja template to remove that limitation. With that in place opencode happily accepts my api endpoint and starts buzzing along.
2. I am also running OpenWebUI pointed to my llama-server and things work okay, but I started noticing that thinking output seems not work as expected.

The second part is what led me down a rabbit hole of trying to get thinking output to work …

I am going to stick to my example with Devstral 2 but I think some of the issues I am describing will be applicable to other models and combinations there of.

So according to the the chat template [THINK] and [/THINK] should be the tags with which to wrap the thinking output

But in my testing I did not get this to work, It only started working after I manually changed the tags to use <think> and </think> (patching the code in llama-cpp to use different prefixes) … I suspect this is related to me using a quantized version of the model that was further fine tuned and may with the training set containing different thinking tags than the original tags the model should be using. Another thing I noticed is that when trying to use [THINK] the model would often generate [TOOL_CALL] instead. So the issue might also be that the tags are quite similar and [TOOL_CALL] is has a higher probability of being selected.

But here is the thing: I can't solve this without patching llama-cpp and I think this is the biggest issue. Open Weights models are distrubuted and repackaged by different organizations with different needs and it is simply impossible for llama-cpp to provide a chat template parser for every combination under the sun. The current implementation of chat template handling simply involves way to much hard coding.

Note: There is also a lot of code in the chat templates part that has no explanation what so ever and chat templates are widely different in their formatting making it hard to reason about if it would even be feasible to get them to behave similarly enough in order to add more control to the user. For example common_chat_templates_support_enable_thinking seems to be used to check if a chat template supports thinking mode, but there is simply not enough information in order for me to understand how to get a chat template and corresponding parser to correctly work with this!

So with my current setup I am stuck accepting either that thinking is not going to work, or that I will have run a forked version, because upstreaming yet another special case to the current state of affairs will not improve the overall situation any bit … But given that this is a complex issue with many moving parts I don't know any easy answer.

At the very least I hope this report helps other people facing similar issues …

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

Problem description & steps to reproduce

See description

First Bad Commit

No response

Relevant log output

No response

[pwilkin]

Please try running on #18675 and tell me if it works correctly there.

[ju6ge]

Please try running on #18675 and tell me if it works correctly there.

Oh that PR sounds very nice indeed and exactly what I am looking for, I will give your PR a shot. Props for pointing me to it!

[aldehir]

Hi, can you point me to the docs that state Devstral 2 is a reasoning model? I am under the impression it is not.

[ju6ge]

Hi, can you point me to the docs that state Devstral 2 is a reasoning model? I am under the impression it is not.

It is not explicitly stated anywhere as far as I can tell but it does have [THINK] tokens in it's tokenset. Make of that what you want … I get that if it was not specifically trained for reasoning then the probability that it will generate reasoning tokens is lower. But does it really matter all that much? Ultimately reasoning is just as much an illusion as the user assistant interaction is. Just a trick to mark certain sections of the output and serve them differently. And then again open weight models might have been post trained / finetuned on a dataset that includes reasoning. Given this complexity I find the current way chat templates are working very limiting …

[aldehir]

It is not explicitly stated anywhere as far as I can tell but it does have [THINK] tokens in it's tokenset. Make of that what you want

This is because it shares the same tokenizer as Ministral 3, which has a reasoning variant. I assume it's easier to share a tokenizer can create a custom one for both.

Just a trick to mark certain sections of the output and serve them differently. And then again open weight models might have been post trained / finetuned on a dataset that includes reasoning.

I think you just answered your own question here. Without insight in their training, you cannot make such assumptions. Presumably, however, the non-reasoning variants were not trained to use the special think tokens.

Lastly, the Ministral 3 reasoning models are trained to only produce a CoT with their specialized system prompt.

[ju6ge]

This is because it shares the same tokenizer as Ministral 3, which has a reasoning variant. I assume it's easier to share a tokenizer can create a custom one for both.

Probably they are also derived from some common check point …

So let's skip guesswork. By experimenting with the derived model I am using I noticed I can get it to generate thinking tags, just that those thinking tags do not look like the ones form the original model. But I can not just modify the chat template or use some cmd parameters to make llama cpp aware of these.

My point is that the current approach of making a lot of assumptions based on hardcoded "heuristics" of what the the parts of the chat template contain is fundamentally flawed. In a world where a model might have been derived from another these heuristics will just not hold up. Post Trained / Finetuned and Quantized models do not behave like the original anymore even though they might share properties in the chat template.

[aldehir]

My point is that the current approach of making a lot of assumptions based on hardcoded "heuristics" of what the the parts of the chat template contain is fundamentally flawed.

Disagree. Parsers are inherently coupled to the model its chat template. Overly aggressive generalizations lead to poor model performance all around. This is already seen in some existing parsers, such as the generic parser.

In a world where a model might have been derived from another these heuristics will just not hold up. Post Trained / Finetuned and Quantized models do not behave like the original anymore even though they might share properties in the chat template.

Then they need a dedicated parser to match.

You can either fork and implement your own parser, or you can write a piece of middleware that uses the /completions endpoint to circumvent the builtin parsing.

Alternatively, the autoparser branch will adapt to most common output formats.

[ju6ge]

Alternatively, the autoparser branch will adapt to most common output formats.

I am already test driving that ;) and it does seem to look like the solution I am looking for.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.