I can't get Minimax-m2.7 (nor m2.5) to work properly with ik_llama.cpp (it does work fine in llama.cpp)

[glsup]

I run multiple models like deepseek (v3.1/v3.2), qwen3.5-397b, Kimi-k2/k2.5, GLM-*, etc but when I try to run Minimax-m2.7 (or m2.5) with ik_llama.cpp, I just get gibberish, while with llama.cpp it runs fine (using the very same flags).

this are the flags when I try with ik_llama.cpp (I prefix the command with "GGML_CUDA_NO_PINNED=1"):

--temp 1.0 --top-k 40 --top-p 0.95 -ngl 99 -fa on --min-p 0.01 --jinja -ot "blk.[0-9][0-9].ffn_.*_exps.weight=CPU" -c 24576 -n 24576 -np 1

I tried Bartowski and Unsloth Q4KM quants. Both have the same issues in ik_llama.cpp (and both work well in llama.cpp).

Anything I'm missing?

edit: forgot to add that I always keep ik_llama.cpp updated

[ikawrakow]

I cannot reproduce the issue. Granted, I'm testing with the IQ2_KS model from Ubergarm as I don't have enough disk space left to download a Q4_K_M variant, but I wouldn't know why it would work with one quantization type but wouldn't wok with another.

[glsup]

I've just tested IQ2KS and only once entered a loop on the built-in web-gui, but it was in a chat I used before with Q4KM, I started a new chat with the same prompt and it was fine. Also tested it with Open Webui 2 times and all fine.

So I guess is something about non-Ubergarm quants? (based on my notes, I had also tested Unsloth's UD-IQ3_XXS and I've just now tested the "old" MiniMax-M2.5-Q4_K_M, with both having the same issue (gibberish))

[ikawrakow]

No, some of the tests below were run with Q4_K_M from Bartowski, so not related to Ubergarm or not. Depending on when you downloaded the models, they may be broken (contain NaNs), see my comment below.

[FNsi]

What I can think in mind would be template. Even though I think ub and others use the same template from official. Maybe you should try ubs quant or download template from official repo then manually load it?

[FNsi]

Thanks for the suggestion! but I've just tried both Bartowski and Unsloth quants with the chat_template.jinja and still the same.

Sorry about that..maybe you could compare md5 between your local and source? You can check that in hf repo. If they are equal then something is wrong.

[glsup]

thanks but they are fine. They both (unsloth and bartowski quants) work fine with llama.cpp.
And it's becoming my go-to model.
I've also tested it with turboquant (which I might keep testing it with, as it reduces the VRAM usage), and they work fine.

[FNsi]

I've also tested it with turboquant (which I might keep testing it with, as it reduces the VRAM usage), and they work fine.

Then, did you clean your build file of ikllama,before update? It probably be massed sometimes.

[glsup]

yes, and I've just did it again (rm -rf build ; and then built it again), I still get gibberish.
Thanks for trying to help!
and nevermind, I'll do what I was recommended to do.

[FNsi]

No clue man, in case we knew MiniMax 2.7 is not worth for it... especially seems no one can reproduce that problem during their tries... also, we don't even know your hardware, your system, detail log...only a simple bug report so basically can't do much except words...

[magikRUKKOLA]

Ha. This is interesting. I am having a looping behaviour in case the prompt contains a newlines.

/opt/bartowski/MiniMax-M2.7/Q4_K_M/

high "Imagine a runaway trolley is hurtling down a track tow
ards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you
pull the lever?"

response:

Thus answer: No, you should not pull lever.

We can also discuss the possibility that the problem is a trick: The five are dead, so you cannot save them; you would be causing a death. So you should not pull lever.

Thus answer: No.

We can also discuss the moral frameworks: Utilitarian (greatest good) would weigh the living person's life vs dead bodies, so no. Deontological (duty not to kill) also says no. So both say no.

Thus answer: No.

We can also discuss the nuance: If the five were alive, then utilitarian says pull lever, deontological says no. But here it's reversed.

Thus answer: No.

We can also discuss the possibility of alternative actions: Could you try to stop the trolley? But the problem says you have a lever.

Thus answer: No.
...

[FNsi]

I test in api (while I cannot control the system prompt in that case)

Then I start with hi.

Then I send your words

It works normal.

high "Imagine a runaway trolley is hurtling down a track tow

ards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you

pull the lever?"

MODEL:

Thought process

This is the classic Trolley Problem - one of the most famous thought experiments in ethics!

There's no single "right" answer, and philosophers have debated this for decades. Here are the main perspectives:

Consequentialist view (pull the lever):

- You save 5 lives at the cost of 1 = net gain of 4 lives
- Outcome maximization: the greatest good for the greatest number
- This is often the "calculus" most people intuitively make
Deontological view (don't pull):

- You are actively causing the death of the 1 person
- Using someone as a means to save others is morally wrong
- inaction vs. action - you didn't tie them there
Variations that complicate it:

- Loop variant : What if the track loops back and the one person would stop the trolley naturally? Does knowledge change the ethics?
- Push variant : Would you push a large person off a bridge to stop the trolley? Many say "no" even though the math is identical.
- Fat man variant : Same dilemma but pushing instead of lever - reveals how physical contact matters to many people's moral intuitions.
Philosophers like Judith Jarvis Thomson have written extensively on this, showing how our intuitions often contradict each other depending on framing.

What's your gut reaction? Do you pull the lever, or let fate run its course?

[magikRUKKOLA]

@FNsi

The really funny part is that "via api" it fails to answer the question. While via the current regular run of, say, Q4_K_M it seems to find the right answer but its unable to get out of a loop.

[FNsi]

@FNsi

The really funny part is that "via api" it fails to answer the question. While via the current regular run of, say, Q4_K_M it seems to find the right answer but its unable to get out of a loop.

😂😂😂so that api model is likely quantised to lower level or quantity of quantised model can handle more campare to mass rl 2.7...

[FNsi]

I have to say base on my experience, Qwen Coder next is the winner of moe below 397B...with manually turn on thinking.

Given that simple test in stepfun, MiniMax 2.5,(long term reasoning then give an answer for scenarios dead and alive) MiniMax 2.7 (simply fail, ignore dead)

Qwen coder next solve it short and precise.

Oh, forgot test 122B Qwen, but I already know that model's long reasoning...

[ikawrakow]

@magikRUKKOLA

You are running this in the server? If using llama-cli with the exact same prompt as you, I don't get the looping, but the answer is not good:

<think>

Hmm, this is the classic trolley problem, a well-known ethical dilemma. The user is asking whether I would pull the lever in this scenario. 

I should approach this carefully, acknowledging the moral complexity rather than giving a simplistic yes or no. The scenario involves choosing between inaction (letting five die) and action (diverting the trolley to kill one person). 

I recall that utilitarian philosophy would argue for pulling the lever to minimize total harm, while deontological ethics might object to actively causing someone's death. Since the user didn't specify a moral framework, I should present both perspectives neutrally. 

The response should be balanced, showing understanding of the dilemma's depth while avoiding judgment. I'll structure it by first acknowledging the dilemma, then presenting the utilitarian and deontological viewpoints, and finally noting how real-world contexts might influence decisions.
</think>

This is the classic **trolley problem**, a famous ethical dilemma. There is no universally "correct" answer—it hinges on the moral framework you apply.

Here are the two dominant perspectives:

1.  **Utilitarian/Consequentialist View (Pull the lever):** From this perspective, the morally right action is the one that maximizes overall well-being or minimizes overall harm. Five lives lost is objectively greater harm than one life lost. Therefore, diverting the trolley is the **obligatory** choice, as it minimizes total deaths.
2.  **Deontological/Rule-Based View (Do not pull the lever):** From this perspective, the morality of an action is based on the action itself and the duty to obey a moral rule (like "do not kill"), rather than its consequences. Actively diverting the trolley makes you **directly responsible** for the death of the one person, which is morally impermissible, even to save five others. Letting the five die is considered a passive omission, not an act of killing.

**In short:** A utilitarian would pull the lever. A strict deontologist would not. Most people's intuitions fall somewhere between, often influenced by factors like the level of direct involvement and the nature of the harm.

The dilemma is designed to highlight the conflict between outcome-based and duty-based ethics.

[magikRUKKOLA]

@ikawrakow

You are running this in the server?

Yeah. The only difference is that I had been using the high TUI client. We can dump the raw input that the llama-server is receiving to know what the exact difference is. But I am not sure if that is related to the issue.

[EDIT]: Anyways, let me check.

[magikRUKKOLA]

Raw input:

]~!b[]~b]system\nYou are a helpful assistant.[e~[\n]~b]user\nImagine a runaway trolley is hurtling down a track tow\nards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you\npull the lever?[e~[\n]~b]ai\n<think>\n

Raw logs:

render_message_to_json: Neither string content nor typed content is supported by the template. This is unexpected and may lead to issues.
init: chat template, example_format: ']~!b[]~b]system
You are a helpful assistant[e~[
]~b]user
Hello[e~[
]~b]ai
Hi there[e~[
]~b]user
How are you?[e~[
]~b]ai
<think>
'
render_message_to_json: Neither string content nor typed content is supported by the template. This is unexpected and may lead to issues.
render_message_to_json: Neither string content nor typed content is supported by the template. This is unexpected and may lead to issues.
srv          init: init: chat template, thinking = 1
INFO [                    main] model loaded | tid="139640826744832" timestamp=1776358728
INFO [                    main] HTTP server listening | tid="139640826744832" timestamp=1776358728 n_threads_http="127" port="8080" hostname="0.0.0.0"
VERB [              start_loop] new task may arrive | tid="139640826744832" timestamp=1776358728
VERB [              start_loop] update_multitasks | tid="139640826744832" timestamp=1776358728
VERB [              start_loop] callback_update_slots | tid="139640826744832" timestamp=1776358728
INFO [              slots_idle] all slots are idle | tid="139640826744832" timestamp=1776358728
VERB [          kv_cache_clear] clearing KV cache | tid="139640826744832" timestamp=1776358728
VERB [              start_loop] wait for new task | tid="139640826744832" timestamp=1776358728
INFO [      log_server_request] request | tid="139588942696448" timestamp=1776358735 remote_addr="127.0.0.1" remote_port=34388 status=200 method="GET" path="/v1/models" params={}
VERB [      log_server_request] request | tid="139588942696448" timestamp=1776358735 request="" response="{\"object\":\"list\",\"data\":[{\"id\":\"MiniMax-M2.7/Q4_K_M\",\"object\":\"model\",\"created\":1776358735,\"owned_by\":\"llamacpp\",\"meta\":{\"vocab_type\":2,\"n_vocab\":200064,\"n_ctx_train\":196608,\"n_embd\":3072,\"n_params\":466064457216,\"size\":285531012096}}]}"
render_message_to_json: Neither string content nor typed content is supported by the template. This is unexpected and may lead to issues.
VERB [              start_loop] new task may arrive | tid="139640826744832" timestamp=1776358735
VERB [      get_available_slot] selected slot by lru | tid="139640826744832" timestamp=1776358735 id_slot=0 t_last=-1
======== Prompt cache: cache size: 0, n_keep: 0, n_discarded_prompt: 0, cache_ram_n_min: 0, f_keep: 0.00, cache_ram_similarity: 0.50
INFO [   launch_slot_with_task] slot is processing task | tid="139640826744832" timestamp=1776358735 id_slot=0 id_task=0
VERB [              start_loop] update_multitasks | tid="139640826744832" timestamp=1776358735
VERB [              start_loop] callback_update_slots | tid="139640826744832" timestamp=1776358735
VERB [            update_slots] posting NEXT_RESPONSE | tid="139640826744832" timestamp=1776358735
VERB [    batch_pending_prompt] tokenizing prompt | tid="139640826744832" timestamp=1776358735 id_slot=0 id_task=0
VERB [    batch_pending_prompt] prompt tokenized | tid="139640826744832" timestamp=1776358735 id_slot=0 id_task=0 n_ctx=262144 n_keep=0 n_prompt_tokens=69 prompt_tokens="]~!b[]~b]system\nYou are a helpful assistant.[e~[\n]~b]user\nImagine a runaway trolley is hurtling down a track tow\nards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you\npull the lever?[e~[\n]~b]ai\n<think>\n"

[magikRUKKOLA]

@ikawrakow

, I don't get the looping, but the answer is not good:

Yes, that is expected. Its a very stupid model. The xcreate had the same response with his Mac.

[ikawrakow]

@glsup

In this HN thread it was mentioned that the Unsloth and Bartowski MiniMax-2.7 quants contained NaNs. As no-one else here can reproduce the issue, you have most likely one of these broken models.

[glsup]

Yes, I know, but Q4KM is confirmed. by both Unsloth and Bartowski, NOT to be affected by it.
I also run llama-perplexity on them, and no NaNs were found.

Again, it works fine (very well, in fact) in llama.cpp and I've just tested it on TheTom turboquant fork on both UIs (llama.cpp and Open Webui), and no issues (no gibberish/loops).

[ikawrakow]

Sorry about that..maybe you could compare md5 between your local and source? You can check that in hf repo. If they are equal then something is wrong.

Yes, corruption during download is one way to get a broken model. What I'm referring to is this:

Taken from a HackerNews Qwen-3.6 discussion from yesterday. Which is the more important way to get a broken model, when the model is broken at the source.

[glsup]

Not my case, I've been running them for the last few days, without a single issue in llama.cpp

[ikawrakow]

Well, in that case you just use this model with llama.cpp. Or whatever your favorite TurboQuant fork of the day might be. Nobody can reproduce your issue here, so not much we can do.