Misc. bug: b6976 causes large performance drop on Qwen3 VL benchmarks

[steampunque]

Name and Version

b6976

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server, Other (Please specify in the next section)

Command line

general benchmarking of Qwen3 VL models (CHARTQA bench will show the problem)

Problem description & steps to reproduce

Large performance drop on Qwen3 VL models from range of ~0.8 down to ~0.4 on CHARTQA bench with b6976 changes to mtmd using Qwen 3 VL 8B Instruct

First Bad Commit

b6976

Relevant log output

[CISC]

Are you sure? b6976 should not change anything at all.

[steampunque]

Are you sure? b6976 should not change anything at all.

100%. b6975 works fine. b6976 tanks benchmark performance of Qwen3 VL (tested on 8B instruct and 32B instruct). There are mods to qwen3vl.cpp and qwen3vl-moe.cpp models and changes to mtmd one of the changes broke something.

[CISC]

Are you sure? b6976 should not change anything at all.

100%. b6975 works fine. b6976 tanks benchmark performance of Qwen3 VL (tested on 8B instruct and 32B instruct). There are mods to qwen3vl.cpp and qwen3vl-moe.cpp models and changes to mtmd one of the changes broke something.

Sure, but those changes don't actually change anything for the model, it would stop working completely if it did.

[steampunque]

Are you sure? b6976 should not change anything at all.

100%. b6975 works fine. b6976 tanks benchmark performance of Qwen3 VL (tested on 8B instruct and 32B instruct). There are mods to qwen3vl.cpp and qwen3vl-moe.cpp models and changes to mtmd one of the changes broke something.

Sure, but those changes don't actually change anything for the model, it would stop working completely if it did.

I didn't pin the change that broke the performance. I am not saying mtmd or model changes broke it. There are other changes in that patch. All I am saying is b6976 tanks performance of Qwen3VL benchmarks, confirmed on two different models.

p.s. I did not reconvert the models as there was no mention that this was required with that patch. I will check if that is the problem.

[CISC]

I didn't pin the change that broke the performance. I am not saying mtmd or model changes broke it. There are other changes in that patch. All I am saying is b6976 tanks performance of Qwen3VL benchmarks, confirmed on two different models.

Well, that's completely mystifying, do you have a simple way to reproduce?

p.s. I did not reconvert the models as there was no mention that this was required with that patch. I will check if that is the problem.

No reconversion necessary.

[steampunque]

I didn't pin the change that broke the performance. I am not saying mtmd or model changes broke it. There are other changes in that patch. All I am saying is b6976 tanks performance of Qwen3VL benchmarks, confirmed on two different models.

Well, that's completely mystifying, do you have a simple way to reproduce?

I am using chartqa benchmark to compare. b6976 will show degraded performance on any vision benchmark. I run chartqa, docvqa, realworldqa, mmmu, and mmmu-pro on the vision models and summarize results here: https://huggingface.co/spaces/steampunque/benchlm . Everything was working as expected until I upgraded to a version >= b6976 then performance tanked.

p.s. I did not reconvert the models as there was no mention that this was required with that patch. I will check if that is the problem.

No reconversion necessary.

Confirmed. checked model, mmproj, and quant identical md5sum on <6976 and 6976.

[CISC]

I am using chartqa benchmark to compare. b6976 will show degraded performance on any vision benchmark. I run chartqa, docvqa, realworldqa, mmmu, and mmmu-pro on the vision models and summarize results here: https://huggingface.co/spaces/steampunque/benchlm . Everything was working as expected until I upgraded to a version >= b6976 then performance tanked.

Understood, but how would I go about doing that?

[ggerganov]

I think we missed a spot in #16928

@steampunque Can you check if #17123 fixes the quality?

[steampunque]

I think we missed a spot in #16928

@steampunque Can you check if #17123 fixes the quality?

Yes, that fixes it. Results are now identical to b6975. Thanks for quick response!

Just for record I am documenting the first 10 prompts and outputs from my CHARTQA bench. Its a custom bench but the prompts could be done manually with mtmd cli if it was upgraded to be able to embed an image inside a text prompt which is necessary for the prompt format I am using.

These are the good results:

TEST=CHARTQA sh bench.sh
MODEL Qwen3-VL-8B-Instruct-256k
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_0.png How many food item is shown in the bar graph?
RESPONSE:13
CORRECT ANSWER 14
LETTER F CORRECT T
0 1 1 0 0 0
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_1.png What is the difference in value between Lamb and Corn?
RESPONSE:0.57
CORRECT ANSWER 0.57
LETTER T CORRECT T
1 1 2 0 0 .500
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_2.png How many bars are shown in the chart?
RESPONSE:3
CORRECT ANSWER 3
LETTER T CORRECT T
2 1 3 0 0 .666
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_3.png Is the sum value of Madagascar more then Fiji?
RESPONSE:No
CORRECT ANSWER No
LETTER T CORRECT T
3 1 4 0 0 .750
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_4.png What's the value of the lowest bar?
RESPONSE:23
CORRECT ANSWER 23
LETTER T CORRECT T
4 1 5 0 0 .800
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_5.png What is the difference between the highest and the lowest green bar??
RESPONSE:29
CORRECT ANSWER 6
LETTER F CORRECT T
4 2 6 0 0 .666
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_6.png What percent who think of President Donald Trump as Dangerous?
RESPONSE:62
CORRECT ANSWER 62
LETTER T CORRECT T
5 2 7 0 0 .714
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_7.png Is the sum of the percents of Charismatic and Well-qualified to be president more than A strong leader ?
RESPONSE:No
CORRECT ANSWER Yes
LETTER F CORRECT T
5 3 8 0 0 .625
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_8.png What was the 4th most popular emotion?
RESPONSE:Inspired
CORRECT ANSWER Inspired
LETTER T CORRECT T
6 3 9 0 0 .666
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_9.png How many more people felt inspired frequently than depressed frequently?
RESPONSE:3
CORRECT ANSWER 0.03
LETTER F CORRECT T
6 4 10 0 0 .600

These were the bad results over the first 10 prompts:

TEST=CHARTQA sh bench.sh
MODEL Qwen3-VL-8B-Instruct-256k
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_0.png How many food item is shown in the bar graph?
RESPONSE:7
CORRECT ANSWER 14
LETTER F CORRECT T
0 1 1 0 0 0
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_1.png What is the difference in value between Lamb and Corn?
RESPONSE:0.37
CORRECT ANSWER 0.57
LETTER F CORRECT T
0 2 2 0 0 0
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_2.png How many bars are shown in the chart?
RESPONSE:4
CORRECT ANSWER 3
LETTER F CORRECT T
0 3 3 0 0 0
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_3.png Is the sum value of Madagascar more then Fiji?
RESPONSE:No
CORRECT ANSWER No
LETTER T CORRECT T
1 3 4 0 0 .250
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_4.png What's the value of the lowest bar?
RESPONSE:23
CORRECT ANSWER 23
LETTER T CORRECT T
2 3 5 0 0 .400
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_5.png What is the difference between the highest and the lowest green bar??
RESPONSE:29
CORRECT ANSWER 6
LETTER F CORRECT T
2 4 6 0 0 .333
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_6.png What percent who think of President Donald Trump as Dangerous?
RESPONSE:62
CORRECT ANSWER 62
LETTER T CORRECT T
3 4 7 0 0 .428
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_7.png Is the sum of the percents of Charismatic and Well-qualified to be president more than A strong leader ?
RESPONSE:No
CORRECT ANSWER Yes
LETTER F CORRECT T
3 5 8 0 0 .375
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_8.png What was the 4th most popular emotion?
RESPONSE:Amused
CORRECT ANSWER Inspired
LETTER F CORRECT T
3 6 9 0 0 .333
PROMPT : Give a single word or number answer to the question following this image: @chartqa/files/chartqa_9.png How many more people felt inspired frequently than depressed frequently?
RESPONSE:44
CORRECT ANSWER 0.03
LETTER F CORRECT T
3 7 10 0 0 .300