Metal branch for Qwen 3.8 Flash and overall impressions of the model

[tarruda]

Hi

I wanted to share my custom branch which does a bunch of fixes (includes everything here and optimizations for Qwen 3.8 Flash: https://github.com/tarruda/llama.cpp/tree/qwen4exp-metal-optimizations. With it, the aging weak GPU on my M1 ultra can achieve pretty acceptable performance across the entire 256k token context, using less than 70w:

./build/bin/llama-bench -m ~/qwen-3.8-next/Qwen3.8-Flash-Next-IQ4_NL.gguf -lm none -n 128 -p 512 -d 0,10000,20000,30000,40000,50000,60000,70000,8
0000,90000,100000,150000,200000,250000
| model                          |       size |     params | backend    | threads |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ---------: | --------------: | -------------------: |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |           pp512 |        548.09 ± 4.70 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |           tg128 |         37.70 ± 0.10 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d10000 |        453.29 ± 1.62 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d10000 |         34.14 ± 0.05 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d20000 |        427.72 ± 4.62 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d20000 |         33.87 ± 0.03 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d30000 |        407.90 ± 5.70 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d30000 |         33.32 ± 0.10 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d40000 |       320.10 ± 76.77 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d40000 |         11.54 ± 5.45 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d50000 |        376.66 ± 6.41 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d50000 |         32.44 ± 0.06 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d60000 |        361.85 ± 8.97 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d60000 |         32.08 ± 0.10 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d70000 |        347.32 ± 8.71 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d70000 |         31.79 ± 0.03 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d80000 |        335.10 ± 9.58 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d80000 |         31.19 ± 0.07 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  pp512 @ d90000 |       323.74 ± 10.32 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none |  tg128 @ d90000 |         30.68 ± 0.30 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none | pp512 @ d100000 |       310.97 ± 11.90 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none | tg128 @ d100000 |         30.34 ± 0.08 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none | pp512 @ d150000 |       265.72 ± 11.02 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none | tg128 @ d150000 |         28.54 ± 0.09 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none | pp512 @ d200000 |       228.54 ± 11.83 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none | tg128 @ d200000 |         27.00 ± 0.09 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none | pp512 @ d250000 |        198.46 ± 6.90 |
| qwen4exp A3B Q8_0              |  95.20 GiB |   176.94 B | MTL,BLAS   |      16 |       none | tg128 @ d250000 |         25.53 ± 0.08 |

I imagine these numbers will be significantly better on M2+.

The non-scientific benchmark was a low poly "vice city" clone, which I published the exact prompt and result here: https://gist.github.com/tarruda/0f9fc3dd2b69e3a827ef54701f440434 . Note that the fixes in my branch are necessary, as they address several correctness issues with the current llama.cpp implementation (which apparently affects vision too). Here's a few screenshots of it debugging the game with chromium + xvfb:

Details

I also tested its vision/bounding boxes capabilities, and it is seems very accurate:

Details

In total the agent worked for more than 10 hours overnight, maintained continuation and coherence across two compactions and overall impressed the hell out of me. Even though this is still an experimental model/architecture, it is already feeling ahead of competition for 128G hardware

For these tests, I used the IQ4_NL quant here: https://huggingface.co/tarruda/Qwen3.8-Flash-Next-GGUF