Repository navigation
Replies: 6 comments 10 replies
|
Thank you for this, PPL with the paired chunks is much more useful than one number. I'm on M2 Max 38-core, 64 GB, and use One thing from my agent replays (coding agent, 32K to 125K context, temperature 0) which may help with the choice: the decode gap between formats gets smaller at long context, because attention over the history costs the same for every format. There UD-Q4_K_M is ~6-7% behind MLX 4-bit per cycle instead of ~10% on short prompts, and prefill is only ~2% slower. So for long agent sessions UD-Q4_K_M costs less than the short numbers suggest. Your PPL numbers make me want to try it, and GSQ IQ3_S too. I also tried Bonsai ( |
|
@ghisguth thanks, the long-context point is a good one, my decode numbers are short prompts only, so at 32K+ the gap to UD-Q4_K_M is smaller than in my table. Same probably for IQ3_S, it was ~25% slower per cycle on short prompts here, would be interesting to see how much of it is left in your replays. Small follow-up with two more things. 1. Q4_0. I added
Paired per chunk:
So Q4_0 is the weakest one here, even behind UD-IQ3_S which is 4 GB smaller. On chat it's ~10% worse than MLX 4-bit, and this is significant even with the noisy chat corpus. I guess the symmetric grid (only a scale per 32, no min) costs more than imatrix gives back on this model. So "Q4_0 is almost as fast as MLX 4-bit" is true, but quality-wise it's not a replacement for it. 2. Splash computes the same as llama.cpp. In the first post I had a caveat that PPL was measured outside Splash. Thanks to jianc99's pointer in incoai#540 I used
Within 0.3% everywhere, and the per-chunk Q4_0 vs MLX 4-bit gap correlates 0.99 between Splash and the other engines. So the numbers above hold for Splash too, the caveat can be dropped. Inside Splash directly, Q4_0 vs MLX 4-bit: top-1 agreement 90.2% on wiki and 86.4% on chat, mean KL 0.060 / 0.192. One run per window takes ~53 s on M1 Pro (model load ~15 s, prefill ~15 s, 127 verify cycles ~23 s), so 30 windows per text is about half an hour per quant. Full logits are 1 GB per window, so I reduce each to per-position NLL, top-1 and every 8th row of log-probs right after it's written. |
|
@sady4850 thanks, the Q4_0 result is useful. In speed it is the closest GGUF to MLX 4-bit on my M2 Max too, so good to know it is not a free replacement. GSQ IQ3_S in my replay (M2 Max, Splash 1.4.0-m1, the same 78 agent turns sent twice, temperature 0), next to MLX 4-bit:
(The 32k row is only 5 requests. tok/s depends on the text, cycles/s is the engine itself.) Prefill on the same 120 requests is the same as MLX 4-bit (-1%). Whole replay: 2489 s against 2324 s. So the gap gets a bit smaller with context, but IQ3_S stays clearly slower. UD-Q4_K_M in the same replay is -6 to -7%. |
|
@sady4850 a follow-up on GSQ-RCO IQ3_S speed. It was slow because of how the IQ3 kernels decode each weight on M1/M2, not the bandwidth. I made a PR for it: #35. Same outputs, bit for bit. bench-all on my M2 Max, decode tok/s at 1 / 2 / 4 lanes (A = 1.4.0-m1, B = with the PR):
So against MLX 4-bit it is now -16 / -10 / -6% instead of -21 / -13 / -7%. One more change for the IQ3 coefficient decode is in testing, I will post when it is ready. |
|
@ghisguth thanks for #35, nice find that it was the grid conversion and not the bandwidth. I tried the other way: I tried to make MLX 4-bit itself closer to bf16, without changing anything Splash sees. Checkpoint: https://huggingface.co/sady4850/Qwen3.8-27B-4bit-GPTQ It is Distance to bf16 (torch on A100, both checkpoints dequantized into the same bf16 model, 30 x 2048 per text, scored on the second half like llama-perplexity):
So -34% KLD on wiki, -46% on chat, and ~25-30% less tokens where top-1 differs from bf16. Calibration was ultrachat train + calibration_v5, chat eval is ultrachat test, so chat is a bit in-domain for it, wiki is the cleaner number. Inside Splash (
GPTQ vs MLX 4-bit inside Splash is the same picture as in torch: -29% KLD on wiki, -47% on chat. And here I have to correct myself about Q4_0. In my last comment I called it the weakest quant because of PPL. By KLD it's actually closer to bf16 than MLX 4-bit on both texts, and on wiki it's even a bit ahead of GPTQ. Same with MLX 4-bit on chat: PPL 8.14 while bf16 itself is 8.72. And GSQ IQ3_S on wiki had 6.316, bf16 is 6.364. A quant with lower PPL than the original is not "better", it's just different from it in a direction that happens to fit this text, and a higher PPL doesn't mean further from bf16 either. So my PPL tables rank how close a quant is to bf16 pretty badly, KLD is the right number for that, sorry for the noise. I want to redo the GGUFs (GSQ IQ3_S, UD-Q4_K_M) with KLD in Splash too, the bf16 reference for these windows is ready now. @ghisguth could you run your 78 agent turns replay with this checkpoint next to In general I think it would be good to have some decent coding agent benchmark which can be run on Mac. Not crazy tasks like rebuilding a browser from scratch, but the tasks one would expect to solve with a local model. Then things like loops or not finished tasks would show up too. Do you know something like this, or would your replays work as a base for it? |
|
@sady4850 thanks, interesting. I will pull your checkpoint and run my replay with it next to As for the benchmark, my replay is a recording of my own agent session, so it needs some cleaning before I can share it. I will look into open sourcing it. I think I will also capture a new session that goes up to 256K, mine ends at 124K. It will take some time. |
|
@sady4850 here is the replay with your checkpoint. Same 78 agent turns, each one sent with the same recorded prompt whatever the model answered before, temperature 0, Splash 1.4.0-m1 on my M2 Max. bf16 doesn't fit here with context, so Q8_0 is the reference. Per turn against Q8_0 (same calls = same tools with the same arguments, args similarity = how close the arguments are when the tools match, 1 is identical):
For scale: Q8_0 itself makes the same calls only 82% of the time when the same turn is sent twice, and MLX 4-bit on two Splash builds where only rounding differs 71%. Most differences are small, like an extra So in my agent turns GPTQ is closer to Q8_0 than MLX 4-bit, 44% vs 31% same calls. MLX 4-bit vs GPTQ is the clean comparison here, same kernels. Q8_0 runs through the GGUF kernels, so the GGUF quants may look a bit closer to it partly for that reason. No quant broke a tool call, and answer length is the same for all. MLX 4-bit and Q8_0 each had one loop that the bench cut, GPTQ none, but that is one case. Speed is the same as MLX 4-bit, cycles/s within 2%. |
|
@ghisguth great, thanks! per turn diff is exactly what I wanted to see, no hurry. And 256K session would be very interesting, in my runs nothing goes beyond 2K so I have no idea what happens with quant error so deep in context. Meanwhile I did the GGUFs with KLD inside Splash too, you mentioned you want to try UD-Q4_K_M and GSQ IQ3_S, so better you see this before. Same 60 windows and same bf16 reference as for GPTQ above (
UD-Q4_K_M is the closest to bf16 on everything, by a lot on wiki: mean KLD ~4x less than MLX 4-bit, p99 6x less, top-1 flips 3.9% vs 9.6%. On chat the gap to GPTQ is smaller (-17% mean KLD), but GPTQ was calibrated on ultrachat train, so chat is a bit in-domain for it, wiki is more fair number. Not a big surprise I think, K-quants with imatrix and more bits for sensitive tensors, and it's 1.5 GB bigger than MLX 4-bit. But it means for your long agent sessions UD-Q4_K_M is a real quality upgrade over MLX 4-bit, not only "a bit different". And GSQ IQ3_S, which was the bold winner in my first PPL table, is not a winner at all. Its PPL is below bf16 itself on both texts (6.307 vs 6.364, 8.037 vs 8.717), but mean KLD is the same as MLX 4-bit and p99 is the worst of all, on chat 3.43 vs 2.72. Means most tokens are fine, but when it goes wrong it goes wrong more. Same lesson as with Q4_0, just from other side: low PPL is not "close to bf16". For 11.8 GB it's still not bad, it's MLX 4-bit quality on average at 3 GB less, but I wouldn't take it for agent work over MLX 4-bit, tails is what breaks tool calls I think. Speed, median per window on M1 Pro (prefill 1016 tokens, then 127 verify cycles):
So UD-Q4_K_M is ~11% slower per cycle here, short context only, by your numbers at 32K+ it's ~6-7%. For me it looks like a good trade if memory allows. |
|
@ghisguth oh, I was writing mine comment without seeing your replay) Nice that the replay and KLD say mostly the same thing, and from different sides: your reference is Q8_0 through the GGUF kernels, mine is bf16 from torch, so the GGUF-kernel bias you mention can't be the whole story at least for UD-Q4_K_M, it's first in both. Side by side:
GSQ IQ3_S is the only one which doesn't fit so well, same calls like GPTQ but same tools is the lowest of the 4-bit ones. Maybe this is the worse tail (p99) showing up, but with 78 turns I wouldn't read much into it. And the clean one, MLX 4-bit vs GPTQ with the same kernels and same speed, 31% vs 44%, is what I hoped to see. Thanks again for running it |
|
@sady4850 thanks for putting it all in one place in #43. I was playing with Flash Next quants on my M2 Max 64 GB, trying to find the one which works best for me:
BTW the Coder "IQ1_M" is not IQ1_M at all. it is ~3.3 bpw. It is smaller only because half of experts are gone. Splash 1.5.0-m1 and
Q2_0 works great at 46G next to my dev apps. IQ3_XXS needs at least 52G for all experts with 256K context, so only when the Mac is just the LLM server. And in agent work it was even slower than the table: in one of my tests it took 2.6x the time of Q2_0, because the ~48 GB fixed part leaves no room for the prefix cache and it kept re-prefilling (479K prompt tokens vs 97K). With 46G and part of experts on SSD decode is ~9% slower than all resident, and short prompt prefill is ~2x slower, every new prompt pulls experts from SSD. Coder fits in 40G with 256K + 128K for a subagent, decode is ~10-14% slower than Q2_0. But in a real coding task it thinks a lot more, 2x output tokens, so the task took 5h40 vs 3h47 for Q2_0. It also loops on prompts with Russian language every time, I guess languages were in the experts that were cut. Now the quality. I tried to find a difference: a reasoning and tool-use agent loop in one of my projects; real coding tasks with tests in my project, same task on each model. Neither IQ3_XXS nor Coder was better in reasoning, precision or tool calls. The reasoning and tool-use agent loop was failing different cases for Q2_0 and IQ3_XXS, same pass rate. On the coding task Coder wrote the same code and made the same decision as Q2_0, but made more tool errors and lost track of things after compactions. So same quality, just slower. I know KLD matters, and the Coder card shows LiveCodeBench 81 -> 87 which should be significant. But I can't see it in day to day use. Same card also shows SWE-bench Verified 79.2 vs 82.8 for BF16, which is closer to what I do. It was n = 1 per task though, so a small gain could still be there. |
|
@ghisguth thanks, would you like to add it to docs/quants.md as a Qwen3.8-Flash-Next section? Useful info which otherwise could be lost in this discussion |
|
@sady4850 sure, please do. Here are the speed numbers for the three quants too, so the section has speed next to memory. All on my M2 Max 38-core 64 GB, Splash 1.5.0-m1, Decode, GPU ms per decode cycle at 1 / 2 / 3 / 4 lanes, 2048-token prompt, lower is better. The verify budget picks the rows like in the server:
Long agent chat replay in one of my tests (78 turns from 32K to 125K context, each sent twice), server log. Cycles/s is decode tok/s ÷ tok/cycle, so it doesn't depend on how many draft tokens get accepted:
Tok/cycle goes from ~3.2 at 32K to ~5.1 at 128K, the same for both. IQ3_XXS didn't fit this replay's context at 46G, so no numbers for it. Prefill is the same for Q2_0 and Coder: a cold 14.9K prompt takes 27.2 s (549 tok/s) on Q2_0, and in the replay appends of 5K+ tokens run at ~507 tok/s and appends under 1K at ~360 tok/s on both. And with #45 on top (MoE gate and up in one pass), decode is faster on all three, prefill doesn't change:
With the budget's rows the two builds verify different rows, so that row is noisier. The 8-row one is the same work on both. |
|
i was using the swift1.5 27b 4 bit mlx..but it did show up the low quant issues with long running tasks with lots of compression and UI setups... Then i switched to Swift-1.5-Qwen3.8-27b-oQ6e-bf16-mtp-MLX . this is essentially 6bit, and i have to say the quality is noticable, especially at long running tasks, multiple issues getting solved in one prompt/go.. the speed did take a small hit compared to 4 bit, but its workable for me... I did ask for the gguf vs mlx speed once in the issues and dev replied that mlx still has the edge in terms of speed. I suspected ggufs would have better quant quality, which your tests show... |
|
@Rdx-ai-art interesting, have you tried stock Qwen3.8-27B before you started with fine tuned swift1.5? What's your impression, did you notice difference just from usage perspective? Swift card says it thinks ~58% less, did you see it in practice on xhigh, like shorter thinking and faster finished tasks, or more the quality side? |
|
@sady4850 ive tried stock 27b first.. it is indeed very much high thinker.. even for simple hi/hello, or web search tasks and sometimes at long ctx or with harder tasks it just repeats testing or reinterpreting its own solutions again and again, basically wasting tokens and taking more time to reach the answer. swift1.5 does use less thinking, u can see it from its thinking traces sometimes.. like instead going on and on and doubting itself, swift, skips it after around 2 times or so mostly... that said for some tasks it does think a lot like the base as well.. 6 bit is defenetly better than 4 bit, that i am sure.. ( oh, and i tested the base 27b at 4 bit only..) |
Uh oh!
There was an error while loading. Please reload this page.
I was choosing a quant for my 32 GB M1 Pro and was curious how they differ in quality, not only in speed. So I ran perplexity on 4 quants of Qwen3.8-27B and measured them in Splash on the same machine. Sharing the numbers in case it's useful for somebody.
Setup
incoai/Qwen3.8-27B-DFlash2)--language-only --max-context 64K --max-memory 21G --idle-release off --ane-ffn-share 0.68 --ane-ffn-minimum-rows 256, default int8 KVmlx-community/Qwen3.8-27B-4bitunsloth/Qwen3.8-27B-GGUF:UD-Q4_K_Munsloth/Qwen3.8-27B-GGUF:UD-IQ3_SISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_SQuality: perplexity
llama-perplexity method: 30 chunks of 2048 tokens, scoring the second half of every chunk (30 x 1023 tokens). GGUF with
llama-perplexity(llama.cpp dd7f73b, Metal). MLX 4-bit with a small mlx-lm script which does exactly the same chunking and scoring. I checked that HF tokenizer and llama.cpp give identical tokens on the text (297,193 tokens on wikitext-2 test), no BOS for both.Texts: wikitext-2 raw test, and first 300 conversations of ultrachat test as plain text ("chat").
Final PPL alone hides how noisy it is, so here are paired per-chunk ratios (same chunks for both quants; < 1 means first quant is better; interval is mean ± 2 se over 30 chunks; last column is in how many chunks the first one wins):
On chat every pair is inside ± 7-15%, so chat corpus is too noisy to separate anything with 30 chunks, I don't read anything from it.
What I read from wiki:
Speed in Splash
TTFT: real first request of omp (oh-my-pi), 7.9K tokens (system prompt + 11 tools), cache busted with a nonce, 2 runs. Decode: 3 short prompts (Python function, B-tree explanation, a text in Russian), temperature 0, thinking off, up to 512 tokens. One request at a time.
tok/s jumps with draft acceptance (on prompt 2 UD-Q4_K_M looks faster only because draft accepted 3.4 vs 3.1 per cycle), so cycles/s is the cleaner number for the target itself:
Takeaways (for M1 Pro 32 GB)
mlx-community/Qwen3.8-27B-4bitis still the one, as paperniuk said in NOT AN ISSUE. just wanna know gguf vs mlx performance #4.Caveats
Scripts (perplexity on MLX matching llama-perplexity, Splash bench) I can share if somebody wants to repeat on other chips or other quants.
And question to everybody: which quants do you run in Splash day to day, and why this one? Did you notice difference in quality between them in real work (coding, agents, long thinking), or you choose only by speed and memory? Would be nice to collect your chip, quant and tok/s here, and any quality numbers if you have.
All reactions