Evaluation of various MLX quantizations

[deepsweet]

KLD measures the "distance" between two probability distributions. It doesn't just look at one token; it compares the entire vector of probabilities from the original model to the quantized model. It asks, "How much information is lost when we use the quantized distribution to approximate the original one?"

See detailed results for more information:

[deepsweet]

I would appreciate it if someone with enough RAM could provide Qwen3.6-27B results, and I'd handle the chart part. AFAIK, MoE models are more sensitive to quantization than dense ones.

[deepsweet]

It turns out that quantization of a sensitivity model itself is way more important than I thought, especially for producing oQ4 and below.

Re-quantizing and re-uploading my Qwen3.6 collection, have updated the chart and data.

It's not like something was dramatically wrong, but quick test shows that more precise affine Q8 forces oQ algorithm to make slightly different choice on some tensors in comparison to MXFP4 I used previously. All model-00001-of-*.safetensors were touched.

[crazyi]

What kind of model can be used as a sensitivity model? What impact does using or not using it have on oQ quantization?

[deepsweet]

Ideally you should use no sensitivity model at all, given that you have enough RAM. But if you do use it then try to squeeze the closest quant to the original model as possible.

The way I understand it the whole point of dynamic oQ quantization is to measure how much certain bits would hurt a certain layer. And if you use an already lower bits quantized sensitivity model, like Q4, then the quantization effects will accumulate on top of each other, leading to potentially wrong bits allocation during the final oQ process step.

I use affine Q8 as sensitivity model:

mlx_vlm.convert --hf-path ~/.cache/huggingface/models/Qwen/Qwen3.6-27B --mlx-path ./Qwen3.6-27B-MLX-VL-Q8 -q --q-bits 8

[deepsweet]

Dear diary, it's me again.

Have switched to Aes Sedai's combined_all_micro.txt prompt dataset instead of a pure "prose", ran it with 16384 tokens context, and now KLD results seem to be of the same magnitude as Unsloth UD MLX Dynamic Quants (yes, they also provide MLX quants). So I might come up with the third scatterplot data layer.

WIll continue digging, it's been an interesting journey.

[nmqanh]

@deepsweet I followed the guide on your mlx-kld and here are the results I got

• Model: Qwen3.6-27B
• My machine is M2 Max 96gb (38 core).
• All oQ quants are generated by oMLX 0.3.8rc1 - Sensitive model is original source Qwen3.6-27B.
• All Q quants are generated by

uv run mlx_lm.convert \
  --hf-path ~/.omlx/models/Qwen3.6-27B \
  --mlx-path ~/test-models/Qwen3.6-27B-MLX-<NUMBER_OF_BITs>bit-FP16 -q --q-bits <NUMBER_OF_BITs> --dtype float16

KLD benchmark for MLX oQ quant

	Qwen3.6-27B-oQ8-FP16	Qwen3.6-27B-oQ6-FP16	Qwen3.6-27B-oQ5-FP16	Qwen3.6-27B-oQ4-FP16	Qwen3.6-27B-oQ3.5-FP16	Qwen3.6-27B-oQ3-FP16	Qwen3.6-27B-oQ2-FP16
Qwen3.6-27B-MLX-BF16	0.079178 - 26.62 GB	0.166850 - 20.84 GB	0.343887 - 17.69 GB	0.439214 -  14.68 GB	0.815383 - 12.70 GB	0.818882 - 11.75 GB	2.026183 - 9.74 GB
Qwen3.6-27B-MLX-FP16	0.041077 - 26.62 GB	0.164307 - 20.84 GB	0.350098 -  17.69 GB	0.439697 - 14.68 GB	0.818359 - 12.70 GB	0.826172 - 11.75 GB	2.033203 - 9.74 GB

KLD benchmark for MLX Q quant

	Qwen3.6-27B-MLX-8bit-FP16	Qwen3.6-27B-MLX-6bit-FP16	Qwen3.6-27B-MLX-5bit-FP16	Qwen3.6-27B-MLX-4bit-FP16	Qwen3.6-27B-MLX-3bit-FP16	Qwen3.6-27B-MLX-2bit-FP16
Qwen3.6-27B-MLX-BF16	0.079178 - 26.62 GB	0.167697 - 20.36 GB	0.334142 - 17.23 GB	0.464763 - 14.09 GB	0.917167 - 10.96 GB	2.695468 - 7.83 GB
Qwen3.6-27B-MLX-FP16	0.041077 - 26.62 GB	0.166870 - 20.36 GB	0.336426 - 17.23 GB	0.472168 - 14.09 GB	0.924805 - 10.96 GB	2.701172 - 7.83 GB

Thoughts

I thought that dense model is less sensitive to quant than MoE model, but why we are seeing the opposite thing? The results for MoE is much more impressive.

[deepsweet]

Hello everyone.

I've just updated the comparison with MLX UD quantizations, so we have 4 layers now:

• oQ
• Q
• MXFP
• UD

See detailed results. Everything has been measured against the most normal multimodal BF16 version.

The KLD metric itself has been expanded to KLD mean, KLD p95 and KLD p99.

@nmqanh thanks for the data! I'll try to visualize it and we'll see. Just a quick note: have you also added --dtype float16 when converting the original "reference" model? Like this:

mlx_lm.convert \
  --hf-path ~/.cache/huggingface/models/Qwen/Qwen3.6-27B \
  --mlx-path /path/to/Qwen3.6-27B-MLX \
  --dtype float16

It's important to have everything as identical and comparable as possible. It's not that FP16 narrower dynamic range could possibly be a real issue here, but just in case.

Also, could you please try it again with the most unexpected values in your opinion with the updated mlx-kld? And look at KLD p95 and KLD p99 specifically.

UPD: just noticed that you've provided both BF16 and FP16 already. Make sure that you've also regenerated reference data: logprobs.npy for BF16 and for FP16 should be different, as well as prompt.npy. I'll probably add a CLI argument to specify explicit file paths for it.

[deepsweet]

@nmqanh it feels a bit odd indeed, I'll try to squeeze some low quants into my M2 Max 64GB and double check everything.

[deepsweet]

Other options to consider:

mlx_lm.convert --help

…

  --quant-predicate {mixed_2_6,mixed_3_4,mixed_3_5,mixed_3_6,mixed_3_8,mixed_4_6,mixed_4_8}
                        Mixed-bit quantization recipe.

[nmqanh]

@deepsweet I tested a bit more and UD quant > oQ > Q . But the results does not match the quality on UD official MLX benchmark.

	Qwen3.6-27B-UD-MLX-4bit-FP16	Qwen3.6-27B-oQ4-FP16	Qwen3.6-27B-MLX-4bit-FP16
Qwen3.6-27B-MLX-FP16	Mean:  0.264648 p95:  0.383240 p99:  8.821484 23.53 GB	Mean: 0.439697 p95: 0.965454 p99: 14.501172 14.68 GB	Mean: 0.472168 p95: 1.177490 p99: 14.577734 14.09 GB

[deepsweet]

What actually matters more is the relative numbers across one test suite using the same data.

I've used a single 16384-token prompt using Aes Sedai's combined_all_micro.txt, which looks challenging and would unlikely to ever match the absolute numbers of other KLD tests. People often use WikiText-2, but I find it rather lower-entropy and monotonous. See wikitext-2-raw-v1 test split.

The benchmark used wikitext2_test.txt at 512 token context with -ncmoe 22 and -ngl 999. But here’s a pro tip from the discussion: imatrix contamination is real. If your quantizer used wikitext2 in their calibration data (and many do), you’re measuring on training data, which skews results. For true validation, construct a fresh corpus from recent sources, podcast transcripts, recent news, anything the model wasn’t explicitly calibrated on.

From https://www.banandre.com/blog/quantization-fidelity-benchmarking-kld-and-ppl-as-metrics-for-gguf-model-selection

It's all very debatable, and I'm open to re-running my tests against a different dataset.

[johnlovesgoats]

KLD measures the "distance" between two probability distributions. It doesn't just look at one token; it compares the entire vector of probabilities from the original model to the quantized model. It asks, "How much information is lost when we use the quantized distribution to approximate the original one?"

Read "Why Maybe We're Measuring LLM Compression Wrong" and "oQ: oMLX Universal Dynamic Quantization" for details.

See mlx-kld for details.

I'd say that oQ5 is the sweet spot:

See detailed results.

can you also test nvfp4 since mlx can run it (pls)

[deepsweet]

@johnlovesgoats

can you also test nvfp4 since mlx can run it (pls)

But it feels wrong to convert and compare, I'm not even sure that it's working properly.

[johnlovesgoats]

Weird that its performing worse. Maybe its not implemented correctly yet...

[deepsweet]

❤️‍🔥

Following "The ‘Q4_K_M’ Illusion: Why KL Divergence and Perplexity Are Your Only Friends in the GGUF Wild West":

High PPL with low KLD suggests the model is fundamentally sound but might need fine-tuning. High KLD with low PPL? You’ve got a distribution mismatch that’s probably going to bite you in production.

Prioritize KLD over PPL for recipe comparison. Use PPL as a sanity check, but trust KLD for structural fidelity.

We are going to have extra PPL mean and PPL delta columns as soon as I run everything over again.

So far, reference PPL mean: 5.906250

oQ5:

KLD mean: 0.010254
KLD p95: 0.096191
KLD p99: 0.132812
PPL mean: 5.937500
PPL delta: +0.031250
RAM: 22.76

Again, absolute values are not as interesting as the relative difference within one test suite using the same data.

[deepsweet]

I've added PPL mean and PPL delta to the results.

Interesting observation that is actually pretty much aligned with "Why Maybe We're Measuring LLM Compression Wrong":

Zero perplexity delta alone doesn't mean that a quantization is flawless. On the contrary: PPL only cares about a "true token" prediction, but not the rest of the probability distribution. So a quantized model might predict all the tokens with perfect accuracy, but the "whole picture" might be wrong.

Thus, oQ6+ and Q5+ have zero perplexity delta, but their KLD still show us a slightly off-road path.

I'm glad to see that nuance in my own practical results.

@nmqanh I finally had time to try out 27B, and the results are pretty wild. I'll definitely dive into it deeper, but so far it feels like the dense model just really falls off on such a diverse prompt.

[nmqanh]

@deepsweet nice. when things get more settle. I can help generate data for 27B again.

[deepsweet]

I've come to a very interesting conclusion regarding the Qwen3.6-27B evaluation.

It gets completely blown away by the very diverse and unpredictable combined_all_micro.txt prompt:

• Reference mean PPL is 16.875000 (expected <5). The model really struggles to predict the correct next tokens.
• Affine Q4 has a high KLD 0.456142 (expected <0.1), lower mean PPL and a negative PPL delta of -3.937500. This suggests that the noise introduced by quantization acts as a "regularization filter", actually helping the model on this specific prompt.
• This highlights and stresses out the generalization limitations of the dense 27B architecture: the model likely wasn't trained on such a diverse mixture of domains simultaneously.
• In contrast, the 35B‑A3B MoE feels robust. Its expert architecture seems to allow it to switch and route between multiple domains easily.
• The evaluation itself is correct, because when I switch to the "community‑standard" but rather monotonous WikiText‑2, reference mean PPL becomes 4.593750, Q4 mean KLD 0.036133 and PPL delta +0.062500 (expected degradation). Also, on Edgar Poe's prose, mean PPL drops even lower to 1.359375.

My intention is to keep this flaw exposed as is, because a true evaluation should not aim to please its target. One could try to get nicer and more community-aligned numbers by using wikitext-2-raw-v1 test split.

[deepsweet]

👋

1. Tried to incorporate The Pile into the pipeline:

• https://en.wikipedia.org/wiki/The_Pile_(dataset)
• https://huggingface.co/datasets/EleutherAI/pile_val_test

Carefuly picked 6 different dataset domains, processed 16384 tokens of each, calculated per-domain mean metrics, and then the overall mean of everything altogether… And I found it being fragile and not reliable at all the moment I provide a different random seed to re-shuffle domain dataset records.

On the contrary, I like the combined_all_micro.txt more and more. Dense gemma-4-31B (not -it – "instruction-tuned" but the base pre-trained one) showed a very good reference mean PPL 3.515625. So it's really Qwen3.6-27B architecture collapses, which still doesn't hurt our main goal – relative difference between quantizations. Here is the incomplete results draft.

2. Implemented a new metric called "Top-1 accuracy":

• PPL - how much the model's confidence in the true token (not necessarily the chosen one) has changed
• Acc@1 – how often the model's top prediction matches the true next token
• KLD - how much the whole probability distribution for the next token over the vocabulary has shifted

Will add it to the results later.

3. Fixed the enormous peak memory pressure, so now we can compute even more useful numbers.

[deepsweet]

Long time no see!

Finished Qwen3.6-27B:

And Qwen3.6-35B-A3B:

New quantizations:

• ParoQuant
• JANG
• OptiQ

And lots of methodology and metrics explanation, as well as detailed results.