Benefits of Using QAT Before GGUF Quantization?

[kiyoonyoo]

Hi,
thank you for the amazing project.

I have a question regarding quantization workflows. Does applying QAT before convering to GGUF format (e.g. using Q4, Q4_K_M) result in better quality fompared to directy quantizing with GGUF alone?

I'm planning to serve my model using llama.cpp, so converting to GGUF is required. I’ve noticed a noticeable quality drop when using methods provided by llama.cpp, so I’m considering trying QAT to mitigate this.

Has anyone experimented with this approach or have any insights to share?

Thanks.

[andrewor14]

Hi @kiyoonyoo, applying QAT before quantization should help with the accuracy of the quantized model, as long as the QAT configs match the GGUF formats. I'm not super familiar with the GGUF formats but my understanding is Q4_K_M uses mixed dtypes (4 bits and 6 bits?), so you may have to fake quantize specific layers in your model to match that format exactly when you do QAT. Please give it a try and let me know if I can help in any way :)

[jerryzh168]

@kiyoonyoo yeah we should be able to improve accuracy with QAT or other accuracy preserving techniques like AWQ, GPTQ, one thing for QAT is that we'd need to make sure we also have gguf compatible fake quant ops

I'm planning to do this in a few months, we have WIP PR here to add gguf format support: #2042, and then we'll compose with other techniques available in torchao to improve accuracy, let me know if you want to contribute to this, either with development or testing

[kiyoonyoo]

Hi @andrewor14 and @jerryzh168, thanks for the helpful responses.

I attempted to convert a QAT model to Q4 GGUF format, but unfortunately didn’t observe any improvement in performance. This might be due to incompatibilities between the quantization techniques, such as the mixed dtypes you mentioned.

Here are the perplexity (PPL) results on the wiki-test dataset using Qwen-3-1.7B-Instruct:

-	Plain Lora	QAT-lora
BF16	11.02 (0.08)	11.11 (0.08)
Q4	11.57 (0.09)	11.62 (0.09)

---

@jerryzh168
It would be very helpful for my project to have GGUF-compatible fake quantization ops. I’ll take a look at the PR when I get some time and will let you know if there’s any way I can contribute.

Thanks again!

[jerryzh168]

@kiyoonyoo that's great. I think it's possible that QAT in torchtune is not compatible with GGUF format, that is for 8da4w format I think. we'd need GGUF specific fake quant ops for QAT to be useful here.

[kiyoonyoo]

Thanks for the input.
I will close the issue for now.