CPU: add torch.compile for F.double_quant and F.quantize_4bit

[Xia-Weiwen]

We didn't use torch.compile for these two functions because of some issue. These issues are now fixed in PyTorch.
Using torch.compile brings performance gain and also avoid some issues during finetuning.

[Titus-von-Koeller]

Cool! Is this ready? Because it's still marked as draft.

[Xia-Weiwen]

Cool! Is this ready? Because it's still marked as draft.

Thanks. I think @jiqing-feng will collect some data before we mark it ready.

[jiqing-feng]

Hi @Titus-von-Koeller

For quantize_4bit_impl, it only happens in the load model, so these are no difference with nf4/fp4 performance before and after we applied torch.compile.

For double_quant_impl, I have tested it on llama-7b inference and opt-6.7b finetune;found no difference in performance.

We plan to apply the torch.compile in quantization op but didn't integrate it into the first version; this is because the issue in the comment:Don't use a torch.compile for now due to PyTorch issue https://github.com/pytorch/pytorch/issues/124382. Now the issue has been fixed, so we can apply torch.compile again. The torch.compile will be optimized constantly in pytorch :)

Thx!

[Titus-von-Koeller]

Thanks @jiqing-feng for explaining. Then it's ready to merge, @Xia-Weiwen, right?

[Xia-Weiwen]

Thanks @jiqing-feng for explaining. Then it's ready to merge, @Xia-Weiwen, right?

Yes. Please. Thanks.