Can I get inference speed gain using quantization and pruning.

[WilliamZhaoz]

Hi,

1. Can I get inference run time speed gain using quantization in WinMLTools.
2. Can I get inference run time speed gain after I pruned my model.(weight prune, in other words, there are more 0 in weight.)
   envs:
   OS: windows server 2019
   processor: Intel(R) Xeon(R) CPU E5-2673 v4 @2.30GHz 2.29GHz
   python env: a anaconda virtual env.
   Thanks.

[martinb35]

Hi @WilliamZhaoz , thanks for your questions.

1. Quantization from WinMLTools is primarily for reducing the size of the model. It can actually make the inference run slightly slower. Please use winmlrunner to verify, though.
2. I think you're asking about sparse tensor support, which is not currently in WinML.

-Brian

[nlml]

It seems in TensorFlow, quantization can afford speed ups on CPU (https://www.tensorflow.org/lite/performance/post_training_quantization). I have also seen this in academic literature.

Are there any plans to improve the inference of quantized models on CPU to afford such speedups?

Cheers
Liam