Quantization TOCO implementation

[lorenz0890]

Hi,

i am trying to understand how exactly TOCO does quantize an annotated model. Assume we have a keras model:

Conv2D
MaxPool
Conv2D
MaxPool
Flatten
Dense
Dense(SoftMax)

where the Conv2D layers have been annotated with for post-training quantization using a QuantizeConfig for quantizing weights and activations based on the following Quantizer:

LastValueQuantizer(num_bits=8, symmetric=True,
 narrow_range=False,
per_axis=False)

As far as i understand, converting a such annotated keras model to TFLite using TOCO should yield a a model where the Conv2D layers have been quantized to 8bit integers while the rest stays at 32bit float. Is that correct? If yes, how is the rounding from the input keras Conv2D layers in 32bit to int8 accomplished? Is the situation the same when using float16 instead of int8? What abot the siutuation where i set num_bits=4?

Thanks for your help!

[jianlijianli]

Thanks for using the tooling and reaching out.

You are correct that the quantization tooling you are using is able to selectively quantize certain layers while leaving the rest in floating point (this offers well controlled trade-off between accuracy, speed, and model size). When a model has both int8 op and float op, there will be a "quantization" op inserted in between to do the conversion of int8 and float32.

Hope this helps.

[lorenz0890]

Hi,

thank for your reply and confirming my assumptions, this does indeed help me.

I am working on a research project using TF about analyzing quantization errors in CNN topologies and i need to know exactly what happens, but the converter optimizations are still an unknown factor. Can you tell me what exactly happens when the optimization flag below is used or can you point me to a relevant paper?

converter.optimizations = [tf.lite.Optimize.DEFAULT]

Thanks a lot for your help!

[ravikyram]

@lorenz0890

Please, see the link for the description of converter optimizations.Thanks!

[lorenz0890]

@lorenz0890

Please, see the link for the description of converter optimizations.Thanks!

Thanks!