Noisy inference results with TF2 int8 tflite models

[Rahn80643]

Hi,

I'm migrating our segmentation model from TensorFlow1 to TensorFlow2, and the model is retrained with TensorFlow quantization aware training technique; however, the inference results of TF2 tflite model from ArmNN are not ideal, the comparisons and environment settings are shown below:

• ArmNN 22.02
• Compute Library 22.02
• FlatBuffers 1.12.0
• Protobuf 3.12.0
• TensorFlow 2.4.0

Based on the inference results from TensorFlow tflite API, the weights in tflite models seems normal, but the inference results from ArmNN contain many noisy pixels, I want to ask:

1. Is this issue resulted from the version of TensorFlow or tflite?
2. How to effectively improve the inference results in ArmNN?

Best Regards,
Rahn

[TeresaARM]

Hi @Rahn80643,

thank you very much for your comment. which backend are you using when you run ArmNN, CpuAcc, GpuAcc or CpuRef?

[Rahn80643]

Hi @TeresaARM,

I've tested with CpuAcc and GpuAcc backend, and the inference results are similar

[TeresaARM]

Hi @Rahn80643,

could you give it a go to CpuRef? and see if the results are closer to TensorFlow? The performance will be worse than CpuAcc and GpuAcc, as CpuRef is just a "reference", but this information will help us to find out where the issue is.

Thank you!

[Rahn80643]

Hi @TeresaARM,

The followings are the inference results using CpuRef, and the results are not closer to those from TensorFlow

Thank you

[Rahn80643]

Hi,

I compared two int8 tflite models trained in TensorFlow1 and TensorFlow2 respectively, and I found the input zero point values are different. Is this the main reason why inferring TensorFlow2 int8 tflite model generates noisy results? If so, how to include the input zero point into ArmNN inference code to solve this issue?

Rahn

[MatthewARM]

yes, this could be the problem. In your application code, when you convert the input image into a tensor for Arm NN, are you taking the zero-point into account?

[MatthewARM]

If this is the problem, there is some good explanation of it here: #161

[Rahn80643]

Hi,

I've tried the following modifications but the inference results still contain noisy results:

1. Based on thread #161, the author had tried not to do preprocess the image and divide the output by 255. I tried the same process in ArmNN; however, since the values of output results are within the range of [0, 255], if the values are divided by 255, the results will be all 0s.

2. the input images are preprocessed based on the following formula in ArmNN,
   new_value = ((old_value/scale_factor) - mean)/ stddev. Therefore, I've tried changing mean values, but the inference results still contain noisy results:

Apart from modifying the mean values in preprocess stage, are there other approaches to take the zero-point into account in ArmNN settings?

[morgolock]

I'm closing this due to inactivity, if you require support please reopen.