Int8 Quantized model same output on different inputs

[xyrod6]

System Information

OpenCV python version: 4.7.0.72
Operating System / Platform: Windows 10
Python version: 3.10.6

Detailed description

I am trying to run an int8 quantized model exported to ONNX with opencv backend.

The problem is with the output of the model. Running inference on different images gives the same output tensor.

I tried the inference with pytorch (quantized .pth model) and with openvino (same onnx file) and they give good results.

OpenCV output:

OpenVino output:

Steps to reproduce

Note: This is not a fully trained model, so it should not return the correct label!

The model:
int8_quantized.zip
Test images:
test.zip

import openvino.runtime as vino
import cv2
import argparse
import numpy as np
import random
import time

def postprocess(output_blob,_labels):
        _labels=['beige', 'black', 'blue', 'brown', 'golden', 'green', 'grey', 'red', 'silver', 'white']
        batched_class_id_list = []
        for o in output_blob:
            #o=softmax(o)
            class_id_list = o.argsort()[::-1][:1]
            batched_class_id_list.append(class_id_list)
        if len(_labels) > 0:
            batched_predicted_labels = []
            for class_id_list in batched_class_id_list:
                predicted_labels = []
                for class_id in class_id_list:
                    predicted_labels.append(_labels[class_id])
                batched_predicted_labels.append(predicted_labels)
            return batched_predicted_labels
        else:
            return batched_class_id_list

ie = vino.Core()
model = ie.read_model(r"model_path.onnx")
cv_model = cv2.dnn.readNet(r"model_path.onnx")

compiled_model = ie.compile_model(model=model, device_name="CPU")
input_layer = compiled_model.input(0)
img_size = (112, 112) 

src_img = cv2.imread(r"img_path.jpg")
src_size = src_img.shape[:2]

img = cv2.resize(src_img,img_size, interpolation=cv2.INTER_LINEAR)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = img.astype(dtype=np.float32)
img /= 255.0
input_image = np.expand_dims(img.transpose(2, 0, 1), 0)
cv_model.setInput(input_image)
cv_outputs = cv_model.forward()
cv_pred=postprocess(cv_outputs,None)
pred = compiled_model([input_image])

print("OPENCV OUTPUT TENSOR:")
print(cv_outputs)
print("OPENVINO OUTPUT TENSOR:")
print(pred)
tmp_out=pred[compiled_model.output(0)]

#print(postprocess(tmp_out,None))

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[zihaomu]

Hi @xyrod6, I have checked your int-8 model. The int-8 model is QDQ model. Currently, OpenCV can not work well on QDQ model.
Please try to quantize the model by onnx with following code:

quantize_static(_, _, _, 
    QuantFormat.QOperator, # start from onnxruntime==1.11.0, quant_format is set to 
                           #  QuantFormat.QDQ by default, which performs fake quantization
    )

For more info, please refer https://github.com/opencv/opencv_zoo/blob/master/tools/quantize/quantize-ort.py#L78

[r-Harshita]

I have optimized the existing code for inference using OpenCV and OpenVINO by removing redundant imports and variables, using numpy operations instead of Python loops, and avoiding redundant computations. The updated code is more efficient and should provide faster inference results.

Please find the optimized code below:

[ import openvino.runtime as vino
import cv2
import argparse
import numpy as np
import time

def postprocess(output_blob):
_labels=['beige', 'black', 'blue', 'brown', 'golden', 'green', 'grey', 'red', 'silver', 'white']
class_ids = np.argsort(output_blob, axis=1)[:, -1:]
predicted_labels = np.array(_labels)[class_ids]
return predicted_labels.tolist()

ie = vino.Core()
model = ie.read_model(r"model_path.onnx")
cv_model = cv2.dnn.readNet(r"model_path.onnx")

compiled_model = ie.compile_model(model=model, device_name="CPU")
input_layer = compiled_model.input(0)
img_size = (112, 112)

src_img = cv2.imread(r"img_path.jpg")
resized_img = cv2.resize(src_img, img_size, interpolation=cv2.INTER_LINEAR)
resized_img = cv2.cvtColor(resized_img, cv2.COLOR_BGR2RGB)
resized_img = resized_img.astype(dtype=np.float32)
resized_img /= 255.0
input_image = np.expand_dims(resized_img.transpose(2, 0, 1), 0)
cv_model.setInput(input_image)
cv_outputs = cv_model.forward()
cv_pred = postprocess(cv_outputs)
pred = compiled_model([input_image])
optimised_pred = postprocess(pred[compiled_model.output(0)])

print("OPENCV OUTPUT TENSOR:")
print(cv_outputs)
print("OPENVINO OUTPUT TENSOR:")
print(pred)
print("OPTIMISED OPENVINO OUTPUT TENSOR:")
print(optimised_pred) ]

If this helps, please assign me this project so i can show my skills.