[DN-DAB-DETR] The output of ONNX's Mul OP is different from the TFLite's output. #327
Comments
PINTO0309
changed the title
|
When I test the only single Mul OP in real numpy data, the output of Mul OP is all matched.
import numpy as np
import tensorflow as tf
TFLITE_PATH = 'ScaleMul_float32.tflite'
POS_PATH = 'pos.npy'
POS_SCALE_PATH = 'pos_scales.npy'
# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path=TFLITE_PATH)
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
interpreter.set_tensor(
input_details[0]['index'],
np.load(POS_PATH).transpose(0,2,1)
)
interpreter.set_tensor(
input_details[1]['index'],
np.load(POS_SCALE_PATH).transpose(0,2,1)
)
interpreter.invoke()
out = interpreter.get_tensor(output_details[0]['index'])
res = np.allclose(
np.load('/data/ojw/convert/samples/res_mul.npy'),
out.transpose(0,2,1)
)
print(res)However, When I used the |
|
Thank you. I will be very busy until the second week of May and will not have enough time to debug and maintain at the same pace as before.
If the accuracy check is Probably related to this issue as well. By the way,
|
|
|
Thank you for your detail answer. I get the
While I'm trying to find the reasons of the problem, I noticed strange things. The original model(tflite) was only taking one input for the Mul OP instead of two. The other inputs appeared to be simply stored as a binary file. Is this the reason for this problem?
|
To me this is not a strange phenomenon at all. One of the two inputs of
This In other words, it is very likely that the very large error in the simple Usually it is very difficult to identify this problem as it is in this case. We need to trace the conversion logs in order starting with the Resize OP that is generating the constants and identify where the transposition error is occurring. |
|
The value is already broken in |
|
Thank you for kindly responding despite your busy schedule. For the rest of the issues, I will try to find the problem myself so as not to bother you. |
|
Note that only the https://github.com/PINTO0309/onnx2tf#parameter-replacement {
"format_version": 1,
"operations": [
{
"op_name": "/backbone/backbone.0/Gather",
"param_target": "attributes",
"param_name": "axis",
"values": 3
},
{
"op_name": "/backbone/backbone.1/Unsqueeze",
"param_target": "op",
"new_shape": [1,15,15,1]
},
{
"op_name": "/backbone/backbone.1/Unsqueeze_1",
"param_target": "op",
"new_shape": [1,15,15,1]
}
]
}
|
|
I made modifications to the model as follows and finally got
Thank you for your help! |
|
By the way, there seems to be some error in the Placeholder operator here and there, is it okay to ignore them?
|
|
It is hard to know what is going on without tracing it in detail, but it may be related to the following issue. |
|
Thank you for your fast response. I will check the issue that you mentioned. Can I close this issue? |
|
It is up to you to decide whether to close the issue. |
|
I am currently having a problem with the Gemm operator. All other outputs are correct, but only the output of the Gemm operator shows an
ONNX Link replace.json{
"format_version": 1,
"operations": [
{
"op_name": "/backbone/backbone.1/Unsqueeze",
"param_target": "op",
"new_shape": [1,15,15,1]
},
{
"op_name": "/backbone/backbone.1/Unsqueeze_1",
"param_target": "op",
"new_shape": [1,15,15,1]
}
]
}The strange thing here is that when I checked the onnx2tf -i new_dn_dab_detr_480x480.onnx \
-prf replace.json \
-onimc /transformer/encoder/layers.0/self_attn/Gemm_output_0 \
-cotof -cotoa 1e-1
Should I modify the replace.json file? |
|
The ONNX file you shared with me was corrupted and unreadable when I downloaded it. |
|
Oh, I'm sorry. That's my mistake. Here is the new ONNX file link. |
Please give me as much info as possible. This command you are posting does not include the Also, how did you generate the model for this image? If I can't reproduce the model you have on my PC, the man-hours required for testing will be very large, and it will be very difficult for me. |
1. Model shown in image
First, the part shown in the image is the MultiHeadAttention process, and below is the link to the corresponding model. For this part of model,
2. The part I think is weird2-1.
|
|
Hi, @PINTO0309! I found a very strange issue while experimenting with
scale=1e-2
( ... )
print(len(res[res==False]))
scale=1e+1
( ... )
print(len(res[res==False]))Model
class MHAtn(nn.Module):
def __init__(self):
super().__init__()
self.self_attn = nn.MultiheadAttention(256, 8, 0.0)
def forward(self, q, k, src):
return self.self_attn(q, k, src) |
|
Thank you. I am allocating some time to implement the research (Private implementation as a research position), so I will be a little slow in responding for a while. I am not ignoring this issue. |
|
I thought this issue was related to Matmul, so I read the issue you wrote below.
On the other hand, wouldn't it be possible to adjust the dimension of the weight that is passed to the input of
If the dimension of the weight in |
This is an incorrect test pattern because it generates different random values for ONNX and TFLite. onnx_dummy_input = np.random.rand(625,1,256).astype(np.float32) * scale
tflite_dummy_input = np.random.rand(625,256,1).astype(np.float32) * scaleHowever, you are correct, increasing the value of scale seems to break the value for some reason. scale=1e+1 # Error
query = np.random.rand(225,1,256).astype(np.float32)*scale
key = np.random.rand(225,1,256).astype(np.float32)*scale
value = np.random.rand(225,1,256).astype(np.float32)*scale
key_padding_mask = np.random.rand(1,225).astype(np.float32)*scale
# Set ONNX model
ort_session = onnxruntime.InferenceSession(
onnx_path,
providers=['CUDAExecutionProvider']
)
# Set TFLite model
interpreter = tf.lite.Interpreter(model_path=tflite_path)
interpreter.allocate_tensors()
tf_lite_model = interpreter.get_signature_runner()
ort_inputs = {}
ort_inputs['query'] = query
ort_inputs['key'] = key
ort_inputs['value'] = value
ort_inputs['key_padding_mask'] = key_padding_mask
# Get output of ONNX
onnx_out_list = ort_session.run(None, ort_inputs)
# Get output of TFLite
tt_lite_output = tf_lite_model(
query=tf.constant(query, dtype=tf.float32),
key=tf.constant(key, dtype=tf.float32),
value=tf.constant(value, dtype=tf.float32),
key_padding_mask=tf.constant(key_padding_mask, dtype=tf.float32),
)
Despite using the same model, both outputs were nearly identical when small values were entered. scale=1e-2
query = np.random.rand(225,1,256).astype(np.float32)*scale
key = np.random.rand(225,1,256).astype(np.float32)*scale
value = np.random.rand(225,1,256).astype(np.float32)*scale
key_padding_mask = np.random.rand(1,225).astype(np.float32)*scale
# Set ONNX model
ort_session = onnxruntime.InferenceSession(
onnx_path,
providers=['CUDAExecutionProvider']
)
# Set TFLite model
interpreter = tf.lite.Interpreter(model_path=tflite_path)
interpreter.allocate_tensors()
tf_lite_model = interpreter.get_signature_runner()
ort_inputs = {}
ort_inputs['query'] = query
ort_inputs['key'] = key
ort_inputs['value'] = value
ort_inputs['key_padding_mask'] = key_padding_mask
# Get output of ONNX
onnx_out_list = ort_session.run(None, ort_inputs)
# Get output of TFLite
tt_lite_output = tf_lite_model(
query=tf.constant(query, dtype=tf.float32),
key=tf.constant(key, dtype=tf.float32),
value=tf.constant(value, dtype=tf.float32),
key_padding_mask=tf.constant(key_padding_mask, dtype=tf.float32),
)
|
|
We already know that the |
|
You mean, this issue is related to a third party? |
|
Test data is not properly transposed from NHWC to NCHW. I am trying to figure out how to deal with this. Essentially, the Numpy array of test data should be NHWC. https://github.com/PINTO0309/onnx2tf#cli-parameter I even identified the problem here. onnx2tf/onnx2tf/utils/common_functions.py Lines 3522 to 3529 in b1cf6e7 |
PINTO0309
added
Parameter replacement
|
I beat the bug. 👍
|
|
Thank you for your effort. I appreciate it.😊
|
|
Please wait a little longer. Right now I am in the middle of a regression test with my CI. I will upgrade to v1.10.0 when it is all green. |
|
I understand it. Thank you so much. |
|
https://github.com/PINTO0309/onnx2tf/releases/tag/1.10.0 The following commands can be used for conversion. Please try it. Originally, the problem was quite complicated by the combination of multiple problems with the way the
|
|
Thank you for your hard work. When I inference the converted tflite model, I still get the
import numpy as np
import tensorflow as tf
TFLITE_PATH = 'my/tflite/path/final_model_float32.tflite'
# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path=TFLITE_PATH)
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
interpreter.set_tensor(
input_details[0]['index'],
np.ones(list(input_details[0]['shape'])).astype(np.float32)
)
interpreter.invoke()
out_list = []
for i in range(len(output_details)):
out_list.append(interpreter.get_tensor(output_details[i]['index']))
print(out_list[0]) |
|
I still get
My library version is exactly same with you. @PINTO0309 Also, in the |
|
The ONNX file you shared with me is corrupt and cannot be displayed on Netron.
If the accuracy check after conversion is normal, I can only surmise that either a problem on the runtime side or the fact that all input data is 1 could be the cause of the problem. To begin with, is the mod_dn_dab_detr.onnx file you are using for the conversion the same as the mod_dn_dab_detr.onnx you shared with me the other day? This is a completely different issue from the story of the result being I can attest to you that there is nothing wrong with my environment.
If you see |
|
I will use your docker envirionments. Thank you for your answer. |
|
Always thank you for your hard work. I tested in the docker environment you shared and was able to confirm that the matched message is displayed in the
import os
import pickle
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=Warning)
warnings.simplefilter(action='ignore', category=DeprecationWarning)
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=RuntimeWarning)
import random
random.seed(0)
import numpy as np
np.set_printoptions(
precision=6,
floatmode='fixed',
suppress=True,
edgeitems=3,
linewidth=100,
)
np.random.seed(0)
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
from tensorflow.lite.python import interpreter as iw
TFLITE_PATH = 'saved_model/mod_dn_dab_detr_float32.tflite'
interpreter = iw.Interpreter(
model_path=TFLITE_PATH,
num_threads=4,
)
input_details = interpreter.get_input_details()
input_shape_1 = input_details[0]['shape']
output_details = interpreter.get_output_details()
test_data1 = np.random.randn(*input_shape_1).astype(np.float32)
interpreter.allocate_tensors()
interpreter.set_tensor(
input_details[0]['index'],
test_data1,
)
interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])
print('')
print('tflite ===============================================')
print(output_data)
# with open('output/tflite_out.pkl', 'wb') as f:
# pickle.dump(out_list, f) |
|
I can't say whether the following is really the correct test data to input into this model, because I don't know. Is it correct to enter random values? Since the accuracy check of the model is normal, it is better to suspect first the parts of the model other than its structure. test_data1 = np.random.randn(*input_shape_1).astype(np.float32)You may want to find out which operations have divergent values. |
|
I have tried various methods other than the test dummy data mentioned below, but I always got a value of np.ones(input_shape_1).astype(np.float32) # unmatched
torch.rand(input_shape_1, dtype=torch.float32).numpy() # unmatched
np.random.rand(1,800,800,3).astype(np.float32) # unmatched
# etc ...Then, does your opinion mean that there is a problem with the tflite runtime? |
|
https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_327/test/mod_dn_dab_detr.onnx I cannot reproduce it. import os
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=Warning)
warnings.simplefilter(action='ignore', category=DeprecationWarning)
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=RuntimeWarning)
import random
random.seed(0)
import numpy as np
np.set_printoptions(
precision=6,
floatmode='fixed',
suppress=True,
edgeitems=3,
linewidth=100,
)
np.random.seed(0)
os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
import onnxruntime
ONNX_PATH = 'mod_dn_dab_detr.onnx'
ort_session = onnxruntime.InferenceSession(ONNX_PATH, providers=['CPUExecutionProvider'])
input_name = ort_session.get_inputs()[0].name
input_shape = ort_session.get_inputs()[0].shape
output_num = len(ort_session.get_outputs())
test_data = np.ones(input_shape, dtype=np.float32)
ort_inputs = {
ort_session.get_inputs()[0].name: test_data,
}
onnx_out = ort_session.run(
['7370'],
ort_inputs,
)
TFLITE_PATH = 'mod_dn_dab_detr_float32.tflite'
interpreter = tf.lite.Interpreter(model_path=TFLITE_PATH)
tf_lite_model = interpreter.get_signature_runner()
tflite_output_float32 = tf_lite_model(
input_1=test_data.transpose(0,2,3,1),
)
TFLITE_PATH = 'mod_dn_dab_detr_float16.tflite'
interpreter = tf.lite.Interpreter(model_path=TFLITE_PATH)
tf_lite_model = interpreter.get_signature_runner()
tflite_output_float16 = tf_lite_model(
input_1=test_data.transpose(0,2,3,1),
)
print('=====================================================')
print(onnx_out[0].shape)
print(tflite_output_float32['7370'].shape)
print(tflite_output_float16['7370'].shape)
print('onnx =================================================')
print(onnx_out[0])
print('')
print('tflite float32 =======================================')
print(tflite_output_float32['7370'])
print('')
print('tflite float16 =======================================')
print(tflite_output_float32['7370']) |
I'm so sorry to bothering you. But, how can I get the input name?
|
|
https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_327/test2/mod_dn_dab_detr.onnx import os
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=Warning)
warnings.simplefilter(action='ignore', category=DeprecationWarning)
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=RuntimeWarning)
import random
random.seed(0)
import numpy as np
np.set_printoptions(
precision=6,
floatmode='fixed',
suppress=True,
edgeitems=3,
linewidth=100,
)
np.random.seed(0)
os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
import onnxruntime
ONNX_PATH = 'mod_dn_dab_detr.onnx'
ort_session = onnxruntime.InferenceSession(ONNX_PATH, providers=['CPUExecutionProvider'])
input_name = ort_session.get_inputs()[0].name
input_shape = ort_session.get_inputs()[0].shape
output_num = len(ort_session.get_outputs())
test_data = np.ones(input_shape, dtype=np.float32)
ort_inputs = {
ort_session.get_inputs()[0].name: test_data,
}
onnx_out = ort_session.run(
['7370'],
ort_inputs,
)
TFLITE_PATH = 'mod_dn_dab_detr_float32.tflite'
interpreter = tf.lite.Interpreter(model_path=TFLITE_PATH)
tf_lite_model = interpreter.get_signature_runner()
inputs = {'input.1': test_data.transpose(0,2,3,1)}
tflite_output_float32 = tf_lite_model(
**inputs
)
TFLITE_PATH = 'mod_dn_dab_detr_float16.tflite'
interpreter = tf.lite.Interpreter(model_path=TFLITE_PATH)
tf_lite_model = interpreter.get_signature_runner()
inputs = {'input.1': test_data.transpose(0,2,3,1)}
tflite_output_float16 = tf_lite_model(
**inputs
)
print('=====================================================')
print(onnx_out[0].shape)
print(tflite_output_float32['7370'].shape)
print(tflite_output_float16['7370'].shape)
print('onnx =================================================')
print(onnx_out[0])
print('')
print('tflite float32 =======================================')
print(tflite_output_float32['7370'])
print('')
print('tflite float16 =======================================')
print(tflite_output_float32['7370']) |
|
Your code works very well. I want to express my gratitude for always helping me. As a token of my appreciation, I have decided to become your sponsor. Although I may not be able to provide a large amount of support as a student, I still want to show my gratitude to you in this way. I will close this issue. Thank you again @PINTO0309 :) |
You don't have to strain yourself. I earned all of my tuition money on my own and attended college. I am well aware that there are times in our lives when we struggle financially. When you graduate and earn a lot of money, please buy me a beer. |
|
Thank you for saying so. I will definitely do that 😊 |
Issue Type
Others
onnx2tf version number
1.9.12
onnx version number
1.13.1
onnxruntime version number
1.13.1
onnxsim (onnx_simplifier) version number
0.4.17
tensorflow version number
2.12.0
Download URL for ONNX
https://drive.google.com/file/d/1rjqhNfn85we2IG6YwKOJlhKv-fMRFha7/view?usp=sharing
Parameter Replacement JSON
{ "format_version": 1, "operations": [ { "op_name": "/backbone/backbone.1/Unsqueeze", "param_target": "outputs", "param_name": "/backbone/backbone.1/Unsqueeze_output_0", "post_process_transpose_perm": [3,0,1,2] }, { "op_name": "/backbone/backbone.1/Unsqueeze_1", "param_target": "outputs", "param_name": "/backbone/backbone.1/Unsqueeze_1_output_0", "post_process_transpose_perm": [3,0,1,2] } ] }Description
Hi! @PINTO0309. I'm trying to convert DN-DAB-DETR to TFLite. However, when I multiply two tensors, the output of ONNX is different from the TFLite's output. Below is my result using
-cotof.-cotofresultShould I avoid tensor's multiply?
The text was updated successfully, but these errors were encountered: