The output is different with the same input on CUDA

[freedenS]

Describe the bug
I trained a MaskRcnn model and convert it to onnx. When i use test images to do inference,it can give the correct output. But if i do the inference with a white image (all 255) first, and then use test images to do inference, it can't recognize correctly with onnxruntime-gpu1.3.0.
if i use onnxruntime1.3.0, it will not happen.

System information

• OS Platform and Distribution :Windows 10 (c++) & Linux Ubuntu 16.04(python3.7)
• ONNX Runtime installed from :binary from pip install
• ONNX Runtime version:1.3.0
• Python version:3.7
• Visual Studio version (if applicable):2017
• GCC/Compiler version (if compiling from source):compiling from binary
• CUDA/cuDNN version:CUDA10.1 & cuDNN7.6.5
• GPU model and memory:2080Ti 11G

To Reproduce

• Describe steps/code to reproduce the behavior.
• Attach the ONNX model to the issue (where applicable) to expedite investigation.
  onnx model and images are here https://drive.google.com/open?id=1c7GDq8AYgneP6mdqFPSH1GKl7Oggn8Mq
  do the inference with a white image (all 255) first, and then use test images to do inference

import os
import onnxruntime
import cv2
import torch #1.5.0
from detectron2.data.transforms as T #just for resize
filelist = os.listdir('./')
sess = onnxruntime.InferenceSession('./segmentation.onnx')
for i in filelist:
    inputs = cv2.imread('./' + i)
    transform_gen = T.ResizeShortestEdge([800,800],1333)
    inputs = transform_gen.get_transform(inputs).apply_image(inputs)
    inputs = torch.as_tensor(inputs.astype('float32').transpose(2,0,1))
    inputs = inputs.unsqueeze(0)
    res = sess.run(None, {sess.get_inputs()[0].name:inputs.numpy()})
    print (res)

Screenshots
If applicable, add screenshots to help explain your problem.

[hariharans29]

Is 1.jpg the fully white image and 2.jpg a normal test image ?

[freedenS]

@hariharans29 Yes

[hariharans29]

Thanks. I ll take a look.

[freedenS]

Thank you!

[freedenS]

@hariharans29 anything new?

[hariharans29]

Sorry - I haven't had time to look at this yet. I ll try and take a look soon.

My guess is that when you feed in all 255's, CUDA/ CuDNN /CuBLAS doesn't like it (maybe triggers a numerical overflow in some operation ?) and the CUDA EP goes into a bad state after this and hence you see issues with subsequent runs.

Also, usually there is a "pre-processing step" for image models which involves subtracting per channel mean and dividing by the per channel std deviation. Not sure, if the all 255 image causes an issue because of this (because clearly it is not "pre-processed").

Does this all 255 image issue block your use-case in any way or is it just that you are curious about it ?

[freedenS]

Thanks for your reply!
i move the preprocess into inference(include subtract mean and divide std deviation), so the preprocess of my model is only image resize.(will it cause the problem?)
This problem will happen only when the model is mask-rcnn, the Conv_3281 is in mask_head layer.
I test some faster-rcnn models, and haven't found this problem.
And i find that some normal images (such as other model's test image) could also cause this problem.

[hariharans29]

Oh so the preprocessing is built into the model ? I see. Okay- I ll take a look at this very soon.

[freedenS]

Yes, the preprocessing(subtracting per channel mean, dividing by the per channel std deviation and padding for fpn) is built into the model.
In addition, I have made some other attempts.Hope it will be useful.

1. if i move the preprocessing out, and add image_size as second input which is used by clip.
   result: I can only get segmentation fault (core dumped)
2. if i move the preprocessing out, and use current input's size for clip (after resize and padding, it may be out of image's boundary)
   result: I got the correct result!
   No matter which way I choose， i can get the correct result as long as i use onnxruntime1.3(cpu)
   Thank you!

[hariharans29]

Taking a look at the model as is (with preprocessing inside it)

[hariharans29]

Hi @freedenS - What is the tensor dimension for height and width ? Is there any aspect ration to be maintained between the two ?

[hariharans29]

Hi @freedenS ,

I couldn't repro this with the height == width == 224. See below - I run all 255s and random valued inputs alternately. I tried with 1.3.0 onnxruntime GPU package (you can uncomment the appropriate lines as you run on your box) . I set log_severity_level to 0 and confirmed that a lot (almost all nodes) got placed on the CUDA EP.

I think I might know why you are running into this issue. Are you using a fixed height and width combination between runs ? My guess is that you are varying height and width. The model is annotated with "dynamic" dim values for height and width - this is a little misleading - while ORT as an engine will accept any dim value, the model might inherently expect that there be some relationship (aspect ratio) between the height and width and the model will work on the assumption that the user has adhered to this requirement and when this is not met the model is bound to fail. You need to check what shapes the model was trained with and check if you need to adhere to any aspect ratio requirements.

import onnxruntime as rt
import numpy as np

opt = rt.SessionOptions()

# Uncomment to see which nodes get placed on CUDA EP
#opt.log_severity_level = 0

sess = rt.InferenceSession(r'segmentation.onnx', opt)

input_name_0 = sess.get_inputs()[0].name

# run 1 - all 255
input_0 = np.zeros((1, 3, 224, 224), dtype = np.float32)
input_0 += 255
#print(input_0)
pred_onnx_0 = sess.run(None, {input_name_0: input_0})
print(pred_onnx_0[0])

# run 2 - random
input_1 = np.random.rand(1, 3, 224, 224).astype(np.float32)
#print(input_1)
pred_onnx_1 = sess.run(None, {input_name_0: input_1})
print(pred_onnx_1[0])

# run 3 - all 255
pred_onnx_2 = sess.run(None, {input_name_0: input_0})
print(pred_onnx_2[0])

# run 2 - random
pred_onnx_3 = sess.run(None, {input_name_0: input_1})
print(pred_onnx_3[0])

print ("Done")

[hariharans29]

Oh I just saw your input script and noticed that you were feeding [800, 800] inputs and I could hit the issue with that. My guess is that there is something in the input combination (input shape + ConvTranspose attributes that CuDNN does not like). This doesn't look like it is an ORT issue in that it doesn't break runs after you feed it all 255s.

I ll debug what shapes the inputs have by the time it reaches the faulty ConvTranspose node in some time.