Tensorrt 8.6 GA : C++ Inference gives diffrence results compared to onnx || pt model python inference

[smyang03]

Description

Hi there

i got a saved model converted to onnx in order to run inference using Tensorrt 8.6 c++ API.
The model is a virtual model from the openai clips model.
(https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/visual.onnx)

checking the embedded value of the image in the model.

but output results are diffrence compared to python
infrence(pt or onnx runtime) to trt in tensorrt c++ api

I checked and found that python inference is correct and tensorrt c++ api is incorrect. (both fp32)

The next log is trtexec verbose log after onnx check. You can check it (
https://drive.google.com/file/d/1carAjQ_oP2xEkia48J0tNztQ5meukaqZ/view?usp=drive_link)

Environment

TensorRT Version:8.6.1GA

NVIDIA GPU:GTX3090

NVIDIA Driver Version:

CUDA Version:12.0

CUDNN Version:8.9.4.25

Operating System:

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

model
(https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/visual.onnx)
log
(https://drive.google.com/file/d/1carAjQ_oP2xEkia48J0tNztQ5meukaqZ/view?usp=drive_link)

Model link:

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

[zerollzeng]

Looks like output between onnxruntime and TRT is good, Could you please check the diff between pytorch and onnxruntime?

[I] onnxrt-runner-N0-09/12/23-15:16:55  | Completed 1 iteration(s) in 31.82 ms | Average inference time: 31.82 ms.
[I] Accuracy Comparison | trt-runner-N0-09/12/23-15:16:55 vs. onnxrt-runner-N0-09/12/23-15:16:55
[I]     Comparing Output: 'output' (dtype=float32, shape=(1, 512)) with 'output' (dtype=float32, shape=(1, 512))
[I]         Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I]         trt-runner-N0-09/12/23-15:16:55: output | Stats: mean=-0.020254, std-dev=0.48418, var=0.23443, median=-0.016851, min=-8.3656 at (0, 92), max=2.1781 at (0, 321), avg-magnitude=0.24284
[I]         onnxrt-runner-N0-09/12/23-15:16:55: output | Stats: mean=-0.020254, std-dev=0.48418, var=0.23443, median=-0.016851, min=-8.3656 at (0, 92), max=2.1781 at (0, 321), avg-magnitude=0.24284
[I]         Error Metrics: output
[I]             Minimum Required Tolerance: elemwise error | [abs=2.3842e-06] OR [rel=0.00084043] (requirements may be lower if both abs/rel tolerances are set)
[I]             Absolute Difference | Stats: mean=4.5608e-07, std-dev=4.0203e-07, var=1.6163e-13, median=3.4645e-07, min=0 at (0, 83), max=2.3842e-06 at (0, 93), avg-magnitude=4.5608e-07
[I]             Relative Difference | Stats: mean=1.0028e-05, std-dev=4.5086e-05, var=2.0327e-09, median=2.1451e-06, min=0 at (0, 83), max=0.00084043 at (0, 394), avg-magnitude=1.0028e-05
[I]         PASSED | Output: 'output' | Difference is within tolerance (rel=1e-05, abs=1e-05)
[I]     PASSED | All outputs matched | Outputs: ['output']
[I] Accuracy Summary | trt-runner-N0-09/12/23-15:16:55 vs. onnxrt-runner-N0-09/12/23-15:16:55 | Passed: 1/1 iterations | Pass Rate: 100.0%
[I] PASSED | Runtime: 15.875s | Command: /home/scratch.zeroz_sw/miniconda3/bin/polygraphy run visual.onnx --trt --onnxrt

[smyang03]

python or onnx runtime reuslt is 512 output is correct

Both output 512 float values, but there is a difference in the values.

What to check
-- sample test image is black image
trt and python, onnxruntime engine input image same
-- normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))

##In the case of cuda 11.6 and tensorrt 8.4.1.6, there are 512 output data from trt, but the values ​​do not match the onnxruntime and python output data, so it was updated to tensorrt 8.6.1 and cuda 12.0.

builder flag = fp32

1. sample onnxruntime or python run output
   [ -1.4941e-01, -2.6343e-01, -5.6592e-01, -7.1899e-02, 1.5564e-01,
   -4.2755e-02, -3.8721e-01, 1.2725e+00, 2.8345e-01, 2.7417e-01,
   5.0586e-01, -1.5564e-01, 3.5376e-01, 1.5152e-02, -1.4221e-01,
   1.1243e-01, 1.1053e-01, 1.5295e-01, 3.1830e-02, 3.7158e-01,
   3.8428e-01, 1.5515e-01, -7.9956e-02, -4.1321e-02, -1.1285e-01,
   -1.9849e-01, -4.2041e-01, -4.7302e-02, 3.5376e-01, -3.2837e-01,
   -1.8896e-01, 1.6541e-01, -2.8076e-01, -2.8809e-02, -5.6299e-01,
   -2.7417e-01, 1.4075e-01, -1.8762e-01, 5.5267e-02, -1.8086e+00,
   -3.0225e-01, 3.6304e-01, -1.9666e-01, 4.7095e-01, -2.6221e-01,
   1.7981e-01, -2.4011e-01, -1.2274e-01, -9.7412e-02, -2.3169e-01,
   -1.1121e-01, -4.0649e-01, 3.9819e-01, 3.4790e-02, 1.0840e-01,
   1.2280e-01, 1.9006e-01, -6.7139e-02, -1.5027e-01, 2.1439e-02,
   -4.5380e-02, -4.9561e-01, 1.6492e-01, 4.1040e-01, 2.6154e-02,
   -7.1777e-02, -8.4229e-02, -4.0234e-01, 1.2566e-02, -3.2764e-01,
   -5.0732e-01, -1.4844e-01, 2.5879e-01, 1.9653e-01, -3.3789e-01,
   -2.0142e-01, 1.2561e-01, 2.0764e-01, 1.7670e-02, -9.3628e-02,
   -7.0007e-02, 5.6458e-02, 2.2375e-01, -3.7891e-01, 2.8882e-01,
   1.5576e-01, 2.1106e-01, 5.2588e-01, 1.1841e-01, -4.1260e-01,
   -3.4082e-01, 8.2275e-02, -9.1016e+00, 5.3613e-01, 3.0835e-01,
   -5.5237e-02, 3.8184e-01, 1.0742e-01, -4.2969e-01, -6.3232e-01,
   -2.2437e-01, 1.4758e-01, 1.0547e-01, -3.7415e-02, -1.3110e-01,
   5.1605e-02, -1.3184e+00, 3.9966e-01, 3.3813e-01, -1.3550e-01,
   -5.5603e-02, -1.2109e-01, -2.1619e-01, -1.8213e-01, -2.2192e-01,
   -3.6792e-01, -1.5356e-01, -2.2485e-01, -1.9458e-01, 6.7822e-01,
   -3.8867e-01, 5.2887e-02, -1.7371e-01, 9.4971e-02, 5.0232e-02,
   -8.5938e-02, -2.7295e-01, -6.7810e-02, 1.7847e-01, 2.3596e-01,
   9.3140e-02, 2.4133e-01, -4.0314e-02, 1.2119e+00, 9.5947e-02,
   3.5449e-01, -2.7295e-01, -1.8494e-01, -9.3872e-02, -4.7455e-02,
   1.8738e-01, -6.4160e-01, 1.5051e-01, 3.9868e-01, -3.7012e-01,
   3.0469e-01, -2.0728e-01, 2.1240e-01, 1.5198e-01, 8.7524e-02,
   7.6111e-02, -1.6174e-02, 4.3750e-01, 7.7759e-02, 4.1943e-01,
   4.6967e-02, 1.1902e-01, -8.9340e-03, -1.1200e-01, 2.3389e-01,
   -1.3904e-01, -1.4050e-01, 5.8708e-03, -1.7603e-01, 2.8687e-01,
   2.5098e-01, -1.9055e-01, -1.3550e-01, 2.0349e-01, -1.2238e-01,
   -9.8694e-02, -8.2016e-03, 1.7480e-01, -4.0710e-02, 2.1729e-01,
   -1.2805e-01, -1.4856e-01, -1.1041e-01, -4.9500e-02, 9.1187e-02,
   1.2561e-01, 6.3049e-02, -3.9978e-02, 1.1597e-01, -6.9275e-02,
   -3.3911e-01, 3.7598e-01, -8.0994e-02, -1.6296e-01, 9.1125e-02,
   1.9458e-01, -2.5537e-01, -3.4741e-01, 1.7932e-01, 4.2212e-01,
   -2.1692e-01, 9.8999e-02, 2.4976e-01, -2.5537e-01, 8.0017e-02,
   1.9177e-01, -3.9673e-02, 2.7686e-01, -5.2856e-02, 2.5488e-01,
   -1.7883e-01, -1.0608e-01, -7.2388e-02, 5.2734e-01, 3.0078e-01,
   1.0992e-01, 6.5576e-01, 3.6255e-01, -2.0190e-01, 3.3545e-01,
   1.6809e-01, 3.6890e-01, 2.5708e-01, -1.4038e-01, -1.0754e-01,
   3.1177e-01, 1.5540e-01, -1.0162e-01, 1.2439e-01, 5.1575e-02,
   1.1310e-01, 1.3928e-01, -8.3740e-02, 8.3191e-02, -3.1948e-04,
   -1.3953e-01, 2.7002e-01, -1.0352e-01, -6.3428e-01, 7.5134e-02,
   3.7988e-01, -8.6670e-02, 1.2627e-02, -4.8877e-01, -3.3276e-01,
   2.5049e-01, 3.4082e-01, -2.9883e-01, 1.0297e-01, -4.4971e-01,
   -5.4102e-01, 2.1210e-02, -1.0095e-01, 3.3423e-01, 4.3640e-02,
   -2.8540e-01, -2.4536e-01, 1.1316e-01, -6.3293e-02, 5.1697e-02,
   1.9470e-01, -8.5297e-03, -2.5513e-01, -1.6089e-01, -7.0740e-02,
   -1.0529e-02, -4.1504e-01, -3.3569e-02, 4.5190e-01, -2.4976e-01,
   2.0166e-01, 4.0991e-01, -5.6201e-01, 4.6478e-02, 1.1279e-01,
   3.2739e-01, 2.6807e-01, 1.2329e-01, 2.2961e-01, -3.4912e-01,
   1.0742e-01, 4.4800e-01, -4.2725e-02, -3.9459e-02, 2.2717e-01,
   1.6125e-01, 3.5010e-01, 4.1718e-02, 3.5889e-02, -1.1328e-01,
   3.1830e-02, -1.1263e-03, -3.1836e-01, 2.5803e-02, -3.6963e-01,
   5.7568e-01, 3.4937e-01, -4.6338e-01, 3.5669e-01, -7.3059e-02,
   3.5889e-01, -2.3157e-01, -1.3123e-01, 2.0667e-01, -6.2134e-02,
   2.0398e-01, -4.1309e-01, -1.2262e-01, 5.1172e-01, -3.4546e-01,
   -8.2373e-01, 1.4648e-01, 4.0680e-02, -6.1722e-03, -1.8164e-01,
   -2.9468e-01, 2.9639e-01, 1.2109e+00, -1.4587e-01, 1.3130e-02,
   5.4150e-01, 2.4817e-01, 7.0007e-02, 1.6162e-01, 1.9312e-01,
   4.5532e-02, 1.2588e+00, 1.3672e-01, 1.5540e-01, 1.2939e-01,
   2.5879e-01, -2.1594e-01, 2.4817e-01, 1.3477e-01, 1.2959e+00,
   -1.2924e-02, 2.1753e-01, 2.1027e-02, -4.3481e-01, 2.3669e-01,
   -5.7831e-02, -1.5881e-01, 2.5537e-01, 2.2266e-01, 3.2764e-01,
   -2.7441e-01, 3.4027e-02, 3.1152e-01, 2.8027e-01, 2.7173e-01,
   1.3855e-02, 2.1423e-01, -1.7700e-01, 1.7236e-01, -2.2766e-01,
   5.7812e-01, -2.5253e-02, 4.7681e-01, -8.0933e-02, -1.6760e-01,
   -6.8604e-01, -4.1626e-01, -2.6196e-01, -7.5195e-02, 1.9263e-01,
   -6.6772e-02, -9.5520e-02, 9.2834e-02, 8.3191e-02, -6.0645e-01,
   -4.2798e-01, 1.7810e-01, -1.5221e-02, 1.6479e-01, 6.5552e-02,
   5.6305e-02, -2.8296e-01, 1.3708e-01, 2.1045e-01, 2.2266e-01,
   2.2754e-01, -7.4272e-03, -1.7715e-02, -5.8008e-01, -3.6499e-01,
   -4.3555e-01, -1.1011e-01, 1.4172e-01, -2.1912e-01, 8.6060e-02,
   -2.7759e-01, 1.6504e-01, -5.5542e-02, 1.2878e-02, 1.0419e-01,
   -5.6836e-01, -7.8979e-02, -1.1530e-01, -2.6660e-01, 3.7402e-01,
   2.5537e-01, 4.3237e-01, -1.7517e-01, -7.2510e-01, -9.8267e-02,
   3.1097e-02, 1.6586e-02, 4.3188e-01, 1.0107e-01, -2.0691e-01,
   -2.9068e-02, 1.6541e-01, 3.1952e-02, 8.5840e-01, -2.6172e-01,
   -3.3154e-01, 1.0223e-01, 1.2915e-01, 9.5398e-02, 1.9360e-01,
   1.9104e-02, -2.4170e-01, -7.2144e-02, 1.2500e-01, -1.2024e-01,
   1.7798e-01, -3.1958e-01, 2.7271e-01, -5.5322e-01, 2.9761e-01,
   5.5298e-02, -1.8420e-01, 7.9712e-02, -5.9033e-01, -4.2603e-02,
   -1.1572e-01, -2.7661e-01, 4.6600e-02, 2.4017e-02, 2.3590e-02,
   -1.0596e-01, -7.4341e-02, -1.2672e-02, -1.4136e-01, -6.9336e-02,
   -6.9046e-03, 3.5962e-01, 1.7346e-01, -1.4624e-01, -6.7810e-02,
   6.7139e-02, -2.6611e-01, -4.9500e-02, -1.2421e-01, -1.8237e-01,
   -5.0262e-02, -3.4204e-01, -4.2847e-02, -3.4058e-01, 2.5562e-01,
   -1.8036e-02, -2.9883e-01, -4.9878e-01, 1.8811e-01, -2.8076e-01,
   2.3486e-01, 4.8431e-02, -2.3035e-01, -2.0340e-02, 4.1406e-01,
   4.1962e-02, 8.5449e-01, -4.0619e-02, 5.5115e-02, -3.2812e-01,
   -2.6489e-01, 4.2505e-01, -7.6332e-03, -3.4961e-01, -5.8960e-02,
   -1.1749e-01, 3.2495e-01, -9.1919e-02, -2.1082e-01, 3.2715e-01,
   -8.7036e-02, -1.0223e-01, 3.1021e-02, 7.1289e-01, -1.8640e-01,
   -3.0029e-01, 6.7578e-01, -1.5808e-01, 6.2408e-02, -3.7988e-01,
   1.2524e-01, 9.3750e-02, 3.5797e-02, 2.8442e-01, 1.0394e-01,
   -3.0249e-01, 3.5181e-01, -1.6797e-01, -2.7856e-01, -4.2944e-01,
   -1.7175e-01, -1.8909e-01, -7.9163e-02, 1.5894e-01, 6.4844e-01,
   9.1431e-02, -2.8610e-02, -3.4888e-01, -1.3464e-01, 7.7197e-01,
   -3.6426e-01, -2.9346e-01

2. trt run output (Likewise, the textual model also shows the same results.)
   0 0 visual value over 1 = -0.081776582
   0 1 visual value over 1 = 0.024557784
   0 2 visual value over 1 = -0.183344185
   0 3 visual value over 1 = -0.236106038
   0 4 visual value over 1 = 0.376788735
   0 5 visual value over 1 = -0.364432037
   0 6 visual value over 1 = 0.224626571
   0 7 visual value over 1 = 1.003672123
   0 8 visual value over 1 = 0.191839829
   0 9 visual value over 1 = 0.129230782
   0 10 visual value over 1 = 0.237609237
   0 11 visual value over 1 = -0.301710635
   0 12 visual value over 1 = -0.281422108
   0 13 visual value over 1 = -0.175923690
   0 14 visual value over 1 = 0.231289104
   0 15 visual value over 1 = 0.108186960
   0 16 visual value over 1 = -0.189509988
   0 17 visual value over 1 = -0.317451119
   0 18 visual value over 1 = -0.008554846
   0 19 visual value over 1 = 0.308098584
   0 20 visual value over 1 = 0.385881186
   0 21 visual value over 1 = -0.347180784
   0 22 visual value over 1 = -0.102019131
   0 23 visual value over 1 = -0.034023255
   0 24 visual value over 1 = 0.180900842
   0 25 visual value over 1 = 0.237769097
   0 26 visual value over 1 = -0.570185304
   0 27 visual value over 1 = 0.210804299
   0 28 visual value over 1 = 0.046871692
   0 29 visual value over 1 = -0.206881344
   0 30 visual value over 1 = 0.070744380
   0 31 visual value over 1 = 0.167558610
   0 32 visual value over 1 = -0.049198851
   0 33 visual value over 1 = 0.167528421
   0 34 visual value over 1 = -0.306383938
   0 35 visual value over 1 = -0.392020226
   0 36 visual value over 1 = -0.172572106
   0 37 visual value over 1 = 0.230519235
   0 38 visual value over 1 = 0.217562810
   0 39 visual value over 1 = -0.976035237
   0 40 visual value over 1 = -0.414280593
   0 41 visual value over 1 = -0.108957112
   0 42 visual value over 1 = -0.188961521
   0 43 visual value over 1 = 0.088423081
   0 44 visual value over 1 = -0.247388810
   0 45 visual value over 1 = 0.044488415
   0 46 visual value over 1 = 0.226941824
   0 47 visual value over 1 = -0.173833132
   0 48 visual value over 1 = 0.279895723
   0 49 visual value over 1 = -0.065127015
   0 50 visual value over 1 = -0.220430821
   0 51 visual value over 1 = -0.231804043
   0 52 visual value over 1 = 0.311237514
   0 53 visual value over 1 = -0.539963365
   0 54 visual value over 1 = -0.232145980
   0 55 visual value over 1 = 0.048041180
   0 56 visual value over 1 = 0.268218577
   0 57 visual value over 1 = -0.609514594
   0 58 visual value over 1 = 0.145302415
   0 59 visual value over 1 = 0.065195642
   0 60 visual value over 1 = -0.077743784
   0 61 visual value over 1 = -0.046050295
   0 62 visual value over 1 = 0.150579453
   0 63 visual value over 1 = 0.148513019
   0 64 visual value over 1 = -0.116065383
   0 65 visual value over 1 = 0.171630204
   0 66 visual value over 1 = -0.212066144
   0 67 visual value over 1 = -0.539296687
   0 68 visual value over 1 = 0.277048200
   0 69 visual value over 1 = 0.119507790
   0 70 visual value over 1 = -0.515325427
   0 71 visual value over 1 = 0.002778262
   0 72 visual value over 1 = 0.237521455
   0 73 visual value over 1 = 0.061807692
   0 74 visual value over 1 = 0.127541989
   0 75 visual value over 1 = -0.444064081
   0 76 visual value over 1 = 0.056793846
   0 77 visual value over 1 = 0.018819995
   0 78 visual value over 1 = 0.262012362
   0 79 visual value over 1 = -0.475130022
   0 80 visual value over 1 = 0.182153970
   0 81 visual value over 1 = -0.187677756
   0 82 visual value over 1 = -0.385241926
   0 83 visual value over 1 = -0.102161534
   0 84 visual value over 1 = 0.110707417
   0 85 visual value over 1 = 0.324309587
   0 86 visual value over 1 = 0.324954659
   0 87 visual value over 1 = 0.181964427
   0 88 visual value over 1 = -0.162437350
   0 89 visual value over 1 = -0.484298587
   0 90 visual value over 1 = 0.107957095
   0 91 visual value over 1 = 0.133363694
   0 92 visual value over 1 = -8.890506744
   0 93 visual value over 1 = 0.238423526
   0 94 visual value over 1 = 0.159745857
   0 95 visual value over 1 = 0.132477254
   0 96 visual value over 1 = 0.160367846
   0 97 visual value over 1 = -0.234191269
   0 98 visual value over 1 = -0.163825154
   0 99 visual value over 1 = -0.336508781
   0 100 visual value over 1 = 0.225079522
   0 101 visual value over 1 = -0.229782581
   0 102 visual value over 1 = 0.251466930
   0 103 visual value over 1 = 0.123222776
   0 104 visual value over 1 = -0.083800495
   0 105 visual value over 1 = -0.154026076
   0 106 visual value over 1 = -2.034134865
   0 107 visual value over 1 = 0.379578650
   0 108 visual value over 1 = -0.011021599
   0 109 visual value over 1 = 0.103692278
   0 110 visual value over 1 = -0.094223082
   0 111 visual value over 1 = -0.536501467
   0 112 visual value over 1 = 0.088537633
   0 113 visual value over 1 = -0.072727732
   0 114 visual value over 1 = -0.274469554
   0 115 visual value over 1 = -0.160410017
   0 116 visual value over 1 = -0.209073722
   0 117 visual value over 1 = -0.290029854
   0 118 visual value over 1 = 0.095706522
   0 119 visual value over 1 = 0.515773237
   0 120 visual value over 1 = 0.028978765
   0 121 visual value over 1 = 0.134880245
   0 122 visual value over 1 = -0.516272545
   0 123 visual value over 1 = 0.038310647
   0 124 visual value over 1 = 0.287553906
   0 125 visual value over 1 = -0.284646600
   0 126 visual value over 1 = -0.294438630
   0 127 visual value over 1 = 0.002952918
   0 128 visual value over 1 = 0.360761106
   0 129 visual value over 1 = 0.613080859
   0 130 visual value over 1 = 0.235581696
   0 131 visual value over 1 = 0.018681012
   0 132 visual value over 1 = -0.245507628
   0 133 visual value over 1 = 1.146387935
   0 134 visual value over 1 = 0.394953787
   0 135 visual value over 1 = 0.361470133
   0 136 visual value over 1 = 0.091869265
   0 137 visual value over 1 = -0.371040553
   0 138 visual value over 1 = -0.264368057
   0 139 visual value over 1 = -0.064111143
   0 140 visual value over 1 = -0.387478769
   0 141 visual value over 1 = -0.326032758
   0 142 visual value over 1 = -0.324655175
   0 143 visual value over 1 = 0.371032089
   0 144 visual value over 1 = -0.586391091
   0 145 visual value over 1 = 0.154773206
   0 146 visual value over 1 = -0.185753405
   0 147 visual value over 1 = 0.374087721
   0 148 visual value over 1 = 0.346637309
   0 149 visual value over 1 = -0.404477298
   0 150 visual value over 1 = 0.047796845
   0 151 visual value over 1 = -0.138880640
   0 152 visual value over 1 = 0.574975729
   0 153 visual value over 1 = -0.052402489
   0 154 visual value over 1 = 0.188804775
   0 155 visual value over 1 = -0.436604142
   0 156 visual value over 1 = 0.051795267
   0 157 visual value over 1 = 0.176675975
   0 158 visual value over 1 = -0.012445912
   0 159 visual value over 1 = 0.172566667
   0 160 visual value over 1 = -0.272665262
   0 161 visual value over 1 = -0.000394069
   0 162 visual value over 1 = -0.166220680
   0 163 visual value over 1 = -0.111244015
   0 164 visual value over 1 = 0.395377070
   0 165 visual value over 1 = 0.518562198
   0 166 visual value over 1 = -0.163287878
   0 167 visual value over 1 = -0.140702069
   0 168 visual value over 1 = -0.331950694
   0 169 visual value over 1 = 0.043232024
   0 170 visual value over 1 = -0.131310150
   0 171 visual value over 1 = 0.292327613
   0 172 visual value over 1 = 0.025498360
   0 173 visual value over 1 = -0.049035117
   0 174 visual value over 1 = 0.076727532
   0 175 visual value over 1 = 0.385224551
   0 176 visual value over 1 = 0.234250277
   0 177 visual value over 1 = -0.092905536
   0 178 visual value over 1 = 0.105981320
   0 179 visual value over 1 = -0.193092272
   0 180 visual value over 1 = 0.181595743
   0 181 visual value over 1 = 0.221157968
   0 182 visual value over 1 = -0.121914983
   0 183 visual value over 1 = -0.463677108
   0 184 visual value over 1 = -0.014705554
   0 185 visual value over 1 = -0.386091709
   0 186 visual value over 1 = -0.089709729
   0 187 visual value over 1 = -0.194615930
   0 188 visual value over 1 = -0.233421490
   0 189 visual value over 1 = 0.236557692
   0 190 visual value over 1 = -0.021914583
   0 191 visual value over 1 = -0.243340671
   0 192 visual value over 1 = 0.259532034
   0 193 visual value over 1 = 0.147617668
   0 194 visual value over 1 = 0.384134829
   0 195 visual value over 1 = -0.366001844
   0 196 visual value over 1 = -0.378247559
   0 197 visual value over 1 = -0.058930114
   0 198 visual value over 1 = -0.520613670
   0 199 visual value over 1 = 0.145184830
   0 200 visual value over 1 = 0.098128758
   0 201 visual value over 1 = -0.094376668
   0 202 visual value over 1 = -0.071256660
   0 203 visual value over 1 = 0.265981913
   0 204 visual value over 1 = -0.089416087
   0 205 visual value over 1 = -0.490181565
   0 206 visual value over 1 = -0.187213451
   0 207 visual value over 1 = 0.225264013
   0 208 visual value over 1 = 0.662953794
   0 209 visual value over 1 = 0.025862470
   0 210 visual value over 1 = 0.065862089
   0 211 visual value over 1 = 1.005577922
   0 212 visual value over 1 = -0.218000785
   0 213 visual value over 1 = -0.047760852
   0 214 visual value over 1 = 0.476287484
   0 215 visual value over 1 = -0.229744107
   0 216 visual value over 1 = 0.001327425
   0 217 visual value over 1 = 0.209129035
   0 218 visual value over 1 = -0.499402404
   0 219 visual value over 1 = -0.208212748
   0 220 visual value over 1 = 0.573245764
   0 221 visual value over 1 = -0.281512320
   0 222 visual value over 1 = 0.113574818
   0 223 visual value over 1 = -0.065877646
   0 224 visual value over 1 = -0.054273933
   0 225 visual value over 1 = 0.372651964
   0 226 visual value over 1 = -0.156617194
   0 227 visual value over 1 = -0.066092864
   0 228 visual value over 1 = -0.002779834
   0 229 visual value over 1 = 0.291225642
   0 230 visual value over 1 = -0.075065181
   0 231 visual value over 1 = 0.096592188
   0 232 visual value over 1 = -0.193110749
   0 233 visual value over 1 = -0.168912545
   0 234 visual value over 1 = -0.231811315
   0 235 visual value over 1 = -0.317430973
   0 236 visual value over 1 = -0.129435912
   0 237 visual value over 1 = -0.109400600
   0 238 visual value over 1 = -0.478563070
   0 239 visual value over 1 = -0.300585777
   0 240 visual value over 1 = -0.200858176
   0 241 visual value over 1 = 0.271454155
   0 242 visual value over 1 = 0.412119031
   0 243 visual value over 1 = 0.281412452
   0 244 visual value over 1 = -0.250203133
   0 245 visual value over 1 = -0.273244619
   0 246 visual value over 1 = 0.172669485
   0 247 visual value over 1 = -0.216270104
   0 248 visual value over 1 = 0.229822800
   0 249 visual value over 1 = -0.029515222
   0 250 visual value over 1 = 0.022884056
   0 251 visual value over 1 = -0.368755966
   0 252 visual value over 1 = -0.180217937
   0 253 visual value over 1 = -0.037977815
   0 254 visual value over 1 = 0.078131586
   0 255 visual value over 1 = -0.108892836
   0 256 visual value over 1 = 0.336257935
   0 257 visual value over 1 = -0.231972605
   0 258 visual value over 1 = -0.922894657
   0 259 visual value over 1 = -0.279423147
   0 260 visual value over 1 = 0.108540528
   0 261 visual value over 1 = 0.200152740
   0 262 visual value over 1 = -0.356449097
   0 263 visual value over 1 = 0.321491390
   0 264 visual value over 1 = 0.029282060
   0 265 visual value over 1 = 0.074907035
   0 266 visual value over 1 = -0.066966020
   0 267 visual value over 1 = 0.092701703
   0 268 visual value over 1 = 0.012898482
   0 269 visual value over 1 = 0.206065983
   0 270 visual value over 1 = 0.123302720
   0 271 visual value over 1 = 0.286424756
   0 272 visual value over 1 = -0.035909772
   0 273 visual value over 1 = -0.182107329
   0 274 visual value over 1 = -0.052833766
   0 275 visual value over 1 = 0.057821631
   0 276 visual value over 1 = -0.029263116
   0 277 visual value over 1 = 0.179720402
   0 278 visual value over 1 = -0.068831235
   0 279 visual value over 1 = -0.102738954
   0 280 visual value over 1 = -0.209884092
   0 281 visual value over 1 = 0.206355765
   0 282 visual value over 1 = -0.031416744
   0 283 visual value over 1 = -0.176986605
   0 284 visual value over 1 = 0.079358213
   0 285 visual value over 1 = 0.399790287
   0 286 visual value over 1 = 0.476008356
   0 287 visual value over 1 = -0.188777447
   0 288 visual value over 1 = 0.526394248
   0 289 visual value over 1 = -0.123176068
   0 290 visual value over 1 = 0.243725449
   0 291 visual value over 1 = 0.360426843
   0 292 visual value over 1 = -0.323001564
   0 293 visual value over 1 = -0.004666999
   0 294 visual value over 1 = -0.271204889
   0 295 visual value over 1 = 0.345959783
   0 296 visual value over 1 = -0.809651434
   0 297 visual value over 1 = 0.014848515
   0 298 visual value over 1 = 0.584776521
   0 299 visual value over 1 = -0.165719062
   0 300 visual value over 1 = 0.253541559
   0 301 visual value over 1 = -0.256987631
   0 302 visual value over 1 = -0.498621672
   0 303 visual value over 1 = 0.186468452
   0 304 visual value over 1 = 0.042557508
   0 305 visual value over 1 = -0.760257602
   0 306 visual value over 1 = -0.186582804
   0 307 visual value over 1 = 0.165693209
   0 308 visual value over 1 = -0.055021144
   0 309 visual value over 1 = -0.667359829
   0 310 visual value over 1 = -0.290651172
   0 311 visual value over 1 = 0.292001694
   0 312 visual value over 1 = 1.146214843
   0 313 visual value over 1 = -0.140186757
   0 314 visual value over 1 = 0.081750497
   0 315 visual value over 1 = 0.641002476
   0 316 visual value over 1 = 0.357029825
   0 317 visual value over 1 = 0.190474540
   0 318 visual value over 1 = 0.121079721
   0 319 visual value over 1 = -0.426536620
   0 320 visual value over 1 = 0.038418084
   0 321 visual value over 1 = 1.936530590
   0 322 visual value over 1 = -0.003911629
   0 323 visual value over 1 = 0.499237657
   0 324 visual value over 1 = 0.296978116
   0 325 visual value over 1 = -0.039242446
   0 326 visual value over 1 = -0.002214178
   0 327 visual value over 1 = -0.022339046
   0 328 visual value over 1 = 0.307342589
   0 329 visual value over 1 = 0.888413072
   0 330 visual value over 1 = -0.304945588
   0 331 visual value over 1 = 0.635916531
   0 332 visual value over 1 = -0.137755722
   0 333 visual value over 1 = -0.441585213
   0 334 visual value over 1 = 0.177278370
   0 335 visual value over 1 = -0.041614890
   0 336 visual value over 1 = 0.229141027
   0 337 visual value over 1 = 0.202542618
   0 338 visual value over 1 = 0.142325908
   0 339 visual value over 1 = 0.450670540
   0 340 visual value over 1 = -0.055149764
   0 341 visual value over 1 = -0.248549789
   0 342 visual value over 1 = -0.069406047
   0 343 visual value over 1 = 0.021039113
   0 344 visual value over 1 = 0.122902960
   0 345 visual value over 1 = -0.240820199
   0 346 visual value over 1 = 0.463473946
   0 347 visual value over 1 = -0.593982518
   0 348 visual value over 1 = 0.229390964
   0 349 visual value over 1 = -0.245744914
   0 350 visual value over 1 = 0.418983877
   0 351 visual value over 1 = 0.194071233
   0 352 visual value over 1 = 0.094454080
   0 353 visual value over 1 = 0.155679673
   0 354 visual value over 1 = 0.219171241
   0 355 visual value over 1 = -0.741509318
   0 356 visual value over 1 = -0.222414255
   0 357 visual value over 1 = -0.305072129
   0 358 visual value over 1 = -0.078185812
   0 359 visual value over 1 = 0.163232177
   0 360 visual value over 1 = -0.029413961
   0 361 visual value over 1 = -0.179588646
   0 362 visual value over 1 = -0.180721849
   0 363 visual value over 1 = 0.183941767
   0 364 visual value over 1 = -0.942063451
   0 365 visual value over 1 = -0.613707542
   0 366 visual value over 1 = -0.026688159
   0 367 visual value over 1 = 0.726849735
   0 368 visual value over 1 = 0.354258686
   0 369 visual value over 1 = 0.014138415
   0 370 visual value over 1 = 0.158838153
   0 371 visual value over 1 = 0.258442044
   0 372 visual value over 1 = -0.331239164
   0 373 visual value over 1 = 0.061262056
   0 374 visual value over 1 = 0.244410276
   0 375 visual value over 1 = 0.148125142
   0 376 visual value over 1 = -0.411402225
   0 377 visual value over 1 = 0.082523227
   0 378 visual value over 1 = -0.705651999
   0 379 visual value over 1 = -0.038929552
   0 380 visual value over 1 = -0.251776695
   0 381 visual value over 1 = -0.018114045
   0 382 visual value over 1 = 0.279260904
   0 383 visual value over 1 = 0.172995746
   0 384 visual value over 1 = 0.023752801
   0 385 visual value over 1 = -0.036558814
   0 386 visual value over 1 = 0.385531306
   0 387 visual value over 1 = -0.413633227
   0 388 visual value over 1 = 0.032380663
   0 389 visual value over 1 = -0.040194690
   0 390 visual value over 1 = -0.834907889
   0 391 visual value over 1 = 0.106882274
   0 392 visual value over 1 = 0.084509969
   0 393 visual value over 1 = -0.053288195
   0 394 visual value over 1 = 0.356140137
   0 395 visual value over 1 = -0.116827592
   0 396 visual value over 1 = 0.221017942
   0 397 visual value over 1 = 0.176051438
   0 398 visual value over 1 = -0.752089262
   0 399 visual value over 1 = -0.085090250
   0 400 visual value over 1 = -0.034791306
   0 401 visual value over 1 = -0.285834551
   0 402 visual value over 1 = 0.411401570
   0 403 visual value over 1 = -0.046421081
   0 404 visual value over 1 = 0.081822746
   0 405 visual value over 1 = 0.220675409
   0 406 visual value over 1 = -0.030317916
   0 407 visual value over 1 = 0.178580970
   0 408 visual value over 1 = 0.518715262
   0 409 visual value over 1 = -0.218482509
   0 410 visual value over 1 = 0.218306333
   0 411 visual value over 1 = -0.020737564
   0 412 visual value over 1 = 0.347726732
   0 413 visual value over 1 = 0.447249889
   0 414 visual value over 1 = 0.265972495
   0 415 visual value over 1 = 0.241365418
   0 416 visual value over 1 = 0.034342080
   0 417 visual value over 1 = -0.260788500
   0 418 visual value over 1 = -0.024644732
   0 419 visual value over 1 = -0.166838437
   0 420 visual value over 1 = -0.044023365
   0 421 visual value over 1 = -0.283562541
   0 422 visual value over 1 = -0.342083007
   0 423 visual value over 1 = -0.296099186
   0 424 visual value over 1 = 0.502427995
   0 425 visual value over 1 = 0.254383534
   0 426 visual value over 1 = -0.095992327
   0 427 visual value over 1 = 0.072240174
   0 428 visual value over 1 = -0.162948012
   0 429 visual value over 1 = -0.273545384
   0 430 visual value over 1 = -0.158075392
   0 431 visual value over 1 = -0.259685427
   0 432 visual value over 1 = -0.446292281
   0 433 visual value over 1 = -0.511785746
   0 434 visual value over 1 = -0.061603770
   0 435 visual value over 1 = 0.169210672
   0 436 visual value over 1 = 0.047403157
   0 437 visual value over 1 = -0.443283528
   0 438 visual value over 1 = -0.026010953
   0 439 visual value over 1 = -0.126797199
   0 440 visual value over 1 = -0.121278986
   0 441 visual value over 1 = -0.012962699
   0 442 visual value over 1 = -0.502521813
   0 443 visual value over 1 = -0.369578123
   0 444 visual value over 1 = 0.143090889
   0 445 visual value over 1 = 0.119973168
   0 446 visual value over 1 = -0.010358043
   0 447 visual value over 1 = -0.042201117
   0 448 visual value over 1 = -0.198975772
   0 449 visual value over 1 = -0.473876774
   0 450 visual value over 1 = -0.099838391
   0 451 visual value over 1 = -0.020818383
   0 452 visual value over 1 = 0.000338774
   0 453 visual value over 1 = -0.677336633
   0 454 visual value over 1 = 0.106147543
   0 455 visual value over 1 = 0.182949603
   0 456 visual value over 1 = -0.178673863
   0 457 visual value over 1 = -0.659576476
   0 458 visual value over 1 = 0.119971335
   0 459 visual value over 1 = -0.185138270
   0 460 visual value over 1 = -0.214721888
   0 461 visual value over 1 = 0.193628997
   0 462 visual value over 1 = -0.222921148
   0 463 visual value over 1 = -0.249578089
   0 464 visual value over 1 = 0.357634723
   0 465 visual value over 1 = -0.191367179
   0 466 visual value over 1 = 0.651169777
   0 467 visual value over 1 = 0.230520532
   0 468 visual value over 1 = 0.016554542
   0 469 visual value over 1 = -0.042892769
   0 470 visual value over 1 = 0.207739919
   0 471 visual value over 1 = -0.007465526
   0 472 visual value over 1 = 0.077094182
   0 473 visual value over 1 = -0.361942530
   0 474 visual value over 1 = -0.105255187
   0 475 visual value over 1 = 0.090649754
   0 476 visual value over 1 = 0.032817990
   0 477 visual value over 1 = -0.229992494
   0 478 visual value over 1 = 0.006844595
   0 479 visual value over 1 = 0.181408182
   0 480 visual value over 1 = -0.054898284
   0 481 visual value over 1 = -0.222125828
   0 482 visual value over 1 = -0.184950098
   0 483 visual value over 1 = 0.457677066
   0 484 visual value over 1 = -0.201888129
   0 485 visual value over 1 = -0.195521101
   0 486 visual value over 1 = 0.135140970
   0 487 visual value over 1 = 0.132548019
   0 488 visual value over 1 = -0.293721735
   0 489 visual value over 1 = -0.422884226
   0 490 visual value over 1 = 0.289520025
   0 491 visual value over 1 = -0.261062026
   0 492 visual value over 1 = 0.244733989
   0 493 visual value over 1 = 0.243325680
   0 494 visual value over 1 = -0.065501414
   0 495 visual value over 1 = 0.084683537
   0 496 visual value over 1 = 0.093555599
   0 497 visual value over 1 = 0.097806819
   0 498 visual value over 1 = -0.262689382
   0 499 visual value over 1 = -0.385075688
   0 500 visual value over 1 = -0.011496395
   0 501 visual value over 1 = -0.386213124
   0 502 visual value over 1 = -0.099921927
   0 503 visual value over 1 = -0.090658046
   0 504 visual value over 1 = 0.881528854
   0 505 visual value over 1 = 0.176057681
   0 506 visual value over 1 = -0.007641792
   0 507 visual value over 1 = -0.222156763
   0 508 visual value over 1 = -0.241174474
   0 509 visual value over 1 = 0.774220169
   0 510 visual value over 1 = -0.311260104
   0 511 visual value over 1 = 0.308889836

[smyang03]

Found the problem. Something went wrong in preprocessing. 224 224 This problem occurred because the padding value was entered when performing cudamemcpy on the image. thanks for your kind

[PhilChina]

Cudamemcpy is migrating data from host to device, what is the specific cause of the error? And how to solve it? Can give some solutions