Non-zero status code returned

[pythondever]

Describe the bug
I exported an onnx model can run in CPUExecutionProvider，But when I use GPUExecutionProvider got Error

2022-05-17 17:30:35.309693323 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=pc-Z390-GAMING-X ; expr=cudnnAddTensor(Base::CudnnHandle(), &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data);
2022-05-17 17:30:35.309720084 [E:onnxruntime:, sequential_executor.cc:364 Execute] Non-zero status code returned while running FusedConv node. Name:'Conv_34' Status Message: CUDNN error executing cudnnAddTensor(Base::CudnnHandle(), &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data)
Traceback (most recent call last):
File "pred_onnx.py", line 97, in
pred_onnx(model, im_path)
File "pred_onnx.py", line 80, in pred_onnx
pred = session.run([output_name], {input_name: image})[0]
File "/home/pc/anaconda3/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
return self.sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running FusedConv node. Name:'Conv_34' Status Message: CUDNN error executing cudnnAddTensor(Base::CudnnHandle(), &alpha, Base::s.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data)

how can i fix this error ？

System information

• OS Platform and Distribution: ubuntu18.04
• ONNX Runtime installed from (source or binary):
• ONNX Runtime version:1.9.0
• Python version:3.8.8
• Visual Studio version (if applicable):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version:11.1/8.0
• GPU model and memory:Nvidia Geforce 3070

[yuslepukhin]

It may be a bug. Try the latest ORT release, or share with us your model to investigate.

[pythondever]

It may be a bug. Try the latest ORT release, or share with us your model to investigate.

I've tried that，but it doesn't work

[pythondever]

It may be a bug. Try the latest ORT release, or share with us your model to investigate.

model file https://github.com/pythondever/_onnx_demo_error

[pythondever]

@yuslepukhin Is there anything update？

[yuslepukhin]

@yuslepukhin Is there anything update？

I did not get a chance to look at it yet.

[beyonehan]

@yuslepukhin Is there anything update？

I did not get a chance to look at it yet.

I have the same problem as well. I have tried everything I can do but still doesn't work. Could you give me some suggestions? Many thanks.

[yuslepukhin]

Please, follow the issue reporting template and submit a program that reproduces the behavior, that would speed up things a lot. Please, attach the input data and your model if you are able to. Then we would be able to determine what the issue is and the best course of action.

[pythondever]

@yuslepukhin thanks reply！
I push my demo program please follow this url https://github.com/pythondever/_onnx_demo_error
and just run python pred_onnx.py， you can reproduce the error

[yuslepukhin]

@yuslepukhin thanks reply！ I push my demo program please follow this url https://github.com/pythondever/_onnx_demo_error and just run python pred_onnx.py， you can reproduce the error

I have seen the link, it shows up as empty folder.

[pythondever]

@yuslepukhin thanks reply！ I push my demo program please follow this url https://github.com/pythondever/_onnx_demo_error and just run python pred_onnx.py， you can reproduce the error

I have seen the link, it shows up as empty folder.

really sorry，I copied the address again，It's ready to go，

https://github.com/pythondever/_onnx_demo_error

[Eliza-and-black]

I got a similar error:
"onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Mul node. Name:'Mul_710' Status Message: Mul_710: right operand cannot broadcast on dim 3 LeftShape: {1,3,15,25,2}, RightShape: {1,3,20,20,2}"

[beyonehan]

2022-05-17 17:30:35.309693323 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=pc-Z390-GAMING-X ; expr=cudnnAddTensor(Base::CudnnHandle(), &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data)

Hi , I am really looking forward your answer.Many thanks. I have same problem with pythondever. I have tired everything that I could do for that problems, but it still remains.

[hosea7456]

@yuslepukhin
I recently came across the same issue when I running my code.
Describe the bug:
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running FusedConv node. Name:'p2o.Conv.68' Status Message: CUDNN error executing cudnnAddTensor(Base::CudnnHandle(), &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data)

And my model can be get by the following link:
https://drive.google.com/file/d/1GiNLsQropC2YJtlmWW2z6RWBVwcyDEQh/view?usp=sharing

Any help will be gratefully appreciated!

[yuslepukhin]

I got a similar error: "onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Mul node. Name:'Mul_710' Status Message: Mul_710: right operand cannot broadcast on dim 3 LeftShape: {1,3,15,25,2}, RightShape: {1,3,20,20,2}"

This is a very different issue and problem is in the model. The two shapes can not be broadcasted. Please, report this separate.

[yuslepukhin]

@yuslepukhin thanks reply！ I push my demo program please follow this url https://github.com/pythondever/_onnx_demo_error and just run python pred_onnx.py， you can reproduce the error

I have seen the link, it shows up as empty folder.

really sorry，I copied the address again，It's ready to go，

https://github.com/pythondever/_onnx_demo_error

This still does not open.

[yuslepukhin]

@yuslepukhin I recently came across the same issue when I running my code. Describe the bug: onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running FusedConv node. Name:'p2o.Conv.68' Status Message: CUDNN error executing cudnnAddTensor(Base::CudnnHandle(), &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data)

And my model can be get by the following link: https://drive.google.com/file/d/1GiNLsQropC2YJtlmWW2z6RWBVwcyDEQh/view?usp=sharing

Any help will be gratefully appreciated!

Got this, would be nice if you shared a script to drive this with inputs. Does this work on CPU?

[hosea7456]

@yuslepukhin I recently came across the same issue when I running my code. Describe the bug: onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running FusedConv node. Name:'p2o.Conv.68' Status Message: CUDNN error executing cudnnAddTensor(Base::CudnnHandle(), &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data)
And my model can be get by the following link: https://drive.google.com/file/d/1GiNLsQropC2YJtlmWW2z6RWBVwcyDEQh/view?usp=sharing
Any help will be gratefully appreciated!

Got this, would be nice if you shared a script to drive this with inputs. Does this work on CPU?

Many thanks for your immediate reply!
Yes, it runs well on CPU but filed to run on GPU. And my environment information is: ONNXRuntime 1.11.1, CUDA11.4, CUDNN8.2
Here is the code for running inference.
https://drive.google.com/drive/folders/1fWoIE2q1t6DJYgyvCIPQcwrsZjdXR4rY?usp=sharing

[yuslepukhin]

I have been able to reproduce your issue. For an immediate workaround you enable only basic optimizations for CUDA runs.
sess_opt = ort.SessionOptions()
sess_opt.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC
session = ort.InferenceSession(args.onnx_file_path, sess_options=sess_opt, providers=providers)

This will likely result in some performance loss, while you are waiting for a fix.

[pythondever]

ort.GraphOptimizationLevel.ORT_ENABLE_BASIC

Yes, it works now thanks!

[hosea7456]

I have been able to reproduce your issue. For an immediate workaround you enable only basic optimizations for CUDA runs. sess_opt = ort.SessionOptions() sess_opt.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC session = ort.InferenceSession(args.onnx_file_path, sess_options=sess_opt, providers=providers)

This will likely result in some performance loss, while you are waiting for a fix.

The problem was solved, thanks a lot. And looking forwaord the fix version!

[gcunhase]

Has this been fixed? I'm getting the following error on mine:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Conv node. Name:'/stages.0/stages.0.1/dwconv/Conv' Status Message: X num_dims does not match W num_dims. X: {1,96,128,256} W: {96}