Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Denoise issue in Graxpert 3.0.0. #133

Open
sam-pedraglio opened this issue Apr 23, 2024 · 23 comments
Open

Denoise issue in Graxpert 3.0.0. #133

sam-pedraglio opened this issue Apr 23, 2024 · 23 comments

Comments

@sam-pedraglio
Copy link

Hallo, I've update my Graxpert installation to the new 3.0.0
I tried to walk through the complete workflow but itfinish with an error message after the denoise step.
I tried to unistall and install again the sw but the error message was always the same (see the attached file with the log obtained after a new installation - AI and denoise models downloaded)
graxpert.log

@schmelly
Copy link
Collaborator

Hi Sam, thanks for the report.

We have received similar issue reports for Windows 10 and are investigating. This will probably take a bit more time unfortunately since we could not reproduce the issue yet. For me it looks like a dependency issue between onnxruntime (a GraXpert dependency we use to execute ml models) and underlying system libraries onnxruntime requires (e.g., DirectML, DirectX 12).

Can you provide some more information for us please: operating system + version and graphics card + driver version?

Thanks a lot and best regards
David

@ljuraszek
Copy link

ljuraszek commented Apr 24, 2024

Hello @schmelly I can provide more information as I have similar issue.
Operating system: Windows 10 Pro
Operating system version: 22H2
Graphic card: Intel HD Graphics 5500
Graphic driver version 20.19.15.4624

2024-04-24 10:41:39,451 MainProcess root ERROR
Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 91, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException

@sam-pedraglio
Copy link
Author

Can you provide some more information for us please: operating system + version and graphics card + driver version?

O.S. : Win 10 Home rel. 22H2 build 19045.4291
CPU: Intel(R) Core(TM) i7-4500U
Graphic Card: NVIDIA GeForce GT 740M (driver: 25.21.14.2205)
DirectX 12.0

If you need more info, let me know.

Thanks for your reply

@Markydotcom
Copy link

Hi,
Have used previous version for some time now with no issues at all.
Recently upgraded to GraXpert 3.
Works as expected on my imaging computer { Mele Quieter 3C } but not on my main PC.
Using as a stand alone program.
I believe all drivers are up to date.
Background extraction works well but when trying to de-noise my computer throws up this error code.

Can you help please?
Attached are details of system and error.

Thanks,
Markydotcom

Device name Astro-Beard
Processor Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz 3.60 GHz
Installed RAM 16.0 GB
Device ID 80E221F1-1CC8-4474-BFD2-C0BE4AAAD59F
Product ID 00330-80000-00000-AA520
System type 64-bit operating system, x64-based processor
Pen and touch No pen or touch input is available for this display

Edition Windows 11 Pro
Version 23H2
Installed on ‎06/‎03/‎2024
OS build 22631.3527
Experience Windows Feature Experience Pack 1000.22700.1003.0

2024-04-24 17:33:47,120 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-24 17:33:47,136 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-24 17:33:47,136 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-24 17:33:47,432 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-24 17:33:47,464 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-24 17:33:47,479 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-24 17:33:52,823 MainProcess root INFO Starting denoising
2024-04-24 17:33:53,776 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:33:53,776 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:33:54,229 MainProcess root ERROR [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAA64C46F: (caller: 00007FFBAA64DC21) Exception(3) tid(98) 80004005 Unspecified error
Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 91, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAA64C46F: (caller: 00007FFBAA64DC21) Exception(3) tid(98) 80004005 Unspecified error

2024-04-24 17:55:45,942 MainProcess root INFO Starting denoising
2024-04-24 17:55:46,848 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:55:46,848 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:55:47,286 MainProcess root ERROR [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAA64C46F: (caller: 00007FFBAA64DC21) Exception(6) tid(98) 80004005 Unspecified error
Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 91, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAA64C46F: (caller: 00007FFBAA64DC21) Exception(6) tid(98) 80004005 Unspecified error

2024-04-24 17:57:20,356 MainProcess root INFO Starting denoising
2024-04-24 17:57:21,247 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:57:21,247 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:57:22,090 MainProcess root ERROR [ONNXRuntimeError] : 1 : FAIL : D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlCommandRecorder.cpp(371)\onnxruntime_pybind11_state.pyd!00007FFBAA606E70: (caller: 00007FFBAA5F0564) Exception(10) tid(98) 887A0020 An internal issue prevented the driver from carrying out the specified operation. The driver's state is probably suspect,
and the application should not continue.
Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 91, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlCommandRecorder.cpp(371)\onnxruntime_pybind11_state.pyd!00007FFBAA606E70: (caller: 00007FFBAA5F0564) Exception(10) tid(98) 887A0020 An internal issue prevented the driver from carrying out the specified operation. The driver's state is probably suspect,
and the application should not continue.

2024-04-24 17:57:27,659 MainProcess root INFO Starting denoising
2024-04-24 17:57:28,550 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:57:28,550 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:57:28,847 MainProcess root ERROR [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAA64C46F: (caller: 00007FFBAA64DC21) Exception(13) tid(98) 80004005 Unspecified error
Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 91, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAA64C46F: (caller: 00007FFBAA64DC21) Exception(13) tid(98) 80004005 Unspecified error

2024-04-24 17:57:33,633 MainProcess root INFO Starting denoising
2024-04-24 17:57:34,476 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:57:34,476 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:57:35,117 MainProcess root ERROR [ONNXRuntimeError] : 1 : FAIL : D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlCommandRecorder.cpp(371)\onnxruntime_pybind11_state.pyd!00007FFBAA606E70: (caller: 00007FFBAA5F0564) Exception(17) tid(98) 887A0020 An internal issue prevented the driver from carrying out the specified operation. The driver's state is probably suspect,
and the application should not continue.
Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 91, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlCommandRecorder.cpp(371)\onnxruntime_pybind11_state.pyd!00007FFBAA606E70: (caller: 00007FFBAA5F0564) Exception(17) tid(98) 887A0020 An internal issue prevented the driver from carrying out the specified operation. The driver's state is probably suspect,
and the application should not continue.

2024-04-24 17:58:41,390 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-24 17:58:41,390 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-24 17:58:41,390 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-24 17:58:41,390 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-24 17:58:41,390 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-24 17:58:41,390 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-24 17:58:41,686 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-24 17:58:41,686 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-24 17:58:41,686 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-24 17:58:41,686 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-24 17:58:41,702 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-24 17:58:41,702 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-24 17:58:45,421 MainProcess root INFO Starting denoising
2024-04-24 17:58:46,327 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:58:46,327 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-24 17:58:46,749 MainProcess root ERROR [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAA64C46F: (caller: 00007FFBAA64DC21) Exception(20) tid(98) 80004005 Unspecified error
Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 91, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAA64C46F: (caller: 00007FFBAA64DC21) Exception(20) tid(98) 80004005 Unspecified error

2024-04-24 17:58:52,890 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-24 17:58:52,890 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-24 17:58:52,890 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-24 17:58:52,890 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-24 17:58:52,890 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-24 17:58:52,890 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-24 17:58:52,890 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-24 17:58:52,890 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-24 17:58:52,890 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-24 17:58:53,186 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-24 17:58:53,186 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-24 17:58:53,186 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-24 17:58:53,208 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-24 17:58:53,208 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-24 17:58:53,208 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-24 17:58:53,218 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-24 17:58:53,218 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-24 17:58:53,218 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:57:05,340 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:57:05,356 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:57:05,356 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:57:05,684 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:57:05,699 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:57:05,699 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:57:59,960 MainProcess root INFO Progress: 8%
2024-04-25 16:57:59,960 MainProcess root INFO Progress: 16%
2024-04-25 16:57:59,976 MainProcess root INFO Progress: 24%
2024-04-25 16:57:59,976 MainProcess root INFO Progress: 32%
2024-04-25 16:58:00,726 MainProcess root INFO Providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-25 16:58:00,726 MainProcess root INFO Used providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-25 16:58:01,500 MainProcess root INFO Progress: 40%
2024-04-25 16:58:01,512 MainProcess root INFO Progress: 48%
2024-04-25 16:58:01,514 MainProcess root INFO Progress: 56%
2024-04-25 16:58:01,516 MainProcess root INFO Progress: 64%
2024-04-25 16:58:01,531 MainProcess root INFO Progress: 72%
2024-04-25 16:58:01,607 MainProcess root INFO Progress: 80%
2024-04-25 16:58:01,669 MainProcess root INFO Progress: 88%
2024-04-25 16:58:01,904 MainProcess root ERROR WARNING: VerifyWarning: Keyword name 'SMOOTHING' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [warnings]
2024-04-25 16:58:01,904 MainProcess astropy WARNING VerifyWarning: Keyword name 'SMOOTHING' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
2024-04-25 16:58:01,919 MainProcess root ERROR WARNING: VerifyWarning: Keyword name 'CORR-TYPE' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [warnings]
2024-04-25 16:58:01,919 MainProcess astropy WARNING VerifyWarning: Keyword name 'CORR-TYPE' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
2024-04-25 16:58:01,935 MainProcess root ERROR WARNING: VerifyWarning: Keyword name 'BGE-AI-VER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [warnings]
2024-04-25 16:58:01,935 MainProcess astropy WARNING VerifyWarning: Keyword name 'BGE-AI-VER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
2024-04-25 16:58:02,310 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,310 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,310 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,310 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,310 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,310 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,670 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,670 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,670 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,670 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,670 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,670 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,685 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,685 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,685 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,685 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,685 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,685 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,685 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,685 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,685 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,998 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,998 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,998 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,998 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,998 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,998 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,998 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-25 16:58:02,998 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,998 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:02,998 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,013 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:03,013 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:03,013 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:03,013 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-25 16:58:03,092 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,092 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,092 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,092 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:03,092 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:03,092 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:03,092 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-25 16:58:03,342 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,342 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,342 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,342 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,342 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,342 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,342 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,342 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,576 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,576 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,576 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:03,576 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-25 16:58:13,247 MainProcess root INFO Starting denoising
2024-04-25 16:58:14,606 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-25 16:58:14,606 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-25 16:58:15,059 MainProcess root ERROR [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAFA3C46F: (caller: 00007FFBAFA3DC21) Exception(3) tid(12e8) 80004005 Unspecified error
Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 91, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFBAFA3C46F: (caller: 00007FFBAFA3DC21) Exception(3) tid(12e8) 80004005 Unspecified error

@pgfabbri
Copy link

Same config. same error. Following

@schmelly
Copy link
Collaborator

Hi all,

we have prepared a pre-release for GraXpert 3.0.1. Could you please try if this improves the situation for you?
https://github.com/Steffenhir/GraXpert/releases/tag/3.0.1

Best regards
David

@sam-pedraglio
Copy link
Author

Hi all,

we have prepared a pre-release for GraXpert 3.0.1. Could you please try if this improves the situation for you? https://github.com/Steffenhir/GraXpert/releases/tag/3.0.1

Best regards David

Hi David, I tested the beta release but the situation is always the same.
Here below you could find the messages saved in the log:

2024-04-26 01:19:54,907 MainProcess root INFO Starting denoising
2024-04-26 01:19:54,907 MainProcess root INFO mapping batch_size of 3 to 2
2024-04-26 01:19:57,704 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-26 01:19:57,704 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-26 01:19:58,344 MainProcess root ERROR [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FF8EBC3C46F: (caller: 00007FF8EBC3DC21) Exception(3) tid(2c34) 80070057 Parametro non corretto.

Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 100, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FF8EBC3C46F: (caller: 00007FF8EBC3DC21) Exception(3) tid(2c34) 80070057 Parametro non corretto.

@pgfabbri
Copy link

Me too withe beta release same error as before : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFB0E9CC46F: (caller: 00007FFB0E9CDC21) Exception(3) tid(3550) 80070057 Parametro non corretto.

@Rikyf3
Copy link
Contributor

Rikyf3 commented Apr 26, 2024

Hi to all.
@schmelly and I are currently exploring the issue. Our main problem is that we are unable to reproduce it on our computers.

Given that the problem is faced on old hardware I will give some general suggestions.

@Markydotcom
Copy link

Hi,
done as requested and still no luck
Thanks

@sam-pedraglio
Copy link
Author

Hi to all. @schmelly and I are currently exploring the issue. Our main problem is that we are unable to reproduce it on our computers.

Given that the problem is faced on old hardware I will give some general suggestions.

Unfortunately the issue is still there for me too

@Rikyf3
Copy link
Contributor

Rikyf3 commented Apr 27, 2024

@Markydotcom @sam-pedraglio @pgfabbri @ljuraszek we have received some positive feedbacks from the following release (https://github.com/Rikyf3/GraXpert/releases/tag/2.1.19). If that is the case I kindly ask you to report back to us.

@pgfabbri
Copy link

it Works!

@henryslash
Copy link

Hello,
i have the same problem.`

2024-04-27 17:44:07,963 SpawnProcess-2 root INFO stretch.stretch_channel started
2024-04-27 17:44:07,965 SpawnProcess-3 root INFO stretch.stretch_channel started
2024-04-27 17:44:07,988 SpawnProcess-4 root INFO stretch.stretch_channel started
2024-04-27 17:44:08,275 SpawnProcess-3 root INFO stretch.stretch_channel finished
2024-04-27 17:44:08,277 SpawnProcess-2 root INFO stretch.stretch_channel finished
2024-04-27 17:44:08,309 SpawnProcess-4 root INFO stretch.stretch_channel finished
2024-04-27 17:44:12,769 MainProcess root INFO Starting denoising
2024-04-27 17:44:12,770 MainProcess root INFO mapping batch_size of 3 to 2
2024-04-27 17:44:13,704 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-27 17:44:13,704 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider']
2024-04-27 17:44:14,019 MainProcess root ERROR [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFE22ECC46F: (caller: 00007FFE22ECDC21) Exception(3) tid(3da8) 80070057 Falscher Parameter.

Traceback (most recent call last):
File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request
File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 100, in denoise
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node.
Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)
\onnxruntime_pybind11_state.pyd!00007FFE22ECC46F: (caller: 00007FFE22ECDC21) Exception(3) tid(3da8) 80070057 Falscher Parameter.

I have updated:

  • to Python 3.12.3
  • winsdk
  • onnxruntime
  • DirectML

still the same error message.

My system requires:
AMD Ryzen 5 3600 6-Core Processor
WIN 10 Pro - Vers. 22H2
Graphic-Card: NVidia Geforce 650 ( DirektX12 - compatible )

@pgfabbri: what has helped you?

@ljuraszek
Copy link

@Markydotcom @sam-pedraglio @pgfabbri @ljuraszek we have received some positive feedbacks from the following release (https://github.com/Rikyf3/GraXpert/releases/tag/2.1.19). If that is the case I kindly ask you to report back to us.

Yep, it is working

@pgfabbri
Copy link

Hello, i have the same problem.`

2024-04-27 17:44:07,963 SpawnProcess-2 root INFO stretch.stretch_channel started 2024-04-27 17:44:07,965 SpawnProcess-3 root INFO stretch.stretch_channel started 2024-04-27 17:44:07,988 SpawnProcess-4 root INFO stretch.stretch_channel started 2024-04-27 17:44:08,275 SpawnProcess-3 root INFO stretch.stretch_channel finished 2024-04-27 17:44:08,277 SpawnProcess-2 root INFO stretch.stretch_channel finished 2024-04-27 17:44:08,309 SpawnProcess-4 root INFO stretch.stretch_channel finished 2024-04-27 17:44:12,769 MainProcess root INFO Starting denoising 2024-04-27 17:44:12,770 MainProcess root INFO mapping batch_size of 3 to 2 2024-04-27 17:44:13,704 MainProcess root INFO Available inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider'] 2024-04-27 17:44:13,704 MainProcess root INFO Used inference providers : ['DmlExecutionProvider', 'CPUExecutionProvider'] 2024-04-27 17:44:14,019 MainProcess root ERROR [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482)\onnxruntime_pybind11_state.pyd!00007FFE22ECC46F: (caller: 00007FFE22ECDC21) Exception(3) tid(3da8) 80070057 Falscher Parameter.

Traceback (most recent call last): File "D:\a\GraXpert\GraXpert\graxpert\application\app.py", line 345, in on_denoise_request File "D:\a\GraXpert\GraXpert\graxpert\denoising.py", line 100, in denoise File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'StatefulPartitionedCall/generator/tf.operators.add/AddV2' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2482) \onnxruntime_pybind11_state.pyd!00007FFE22ECC46F: (caller: 00007FFE22ECDC21) Exception(3) tid(3da8) 80070057 Falscher Parameter.

I have updated:

  • to Python 3.12.3
  • winsdk
  • onnxruntime
  • DirectML

still the same error message.

My system requires: AMD Ryzen 5 3600 6-Core Processor WIN 10 Pro - Vers. 22H2 Graphic-Card: NVidia Geforce 650 ( DirektX12 - compatible )

@pgfabbri: what has helped you?

this version https://github.com/Rikyf3/GraXpert/releases/tag/2.1.19

@sam-pedraglio
Copy link
Author

@Markydotcom @sam-pedraglio @pgfabbri @ljuraszek we have received some positive feedbacks from the following release (https://github.com/Rikyf3/GraXpert/releases/tag/2.1.19). If that is the case I kindly ask you to report back to us.

Yeaaahhh, it's working!

@henryslash
Copy link

Hello all,
with the version https://github.com/Rikyf3/GraXpert/releases/tag/2.1.19) it works. Yeaaaah
You are all great

Thank you to all.

@Markydotcom
Copy link

Hi,
https://github.com/Rikyf3/GraXpert/releases/tag/2.1.19
This works very well for me too!
Many,many thanks.
Markydotcom.

@schmelly
Copy link
Collaborator

schmelly commented May 6, 2024

Hi all,

we have released version 3.0.2 of GraXpert and also a new denoising model in version 2.0.0.
Please note: as a workaround, 3.0.2 contains a switch to disable hardware acceleration in GraXpert's advanced settings. This needs to be disabled once to workaround your issue on older Win10 / hardware combinations. We are still investigating the actual root cause which appears to be some incompatability between onnxruntime / Win10 / drivers / hardware etc.

The drawback is no hardware acceleration at the moment, the advantage is at least a working denoising.

https://github.com/Steffenhir/GraXpert/releases/tag/3.0.2

Best regards
David

@FilipDS
Copy link

FilipDS commented May 6, 2024

Hi,

Downloaded version 3.02 and on HP desktop with Systeminfo (SysInfo_HPDesktop.jpg) still the same error message as before.

On Lenovo Notebook with Systeminfo (SysInfo_Notebook.jpg) both version 3.01 and 3.02 work.

Remark: version 2.1.1.19 works fine on the HP Desktop

Regards

SysInfo_HPDesktop
SysInfo_Notebook

@Rikyf3
Copy link
Contributor

Rikyf3 commented May 6, 2024

Hi,

Downloaded version 3.02 and on HP desktop with Systeminfo (SysInfo_HPDesktop.jpg) still the same error message as before.

On Lenovo Notebook with Systeminfo (SysInfo_Notebook.jpg) both version 3.01 and 3.02 work.

Remark: version 2.1.1.19 works fine on the HP Desktop

Regards

SysInfo_HPDesktop

SysInfo_Notebook

Have you disabled hardware acceleration from the advanced panel?

@FilipDS
Copy link

FilipDS commented May 6, 2024

Thanks for reminding me - works perfectly. Mea culpa!!!!!!!!!

Meanwhile, I take this opportunity to thank you for the team's efforts to create and maintain a very useful program.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants