Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Converting onnx model to deepsparse #218

Closed
joaanna opened this issue Dec 5, 2021 · 11 comments
Closed

Converting onnx model to deepsparse #218

joaanna opened this issue Dec 5, 2021 · 11 comments
Labels
bug Something isn't working

Comments

@joaanna
Copy link

joaanna commented Dec 5, 2021

Hi,
I'm trying to convert an onnx model to a deepsparse model, here is the code:

from deepsparse import compile_model
from deepsparse.utils import generate_random_inputs
onnx_filepath = "fom.onnx"
batch_size = 1

# Generate random sample input
inputs = generate_random_inputs(onnx_filepath, batch_size)

# Compile and run
engine = compile_model(onnx_filepath, batch_size)
outputs = engine.run(inputs)

**Environment**
Include all relevant environment information:
1. Ubuntu 18.04:
2. Python version 3.7.9. :
3. DeepSparse version 0.8.0 :
4. torch 1.9.0+cu102:
5. Other Python package versions [e.g. SparseML, Sparsify, numpy, ONNX]:
6. CPU {'vendor': 'GenuineIntel', 'isa': 'avx512', 'vnni': True, 'num_sockets': 2, 'available_sockets': 2, 'cores_per_socket': 18, 'available_cores_per_socket': 18, 'threads_per_core': 2, 'available_threads_per_core': 2, 'L1_instruction_cache_size': 32768, 'L1_data_cache_size': 32768, 'L2_cache_size': 1048576, 'L3_cache_size': 25952256}

**Errors**
[     INFO            onnx.py: 128 - generate_random_inputs() ] -- generating random input #0 of shape = [1, 3, 256, 256]
[     INFO            onnx.py: 128 - generate_random_inputs() ] -- generating random input #1 of shape = [1, 10, 2]
[     INFO            onnx.py: 128 - generate_random_inputs() ] -- generating random input #2 of shape = [1, 10, 2]
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.8.0 (68df72e1) (release) (optimized) (system=avx512, binary=avx512)
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.8.0 (68df72e1) (release) (optimized)
Date: 12-05-2021 @ 12:58:29 EST
OS: Linux visiongpu49 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021
Arch: x86_64
CPU: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
Vendor: GenuineIntel
OS: Linux visiongpu49 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021
Arch: x86_64
CPU: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
Vendor: GenuineIntel
Cores/sockets/threads: [36, 2, 72]
Available cores/sockets/threads: [36, 2, 72]
L1 cache size data/instruction: 32k/32k
L2 cache size: 1Mb
L3 cache size: 24.75Mb
Total memory: 507.367G
Free memory: 22.4387G

Assertion at ./src/include/wand/engine/compute/planner.hpp:131

Backtrace:
0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
1# 0x00007F36EB17C234 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
2# 0x00007F36EB185889 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
3# 0x00007F36EB185982 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
4# 0x00007F36EB18AA8A in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
5# 0x00007F36EB18AB00 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
6# 0x00007F36EA7E985D in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
7# 0x00007F36EA7EE443 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
8# 0x00007F36EA76BD6B in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
9# 0x00007F36EA75AB3F in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
10# 0x00007F36EA75C1C1 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
11# 0x00007F36EADA9668 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
12# 0x00007F36EADAC0A2 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
13# 0x00007F36EADAF3B9 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
14# 0x00007F36EA73B76C in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
15# 0x00007F36EA7414C3 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
16# 0x00007F36EA6FB982 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
17# 0x00007F36EA6FBC05 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
18# deepsparse::ort_engine::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, wand::safe_type<wand::parallel::use_current_affinity_tag, bool>, std::shared_ptr<wand::parallel::scheduler_factory_t>) in /data/lib/python3.7/site-packages/deepsparse/avx512/libdeepsparse.so
19# 0x00007F3771031D1B in /data/lib/python3.7/site-packages/deepsparse/avx512/deepsparse_engine.so
20# 0x00007F3771031F39 in /data/lib/python3.7/site-packages/deepsparse/avx512/deepsparse_engine.so
21# 0x00007F377105D5C5 in /data/lib/python3.7/site-packages/deepsparse/avx512/deepsparse_engine.so
22# 0x00007F377104B250 in /data/lib/python3.7/site-packages/deepsparse/avx512/deepsparse_engine.so
23# _PyMethodDef_RawFastCallDict in python

Please email a copy of this stack trace and any additional information to: support@neuralmagic.com
Aborted

Do you have any ideas why the code is failing?
@joaanna joaanna added the bug Something isn't working label Dec 5, 2021
@SageMoore
Copy link
Contributor

Hi, joaanna. Thanks for filing the bug. Can you please post a link to the onnx model?

@joaanna
Copy link
Author

joaanna commented Dec 6, 2021

Hi, there it it: https://drive.google.com/file/d/1W3TS2VNirOeI5bcqqy4a9YdaRmfB7uza/view?usp=sharing. Thank you for your help

@joaanna
Copy link
Author

joaanna commented Dec 6, 2021

Hi, there it it: https://drive.google.com/file/d/1W3TS2VNirOeI5bcqqy4a9YdaRmfB7uza/view?usp=sharing. Thank you for your help

I'm happy to upload it again in case it's no available

@tlrmchlsmth
Copy link
Member

Hi joanna,

Thank you very much for providing us with the model. We've been able to reproduce the bug and plan to release a hotfix to fix the issue soon.

@mgoin
Copy link
Member

mgoin commented Dec 7, 2021

Hi there, we have a fix in the latest nightly build if you would like to try it out.
https://pypi.org/project/deepsparse-nightly/0.10.0.20211206/

pip install deepsparse-nightly==0.10.0.20211206

Otherwise stay posted for a hotfix release, thanks!

@joaanna
Copy link
Author

joaanna commented Dec 7, 2021

Hi,

Thank you for such a fast response!
I did try your updated package, however I get the same error as before:

[     INFO            onnx.py: 128 - generate_random_inputs() ] -- generating random input #0 of shape = [1, 3, 256, 256]
[     INFO            onnx.py: 128 - generate_random_inputs() ] -- generating random input #1 of shape = [1, 10, 2]
[     INFO            onnx.py: 128 - generate_random_inputs() ] -- generating random input #2 of shape = [1, 10, 2]
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.10.0.20211206 (08dec93d) (release) (optimized) (system=avx512, binary=avx512)
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.10.0.20211206 (08dec93d) (release) (optimized)
Date: 12-07-2021 @ 17:33:46 EST
OS: Linux visiongpu49 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021
Arch: x86_64
CPU: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
Vendor: GenuineIntel
Cores/sockets/threads: [36, 2, 72]
Available cores/sockets/threads: [36, 2, 72]
L1 cache size data/instruction: 32k/32k
L2 cache size: 1Mb
L3 cache size: 24.75Mb
Total memory: 507.367G
Free memory: 110.122G

Assertion at ./src/include/wand/engine/compute/planner.hpp:131

Backtrace:
0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
1# 0x00007F36EB17C234 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
2# 0x00007F36EB185889 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
3# 0x00007F36EB185982 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
4# 0x00007F36EB18AA8A in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
5# 0x00007F36EB18AB00 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
6# 0x00007F36EA7E985D in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
7# 0x00007F36EA7EE443 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
8# 0x00007F36EA76BD6B in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
9# 0x00007F36EA75AB3F in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
10# 0x00007F36EA75C1C1 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
11# 0x00007F36EADA9668 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
12# 0x00007F36EADAC0A2 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
13# 0x00007F36EADAF3B9 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
14# 0x00007F36EA73B76C in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
15# 0x00007F36EA7414C3 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
16# 0x00007F36EA6FB982 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
17# 0x00007F36EA6FBC05 in /data/lib/python3.7/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0
18# deepsparse::ort_engine::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, wand::safe_type<wand::parallel::use_current_affinity_tag, bool>, std::shared_ptr<wand::parallel::scheduler_factory_t>) in /data/lib/python3.7/site-packages/deepsparse/avx512/libdeepsparse.so
19# 0x00007F3771031D1B in /data/lib/python3.7/site-packages/deepsparse/avx512/deepsparse_engine.so
20# 0x00007F3771031F39 in /data/lib/python3.7/site-packages/deepsparse/avx512/deepsparse_engine.so
21# 0x00007F377105D5C5 in /data/lib/python3.7/site-packages/deepsparse/avx512/deepsparse_engine.so
22# 0x00007F377104B250 in /data/lib/python3.7/site-packages/deepsparse/avx512/deepsparse_engine.so
23# _PyMethodDef_RawFastCallDict in python

Please email a copy of this stack trace and any additional information to: support@neuralmagic.com
Aborted

I am looking forward to the hotfix release :)

@mgoin
Copy link
Member

mgoin commented Dec 10, 2021

Hi @joaanna sorry for the delay. Thanks for trying and you are right to see the same error, that nightly was just too old to include the fix.

If you'd like to give it a try with our initial fix, please feel free to try the new nightly build with pip install deepsparse-nightly==0.10.0.20211208.

An official hotfix release will be coming likely next week. Thanks!

@prasanth-pivotchain
Copy link

prasanth-pivotchain commented Dec 16, 2021 •
edited
Loading

Hi @mgoin, I am facing a similar issue

DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.10.0.20211208 (811251bd) (release) (optimized) (system=avx2, binary=avx2)
WARNING: Running quantized (INT8) operations with only AVX2 instructions available. Sparse optimizations disabled and performance is greatly degraded.
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.10.0.20211208 (811251bd) (release) (optimized)
Date: 12-16-2021 @ 11:55:29 IST
OS: Linux support-ThinkPad-E490 5.11.0-40-generic #44~20.04.2-Ubuntu SMP Tue Oct 26 18:07:44 UTC 2021
Arch: x86_64
CPU: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
Vendor: GenuineIntel
Cores/sockets/threads: [4, 1, 8]
Available cores/sockets/threads: [4, 1, 8]
L1 cache size data/instruction: 32k/32k
L2 cache size: 0.25Mb
L3 cache size: 6Mb
Total memory: 17.3526G
Free memory: 0.371181G

Assertion at ./src/include/wand/jit/pooling/common.hpp:239

Backtrace:
 0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 1# wand::detail::assert_fail(char const*, char const*, int) in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 2# 0x00007FCD9390154F in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 3# 0x00007FCD93900E53 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 4# 0x00007FCD93901405 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 5# 0x00007FCD93901A32 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 6# 0x00007FCD93905330 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 7# 0x00007FCD939056FA in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 8# 0x00007FCD93905CC8 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
 9# 0x00007FCD929F7A49 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
10# 0x00007FCD929FB675 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
11# 0x00007FCD929FD98F in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
12# 0x00007FCD929FDF44 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
13# 0x00007FCD92682EDF in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
14# 0x00007FCD92677E9C in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
15# 0x00007FCD9266525F in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
16# 0x00007FCD92B989A9 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
17# 0x00007FCD92697EBF in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
18# 0x00007FCD926B317D in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
19# 0x00007FCD926B32A3 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
20# 0x00007FCD926B3913 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
21# 0x00007FCD93A61130 in /home/support/anaconda3/envs/sparsify/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.8.0
22# 0x00007FCDFAA5E609 in /lib/x86_64-linux-gnu/libpthread.so.0
23# clone in /lib/x86_64-linux-gnu/libc.so.6

I have deepsparse-nightly==0.10.0.20211208 installed. The engine runs fine when using the original image size for yolo, however, it fails with the above stack trace when I modify the model input size using deepsparse_utils.modify_yolo_onnx_input_shape . Please help.
TY

@mgoin
Copy link
Member

mgoin commented Dec 16, 2021

Hi @joaanna our hotfix release 0.9.1 is out which includes a fix for your model, thanks for your help.

Also hi @prasanth-pivotchain , unfortunately this seems to be a separate issue that we're having trouble replicating. Another user just posted an issue (#226) that seems to the same though, so please share details there if you'd like to help out. Thanks!

@prasanth-pivotchain
Copy link

Okay, Thanks

@jeanniefinks
Copy link
Member

jeanniefinks commented Jan 24, 2022 •
edited
Loading

Hello @prasanth-pivotchain and @joaanna
It appears we can close our this thread and I am going to go ahead and close this comment. Please re-open if you have a follow-up. Also, I invite you to "star" our DeepSparse repo if you like and have not already! We always like seeing the community support!
https://github.com/neuralmagic/deepsparse/

Thank you!
Jeannie / Neural Magic

@edwardnguyen1705 edwardnguyen1705 mentioned this issue Mar 8, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@tlrmchlsmth @mgoin @SageMoore @joaanna @jeanniefinks @prasanth-pivotchain