Converting nanodet onnx model to deepsparse

[edwardnguyen1705]

Dear @jeanniefinks ,

Firstly, thanks for sharing your work.
We are trying to apply sparseml to NanoDet-Plus-m that is considered the most suitable for edge devices til now.

Here are some steps I have been trying:

1. Download pytorch (.pth) model then convert to onnx model. I even tried: sparseml.onnx_export, I was able to convert to model.onnx, but still failed in the next step.
2. Convert onnx model to deepsparse. It is similar to the issue Converting onnx model to deepsparse #218
3. Use both deepsparse and sparsify

I already tried on varying environments:

OS: Ubuntu16.4/18.4
CPU: avx avx2, grep -o 'avx[^ ]*' /proc/cpuinfo
Varying deepsparse, onnx/onnxruntine versions
torch: 1.8.2+cpu

Code to produce error:

>>> from deepsparse import compile_model
>>> from deepsparse.utils import generate_random_inputs
>>> batch_size = 1
>>> onnx_filepath = "checkpoints/nanodet-plus-m_320.onnx"
>>> inputs = generate_random_inputs(onnx_filepath, batch_size)
[INFO            onnx.py:176 ] Generating input 'data', type = float32, shape = [1, 3, 320, 320]
>>> engine = compile_model(onnx_filepath, batch_size)

Error:

DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.10.0 (c2458ea3) (release) (optimized) (system=avx2, binary=avx2)
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.10.0 (c2458ea3) (release) (optimized)
Date: 03-08-2022 @ 08:50:06 UTC
OS: Linux 7a5617e3c49d 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021
Arch: x86_64
CPU: Intel(R) Xeon(R) CPU E5-2623 v3 @ 3.00GHz
Vendor: GenuineIntel
Cores/sockets/threads: [8, 2, 16]
Available cores/sockets/threads: [8, 2, 16]
L1 cache size data/instruction: 32k/32k
L2 cache size: 0.25Mb
L3 cache size: 10Mb
Total memory: 127.793G
Free memory: 10.5767G

Assertion at ./src/include/wand/utility/pyramidal/task_graph_utils.hpp:133

Backtrace:
 0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 1# 0x00007F2DCE3D0D08 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 2# 0x00007F2DCE3D5487 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 3# 0x00007F2DCE3D9B76 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 4# 0x00007F2DCE311F6F in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 5# 0x00007F2DCE3140A5 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 6# 0x00007F2DCE315444 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 7# 0x00007F2DCE315819 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 8# 0x00007F2DCE2C6E1B in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 9# 0x00007F2DCE228704 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
10# 0x00007F2DCE228A32 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
11# 0x00007F2DCE228B78 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
12# 0x00007F2DCE228D5D in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
13# 0x00007F2DCE228FA8 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
14# 0x00007F2DCE229010 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
15# 0x00007F2DCD82BD47 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
16# 0x00007F2DCD8320CF in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
17# 0x00007F2DCD7AA52B in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
18# 0x00007F2DCD79A109 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
19# 0x00007F2DCD79B2C1 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
20# 0x00007F2DCDE266B8 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
21# 0x00007F2DCDE290CC in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
22# 0x00007F2DCDE2C399 in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
23# 0x00007F2DCD77B9AB in /usr/local/lib/python3.6/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0

Please email a copy of this stack trace and any additional information to: support@neuralmagic.com
Aborted

It seems that you have your own onnxruntime?
Could you examine the NanoDet-Plus-m model? I really appreciate your time.

[mgoin]

Hi @edwardnguyen1705 thanks for telling us about this new model and your issue.

We were able to reproduce your issue and are working on a fix with the nanodet-plus-m_320.onnx pre-exported from this release: https://github.com/RangiLyu/nanodet/releases/tag/v1.0.0-alpha-1

We'll let you know when we have a release ready for you to test so you can be unblocked. Thanks!

[edwardnguyen1705]

Hi @mgoin thanks for letting me know. It is great to hear you said that.

[tlrmchlsmth]

Hi @edwardnguyen1705 we just released DeepSparse 0.11.1, which contains a fix for the assertion that you ran into. Hopefully this fixes your issue, and if you try it, please let us know how it goes. Thank you!

[edwardnguyen1705]

Hi @tlrmchlsmth ,
Thanks for the fix. After install new verion:

deepsparse           0.11.2
sparseml             0.11.1
sparsezoo            0.11.0
sparsify             0.11.0

It seems that the old err is solved, but I got another err, please the the attached image.

r

[edwardnguyen1705]

Hi @tlrmchlsmth ,
I am able to fix the error. However, It would be nice if sparsify supports pytorch model (pth). It is really hard to find the mapping between onnx and pytorch. Thanks for your time.

[markurtz]

Hi @edwardnguyen1705, I've opened up neuralmagic/sparseml#670 to continue the discussion on this. These issues will happen outside of DeepSparse and we'll continue to support through that new thread.

Thanks!