[Tutorial - QNN] Prequantized MXNet model compilation. #5362

anijain2305 · 2020-04-17T21:14:28Z

@masahi Earlier the plan was to have a single tutorial. However, I get seg faults if I import torch and mxnet in the same file. So, I am breaking it down into 3 parts - PyTorch, MXNet and TFLite. I added for Part 2 and massaged the intro to part 1.

anijain2305 · 2020-04-17T21:15:42Z

@jwfromm @siju-samuel You might also be interested

masahi · 2020-04-17T22:44:57Z

tutorials/frontend/deploy_prequantized_mxnet.py

+# print(mod)
+
+# Compile Relay module. Set the target platform. Replace the target with the your target type.
+target = 'llvm -mcpu=cascadelake'


this doesn't work on CI

siju-samuel

A few nits.

siju-samuel · 2020-04-18T04:13:47Z

tutorials/frontend/deploy_prequantized_mxnet.py

+###############################################################################
+# Helper functions
+# ----------------
+def download_calib_dataset(dataset_url, calib_dataset):


Can you use from tvm.contrib.download import download_testdata' as in other tutorials to download the calib record, currently its downloading to ./data`

I need more images to perform the calibration. I moved the dataset to /tmp folder for now.

siju-samuel · 2020-04-18T04:21:56Z

tutorials/frontend/deploy_prequantized_pytorch.py

+In this series of tutorials, we demonstrate how to load and run models quantized by PyTorch (Part
+1), MXNet (Part 2), and TFLite (Part 3). Once loaded, we can run compiled, quantized models on any
+hardware TVM supports.
+
+This is part 1 of the tutorial, where we will focus on PyTorch-prequantized models.


Since the 3 tutorials are in different files, suggest we can remove the references to MxNet and TFLite here. May be the below line is enough.

Here, we demonstrate how to load and run models quantized by PyTorch. Once loaded, we can run compiled, quantized models on any hardware TVM supports.

siju-samuel · 2020-04-18T04:25:48Z

tutorials/frontend/deploy_prequantized_mxnet.py

+def get_mxnet_fp32_model():
+    """ Read the MXNet symbol. """
+    model_name = 'resnet50_v1'
+    dir_path = os.path.dirname(os.path.realpath(__file__))


dir_path not used, can be removed.

siju-samuel · 2020-04-18T04:29:29Z

tutorials/frontend/deploy_prequantized_mxnet.py

+
+import mxnet as mx
+from gluoncv.model_zoo import get_model
+from mxnet.contrib.quantization import *


if possible, remove wildcard imports

siju-samuel · 2020-04-18T04:33:07Z

tutorials/frontend/deploy_prequantized_mxnet.py

+# affected. Output of the following code is as follows
+#
+# TVM Top-5 labels: [236 211 178 165 168]
+# MXNet Top-5 labels: [236 211 178 165 168]


This can be removed, while document is rendered, it will print.

anijain2305 · 2020-04-19T05:21:11Z

@tqchen This tutorial requires mxnet-mkl package.

Currently, the CI failure is

  File "/usr/local/lib/python3.6/dist-packages/sphinx_gallery/gen_rst.py", line 480, in _memory_usage

    out = func()

  File "/usr/local/lib/python3.6/dist-packages/sphinx_gallery/gen_rst.py", line 465, in __call__

    exec(self.code, self.globals)

  File "/workspace/tutorials/frontend/deploy_prequantized_mxnet.py", line 46, in <module>

    from mxnet.contrib.quantization import quantize_model_mkldnn

ImportError: cannot import name 'quantize_model_mkldnn'

There is no workaround here, I am using MXNet-MKL quantizer to quantize the model.

If we want to have this tutorial anyways, I can wrap the tutorial into a function and comment its invocation. And when in future, we have the package, I can remove the comment.

masahi · 2020-04-21T11:53:00Z

Can we skip calibration (uncomment to encourage testing locally) and still get meaningful output? I think it is unlikely we can update our CI for the sake of this tutorial.

tqchen · 2020-04-21T14:58:57Z

We can just use the floating pt model for the reference pt

anijain2305 · 2020-04-21T16:48:57Z

@masahi @tqchen The MXNet quantized models has operators that can only work with MKLDNN. Example is as follows. Note the "op": "_sg_mkldnn_conv",

{
      "op": "_sg_mkldnn_conv",
      "name": "quantized_sg_mkldnn_conv_bn_act_18",
      "attrs": {
        "max_calib_range": "2.660447",
        "min_calib_range": "0.000000",
        "quantized": "true",
        "with_act": "true",
        "with_bn": "true"
      },
      "inputs": [[110, 0, 0], [111, 0, 0], [112, 0, 0], [113, 0, 0], [114, 0, 1], [115, 0, 1], [110, 1, 0], [110, 2, 0]],
      "subgraphs": [
        {
          "nodes": [
            {
              "op": "null",
              "name": "sg_mkldnn_conv_bn_add_act_15_output0",
              "inputs": []
            },
            {
              "op": "null",
              "name": "resnetv10_stage4_conv3_weight0",
              "inputs": []
            },
            {
              "op": "Convolution",
              "name": "resnetv10_stage4_conv3_fwd",
              "attrs": {
                "dilate": "(1, 1)",
                "kernel": "(3, 3)",
                "layout": "NCHW",
                "no_bias": "True",
                "num_filter": "512",
                "num_group": "1",
                "pad": "(1, 1)",
                "stride": "(1, 1)"
              },
              "inputs": [[0, 0, 0], [1, 0, 0]]
            },
            {
              "op": "null",
              "name": "resnetv10_stage4_batchnorm3_gamma0",
              "inputs": []
            },
            {
              "op": "null",
              "name": "resnetv10_stage4_batchnorm3_beta0",
              "inputs": []
            },
            {
              "op": "null",
              "name": "resnetv10_stage4_batchnorm3_running_mean0",
              "inputs": []
            },
            {
              "op": "null",
              "name": "resnetv10_stage4_batchnorm3_running_var0",
              "inputs": []
            },
            {
              "op": "BatchNorm",
              "name": "resnetv10_stage4_batchnorm3_fwd",
              "attrs": {
                "axis": "1",
                "eps": "1e-05",
                "fix_gamma": "False",
                "momentum": "0.9",
                "use_global_stats": "False"
              },
              "inputs": [[2, 0, 0], [3, 0, 0], [4, 0, 0], [5, 0, 0], [6, 0, 0]]
            },
            {
              "op": "Activation",
              "name": "resnetv10_stage4_relu1_fwd",
              "attrs": {"act_type": "relu"},
              "inputs": [[7, 0, 0]]
            }
          ],
          "arg_nodes": [0, 1, 3, 4, 5, 6],
          "node_row_ptr": [0, 1, 2, 3, 4, 5, 6, 7, 10, 11],
          "heads": [[8, 0, 0]]
        }
      ]
    },

Even if I quantize the model outside of this tutorial. I would still need mxnet-mkl to read the MXNet quantized model in the Relay parser.

tqchen · 2020-04-21T18:36:03Z

OK, let us wait for the mxnet-mkl then, currently blocked by #5396 hopefully we can land this week

masahi · 2020-05-11T20:07:54Z

@anijain2305 @tqchen What is the status on this? I see there have been a recent CI change. Are we ready to have mxnet-mkl?

tqchen · 2020-05-11T21:00:15Z

see #5458

[Tutorial - QNN] Prequantized MXNet model compilation.

10236cd

masahi reviewed Apr 17, 2020

View reviewed changes

Sphinx fis.

0520b36

siju-samuel reviewed Apr 18, 2020

View reviewed changes

Reviews address.

4aa193f

masahi self-assigned this May 11, 2020

siju-samuel mentioned this pull request May 14, 2020

[TUTORIAL]TFLite QNN Tutorial #5595

Merged

masahi mentioned this pull request May 16, 2020

[DOC] Documentation on Quantization #4435

Closed

tqchen closed this Oct 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tutorial - QNN] Prequantized MXNet model compilation. #5362

[Tutorial - QNN] Prequantized MXNet model compilation. #5362

anijain2305 commented Apr 17, 2020

anijain2305 commented Apr 17, 2020

masahi Apr 17, 2020

siju-samuel left a comment

siju-samuel Apr 18, 2020

anijain2305 Apr 19, 2020

siju-samuel Apr 18, 2020

siju-samuel Apr 18, 2020

siju-samuel Apr 18, 2020

siju-samuel Apr 18, 2020

anijain2305 commented Apr 19, 2020 •

edited

masahi commented Apr 21, 2020 •

edited

tqchen commented Apr 21, 2020

anijain2305 commented Apr 21, 2020 •

edited

tqchen commented Apr 21, 2020 •

edited

masahi commented May 11, 2020

tqchen commented May 11, 2020

[Tutorial - QNN] Prequantized MXNet model compilation. #5362

[Tutorial - QNN] Prequantized MXNet model compilation. #5362

Conversation

anijain2305 commented Apr 17, 2020

anijain2305 commented Apr 17, 2020

masahi Apr 17, 2020

Choose a reason for hiding this comment

siju-samuel left a comment

Choose a reason for hiding this comment

siju-samuel Apr 18, 2020

Choose a reason for hiding this comment

anijain2305 Apr 19, 2020

Choose a reason for hiding this comment

siju-samuel Apr 18, 2020

Choose a reason for hiding this comment

siju-samuel Apr 18, 2020

Choose a reason for hiding this comment

siju-samuel Apr 18, 2020

Choose a reason for hiding this comment

siju-samuel Apr 18, 2020

Choose a reason for hiding this comment

anijain2305 commented Apr 19, 2020 • edited

masahi commented Apr 21, 2020 • edited

tqchen commented Apr 21, 2020

anijain2305 commented Apr 21, 2020 • edited

tqchen commented Apr 21, 2020 • edited

masahi commented May 11, 2020

tqchen commented May 11, 2020

anijain2305 commented Apr 19, 2020 •

edited

masahi commented Apr 21, 2020 •

edited

anijain2305 commented Apr 21, 2020 •

edited

tqchen commented Apr 21, 2020 •

edited