[Feature]: format README, fix doc link (#88)

* [Feature]: format README, fix doc link

* [Feature]: update PTQ and QAT docs

* [Feature]: update Detection docs

* [Feature]: add NLP docs and  uninstall guide

Co-authored-by: 董志伟 <dongzhiwei1@sensetime.com>

README.md

@@ -1,6 +1,5 @@
 <div align="center">
-  <img src="resources/logo.png" width="700"/>
-
+    <img src="resources/logo.png" width="700" />
 </div>
 
 ------------
@@ -13,31 +12,29 @@
 MQBench is an open-source model quantization toolkit based on PyTorch fx.
 
 The envision of MQBench is to provide:
-- **SOTA Algorithms**. With MQBench, the hardware vendors and researchers can benefit from the latest research progress in academia.
-- **Powerful Toolkits**. With the toolkit, quantization node can be inserted to the original PyTorch module automatically with respect to the specific hardware. After training, the quantized model can be smoothly converted to the format that can inference on the real device.
 
+- **SOTA Algorithms**. With MQBench, the hardware vendors and researchers can benefit from the latest research progress in academic.
+- **Powerful Toolkits**. With the toolkit, quantization node can be inserted to the original PyTorch module automatically with respect to the specific hardware. After training, the quantized model can be smoothly converted to the format that can inference on the real device.
 
 ## Installation
-```
 
+```shell
 git clone git@github.com:ModelTC/MQBench.git
 cd MQBench
 python setup.py install
 ```
 
-
 ## Documentation
 
 MQBench aims to support (1) various deployable quantization algorithms and (2) hardware backend libraries to facilitate the development of the community.
 
-For the detailed information, please refer to [mqbench documentation](http://mqbench.tech/assets/docs/html/) or [latest version](https://mqbench.readthedocs.io/en/main/).
-
+For the detailed information, please refer to [MQBench documentation](https://mqbench.readthedocs.io/en/latest/).
 
 ## Citation
 
 If you use this toolkit or benchmark in your research, please cite this project.
 
-```
+```latex
 @article{MQBench,
   title   = {MQBench: Towards Reproducible and Deployable Model Quantization Benchmark},
   author  = {Yuhang Li* and Mingzhu Shen* and Jian Ma* and Yan Ren* and Mingxin Zhao* and
@@ -47,7 +44,6 @@ If you use this toolkit or benchmark in your research, please cite this project.
 }
 ```
 
-
 ## License
 
-This project is released under the [Apache 2.0 license](LICENSE).
+This project is released under the [Apache 2.0 license](LICENSE).

docs/source/benchmark/NaturalLanguageProcessing/Benchmark.rst

@@ -1,4 +1,31 @@
 Natural Language Processing Benchmark
 =====================================
+Based on MQBench, we provide a sentence classification benchmark on GLUE.
+We provide the results of bert-base-uncased on 8 tasks of GLUE.
 
-To be finished.
+Post-training Quantization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Backend: Academic
+- W_calibration: MinMax
+- A_calibration: EMAQuantile
+
++------------------+----------------------------+--------------------------------------+-------------------------------+
+| Task             | Metrics                    | FP32 results                         | int8 results                  |
++==================+============================+======================================+===============================+
+| **mrpc**         | **acc/f1**                 | **87.75/91.35**                      | **87.75/91.2**                |
++------------------+----------------------------+--------------------------------------+-------------------------------+
+| **mnli**         | **acc m/mm**               | **84.94/84.76**                      | **84.69/84.59**               |
++------------------+----------------------------+--------------------------------------+-------------------------------+
+| **cola**         | **Matthews corr**          | **59.6**                             | **59.41**                     |
++------------------+----------------------------+--------------------------------------+-------------------------------+
+| **sst2**         | **acc**                    | **93.35**                            | **92.78**                     |
++------------------+----------------------------+--------------------------------------+-------------------------------+
+| **stsb**         | **Pearson/Spearman corr**  | **89.70/89.28**                      | **89.36/89.22**               |
++------------------+----------------------------+--------------------------------------+-------------------------------+
+| **qqp**          | **f1/acc**                 | **87.82/90.91**                      | **87.46/90.72**               |
++------------------+----------------------------+--------------------------------------+-------------------------------+
+| **rte**          | **acc**                    | **72.56**                            | **71.84**                     |
++------------------+----------------------------+--------------------------------------+-------------------------------+
+| **qnli**         | **acc**                    | **91.84**                            | **91.32**                     |
++------------------+----------------------------+--------------------------------------+-------------------------------+

docs/source/get_started/index.rst

@@ -7,6 +7,6 @@ This tutorial will give details about the whole work-through to do quantization
    :titlesonly:
 
    setup
-   quick_start_academic
    quick_start_deploy
+   quick_start_academic
    support_matrix

docs/source/get_started/quick_start_academic.rst

@@ -1,15 +1,12 @@
 Quick Start -- Embrace Best Research Experience
 =================================================
 
-This page is for researchers **who want to validate their marvelous quantization idea using MQBench**, 
-if you want to get started with deployment using MQBench, check :doc:`quick_start_deploy`.
+This page is for researchers **who want to validate their marvelous quantization idea using MQBench**, if you want to get started with deployment using MQBench, check :doc:`quick_start_deploy`.
 
-MQBench is a benchmark, a framework and a good tool for researchers. MQBench is designed easy-to-use for researchers, 
-for example, you can easily custom Academic Backend by provide a extra config dict to conduct any experiment.
+MQBench is a benchmark, a framework and a good tool for researchers. MQBench is designed easy-to-use for researchers, for example, you can easily custom Academic Backend by providing an extra config dict to conduct any experiment.
 We provide step-by-step instructions and detailed comments below to help you finish deploying the **PyTorch ResNet18** model to a **Custom Academic** Backend.
 
-Before starting, you should install MQBench first. Now we start the tour.
-
+Before starting, you should have done the MQBench setup in :doc:`setup`. Now we start the tour.
 
 **1**. **To begin with, let's import MQBench and prepare FP32 model.**
 
@@ -27,7 +24,7 @@ Before starting, you should install MQBench first. Now we start the tour.
 **2**. **Then we learn the extra configration to custom Academic Backend.**
 
 You can also learn this section through MQBench `source code <https://github.com/ModelTC/MQBench/blob/main/mqbench/prepare_by_platform.py#L125>`_.
-Learn which option you can choose below config through our :doc:`../user_guide/internal/learn_config`
+Learn all options through our :doc:`../user_guide/internal/learn_config`
 
 .. code-block:: python
 
@@ -52,7 +49,7 @@ Learn which option you can choose below config through our :doc:`../user_guide/i
         }
     }
 
-**3**. **The next step prepares to conduct the experiment, take PTQ as example.**
+**3**. **The next step prepares to conduct the experiment, take PTQ as the example.**
 
 .. code-block:: python
 
@@ -73,6 +70,6 @@ Learn which option you can choose below config through our :doc:`../user_guide/i
         # do forward procedures
         ...
 
-**You already know all basics about how to validate your marvelous quantization idea with MQBench, congratulations!**
+**You have already known all basics about how to validate your marvelous quantization idea with MQBench, congratulations!**
 
 Now you can follow our advanced :doc:`user guide <../developer_guide/index>` and :doc:`developer guide <../user_guide/index>` to know more about MQBench.

docs/source/get_started/quick_start_deploy.rst

@@ -27,14 +27,30 @@ Before starting, you should install MQBench first. Now we start the tour.
     model = models.__dict__["resnet18"](pretrained=True)          # use vision pre-defined model
     model.eval()
 
-**2**. **The next step prepares to quantize the model.**
+**2**. **Choose your backend.**
 
 .. code-block:: python
 
-    model = prepare_by_platform(model, BackendType.Tensorrt)      #! line 1. trace model and add quant nodes for model on Tensorrt Backend
-    enable_calibration(model)                                     #! line 2. turn on calibration, ready for gathering data
+    # backend options
+    backend = BackendType.Tensorrt
+    # backend = BackendType.SNPE
+    # backend = BackendType.PPLW8A16
+    # backend = BackendType.NNIE
+    # backend = BackendType.Vitis
+    # backend = BackendType.ONNX_QNN
+    # backend = BackendType.PPLCUDA
+    # backend = BackendType.OPENVINO
+    # backend = BackendType.Tengine_u8
+    # backend = BackendType.Tensorrt_NLP
+
+**3**. **Prepares to quantize the model.**
+
+.. code-block:: python
+
+    model = prepare_by_platform(model, backend)                   #! line 1. trace model and add quant nodes for model on Tensorrt Backend
 
     # calibration loop
+    enable_calibration(model)                                     #! line 2. turn on calibration, ready for gathering data
     for i, batch in enumerate(data):
         # do forward procedures
         ...
@@ -46,9 +62,13 @@ Before starting, you should install MQBench first. Now we start the tour.
         # do forward procedures
         ...
 
+**4**. **Export quantized model.**
+
+.. code-block:: python
+
     # define dummy data for model export.
     input_shape={'data': [10, 3, 224, 224]}
-    convert_deploy(model, BackendType.Tensorrt, input_shape)      #! line 4. remove quant nodes, ready for deploying to real-world hardware
+    convert_deploy(model, backend, input_shape)                   #! line 4. remove quant nodes, ready for deploying to real-world hardware
 
 If you want to know more about deploying to a customize backend, check :doc:`../user_guide/internal/learn_config` and :doc:`../user_guide/howtodeploy`
 

docs/source/get_started/setup.rst

@@ -1,7 +1,9 @@
-Installation
-============
+Setup MQBench
+=============
 
-MQBench only depend on PyTorch 1.8.1，following `pytorch.org <http://pytorch.org/>`_ or use requirements file to install.
+**1**. **Install**
+
+MQBench only depend on PyTorch 1.8.1, following `pytorch.org <http://pytorch.org/>`_ or use requirements file to install.
 
 .. code-block:: shell
     :linenos:
@@ -10,4 +12,22 @@ MQBench only depend on PyTorch 1.8.1，following `pytorch.org <http://pytorch.or
     pip install -r requirements.txt  # install MQBench dependencies
     python setup.py install          # install MQBench
 
-You have done installation of MQBench, check :doc:`quick_start_academic` or check :doc:`quick_start_deploy` to get started with MQBench.
+**2**. **Validate**
+
+Validate by executing following with no errors.
+
+.. code-block:: shell
+    :linenos:
+
+    python -c 'import mqbench'
+
+**3**. **Uninstall**
+
+Remove MQBench by executing following shell script.
+
+.. code-block:: shell
+    :linenos:
+
+    pip uninstall mqbench
+
+You have done setup of MQBench, check :doc:`quick_start_deploy` or check :doc:`quick_start_academic` to get started with MQBench.

docs/source/get_started/support_matrix.rst

@@ -1,16 +1,21 @@
 Support Matrix
 ==============
 
-**Now we support this hardwares and algorithms, updating.**
+**Now we support following hardwares and algorithms, still in updating.**
 
-Hardware & Backend
-^^^^^^^^^^^^^^^^^^
+Hardware/Backend
+^^^^^^^^^^^^^^^^
 
 1. :doc:`../user_guide/deploy/tensorrt`
 2. :doc:`../user_guide/deploy/snpe`
 3. NNIE
 4. Vitis
 5. ONNX_QNN
+6. PPLW8A16
+7. PPLCUDA
+8. OPENVINO
+9. Tengine_u8
+10. Tensorrt_NLP
 
 Algorithm
 ^^^^^^^^^

docs/source/user_guide/PTQ/advanced.rst

@@ -1,14 +1,11 @@
 Advanced PTQ
 ============
 
-Code Snippets
-^^^^^^^^^^^^^
-
-You can follow this snippet to start your mission with MQBench! You can find config details in `here <https://github.com/ModelTC/MQBench/tree/main/application/imagenet_example/PTQ/configs>`_, and you can find algorithm details in :doc:`../algorithm/advanced_ptq`.
+MQBench provides a simple API for advanced PTQ, learn our step-by-step instructions to quantize your model.
 
+**1**. **Prepare FP32 model firstly.**
 
 .. code-block:: python
-    :linenos:
 
     import torchvision.models as models
     from mqbench.convert_deploy import convert_deploy
@@ -18,54 +15,71 @@ You can follow this snippet to start your mission with MQBench! You can find con
 
     # first, initialize the FP32 model with pretrained parameters.
     model = models.__dict__["resnet18"](pretrained=True)
+    model.eval()
 
-    # then, we will trace the original model using torch.fx and \
-    # insert fake quantize nodes according to different hardware backends (e.g. TensorRT).
-    model = prepare_by_platform(model, BackendType.Tensorrt)
+**2**. **Configure advanced ptq and backend.**
 
-    # before training, we recommend to enable observers for calibration in several batches, and then enable quantization.
-    model.eval()
+.. code-block:: python
+
+    # configuration
+    ptq_reconstruction_config = {
+        'pattern': 'block',                   #? 'layer' for Adaround or 'block' for BRECQ
+        'scale_lr': 4.0e-5,                   #? learning rate for learning step size of activation
+        'warm_up': 0.2,                       #? 0.2 * max_count iters without regularization to floor or ceil
+        'weight': 0.01,                       #? loss weight for regularization item
+        'max_count': 20000,                   #? optimization iteration
+        'b_range': [20,2],                    #? beta decaying range
+        'keep_gpu': True,                     #? calibration data restore in gpu or cpu
+        'round_mode': 'learned_hard_sigmoid', #? ways to reconstruct the weight, currently only support learned_hard_sigmoid
+        'prob': 0.5,                          #? dropping probability of QDROP
+    }
+
+    # backend options
+    backend = BackendType.Tensorrt
+    # backend = BackendType.SNPE
+    # backend = BackendType.PPLW8A16
+    # backend = BackendType.NNIE
+    # backend = BackendType.Vitis
+    # backend = BackendType.ONNX_QNN
+    # backend = BackendType.PPLCUDA
+    # backend = BackendType.OPENVINO
+    # backend = BackendType.Tengine_u8
+    # backend = BackendType.Tensorrt_NLP
+
+**3**. **Prepare to quantize the model.**
+
+.. code-block:: python
+
+    # trace model and add quant nodes for model on backend
+    model = prepare_by_platform(model, backend)
+
+    # calibration loop
     enable_calibration(model)
-    calibration_flag = True
-
-    # set config
-    config_dict = {
-        pattern: 'block',
-        warm_up: 0.2,
-        weight: 0.01,
-        max_count: 10000,
-        b_range: [20, 2],
-        keep_gpu: True,
-        round_mode: learned_hard_sigmoid,
-        prob: 1.0
-        }
+    for i, batch in enumerate(data):
+        # do forward procedures
+        ...
 
     # ptq_reconstruction loop
     stacked_tensor = []
-    # add calibration data to stack
+    # add ptq_reconstruction data to stack
     for i, batch_data in enumerate(data):
         if i == cali_batchsize:
             break
         stacked_tensor.append(batch_data)
-    # start calibration
-    enable_quantization(model)
-    model = ptq_reconstruction(model, stacked_tensor, config_dict)
+    # start ptq_reconstruction
+    model = ptq_reconstruction(model, stacked_tensor, ptq_reconstruction_config)
 
-    # do evaluation
-    ...
+    # evaluation loop
+    for i, batch in enumerate(data):
+        # do forward procedures
+        ...
 
-    # deploy model, remove fake quantize nodes, and dump quantization params like clip ranges.
-    convert_deploy(model.eval(), BackendType.Tensorrt, input_shape_dict={'data': [10, 3, 224, 224]})
+**4**. **Export quantized model.**
 
-MQBench examples
-^^^^^^^^^^^^^^^^^
+.. code-block:: python
 
-We follow the `PyTorch official example <https://github.com/pytorch/examples/tree/master/imagenet/>`_ to build the example of Model Quantization Benchmark for ImageNet classification task, you can run advanced ptq easily.
+    # deploy model, remove fake quantize nodes, and dump quantization params like clip ranges.
+    input_shape={'data': [10, 3, 224, 224]}
+    convert_deploy(model, backend, input_shape)
 
-1. Clone and install MQBench;
-2. Prepare the ImageNet dataset from `the official website <http://www.image-net.org/>`_ and move validation images to labeled subfolders, using the following `shell script <https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh>`_;
-3. Download pre-trained models from our `release <https://github.com/ModelTC/MQBench/releases/tag/pre-trained>`_;
-4. Check out `/path-of-MQBench/application/imagenet_example/PTQ/configs` and find yaml file you want to reproduce;
-5. Replace `/path-of-pretained` and `/path-of-imagenet` in yaml file;
-6. Change directory, `cd /path-of-MQBench/application/imagenet_example/PTQ/ptq`;
-7. Exec `python ptq.py -\-config /path-of-config.yaml`.
+you can find algorithm details in :doc:`../algorithm/advanced_ptq`. We also provides an example in `here <https://github.com/ModelTC/MQBench/tree/main/application/imagenet_example/PTQ/>`_.