Feature: support nn-Meter builder (#43)

microsoft · Mar 8, 2022 · 76d7f88 · 76d7f88
1 parent dd74162
commit 76d7f88
Show file tree

Hide file tree

Showing 140 changed files with 10,266 additions and 292 deletions.
diff --git a/.gitattributes b/.gitattributes
diff --git a/.github/workflows/integration-test.yml b/.github/workflows/integration-test.yml
@@ -28,7 +28,7 @@ jobs:
         path: |
           ~/.nn_meter
           /home/runner/work/nn-Meter/data/testmodels
-        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test.py')}}
+        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test/test_latency_predictor.py')}}
 
     - name: Install dependencies
       run: |
@@ -42,11 +42,11 @@ jobs:
       run: pip install -U .
 
     - name: Integration test
-      run: python tests/integration_test.py
+      run: python tests/integration_test/test_latency_predictor.py
 
     - name: Diff result with reference
-      run: diff tests/data/reference_result.txt tests/test_result.txt
+      run: diff tests/integration_test/data/reference_result.txt tests/integration_test/test_result.txt
 
     - name: clean env
-      run: rm tests/test_result.txt  
+      run: rm tests/integration_test/test_result.txt  
 
diff --git a/.github/workflows/integration-test_nni_based_torch.yml b/.github/workflows/integration-test_nni_based_torch.yml
@@ -28,7 +28,7 @@ jobs:
         path: |
           ~/.nn_meter
           /home/runner/work/nn-Meter/data/testmodels
-        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test.py')}}
+        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test/test_latency_predictor.py')}}
 
     - name: Install dependencies
       run: |
@@ -40,10 +40,10 @@ jobs:
       run: pip install -U .
 
     - name: Integration test
-      run: python tests/integration_test_torch.py --apply-nni
+      run: python tests/integration_test/test_latency_predictor_torch.py --apply-nni
 
     - name: Diff result with reference
-      run: diff tests/data/reference_result_nni_based_torch.txt tests/test_result_nni_based_torch.txt
+      run: diff tests/integration_test/data/reference_result_nni_based_torch.txt tests/integration_test/test_result_nni_based_torch.txt
 
     - name: clean env
-      run: rm tests/test_result_nni_based_torch.txt
+      run: rm tests/integration_test/test_result_nni_based_torch.txt
diff --git a/.github/workflows/integration-test_onnx_based_torch.yml b/.github/workflows/integration-test_onnx_based_torch.yml
@@ -28,7 +28,7 @@ jobs:
         path: |
           ~/.nn_meter
           /home/runner/work/nn-Meter/data/testmodels
-        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test.py')}}
+        key: ${{hashFiles('nn_meter/configs/predictors.yaml')}}-${{hashFiles('tests/integration_test/test_latency_predictor.py')}}
 
     - name: Install dependencies
       run: |
@@ -41,10 +41,10 @@ jobs:
       run: pip install -U .
 
     - name: Integration test
-      run: python tests/integration_test_torch.py --apply-onnx
+      run: python tests/integration_test/test_latency_predictor_torch.py --apply-onnx
 
     - name: Diff result with reference
-      run: diff tests/data/reference_result_onnx_based_torch.txt tests/test_result_onnx_based_torch.txt
+      run: diff tests/integration_test/data/reference_result_onnx_based_torch.txt tests/integration_test/test_result_onnx_based_torch.txt
 
     - name: clean env
-      run: rm tests/test_result_onnx_based_torch.txt
+      run: rm tests/integration_test/test_result_onnx_based_torch.txt
diff --git a/README.md b/README.md
@@ -1,5 +1,3 @@
-Note: This is an beta (preview) version which is still under refining.
-
 **nn-Meter** is a novel and efficient system to accurately predict the inference latency of DNN models on diverse edge devices. The key idea is dividing a whole model inference into kernels, i.e., the execution units of fused operators on a device, and conduct kernel-level prediction. We currently evaluate four popular platforms on a large dataset of 26k models. It achieves 99.0% (mobile CPU), 99.1% (mobile Adreno 640 GPU), 99.0% (mobile Adreno 630 GPU), and 83.4% (Intel VPU) prediction accuracy.
 
 The current supported hardware and inference frameworks:
@@ -73,7 +71,7 @@ Here is a summary of supported inputs of the two methods.
 |     Tensorflow     |             Checkpoint file dumped by `tf.saved_model()` and end with `.pb`             |                 Checkpoint file dumped by `tf.saved_model` and end with `.pb`                 |
 |       Torch       |                              Models in `torchvision.models`                              |                                   Object of `torch.nn.Module`                                   |
 |        Onnx        | Checkpoint file dumped by `torch.onnx.export()` or `onnx.save()` and end with `.onnx` |           Checkpoint file dumped by `onnx.save()` or model loaded by `onnx.load()`           |
-| nn-Meter IR graph |     Json file in the format of [nn-Meter IR Graph](./docs/input_models.md#nnmeter-ir-graph)     | `dict` object following the format of [nn-Meter IR Graph](./docs/input_models.md#nnmeter-ir-graph) |
+| nn-Meter IR graph |     Json file in the format of [nn-Meter IR Graph](./docs/predictor/input_models.md#nnmeter-ir-graph)     | `dict` object following the format of [nn-Meter IR Graph](./docs/predictor/input_models.md#nnmeter-ir-graph) |
 |    NNI IR graph    |                                              -                                              |                                        NNI IR graph object                                        |
 
 In both methods, users could appoint predictor name and version to target a specific hardware platform (device). Currently, nn-Meter supports prediction on the following four configs: