Validate the generated number of input.pb and output.pb and new node …

…files in CIs (onnx#4514)

* validate the generated number of input.pb and output.pb in CIs

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* --diff-filter=ADR

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* add --clean for generate-data

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* add --clean in CIs

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* remove unreproducible tests

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* add sentences in AddNewOp.md

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* check original directory number

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* typo

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* nit: warning msg

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

.azure-pipelines/Linux-CI.yml

@@ -119,7 +119,7 @@ jobs:
 
   - script: |
       source venv/bin/activate
-      python onnx/backend/test/cmd_tools.py generate-data
+      python onnx/backend/test/cmd_tools.py generate-data --clean
       git status
       # Skip *output_*.pb because NumPy functions might behave differently on different platforms
       # Skip test_log's input.pb because it uses np.random, which might behave differently on different platforms
@@ -128,6 +128,11 @@ jobs:
         echo "git diff for test generation returned failures. Please check updated node test files"
         exit 1
       fi
+      git diff --exit-code --diff-filter=ADR -- . ':!onnx/onnx-data.proto' ':!onnx/onnx-data.proto3'
+      if [ $? -ne 0 ]; then
+        echo "Test generation returned failures. Please check the number of node test files (input_*.pb or output_*.pb)"
+        exit 1
+      fi
 
     displayName: Test backend test data
 

.azure-pipelines/MacOS-CI.yml

@@ -96,12 +96,17 @@ jobs:
         exit 1
       fi
 
-      python onnx/backend/test/cmd_tools.py generate-data
+      python onnx/backend/test/cmd_tools.py generate-data --clean
       git status
       git diff --exit-code -- . ':!onnx/onnx-data.proto' ':!onnx/onnx-data.proto3' ':!*output_*.pb' ':!*input_*.pb'
       if [ $? -ne 0 ]; then
         echo "git diff for test generation returned failures. Please check updated node test files"
         exit 1
       fi
+      git diff --exit-code --diff-filter=ADR -- . ':!onnx/onnx-data.proto' ':!onnx/onnx-data.proto3'
+      if [ $? -ne 0 ]; then
+        echo "Test generation returned failures. Please check the number of node test files (input_*.pb or output_*.pb)"
+        exit 1
+      fi
 
     displayName: 'Run ONNX Tests'

.azure-pipelines/Windows-CI.yml

@@ -65,19 +65,24 @@ jobs:
       python onnx/gen_proto.py -l
       python onnx/gen_proto.py -l --ml
       
-      git diff --exit-code  -- . :(exclude)onnx/onnx-data.proto :(exclude)onnx/onnx-data.proto3
+      git diff --exit-code -- . :(exclude)onnx/onnx-data.proto :(exclude)onnx/onnx-data.proto3
       IF NOT %ERRORLEVEL% EQU 0 (
         @echo "git diff returned failures"
         EXIT 1
       )
 
-      python onnx/backend/test/cmd_tools.py generate-data
+      python onnx/backend/test/cmd_tools.py generate-data --clean
       git status
-      git diff --exit-code  -- . :!onnx/onnx-data.proto :!onnx/onnx-data.proto3 :!*output_*.pb :!*input_*.pb
+      git diff --exit-code -- . :!onnx/onnx-data.proto :!onnx/onnx-data.proto3 :!*output_*.pb :!*input_*.pb
       IF NOT %ERRORLEVEL% EQU 0 (
         @echo "git diff for test generation returned failures. Please check updated node test files"
         EXIT 1
       )
+      git diff --exit-code --diff-filter=ADR -- . :!onnx/onnx-data.proto :!onnx/onnx-data.proto3
+      IF NOT %ERRORLEVEL% EQU 0 (
+        @echo "Test generation returned failures. Please check the number of node test files (input_*.pb or output_*.pb)."
+        EXIT 1
+      )
 
       rm -rf .setuptools-cmake-build
 

docs/AddNewOp.md

@@ -56,7 +56,7 @@ Once the criteria of proposing new operator/function has been satisfied, you wil
     3. Example: [onnx/backend/test/case/node/abs.py](/onnx/backend/test/case/node/abs.py)
 5. Add at least one automatic upgrade test for your operator in [onnx/test/automatic_upgrade_test.py](/onnx/test/automatic_upgrade_test.py) using `_test_op_upgrade`. These tests create a given operator at a given opset version (usually the version the operator was introduced in) and test that the version converter is able to convert them to the highest available version. So for a new operator `_test_op_upgrade` will not test anything, but as soon as the operator gets updated in a future opset the test will automatically become nontrivial.
 6. Update the documentation and generate the test data.
-    1. Running [the script](/tools/update_doc.sh)
+    1. Running [the script](/tools/update_doc.sh). If you have files under `onnx/backend/test/data/node` which cannot be generated by the scripts from `onnx/backend/test/case/node`, please further use `python onnx/backend/test/cmd_tools.py generate-data --clean` to cleanup the directory and only preserve needed test data.
 to update the doc and generate the test data.
 7. Shape Inference function
     1. Please provide a shape inference function in cases where it is meaningful and applicable.

onnx/backend/test/cmd_tools.py

@@ -4,6 +4,7 @@
 import json
 import os
 import shutil
+import warnings
 
 import onnx.backend.test.case.model as model_test
 import onnx.backend.test.case.node as node_test
@@ -19,14 +20,27 @@ def prepare_dir(path: str) -> None:
             shutil.rmtree(path)
         os.makedirs(path)
 
+    # Clean the output directory before generating data for node testcases
+    # It is used to check new generated data is correct in CIs
+    node_root = os.path.join(args.output, "node")
+    original_dir_number = len(
+        [name for name in os.listdir(node_root) if os.path.isfile(name)]
+    )
+    if args.clean and os.path.exists(node_root):
+        shutil.rmtree(node_root)
+        os.makedirs(node_root)
+
     cases = model_test.collect_testcases()
     # If op_type is specified, only include those testcases including the given operator
     # Otherwise, include all of the testcases
     cases += node_test.collect_testcases(args.op_type)
+    node_number = 0
 
     for case in cases:
         output_dir = os.path.join(args.output, case.kind, case.name)
         prepare_dir(output_dir)
+        if case.kind == "node":
+            node_number += 1
         if case.kind == "real":
             with open(os.path.join(output_dir, "data.json"), "w") as fi:
                 json.dump(
@@ -105,14 +119,30 @@ def prepare_dir(path: str) -> None:
                                     output, case.model.graph.output[j].name
                                 ).SerializeToString()
                             )
+    if not args.clean and node_number != original_dir_number:
+        warnings.warn(
+            "There are some models under 'onnx/backend/test/data/node' which cannot not"
+            + " be generated by the script from 'onnx/backend/test/case/node'. Please add"
+            + " '--clean' option for 'python onnx/backend/test/cmd_tools.py generate-data'"
+            + " to cleanup the existing directories and regenerate them.",
+            Warning,
+            stacklevel=2,
+        )
 
 
 def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser("backend-test-tools")
     subparsers = parser.add_subparsers()
 
     subparser = subparsers.add_parser(
-        "generate-data", help="convert testcases to test data"
+        "generate-data", help="convert testcases to test data."
+    )
+    subparser.add_argument(
+        "-c",
+        "--clean",
+        default=False,
+        action="store_true",
+        help="Clean the output directory before generating data for node testcases.",
     )
     subparser.add_argument(
         "-o",