Testing adlik performance (#79)

Closes #79
Signed-off-by: zhangkaili <zhang.kaili@zte.com.cn>

azure-pipelines.yml

@@ -19,3 +19,4 @@ jobs:
   - template: ci/azure-pipelines/jobs/copyright.yml
   - template: ci/azure-pipelines/jobs/flake8.yml
   - template: ci/azure-pipelines/jobs/tox-model-compiler.yml
+  - template: ci/azure-pipelines/jobs/tox-benchmark.yml

benchmark/.flake8

@@ -0,0 +1,2 @@
+[flake8]
+max-line-length = 120

benchmark/.pylintrc

@@ -0,0 +1,21 @@
+[MASTER]
+jobs=0
+
+[MESSAGES CONTROL]
+disable = fixme,
+          no-else-return,
+          too-many-arguments,
+          too-few-public-methods,
+          too-many-locals,
+          too-many-instance-attributes,
+          no-member,
+          unnecessary-pass
+
+[FORMAT]
+max-line-length = 120
+
+[BASIC]
+good-names = i,
+             j,
+             k,
+             o

benchmark/README.md

@@ -0,0 +1,51 @@
+# About the benchmark
+
+The benchmark is used to test the adlik serving performance of different models.
+
+## Test the runtime performance
+
+The parameters of the automatic test framework are as follows:
+
+|abbreviation|   detail           |  type |                         help                  |default|
+|----------- |--------------------|------ |---------------------------------------------- |-------|
+|-d          |--docker-file-path  | str   | The docker file path of the test serving type |       |
+|-s          |--serving-type      | str   | The test serving type                         |       |
+|-b          |--build-directory   | str   | The directory which to build the docker       |       |
+|-a          |--adlik-directory   | str   | The adlik directory                           |Adlik  |
+|-m          |--model-name        | str   | The path of model used for test               |       |
+|-c          |--client-script     | str   | The script used to infer                      |client_script.sh|
+|-ss         |--serving-script    | str   | The serving script                            |serving_script.sh|
+|-ov         |--openvino-version  | str   | The version of the OpenVINO                   |2019.3.344|
+|-tt         |--tensorrt-tar      | str   | The tar version of the TensorRT               |TensorRT-7.0.0.11.Ubuntu<br>-18.04.x86_64-gnu.cuda-<br>10.0.cudnn7.6.tar.gz|
+|-tv         |--tensorrt-version  | str   | The version of TensorRT                       |7.0.0.11|
+|-l          |--log-path          | str   | The path of log directory                     |log     |
+|-tm         |--test-model-path   | str   | The path of test model                        |        |
+|-sj         |--serving-json      | str   | The json of model                             |serving_model.json|
+|-cis        |--client-inference-script|str|The inference script                          |        |  
+|-i          |--image-filename    | str   | Input image                                   |        |
+|-gl         |--gpu-label         | int   | The GPU label                                 | None   |
+|-cs         |--compile-script    | str   | Compile the model script                      |compile_script.sh|
+
+If you want to use the automatic_test.py test the runtime, you need to follow the steps below:
+
+1. Download Adlik code.
+2. Install docker.
+3. Prepare the serving_model.json (required for compile model) trained model, the format of the model file can be:.pb, .h5, .ckpt,
+.onnx, savedModel, and it is recommended to put the model and serving_model.json under the Adlik/benchmark/test/test_model directory.
+4. The writing of serving_model.json can refer to the serving_model.json of each model in Adlki/benchmark/tests/test_model and Adlik/model_compiler/src/model_compiler/config_schema.json.
+5. If there is no required inference code in the Adlik/benchmark/test/client directory of the benchmark, you need to write the inference code
+6. Specify the type of test runtime and version number (if needed, eg: OpenVINO and TensorRT). 
+7. Explicitly test whether a GPU is required.
+8. The environment running the code has python3.7 or above installed.
+9. According to the runtime type of the test, select the dockerfile, serving script and compile script required by the test under the Adlik/benchmark/test directory
+10. Configure parameters for testing, for example, run the follow command in the Adlik directory:
+
+```sh
+python3 benchmark/src/automatic_test.py -d benchmark/test/docker_test/openvino.Dockerfile -s openvino -b . -a . -m mnist -c benchmark/test/client_script/client_script.sh -ss benchmark/test/serving_script/openvino_serving_script.sh -l abspath(log) -tm benchmark/test/test_model/mnist_keras -cis mnist_client.py -i mnist.png -cs benchmark/test/compile_script/compile_script.sh
+```
+
+## NOTE
+1. If the test tensorrt is running, you need to register and download the dependency package from [TensorRT](https://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html). The downloaded dependency package is recommended to be placed in the Adlik directory.
+2. If your local environment has no way to connect to the external network, you need to configure the apt source and pip source that can be used, and add the configuration command to the Dockerfile
+3. When bazel build, if you can't pull the package, you can also download the required packages in advance, and use the --distdir command.
+4. To prevent the computer from occupying too many cores when bazel build, causing jams. During bazel build, you can also use --jobs to set concurrent jobs.

benchmark/bandit.yaml

@@ -0,0 +1,4 @@
+include:
+  - '*.py'
+
+skips: [B404,B603,B101,B110]

benchmark/setup.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+
+# Copyright 2019 ZTE corporation. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+Benchmark test.
+"""
+
+from setuptools import find_packages, setup
+
+_VERSION = '0.0.0'
+
+_REQUIRED_PACKAGES = [
+    'keras==2.2.4',
+    'onnx==1.5.0',
+    'protobuf==3.6.1',
+    'torch==1.3.0',
+    'torchvision==0.4.0',
+    'requests',
+    'tensorflow==1.14.0',
+    'jsonschema==3.1.1',
+    'networkx==2.3',
+    'defusedxml==0.5.0'
+]
+
+_TEST_REQUIRES = [
+    'bandit==1.6.0',
+    'flake8==3.7.7',
+    'pylint==2.3.1',
+    'pytest-cov',
+    'pytest-xdist'
+]
+
+setup(
+    name="benchmark",
+    version=_VERSION.replace('-', ''),
+    author='ZTE',
+    author_email='ai@zte.com.cn',
+    packages=find_packages('src'),
+    package_dir={'': 'src'},
+    description=__doc__,
+    license='Apache 2.0',
+    keywords='Test serving-lite performance',
+    install_requires=_REQUIRED_PACKAGES,
+    extras_require={'test': _TEST_REQUIRES}
+
+)

benchmark/src/automatic_test.py

@@ -0,0 +1,93 @@
+"""
+Automated test runtime performance.
+"""
+
+import subprocess
+import argparse
+import os
+
+
+def _parse_arguments():
+    args_parser = argparse.ArgumentParser()
+    args_parser.add_argument("-d", "--docker-file-path", type=str, required=True,
+                             help="The docker file path of the test serving type")
+    args_parser.add_argument("-s", "--serving-type", type=str, required=True, help="The test serving type",
+                             choices=("openvino", "tensorrt", "tensorflow", "tensorflow_gpu"))
+    args_parser.add_argument("-b", "--build-directory", type=str, required=True,
+                             help="The directory which to build the docker")
+    args_parser.add_argument("-a", "--adlik-directory", type=str, default="Adlik", help="The adlik directory")
+    args_parser.add_argument("-m", "--model-name", type=str, required=True, help="The path of model used for test")
+    args_parser.add_argument("-c", "--client-script", type=str, default="client_script.sh",
+                             help="The script used to infer")
+    args_parser.add_argument("-ss", "--serving-script", type=str, default="serving_script.sh",
+                             help="The serving script")
+    args_parser.add_argument("-ov", "--openvino-version", type=str, default="2019.3.344",
+                             help="The version of the OpenVINO")
+    args_parser.add_argument("-tt", "--tensorrt-tar", type=str,
+                             default="TensorRT-7.0.0.11.Ubuntu-18.04.x86_64-gnu.cuda-10.0.cudnn7.6.tar.gz",
+                             help="The tar version of the TensorRT")
+    args_parser.add_argument("-tv", "--tensorrt-version", type=str, default="7.0.0.11", help="The version of TensorRT")
+    args_parser.add_argument("-l", "--log-path", type=str, default="log", help="The path of log directory")
+    args_parser.add_argument('-tm', '--test-model-path', type=str, required=True, help="The path of test model")
+    args_parser.add_argument("-sj", "--serving-json", type=str, default="serving_model.json", help="The json of model")
+    args_parser.add_argument("-cis", "--client-inference-script", type=str, required=True, help="The inference script")
+    args_parser.add_argument("-i", "--image-filename", type=str, required=True, nargs="?", help="Input image.")
+    args_parser.add_argument("-gl", "--gpu-label", type=int, default=None, help="The GPU label")
+    args_parser.add_argument("-cs", "--compile-script", type=str, default="compile_script.sh",
+                             help="Compile the model script")
+    return args_parser.parse_args()
+
+
+def _get_result(log_path, model_name):
+    calculate_command = ['python3', os.path.join(os.path.dirname(__file__), 'test_result.py'),
+                         '-c', os.path.join(log_path, 'client_time.log'),
+                         '-s', os.path.join(log_path, 'serving_time.log'),
+                         '-m', model_name]
+    with subprocess.Popen(calculate_command) as result_process:
+        print(result_process.stdout)
+
+
+def _docker_build_command(args):
+    build_arg = f'--build-arg SERVING_SCRIPT={args.serving_script} ' \
+                f'--build-arg CLIENT_SCRIPT={args.client_script} ' \
+                f'--build-arg TEST_MODEL_PATH={args.test_model_path} ' \
+                f'--build-arg SERVING_JSON={args.serving_json} ' \
+                f'--build-arg CLIENT_INFERENCE_SCRIPT={args.client_inference_script} ' \
+                f'--build-arg IMAGE_FILENAME={args.image_filename} ' \
+                f'--build-arg COMPILE_SCRIPT={args.compile_script} '
+
+    if args.serving_type == 'openvino':
+        build_arg = build_arg + f'--build-arg OPENVINO_VERSION={args.openvino_version} '
+    elif args.serving_type == 'tensorrt':
+        build_arg = build_arg + f'--build-arg TENSORRT_VERSION={args.tensorrt_version} ' \
+                                f'--build-arg TENSORRT_TAR={args.tensorrt_tar} '
+    else:
+        build_arg = build_arg
+
+    build_command = f'docker build --build-arg ADLIK_DIRECTORY={args.adlik_directory} ' + build_arg + \
+                    f' -f {args.docker_file_path} -t adlik-test:{args.serving_type} {args.build_directory}'
+    return build_command
+
+
+def main(args):
+    """
+    Automated test runtime performance.
+    """
+
+    docker_build_command = _docker_build_command(args)
+
+    if not args.gpu_label:
+        docker_run_command = f'docker run --rm -v {args.log_path}:/home/john/log ' \
+                             f'adlik-test:{args.serving_type}'
+    else:
+        docker_run_command = f'NV_GPU={args.gpu_label} nvidia-docker run --rm ' \
+                             f'-v {args.log_path}:/home/john/log adlik-test:{args.serving_type}'
+
+    test_command = ['sh', '-c', docker_build_command + ' && ' + docker_run_command]
+
+    subprocess.run(test_command)
+    _get_result(args.log_path, args.model_name)
+
+
+if __name__ == '__main__':
+    main(_parse_arguments())

benchmark/src/cmd_script.py

@@ -0,0 +1,28 @@
+"""
+The CMD script
+"""
+
+import subprocess
+import argparse
+
+
+def _main(args):
+    compile_command = ['sh', '-c', args.compile_script]
+    serving_command = ['sh', '-c', args.serving_script]
+    client_command = ['sh', '-c', args.client_script]
+    subprocess.run(compile_command)
+    with subprocess.Popen(serving_command) as process:
+        subprocess.run(client_command)
+        process.kill()
+
+
+if __name__ == '__main__':
+    ARGS_PARSER = argparse.ArgumentParser()
+    ARGS_PARSER.add_argument('-s', '--serving-script', type=str, required=True,
+                             help='The serving script')
+    ARGS_PARSER.add_argument('-c', '--client-script', type=str, required=True,
+                             help='The client script')
+    ARGS_PARSER.add_argument('-cs', '--compile-script', type=str, required=True,
+                             help='The compile script')
+    PARSE_ARGS = ARGS_PARSER.parse_args()
+    _main(PARSE_ARGS)

benchmark/src/compile_model.py

@@ -0,0 +1,42 @@
+"""
+Compile the model.
+"""
+
+import os
+import json
+import argparse
+import model_compiler  # pylint:disable=import-error
+
+
+def _get_request(request_file, test_model_dir):
+    request = json.load(request_file)
+    model_dir = request["input_model"]
+    request["input_model"] = os.path.join(test_model_dir, model_dir)
+    export_dir = request["export_path"]
+    request["export_path"] = os.path.join(test_model_dir, export_dir)
+    return request
+
+
+def compile_model(args):
+    """
+    Compile the model.
+    """
+
+    request_dir = os.path.join(args.test_model_path, args.serving_model_json)
+    try:
+        with open(request_dir, 'r') as request_file:
+            test_model_dir = args.test_model_path
+            request = _get_request(request_file, test_model_dir)
+            result = model_compiler.compile_model(request)
+            print(result)
+    except FileNotFoundError:
+        print(f"Can not compile the model in {os.path.join(test_model_dir, args.model_path)}")
+
+
+if __name__ == '__main__':
+    ARGS_PARSER = argparse.ArgumentParser()
+    ARGS_PARSER.add_argument('-t', '--test-model-path', type=str, required=True, help='The path of test model')
+    ARGS_PARSER.add_argument('-s', '--serving-model-json', type=str, default='serving_model.json',
+                             help='The json of model')
+    PARSE_ARGS = ARGS_PARSER.parse_args()
+    compile_model(PARSE_ARGS)

benchmark/src/test_result.py

@@ -0,0 +1,72 @@
+"""
+The test result of adlik performance
+"""
+import argparse
+
+
+def _speed_of_client(client_log_path, batch_size):
+    with open(client_log_path, 'r') as file:
+        lines = file.readlines()
+        sum_time = []
+        for line in lines:
+            line = line.strip('\n')
+            time = line.split('predict:')[-1]
+            time = float(time.strip(' '))
+            sum_time.append(time)
+        sum_time.pop(0)
+        batch_num = len(sum_time)
+        speed_processing_picture = (batch_num * batch_size) / sum(sum_time)
+    return speed_processing_picture, batch_num
+
+
+def _speed_of_serving(serving_log_path, batch_size):
+    with open(serving_log_path, 'r') as file:
+        lines = file.readlines()
+        runtime = lines[0].partition('found runtime ')[-1]
+        lines = [line.partition('PredictServiceImpl')[-1] for line in lines]
+        sum_time = []
+        for line in lines:
+            if line:
+                line = line.strip('\n')
+                time = line.partition('time (milliseconds):')[-1]
+                time = float(time.strip(' '))
+                sum_time.append(time)
+        sum_time.pop(0)
+        batch_num = len(sum_time)
+        speed_processing_picture = (batch_num * batch_size) / sum(sum_time) * 1000
+    return speed_processing_picture, batch_num, runtime
+
+
+def main(args):
+    """
+    Analyze inference results
+    """
+    speed_processing_picture_client, batch_num = _speed_of_client(args.client_log_path, args.batch_size)
+    speed_processing_picture_serving, batch_num1, serving_runtime = _speed_of_serving(args.serving_log_path,
+                                                                                      args.batch_size)
+    assert batch_num == batch_num1
+    if args.runtime:
+        serving_runtime = args.runtime
+    else:
+        serving_runtime = serving_runtime
+    tail_latency = 1 / speed_processing_picture_client - 1 / speed_processing_picture_serving
+    print(f'Model: {args.model_name}, Runtime: {serving_runtime}')
+    print(f'The speed of processing picture in the client is : {speed_processing_picture_client}')
+    print(f'The speed of processing picture in the serving is : {speed_processing_picture_serving}')
+    print(f'The tail latency of one picture is : {tail_latency}')
+
+
+if __name__ == '__main__':
+    ARGS_PARSER = argparse.ArgumentParser()
+    ARGS_PARSER.add_argument('-c', '--client-log-path', type=str, required=True,
+                             help='The path of client log')
+    ARGS_PARSER.add_argument('-s', '--serving-log-path', type=str, required=True,
+                             help='The path of serving log')
+    ARGS_PARSER.add_argument('-b', '--batch-size', type=int, required=False, default=128,
+                             help='Batch size. Default is 128.')
+    ARGS_PARSER.add_argument('-m', '--model-name', type=str, required=True,
+                             help='The name of model')
+    ARGS_PARSER.add_argument('-r', '--runtime', type=str, required=False, default=None,
+                             help='The serving type')
+    PARSE_ARGS = ARGS_PARSER.parse_args()
+    main(PARSE_ARGS)