Architecture

Benchmark Achitecture

The software consists of the several components.

1. ConfigMaker is a graphical application for generating configuration files for various components of the DLI benchmark. The application is self-contained and does not depend on the rest of the software components.
2. Deployment is a component that provides automatic deployment of infrastructure to the computational nodes using Docker technology. Information about the computational nodes is contained in the configuration file.
3. Benchmark is a component responsible for collecting performance metrics for the inference of a set of models using various deep learning frameworks. Information about models and parameters for executing inference is contained in the component’s configuration file.
4. Inference is a component that implements the inference of deep neural networks using various frameworks. It is used by the BenchmarkApp component to infer neural networks with the specified parameters.
5. AccuracyChecker is a component that provides an assessment of models performance on the public datasets. It is an add-on over a similar component of the OpenVINO toolkit.
6. RemoteController is a component that executes experiments remotely to determine inference performance and accuracy of deep models on the computational nodes.
7. Converters is an auxiliary component that contains various converters for convenient representation of performance and accuracy results. In particular, this component provides conversion of the output data to HTML-format for publishing the experimental results on the project web-page.

The main scenario involves the following actions.

1. Generating configuration files using ConfigMaker for the Deployment, RemoteController, Benchmark and AccuracyChecker components.
2. Deploying test infrastructure to the computational nodes using the Deployment component. All necessary data for experiments is stored on the FTP-server. At this stage, the directory structure on the FTP-server is prepared, the template docker image of the computer is copied to the FTP-server, the docker image is remotely downloaded from the FTP-server to the computational nodes and the docker image is deployed locally, and the experiments configurations are copied per node.
3. Launching experiments remotely on the computational nodes using RemoteController. RemoteController launches AccuracyChecker, where the accuracy of the original models is determined first, and then the accuracy of the models that are converted to the intermediate representation of the OpenVINO toolkit with FP32, FP16 and INT8 weights formats. Next, RemoteController launches Benchmark to determine inference performance (test order is the same as for AccuracyChecker). Benchmark uses the Inference component to infer deep model on the computational node, each model is inferred using the OpenVINO toolkit or another supported inference framework. After all calculations, RemoteController generates a file containing the results of experiments on the FTP-server.
4. Converting the results table to HTML- or XLSX-format using Converters. This component converts tables obtained by the Benchmark and AccuracyChecker components into HTML- or XLSX-format for further publication on the official web-page.