Repository navigation

Release list

v3.1 Latest

Latest

YingkunZhou released this 13 Dec 03:56

v3.1

a307250

add layer-wise operation profiling util-tools

MNN

bin/timeProfile.out .mnn/fp16/LeViT_128.mnn 100 0 1x3x224x224 1 2
bin/timeProfile.out .mnn/int8/LeViT_128.mnn 100 0 1x3x224x224 1 2
cd .mnn; ln -sf fp16/*.mnn .; cd ..
DEF="-DUSE_PERF" FP=16 BACK=x MODEL=128 make run-mnn-perf

cd .mnn; ln -sf int8/*.mnn .; cd ..
DEF="-DUSE_PERF" BACK=z MODEL=128 make run-mnn-perf

TFLite

bazel build -c opt //tensorflow/lite/tools/benchmark:benchmark_model

bin/benchmark_model \
  --graph=.tflite/tinynn-32/LeViT_128.tflite \
  --num_threads=1 \
  --warmup_runs=10 \
  --num_runs=50 \
  --enable_op_profiling=true \
  --use_xnnpack=true \
  --xnnpack_force_fp16=true

bin/benchmark_model \
  --graph=.tflite/tinynn-d8/LeViT_128.tflite \
  --num_threads=1 \
  --warmup_runs=10 \
  --num_runs=50 \
  --enable_op_profiling=true

cd .tflite; ln -sf tinynn-32/*.tflite .; cd ..
DEF="-DUSE_PERF" FP=16 BACK=x MODEL=128 make run-tflite-perf

cd .tflite; ln -sf tinynn-d8/*.tflite .; cd ..
DEF="-DUSE_PERF" BACK=z MODEL=128 make run-tflite-perf

TensorRT

/usr/src/tensorrt/bin/trtexec --batch=1 --loadEngine=.onnx/gpu-fp16/mobilevit_x_small.engine --dumpProfile --separateProfileRun | tee /tmp/fp16
python python/utils/trt-op-statistic.py fp16

/usr/src/tensorrt/bin/trtexec --batch=1 --loadEngine=.onnx/gpu-int8/mobilevit_x_small.engine --dumpProfile --separateProfileRun | tee /tmp/int8
python python/utils/trt-op-statistic.py int8

CANN

python python/cann-perf.py --format fp16 --only-test mobilevit_x_small --profile 2>/dev/null | tee /tmp/fp16
python python/utils/acl-op-statistic.py fp16

python python/cann-perf.py --format int8 --only-test mobilevit_x_small --profile 2>/dev/null | tee /tmp/int8
python python/utils/acl-op-statistic.py int8

OpenVino

export LD_LIBRARY_PATH=xxx
bin/benchmark_app -m .xml/fp16/mobilevit_x_small.xml -t 20 -pc -d NPU -report_type detailed_counters -hint latency -api sync -infer_precision f16
mv benchmark_detailed_counters_report.csv /tmp/fp16.csv; python /tmp/opv-op-statistic.py fp16

bin/benchmark_app -m .xml/int8/mobilevit_x_small.xml -t 20 -pc -d NPU -report_type detailed_counters -hint latency -api sync
mv benchmark_detailed_counters_report.csv /tmp/int8.csv; python /tmp/opv-op-statistic.py int8

the openvino benchmark_app is modified by myself with addition of my NPU config(which has better performance)

diff --git a/samples/cpp/benchmark_app/main.cpp b/samples/cpp/benchmark_app/main.cpp
index 2cfd15b77a..00dbf04752 100644
--- a/samples/cpp/benchmark_app/main.cpp
+++ b/samples/cpp/benchmark_app/main.cpp
@@ -13,6 +13,7 @@
 // clang-format off
 #include "openvino/openvino.hpp"
 #include "openvino/pass/serialize.hpp"
+#include "openvino/runtime/intel_npu/properties.hpp"
 
 #ifndef IN_OV_COMPONENT
 #    define IN_OV_COMPONENT
@@ -742,6 +743,14 @@ int main(int argc, char* argv[]) {
             // ----------------- 7. Loading the model to the device
             // --------------------------------------------------------
             next_step();
+            if (device_name == "NPU") {
+                device_config = {};
+                device_config.insert(ov::hint::performance_mode(ov::hint::PerformanceMode::LATENCY));
+                device_config.insert(ov::intel_npu::compilation_mode_params(ov::Any("optimization-level=2 performance-hint-override=latency")));
+                device_config.insert(ov::intel_npu::turbo(true));
+                device_config.insert(ov::hint::inference_precision(ov::element::f16));
+                device_config[ov::enable_profiling.name()] = FLAGS_pc;
+            }
             startTime = Time::now();
             compiledModel = core.compile_model(model, device_name, device_config);
             duration_ms = get_duration_ms_till_now(startTime);

Assets 5

v3.0

YingkunZhou released this 08 Dec 09:21

v3.0

7d6103d

add android real application

modified from tensorflow/example repo

how to build

export ANDROID_HOME=
cd android-app
# build debug apk
./gradlew assembleDebug #-Dhttp.proxyHost= -Dhttp.proxyPort=-Dhttps.proxyHost= -Dhttps.proxyPort=
# build release apk
./gradlew assembleRelease
# get my-release-key.jks by yourself
# $ANDROID_HOME/build-tools/30.0.3/apksigner sign --ks /tmp/my-release-key.jks --out app-release.apk ./app/build/outputs/apk/release/app-release-unsigned.apk

Assets 12

v2.1

YingkunZhou released this 21 Nov 03:45

v2.1

cf1969b

add x86_64 linux and windows support

tflite (2.16.1) build by myself
MNN (commit id: 630d59310c3e949560866b07d8edc9232e89ec0a) build by myself
onnxruntime v1.20.0 (official link)
ncnn (official link)
openvino (2024.4.0) build by myself with minor modificaiton

diff --git a/src/inference/src/os/cpu_map_info.hpp b/src/inference/src/os/cpu_map_info.hpp
index 497b25c3b6..2cc6cef768 100644
--- a/src/inference/src/os/cpu_map_info.hpp
+++ b/src/inference/src/os/cpu_map_info.hpp
@@ -53,6 +53,39 @@ public:
     std::map<int, int> _numaid_mapping_table;
     std::mutex _cpu_mutex;
     int _socket_idx = 0;
+
+private:
+    /**
+     * @brief      Sort proc_type_table by CPU ID on which application is running. The numa node containing this CPU ID
+     * will move to first row.
+     * @param[in]  _processor_id CPU ID on which application is running.
+     * @param[in] _proc_type_table summary table of number of processors per type
+     * @param[in] _cpu_mapping_table CPU mapping table for each processor
+     * @return
+     */
+    void sort_table_by_cpu_id(const int _processor_id,
+                              std::vector<std::vector<int>>& _proc_type_table,
+                              const std::vector<std::vector<int>>& _cpu_mapping_table) {
+        int current_numa_node = 0;
+        int current_socket = 0;
+
+        for (auto& row : _cpu_mapping_table) {
+            if (_processor_id == row[CPU_MAP_PROCESSOR_ID]) {
+                current_numa_node = row[CPU_MAP_NUMA_NODE_ID];
+                current_socket = row[CPU_MAP_SOCKET_ID];
+                break;
+            }
+        }
+        for (size_t i = 1; i < _proc_type_table.size(); i++) {
+            if ((current_numa_node == _proc_type_table[i][PROC_NUMA_NODE_ID]) &&
+                (current_socket == _proc_type_table[i][PROC_SOCKET_ID])) {
+                std::rotate(_proc_type_table.begin() + 1, _proc_type_table.begin() + i, _proc_type_table.end());
+                break;
+            }
+        }
+    };
+
+    friend class LinuxSortProcTableTests;
 };
 
 CPU& cpu_info();
diff --git a/src/inference/src/os/lin/lin_system_conf.cpp b/src/inference/src/os/lin/lin_system_conf.cpp
index 2dcf9eaa4d..37d6119688 100644
--- a/src/inference/src/os/lin/lin_system_conf.cpp
+++ b/src/inference/src/os/lin/lin_system_conf.cpp
@@ -1,3 +1,4 @@
+
 // Copyright (C) 2018-2024 Intel Corporation
 // SPDX-License-Identifier: Apache-2.0
 //
@@ -12,6 +13,7 @@
 #include <vector>
 
 #include "dev/threading/parallel_custom_arena.hpp"
+#include "dev/threading/thread_affinity.hpp"
 #include "openvino/core/except.hpp"
 #include "openvino/runtime/system_conf.hpp"
 #include "os/cpu_map_info.hpp"
@@ -114,10 +116,11 @@ CPU::CPU() {
     };
 
     auto check_valid_cpu = [&]() {
-        cpu_set_t mask;
-        CPU_ZERO(&mask);
+        ov::threading::CpuSet mask;
+        int ncpus = 0;
+        std::tie(mask, ncpus) = ov::threading::get_process_mask();
 
-        if ((_processors == 0) || (sched_getaffinity(0, sizeof(cpu_set_t), &mask) == -1)) {
+        if ((_processors == 0) || mask == nullptr) {
             return -1;
         }
 
@@ -128,7 +131,7 @@ CPU::CPU() {
 
         numa_node_list.assign(_sockets, std::vector<int>());
         for (int i = 0; i < _processors; i++) {
-            if (CPU_ISSET(i, &mask)) {
+            if (CPU_ISSET(i, mask)) {
                 valid_cpu_mapping_table.emplace_back(_cpu_mapping_table[i]);
                 if (_cpu_mapping_table[i][CPU_MAP_CORE_TYPE] == MAIN_CORE_PROC) {
                     phy_core_list.emplace_back(_cpu_mapping_table[i][CPU_MAP_CORE_ID]);
@@ -280,6 +283,11 @@ CPU::CPU() {
         OPENVINO_THROW("CPU affinity check failed. No CPU is eligible to run inference.");
     };
 
+    if (_proc_type_table.size() > 1) {
+        int cur_processor_id = sched_getcpu();
+        sort_table_by_cpu_id(cur_processor_id, _proc_type_table, _cpu_mapping_table);
+    }
+
     _org_proc_type_table = _proc_type_table;
 
     cpu_debug();
@@ -775,3 +783,4 @@ void update_valid_processor_linux(const std::vector<int> phy_core_list,
 };
 
 }  // namespace ov
+

Assets 7

v2.0

YingkunZhou released this 13 May 06:01

v2.0

b61d0fc

tvm (0.14.0): add TVM related model and runtime files (use python 3.9)
torch.compile add torch 2.4.0 aarch python whl

Assets 7

v1.1

YingkunZhou released this 13 Mar 08:33

v1.1

a3ca6c5

upload Android prebuilt DNN libs binary (shared library)

Assets 7

v1.0

YingkunZhou released this 13 Mar 07:38

v1.0

36d4077

support 5 most popular and famous DNN libs

ncnn: 1012f851396d4d6e502e1f1836b6d997b31ed89f Date: Thu Jan 18 14:08:30 2024 +0800
mnn: d20f37fd7134127dde9201e74d38dc0c08d4a096 Date: Thu Feb 29 17:57:29 2024 +0800
tflite: 53c79c506b9032fa1311ff29ed90fbf1afcba204 Date: Mon Jan 29 03:24:43 2024 -0800
- armnn: fbfa49eeb14c6cb94d47e3c770b0c168e818cf79 Date: Tue Jan 23 11:21:48 2024 +0000
- ComputeLibrary: fb92e22c642985a5ea7906e7e7f46285d1d47718 Date: Wed Nov 22 11:33:46 2023 +0000
- flatbuffer: 129ef422e8a4e89d87a7216a865602673a6d0bf3 Date: Tue Dec 19 23:43:55 2023 +0100
pdlite: 8795a4d3d8d209bd812dee95947a294ff017d63f Date: Fri Mar 8 10:00:08 2024 +0800
onnxruntime: 435e19953ea54115124fd637a67a87681a7fc8eb Date: Mon Feb 5 07:26:24 2024 +0800

and Pytorch 2.2.1: 6c8c5ad5eaf47a62fafbb4a2747198cbffbf1ff0 Date: Thu Feb 15 13:19:00 2024 -0500 v2.2.1

Assets 15

v0.0 Pre-release

Pre-release

YingkunZhou released this 14 Jul 03:08

v0.0

a0a3972

upload imagenet validation subset (1/20 imagenet by selecting the first class for every 20 classes)
upload imagenet validation subset (1/50 imagenet by selecting the first class for every 50 classes)
mnn-vulkan-patch.diff 参照ncnn的形式重新构造了vkCreateInstance（先）和 vkCreateDevice（后）两个函数调用方式
mali-g610-opencl.tar.zst 适用于edge2的Linux环境，支持OpenCL，但是不支持vulkan！
libMali.so 适用于vim3 Linux 环境，支持OpenCL? 但是不支持vulkan！
torch-2.1.0.dev20230825-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 是最后一个onednn+acl能够跑通运行正确的torch二进制
ncnn-eq-int8-table.tar.gz ncnn量化表，使用eq方式得到，而且结合我对ncnn2table这个工具的魔改，具体参考 https://github.com/YingkunZhou/EdgeTransformerPerf/wiki/ncnn#results

Assets 11

Releases: YingkunZhou/EdgeTransformerBench

Release list

v3.1

add layer-wise operation profiling util-tools

MNN

TFLite

TensorRT

CANN

OpenVino

Uh oh!

v3.0

add android real application

Uh oh!

v2.1

add x86_64 linux and windows support

Uh oh!

v2.0

Uh oh!

v1.1

Uh oh!

v1.0

Uh oh!

v0.0

Uh oh!