[Benchmark tool] Support precision evaluation on Android (using imagenet) #7525

zhaoyang-star · 2021-11-01T15:16:03Z

1. 背景

benchmark_bin 工具可以读取真实的验证集数据，进行计算后，给出性能数据和精度数据。

2012 ILSVRC 验证集共 5K 张图片，约 6GB 存储空间，占用空间比较大，为适应端侧应用场景从中筛选出 1K 张图片进行验证。
筛选方法：从验证集中，每个类别挑选一张图片，最后形成一个含有 1K 张图片的验证集子集。

ILSVRC 2012 image classification task，可在此链接下载。

2. 适用场景

目前基于验证集的精度评测功能仅支持安卓系统，对其它系统的支持在开发中。

3. 在 Android 上运行精度评测

3.1 编译

./lite/tools/build_android.sh --toolchain=clang --with_benchmark=ON full_publish

编译完成后，会生成 build.lite.*./lite/api/benchmark_bin 二进制文件、基于 NDK 预编译好的 OpenCV4.1.0 和验证集。

   build.lite.*./third_party
   ├── opencv4.1.0                              预编译好的 OpenCV
   ├── validation_dataset
   │   ├── ILSVRC2012_1000_cls                  ILSVRC 2012 图片数据
   │   ├── ILSVRC2012_1000_cls_label_list.txt   ILSVRC 2012 标签数据

3.2 运行

需要将如下文件通过 adb 上传至手机：

Paddle 模型（combined 或 uncombined 格式均可）或已经 opt 工具离线优化后的 .nb 文件
二进制文件benchmark_bin
ILSVRC 2012 图片数据
ILSVRC 2012 标签数据
数据集的配置文件 config.txt

在 Host 端机器上操作例子如下：

# 获取模型文件
wget https://paddle-inference-dist.bj.bcebos.com/AI-Rank/mobile/MobileNetV1.tar.gz
tar zxvf MobileNetV1.tar.gz

# 上传文件
adb shell mkdir /data/local/tmp/benchmark
adb push MobileNetV1 /data/local/tmp/benchmark
adb push build.lite.android.armv8.clang/lite/api/benchmark_bin /data/local/tmp/benchmark
adb push build.lite.android.armv8.clang/third_party/validation_dataset /data/local/tmp/benchmark
adb push lite/api/tools/benchmark/precision_evaluation/imagenet_image_classification/config.txt /data/local/tmp/benchmark

# 执行性能测试
adb shell "cd /data/local/tmp/benchmark;
  ./benchmark_bin \
    --model_file=MobileNetV1/inference.pdmodel \
    --param_file=MobileNetV1/inference.pdiparams \
    --validation_set=ILSVRC2012 \
    --config_path=config.txt \
    --input_shape=1,3,224,224 \
    --warmup=10 \
    --repeats=20 \
    --backend=arm"

部分输出日志如下：

======= Accurancy Info =======
config: /data/local/tmp/config.txt
label_path:/data/local/tmp/imagenet1k_label_list.txt
ground_truth_images_path:/data/local/tmp/ILSVRC2012_1000_cls/val_list_1k.txt
resize_short_size:256
crop_size:224
mean:0.485,0.456,0.406
scale:1/0.229,1/0.224,1/0.225
topk:2
store_result_as_image:1

Top-1 Accurancy: 0.739
Top-2 Accurancy: 0.121

======= Perf Info =======
Time(unit: ms):
init  = 11.925
first = 119.312
min   = 6.472
max   = 9.091
avg   = 7.785

详细信息请阅读：lite/api/tools/benchmark/precision_evaluation/imagenet_image_classification/README.md

paddle-bot-old · 2021-11-01T15:16:06Z

Thanks for your contribution!

…/zhaoyang-star/Paddle-Lite into benchmark_tool_support_precision

chenjiaoAngel · 2021-11-11T12:00:20Z

CMakeLists.txt

@@ -86,6 +86,7 @@ lite_option(LITE_THREAD_POOL                   "Enable thread pool in lite"
 lite_option(LITE_BUILD_EXTRA                   "Enable extra algorithm support in Lite, both kernels and operators"   OFF)
 lite_option(LITE_BUILD_TAILOR                  "Enable tailoring library according to model"                          OFF)
 # cv build options
+lite_option(LITE_USE_PRECOMPILED_OPENCV        "Use precompiled opencv, mainly for pre/post processing in benchmark"  OFF)


能否用已经支持的CV 加速库？不然lite 里有两个CV，感觉有些混淆
或者benchmark 使用opencv 方法效仿demo/cxx 下的使用方法，不要把他放到主库

已删除该选项

chenjiaoAngel

LGTM

hong19860320 · 2021-11-16T02:31:35Z

lite/api/CMakeLists.txt

 if(NOT IOS AND NOT LITE_ON_TINY_PUBLISH)
    lite_cc_binary(test_model_bin SRCS tools/model_test.cc
        DEPS gflags
        CV_DEPS paddle_cv_arm)
-    lite_cc_binary(benchmark_bin SRCS tools/benchmark.cc tools/flags.cc tools/opt_base.cc
+
+    if(ARM_TARGET_OS STREQUAL "android")


是否能将benchmark的cmake代码都以新的CMakeLists.txt放在tools/benchmark/目录下，然后这里直接include？

可以的，我修改下

hong19860320

LGTM

chenjiaoAngel · 2021-11-17T02:50:18Z

lite/api/tools/benchmark/precision_evaluation/imagenet_image_classification/config.txt

@@ -0,0 +1,8 @@
+label_path:/data/local/tmp/benchmark/validation_dataset/ILSVRC2012_1000_cls_label_list.txt
+ground_truth_images_path:/data/local/tmp/benchmark/validation_dataset/ILSVRC2012_1000_cls/val_list_1k.txt
+resize_short_size:256


config 中冒号是不是用空格隔开，这样看起来好点

嗯，我在下个PR中更新下字符串处理的功能函数。由于这个是样板间，咱们CI执行有些慢，所以考虑基本功能OK的话，就先合入。

后续改为使用 RapidJSON 解析 config.json

chenjiaoAngel · 2021-11-17T02:50:53Z

lite/api/tools/benchmark/precision_evaluation/imagenet_image_classification/prepost_process.cc

+  if (image_files.empty()) return;
+
+  // Read image
+  // std::cout << "image: " << image_files.at(cnt) << std::endl;


没有用的注释能否删除

我在下个PR中删除该调试注释

chenjiaoAngel · 2021-11-17T02:51:57Z

lite/api/tools/benchmark/precision_evaluation/imagenet_image_classification/prepost_process.cc

+  for (int i = 0; i < ele_num; i++) {
+    float score = output_data[i];
+    int index = i;
+    for (int j = 0; j < TOPK; j++) {


这个可以用std::sort 排序

std::partial_sort(vec.begin(),
vec.begin() + topk,
vec.end(),
std::greater<std::pair<float, int>>());

OK 在下个PR中更新此处

zhaoyang-star added 2 commits November 1, 2021 22:57

add opencv

2e2cdb4

test=develop

a693f10

zhaoyang-star added 16 commits November 1, 2021 23:52

v1

83681ee

debug v1

cd039fc

tmp

604edac

v2

dfc1d93

compile pass

db2cd4b

pass

dbbc4fc

pass 2

26e1495

refactor

66f4854

refactor

f461a64

refactor pass

d033e33

refactor pass

618d8d1

Merge branch 'benchmark_tool_support_precision' of https://github.com…

293b435

…/zhaoyang-star/Paddle-Lite into benchmark_tool_support_precision

refactor pass

02817b6

imagenet classification pass

782daa5

grace 1

e32ce91

pass

9670280

zhaoyang-star changed the title ~~support precision benchmark~~ [Benchmark tool] Support precision evaluation on Android (using imagenet) Nov 11, 2021

chenjiaoAngel reviewed Nov 11, 2021

View reviewed changes

zhaoyang-star added 5 commits November 11, 2021 20:30

remove LITE_USE_PRECOMPILED_OPENCV

20b7604

update readme

12c9d68

remove LITE_USE_PRECOMPILED_OPENCV

b014e6a

fix lite/api/CMakeLists.txt

5a9d2e1

fix lite/api/CMakeLists.txt

6bf5e05

chenjiaoAngel previously approved these changes Nov 12, 2021

View reviewed changes

fix undefined reference to 'atof' on armv7

ddeec64

zhaoyang-star dismissed chenjiaoAngel’s stale review via ddeec64 November 15, 2021 14:54

change https to http

b3b19db

hong19860320 reviewed Nov 16, 2021

View reviewed changes

zhaoyang-star added 3 commits November 16, 2021 13:25

add CMakeLists.txt for benchmark_bin

d3dbe2a

test=develop

7f99b91

Merge branch 'develop' into benchmark_tool_support_precision

34c22c4

hong19860320 approved these changes Nov 17, 2021

View reviewed changes

chenjiaoAngel reviewed Nov 17, 2021

View reviewed changes

zhaoyang-star merged commit d956eba into PaddlePaddle:develop Nov 18, 2021

zhaoyang-star deleted the benchmark_tool_support_precision branch November 18, 2021 05:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Benchmark tool] Support precision evaluation on Android (using imagenet) #7525

[Benchmark tool] Support precision evaluation on Android (using imagenet) #7525

zhaoyang-star commented Nov 1, 2021 •

edited

paddle-bot-old bot commented Nov 1, 2021

chenjiaoAngel Nov 11, 2021

zhaoyang-star Nov 11, 2021

chenjiaoAngel left a comment

hong19860320 Nov 16, 2021

zhaoyang-star Nov 16, 2021

zhaoyang-star Nov 16, 2021

hong19860320 left a comment

chenjiaoAngel Nov 17, 2021

zhaoyang-star Nov 17, 2021 •

edited

zhaoyang-star Nov 17, 2021

chenjiaoAngel Nov 17, 2021

zhaoyang-star Nov 17, 2021

chenjiaoAngel Nov 17, 2021

chenjiaoAngel Nov 17, 2021

zhaoyang-star Nov 17, 2021

[Benchmark tool] Support precision evaluation on Android (using imagenet) #7525

[Benchmark tool] Support precision evaluation on Android (using imagenet) #7525

Conversation

zhaoyang-star commented Nov 1, 2021 • edited

1. 背景

2. 适用场景

3. 在 Android 上运行精度评测

3.1 编译

3.2 运行

paddle-bot-old bot commented Nov 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenjiaoAngel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hong19860320 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaoyang-star Nov 17, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaoyang-star commented Nov 1, 2021 •

edited

zhaoyang-star Nov 17, 2021 •

edited