[Hackathon 182 Model] Update PPOCRV3 For RKNPU2 #1403

Zheng-Bicheng · 2023-02-21T11:38:49Z

PR types(PR类型)

Model

Description

Update PPOCRV3 For RKNPU2

Zheng-Bicheng · 2023-02-21T11:39:11Z

关联Issue

Zheng-Bicheng · 2023-02-21T11:40:30Z

PPOCRV3更新细节

FastDeploy cmake部分

更新RKNPU2驱动版本至1.4.2b0
更新RKNPU2 cmake 现在RKNPU2驱动将分为RK3588和RK356X进行下载，不再混合
FastDeploy不在依赖rknnlib_api

Det部分

RKNPU2不支持Normalzie 和 Permute因此添加了DisableNormalize 和DisablePermute
RKNPU2 不支持动态shape，因此参考Rec部分新增了固定shape推理。

Cls部分

RKNPU2不支持Normalzie 和 Permute因此添加了DisableNormalize 和DisablePermute

Rec部分

RKNPU2不支持Normalzie 和 Permute因此添加了DisableNormalize 和DisablePermute

OCR Result部分

在显示result时会出现rec_score为0但是仍然被框出来的情况，这里对VisOcr函数新增了参数score_threshold

导出模型部分

新增三个模型的配置文件
对导出模型的代码进行了优化

Zheng-Bicheng · 2023-02-21T11:40:43Z

结果展示

det boxes: [[276,174],[285,173],[285,178],[276,179]]rec text:  rec score:0.000000 cls label: 1 cls score: 0.766602
det boxes: [[43,408],[483,390],[483,431],[44,449]]rec text: 上海斯格威铂尔曼大酒店 rec score:0.888450 cls label: 0 cls score: 1.000000
det boxes: [[186,456],[399,448],[399,480],[186,488]]rec text: 打浦路15号 rec score:0.988769 cls label: 0 cls score: 1.000000
det boxes: [[18,501],[513,485],[514,537],[18,554]]rec text: 绿洲仕格维花园公寓 rec score:0.992730 cls label: 0 cls score: 1.000000
det boxes: [[78,553],[404,541],[404,573],[78,585]]rec text: 打浦路252935号 rec score:0.983545 cls label: 0 cls score: 1.000000
Visualized result saved in ./vis_result.jpg

Zheng-Bicheng · 2023-02-21T11:40:51Z

目前未实现要求

经过实测，使用RK静态量化工具，PPOCRV3的精度会大幅度下降，这一点可能要和RK的工程师讨论下。 @leiqing1

jiangjiajun · 2023-02-21T11:46:37Z

fastdeploy/vision/ocr/ppocr/det_preprocessor.h

+  /// Set static_shape_infer is true or not. When deploy PP-OCR
+  /// on hardware which can not support dynamic input shape very well,
+  /// like Huawei Ascned, static_shape_infer needs to to be true.
+  void SetStaticShapeInfer(bool static_shape_infer) {


这个接口，与SetDetImageShape()，是否是有重复。例如调用了SetDetImageShape()本身就是要求SetStaticShapeInfer()

这个是参考rec_preprocessor的api结构，如下:

/// Set static_shape_infer is true or not. When deploy PP-OCR /// on hardware which can not support dynamic input shape very well, /// like Huawei Ascned, static_shape_infer needs to to be true. void SetStaticShapeInfer(bool static_shape_infer) { static_shape_infer_ = static_shape_infer; } /// Get static_shape_infer of the recognition preprocess bool GetStaticShapeInfer() const { return static_shape_infer_; } /// Set rec_image_shape for the recognition preprocess void SetRecImageShape(const std::vector<int>& rec_image_shape) { rec_image_shape_ = rec_image_shape; } /// Get rec_image_shape for the recognition preprocess std::vector<int> GetRecImageShape() { return rec_image_shape_; }

api统一会不会更利于用户理解？调用SetDetImageShape()确实是在固定shape推理的情况下才有意义。

yunyaoXYY · 2023-02-22T02:42:06Z

fastdeploy/vision/ocr/ppocr/det_preprocessor.h

+  void DisablePermute() { disable_normalize_ = true; }
+
+  /// Set cls_image_shape for the classification preprocess
+  void SetDetImageShape(const std::vector<int>& det_image_shape) {


这个SetDetImageShape是否不需要暴露给用户使用？我看着这个SetDetImageShape如果暴露后，是由用户自己去决定这个reszie的shape. 但是他们可能也不知道resize到多少比较合适.

还有这里的英文注释，这里不是 cls_image_shape

SetDetImageShape是考虑到用户自己训练Det模型的情况，注释我修改下哈

SetDetImageShape是考虑到用户自己训练Det模型的情况，注释我修改下哈

RK的用户，自己训OCR的用户多吗，因为之前很少有听说能超越官方PP-OCR的情况.
如果是出于这种考虑，需要把SetDetImageShape这种接口的使用情况说明白，可以在文档、或者demo代码里，添加一行，并且注释掉，说明使用情况

有听说重新训练Det模型的，其他模型没听说过重新训练的。因为使用场景不一致，有的用户希望检测特定领域，比如发票啥的，就希望能够自己重新训练。

我将注释修改如下:

/// Set det_image_shape for the detection preprocess. /// This api is usually used when you retrain the model. /// Generally, you do not need to use it.

你好，我发现虽然测试例子能跑通，但是模型未做量化，最后使用npu的处理时间比使用cpu还要高很多，在cpu状态下大概1s,而使用npu需要1.4s。

你好，我发现虽然测试例子能跑通，但是模型未做量化，最后使用npu的处理时间比使用cpu还要高很多，在cpu状态下大概1s,而使用npu需要1.4s。

量化的问题和RK的杜大佬那边讨论过。他们对Rec模型的支持度不太好，所以暂时没有做量化。这个问题已经在沟通了，将考虑使用混合量化来解决这个问题。

至于NPU的处理时间比CPU还多，其实这个问题出在FP32转FP16的这个过程，已经反馈到RK那边了，在等他们进行优化。

你好，我发现虽然测试例子能跑通，但是模型未做量化，最后使用npu的处理时间比使用cpu还要高很多，在cpu状态下大概1s,而使用npu需要1.4s。

您好，我想问以下您的板子型号是什么，目前我在RK3588上的测试结果如下:

# ONNX下 det: 0.52s cls: 0.02 rec: 0.32 # RKNN下 det: 0.15 cls: 0.007 rec: 0.18

和您说的有出入

yunyaoXYY · 2023-02-22T02:48:27Z

还有一个小问题，不知道为什么代码的格式，会改变这么多. Bicheng你是否用了pre-commit，按理说这里先前提交的代码，也应该用了pre-commit的. 😿

Zheng-Bicheng · 2023-02-22T02:50:12Z

还有一个小问题，不知道为什么代码的格式，会改变这么多. Bicheng你是否用了pre-commit，按理说这里先前提交的代码，也应该用了pre-commit的. 😿

我用了pre-commit，有个cpp-lint执行完就这样了

yunyaoXYY · 2023-02-22T02:54:52Z

还有一个小问题，不知道为什么代码的格式，会改变这么多. Bicheng你是否用了pre-commit，按理说这里先前提交的代码，也应该用了pre-commit的. 😿

我用了pre-commit，有个cpp-lint执行完就这样了

Ok 好的

Zheng-Bicheng · 2023-02-22T03:18:46Z

@jiangjiajun @yunyaoXYY 大佬们还有需要我修改的地方吗？

jiangjiajun · 2023-02-23T08:17:12Z

@Zheng-Bicheng 昨天有同事对DetModel的预处理进行了修改，产生了冲突，需要合并一下

Zheng-Bicheng · 2023-02-25T12:59:25Z

已经merge了最新的修改，大佬你再看下 @jiangjiajun

fastdeploy/vision/ocr/ppocr/det_preprocessor.h

3togo · 2023-02-27T15:41:10Z

Could you provide an example on CI-FastDeploy-Android C++ for rknpu2?

Zheng-Bicheng · 2023-02-27T15:43:38Z

Could you provide an example on CI-FastDeploy-Android C++ for rknpu2?

We have only tested rknpu2 on the Linux platform for the time being.

KongXCai · 2023-03-06T01:51:38Z

请问您是在未量化的模型上做的测试吗？  我这边是跑的非量化模型，然后测的时间是你封好的pipline,三个模型一起的时间，使用npu时会有warning提示： [WARNING] fastdeploy/runtime/backends/rknpu2/rknpu2_backend.cc(323)::Infer      The input tensor type != model's inputs type.The input_type need FP16,but inputs[0].type is UINT8

…

------------------ 原始邮件 ------------------ 发件人: "PaddlePaddle/FastDeploy" ***@***.***>; 发送时间: 2023年3月5日(星期天) 上午6:42 ***@***.***>; ***@***.******@***.***>; 主题: Re: [PaddlePaddle/FastDeploy] [Hackathon 182 Model] Update PPOCRV3 For RKNPU2 (PR #1403) @Zheng-Bicheng commented on this pull request. In fastdeploy/vision/ocr/ppocr/det_preprocessor.h: > @@ -54,11 +54,39 @@ class FASTDEPLOY_DECL DBDetectorPreprocessor { /// Get is_scale of the image normalization in detection preprocess bool GetIsScale() const { return is_scale_; } + /// This function will disable normalize in preprocessing step. + void DisableNormalize() { disable_permute_ = true; } + /// This function will disable hwc2chw in preprocessing step. + void DisablePermute() { disable_normalize_ = true; } + + /// Set cls_image_shape for the classification preprocess + void SetDetImageShape(const std::vector<int>& det_image_shape) { 你好，我发现虽然测试例子能跑通，但是模型未做量化，最后使用npu的处理时间比使用cpu还要高很多，在cpu状态下大概1s,而使用npu需要1.4s。您好，我想问以下您的板子型号是什么，目前我在RK3588上的测试结果如下: # ONNX下 det: 0.52s cls: 0.02 rec: 0.32 # RKNN下 det: 0.15 cls: 0.007 rec: 0.18 和您说的有出入 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

update ppocrv3 for rknpu2

cc8c549

Zheng-Bicheng added 2 commits February 21, 2023 19:42

add config

9eceb60

add config

855d367

jiangjiajun requested changes Feb 21, 2023

View reviewed changes

jiangjiajun requested a review from yunyaoXYY February 21, 2023 11:47

Zheng-Bicheng added 2 commits February 21, 2023 20:09

detele unuseful

74bc645

update useful results

8782760

Zheng-Bicheng mentioned this pull request Feb 21, 2023

【PaddlePaddle Hackathon 第四期】任务总览 PaddlePaddle/Paddle#50629

Closed

yunyaoXYY reviewed Feb 22, 2023

View reviewed changes

Zheng-Bicheng added 2 commits February 22, 2023 10:58

Repair note

8e1be23

Repair note

67f24be

Zheng-Bicheng mentioned this pull request Feb 22, 2023

文字识别OCR部分支持端侧平台瑞芯微3588吗？ #1400

Closed

Merge branch 'develop' into ppocr_v3.0

d9f5057

Zheng-Bicheng added 3 commits February 24, 2023 10:58

update

ea37f54

Merge remote-tracking branch 'origin/ppocr_v3.0' into ppocr_v3.0

d79ab4a

fixed bugs

c42f9b9

Zheng-Bicheng requested a review from jiangjiajun February 25, 2023 12:58

yunyaoXYY requested changes Feb 27, 2023

View reviewed changes

fastdeploy/vision/ocr/ppocr/det_preprocessor.h Outdated Show resolved Hide resolved

update

23aed21

yunyaoXYY approved these changes Feb 27, 2023

View reviewed changes

jiangjiajun approved these changes Feb 27, 2023

View reviewed changes

jiangjiajun merged commit 8c3ccc2 into PaddlePaddle:develop Feb 27, 2023

Ligoml mentioned this pull request Mar 7, 2023

【PaddlePaddle Hackathon 第四期】任务总览 PaddlePaddle/Paddle#51281

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Hackathon 182 Model] Update PPOCRV3 For RKNPU2 #1403

[Hackathon 182 Model] Update PPOCRV3 For RKNPU2 #1403

Zheng-Bicheng commented Feb 21, 2023

Zheng-Bicheng commented Feb 21, 2023

Zheng-Bicheng commented Feb 21, 2023

Zheng-Bicheng commented Feb 21, 2023

Zheng-Bicheng commented Feb 21, 2023

jiangjiajun Feb 21, 2023

Zheng-Bicheng Feb 22, 2023

Zheng-Bicheng Feb 22, 2023

yunyaoXYY Feb 22, 2023

yunyaoXYY Feb 22, 2023

Zheng-Bicheng Feb 22, 2023

yunyaoXYY Feb 22, 2023

Zheng-Bicheng Feb 22, 2023

Zheng-Bicheng Feb 22, 2023

yunyaoXYY Feb 22, 2023

KongXCai Feb 27, 2023

Zheng-Bicheng Feb 27, 2023

Zheng-Bicheng Mar 4, 2023

yunyaoXYY commented Feb 22, 2023

Zheng-Bicheng commented Feb 22, 2023

yunyaoXYY commented Feb 22, 2023

Zheng-Bicheng commented Feb 22, 2023

jiangjiajun commented Feb 23, 2023

Zheng-Bicheng commented Feb 25, 2023

3togo commented Feb 27, 2023

Zheng-Bicheng commented Feb 27, 2023 •

edited

KongXCai commented Mar 6, 2023 via email

[Hackathon 182 Model] Update PPOCRV3 For RKNPU2 #1403

[Hackathon 182 Model] Update PPOCRV3 For RKNPU2 #1403

Conversation

Zheng-Bicheng commented Feb 21, 2023

PR types(PR类型)

Description

Zheng-Bicheng commented Feb 21, 2023

关联Issue

Zheng-Bicheng commented Feb 21, 2023

PPOCRV3更新细节

FastDeploy cmake部分

Det部分

Cls部分

Rec部分

OCR Result部分

导出模型部分

Zheng-Bicheng commented Feb 21, 2023

结果展示

Zheng-Bicheng commented Feb 21, 2023

目前未实现要求

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yunyaoXYY commented Feb 22, 2023

Zheng-Bicheng commented Feb 22, 2023

yunyaoXYY commented Feb 22, 2023

Zheng-Bicheng commented Feb 22, 2023

jiangjiajun commented Feb 23, 2023

Zheng-Bicheng commented Feb 25, 2023

3togo commented Feb 27, 2023

Zheng-Bicheng commented Feb 27, 2023 • edited

KongXCai commented Mar 6, 2023 via email

Zheng-Bicheng commented Feb 27, 2023 •

edited