Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

paddleocr with paddle serving on tensorrt #7401

Closed
sybest1259 opened this issue Aug 29, 2022 · 28 comments
Closed

paddleocr with paddle serving on tensorrt #7401

sybest1259 opened this issue Aug 29, 2022 · 28 comments

Comments

@sybest1259
Copy link

环境配置如下：
paddleocr-release2.5
docker_image: registry.baidubce.com/paddlepaddle/paddle:2.1.3-gpu-cuda10.2-cudnn7
tensorrt: 7.2.1.6
paddle-gpu: 2.1.1（用来适配tensorrt7.2）
paddle-serving-app: 0.7.0
paddle-serving-client: 0.7.0
paddle-serving-server-gpu: 0.7.0.post102
问题:
运行paddle serving
运行python pipeline: python web_service.py报错如下:
The input [conv2d_252.tmp_0] shape of trt subgraph is [-1,96,-1,-1], please enable trt dynamic_shape mode by SetTRTDynamicShapeInfo
之前根据https://gitee.com/paddlepaddle/Serving/blob/v0.8.2/doc/TensorRT_Dynamic_Shape_CN.md已在web_service.py中的DetOp和RecOp类加入set_dynamic_shape_info函数，但是无效，依然报错
@andyjiang1116
Copy link
Collaborator

升级下paddle版本，推荐2.3，同时serving也升级一下吧，可以用0.8.3，另外paddleocr已经更新到2.6分支，也可以更新下代码

@sybest1259
Copy link
Author

用了新的serving镜像registry.baidubce.com/paddlepaddle/serving:0.9.0-cuda11.2-cudnn8-devel
运行web_service.py报错Compiled with WITH_GPU, but no GPU found in runtime

@andyjiang1116
Copy link
Collaborator

按照这个文档配置环境吧 https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/deploy/pdserving/README_CN.md#%E7%8E%AF%E5%A2%83%E5%87%86%E5%A4%87

@sybest1259
Copy link
Author

好的，谢谢。我先试试，另外还想问一下，发布的ppocr训练模型和推理模型的关系是什么

@sybest1259
Copy link
Author

怎么修改推理模型使用的算法

@andyjiang1116
Copy link
Collaborator

训练模型是动态图模型，可以用来作预训练模型，推理模型是导出的静态图模型，便于部署推理，可以适配c++，python等不同部署方式

@andyjiang1116
Copy link
Collaborator

怎么修改推理模型使用的算法

具体是哪种使用方式呢？

@sybest1259
Copy link
Author

我用的paddle serving，python http pipeline，模型用的ch_PP-OCRv3_det_slim和ch_PP-OCRv3_rec_slim的推理模型

@andyjiang1116
Copy link
Collaborator

paddle serving部署目前只支持pp系列模型，因为其他算法的前后处理会不太一样，serving需要单独适配，感兴趣的话可以自己编写前后处理的代码适配serving

@sybest1259
Copy link
Author

我用的这个是pp系列模型吧？您的意思是推理模型使用的算法是固定的，不能更换？

@sybest1259
Copy link
Author

文档里这么写的：PP-OCRv3的识别模型默认使用的rec_algorithm为SVTR_LCNet，注意和原始SVTR的区别。
SVTR_LCNet算法不能更换别的吗？

@andyjiang1116
Copy link
Collaborator

是的，PP-OCRv3的识别模型就是SVTR_LCNet，serving部署的话，直接下载对应的pp系列的推理模型就行

@sybest1259
Copy link
Author

那我如果想使用其他算法呢？我看见使用训练模型预测时可以在配置文件修改算法。能否将训练模型转化为推理模型？

@sybest1259
Copy link
Author

我的目的就是在服务器上部署一个模型服务，而且可以更换检测和识别的算法。这个现在可以实现吗？

@andyjiang1116
Copy link
Collaborator

Python预测是支持不同算法的，serving目前不支持pp系列以外的算法

@andyjiang1116
Copy link
Collaborator

我的目的就是在服务器上部署一个模型服务，而且可以更换检测和识别的算法。这个现在可以实现吗？

serving部署更换的话，也只能使用PP系列的模型，比如v2或者v3，其他暂不支持

@sybest1259
Copy link
Author

好的，谢谢。那后续serving会支持算法更换吗？

@sybest1259
Copy link
Author

比如在配置文件里修改

@sybest1259
Copy link
Author

我试了很多种paddle开发镜像和serving镜像部署tensorrt模式的服务，但是都失败了。您能推荐一个版本组合吗？开发/serving镜像+paddlepaddle-gpuXX+tensorrtXX+PaddleOCR-releaseXX

@andyjiang1116
Copy link
Collaborator

目前还没计划，后续可能会有

@sybest1259
Copy link
Author

我刚试了一下
环境如下：
PaddleOCR-release-2.6
镜像：registry.baidubce.com/paddlepaddle/serving:0.9.0-cuda11.2-cudnn8-devel
python: 3.8.9（镜像中是3.6.0，运行webservice提示无法导入包，升级python解决）
python依赖：
paddle-serving-app==0.8.3, paddle-serving-client==0.8.3, paddle-serving-server-gpu==0.8.3.post112
paddlepaddle_gpu-2.3.2.post112-cp38-cp38-linux_x86_64.whl
并且按照https://github.com/PaddlePaddle/Serving/blob/v0.8.3/doc/Install_CN.md，执行了bash Serving/tools/paddle_env_install.sh

运行python http pipeline的webservice报错：
[DAG] Succ init
[PipelineServicer] succ init
2022/09/02 03:59:34 start proxy service
W0902 03:59:37.344923 19405 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime.
W0902 03:59:37.344949 19380 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime.
W0902 03:59:37.355643 19368 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime.
W0902 03:59:37.361737 19354 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime.
W0902 03:59:37.365087 19330 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime.
W0902 03:59:37.378244 19341 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime.
W0902 03:59:37.381970 19391 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime.
W0902 03:59:37.389566 19323 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime.
/home/envs/ppserving/lib/python3.8/site-packages/paddle/fluid/framework.py:300: UserWarning: You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default.
warnings.warn(
failed to create predictor: Log_id: 0 Raise_msg: (External) Cuda error(804), forward compatibility was attempted on non supported HW.
[Advise: Please search for the error code(804) on website( https://docs.nvidia.com/cuda/archive/9.0/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038 ) to get Nvidias official solution about CUDA Error.] (at /paddle/paddle/fluid/platform/gpu_info.cc:99)
ClassName: LocalPredictor.load_model_config..create_predictor_check FunctionName: create_predictor_check
Killed

@sybest1259
Copy link
Author

nvidia-smi命令正常：
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.102.04 Driver Version: 450.102.04 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:5A:00.0 Off | N/A |
| 37% 59C P2 167W / 250W | 8320MiB / 11019MiB | 46% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 208... Off | 00000000:66:00.0 Off | N/A |
| 35% 57C P2 185W / 250W | 8160MiB / 11019MiB | 86% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 GeForce RTX 208... Off | 00000000:BD:00.0 Off | N/A |
| 43% 64C P2 148W / 250W | 8160MiB / 11019MiB | 81% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 GeForce RTX 208... Off | 00000000:C1:00.0 Off | N/A |
| 35% 56C P2 179W / 250W | 8160MiB / 11019MiB | 82% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+

cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR -A 2

#define CUDNN_MAJOR 8
#define CUDNN_MINOR 1
#define CUDNN_PATCHLEVEL 1

#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#endif /* CUDNN_VERSION_H */

@andyjiang1116
Copy link
Collaborator

首先确保gpu版本的paddle安装正确，看报错你的基础环境还没安装成功

@sybest1259
Copy link
Author

我用10.2的镜像就不报cuda的错误，11.2的这个镜像我试了不同python3的版本、paddle-gpu的版本，cuda都驱不起来

@sybest1259
Copy link
Author

另外还想问一下，serving的rpc方式比http速度要快吧？调用rpc返回paddle_serving_server.pipeline.proto.pipeline_service_pb2.Response这个对象，要怎么解析呢？文档里没找到相关说明

@andyjiang1116
Copy link
Collaborator

我用10.2的镜像就不报cuda的错误，11.2的这个镜像我试了不同python3的版本、paddle-gpu的版本，cuda都驱不起来

镜像安装问题可以到paddle的repo下提issue

@andyjiang1116
Copy link
Collaborator

另外还想问一下，serving的rpc方式比http速度要快吧？调用rpc返回paddle_serving_server.pipeline.proto.pipeline_service_pb2.Response这个对象，要怎么解析呢？文档里没找到相关说明

这个是serving相关的问题，可以到PaddleServing的repo下面问问

@sybest1259
Copy link
Author

好的，谢谢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sybest1259 @andyjiang1116