Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

华为 Ascend 310部署yolov3_darknet53_270e_coco_fp32_608问题 #10417

Open
alixiu opened this issue Nov 15, 2023 · 2 comments
Open

华为 Ascend 310部署yolov3_darknet53_270e_coco_fp32_608问题 #10417

alixiu opened this issue Nov 15, 2023 · 2 comments

Comments

@alixiu
Copy link

alixiu commented Nov 15, 2023 •
edited
Loading

在华为Ascend 310 (linux armv8)上，从https://paddlelite-demo.bj.bcebos.com/devices/generic/PaddleLite-generic-demo_v2_12_0.tar.gz 下载 Paddle Lite 通用示例程序，解压后
$ cd PaddleLite-generic-demo/object_detection_demo/shell
$ bash run.sh yolov3_darknet53_270e_coco_fp32_608 yolov3_coco_608.txt test linux arm64 huawei_ascend_npu
报如下错误：
EVENT] DVPP(7924,demo):2023-04-05-07:55:42.949.683 [dvpp_engine.cc:79][ENGINE][GetGraphOptimizerObjs:79][tid:7924]dvpp graph optimizer do not support
[ERROR] GE(7924,demo):2023-04-05-07:55:43.928.641 [error_manager.cc:91]7924 FormatErrorMessage: [INIT][OPS_KER][Check][Param] FormatErrorMessage failed, ret:-1, pattern:%s[FUNC:%s][FILE:%s][LINE:%d]
[ERROR] FE(7924,demo):2023-04-05-07:55:43.928.710 [plugin_manager.cc:62]7924 OpenPlugin:"[FEInit][OpPluginSo] Fail to load so file /usr/local/Ascend/ascend-toolkit/5.0.4.alpha005/arm64-linux/opp/fusion_pass/built_in/libops_fusion_pass_aicore.so, error message is (null)."
[ERROR] FE(7924,demo):2023-04-05-07:55:43.928.827 [fusion_pass_manager.cc:164]7924 OpenPassPath:"[GraphOpt][Init][OpenPassPath] Failed to open the fusion pass so [/usr/local/Ascend/ascend-toolkit/5.0.4.alpha005/arm64-linux/opp/fusion_pass/built_in/libops_fusion_pass_aicore.so]."
[ERROR] FE(7924,demo):2023-04-05-07:55:44.958.882 [fusion_config_parser.cc:345]7924 ParseFusionConfigFile:"[GraphOpt][Init][ParseFusConfig] Read Json content from file:/home/HwHiAiUser/Ascend/ascend-toolkit/5.1.RC2/aarch64-linux/lib64/plugin/opskernel/fe_config/fusion_config.json failed."
[ERROR] FE(7924,demo):2023-04-05-07:55:44.958.948 [fe_graph_optimizer.cc:187]7924 Initialize:"[GraphOpt][Init] FusionPriorityMgr initialize Fail."
[ERROR] GE(7924,demo):2023-04-05-07:55:44.958.990 [ops_kernel_manager.cc:326]7924 InitGraphOptimizers: ErrorNo: 1343250433(Failed to initialize GraphOptimizer.) [INIT][OPS_KER][Init][GraphOptimizer] GraphOptimizer: AIcoreEngine initialize failed.
[ERROR] GE(7924,demo):2023-04-05-07:55:44.959.067 [gelib.cc:177]7924 InnerInitialize: ErrorNo: 1343250433(Failed to initialize GraphOptimizer.) [INIT][OPS_KER][Init][OpsManager]GE ops manager initial failed.
[ERROR] GE(7924,demo):2023-04-05-07:55:44.959.184 [gelib.cc:119]7924 Initialize: ErrorNo: 1343250433(Failed to initialize GraphOptimizer.) [INIT][OPS_KER][Init][GeLib]GeLib initial failed.
[ERROR] GE(7924,demo):2023-04-05-07:55:44.959.235 [ge_ir_build.cc:205]7924 aclgrphBuildInitializeImpl: ErrorNo: 1343250433(Failed to initialize GraphOptimizer.) [INIT][OPS_KER][Init][GELib] failed!
[3 4/ 5 7:55:44.959 .../src/driver/huawei_ascend_npu/utility.cc:247 BuildOMModelToBuffer] input_count: 1
[3 4/ 5 7:55:44.959 .../src/driver/huawei_ascend_npu/utility.cc:248 BuildOMModelToBuffer] output_count: 3
[EVENT] GE(7924,demo):2023-04-05-07:55:44.997.303 [graph_manager.cc:902]7924 PreRun:PreRun start: graph node size 341, session id 0, graph id 0, graph name graph.
[EVENT] TDT(7924,demo):2023-04-05-07:55:44.998.596 [log.cpp:42][TsdClient] set profiling callback success.,[client_manager.cpp:158:SetProfilingCallback]7924
[EVENT] GE(7924,demo):2023-04-05-07:55:45.012.592 [graph_manager.cc:767]7924 PreRunOptimizeOriginalGraph:[GEPERFTRACE] The time cost of GraphManager::stages.preparer.PrepareInitAndUpdateInput is [9895] micro second.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.012.750 [graph_optimize.cc:169]7924 GetValidGraphOptimizer: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][PRE_OPT][Get][GELib] Gelib not init before.
[EVENT] GE(7924,demo):2023-04-05-07:55:45.012.768 [graph_manager.cc:768]7924 PreRunOptimizeOriginalGraph:[GEPERFTRACE] The time cost of GraphManager::stages.optimizer.OptimizeOriginalGraphForQuantize is [109] micro second.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.012.783 [graph_manager.cc:768]7924 PreRunOptimizeOriginalGraph: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][PRE_OPT][Process][GraphManager_stages.optimizer.OptimizeOriginalGraphForQuantize] failed
[ERROR] GE(7924,demo):2023-04-05-07:55:45.012.801 [graph_manager.cc:3322]7924 OptimizeGraph: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][PRE_OPT][Run][PreRunOptimizeOriginalGraph] failed for graph:graph_NPU, session_id:0
[ERROR] GE(7924,demo):2023-04-05-07:55:45.031.992 [graph_manager.cc:3395]7924 ProcessNodeEngineBuild: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][PRE_OPT][Run][OptimizeSubGraph] failed for graph:graph_NPU, session_id:0.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.032.301 [graph_manager.cc:969]7924 StartForRunGraph: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][PRE_OPT][Call][PreRun] Failed, graph_id:0, session_id:0.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.032.828 [graph_manager.cc:1366]7924 BuildGraph: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][PRE_OPT][Call][StartForRunGraph] failed! graph_id:0.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.032.918 [ge_generator.cc:1430]7924 BuildModel: ErrorNo: 1343266819(Graph manager build graph failed.) [COMP][DEFAULT][Build][Graph] fail, graph id: 0
[ERROR] GE(7924,demo):2023-04-05-07:55:45.032.970 [ge_generator.cc:821]7924 GenerateModel: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][DEFAULT][Build][Model] failed, ret:1343229963.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.033.678 [ge_ir_build.cc:733]7924 BuildModel: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][DEFAULT][Generate][OnlineModel] failed!
[ERROR] GE(7924,demo):2023-04-05-07:55:45.036.323 [graph_manager.cc:586]7924 CheckGraphAdded: ErrorNo: 1343242243(Graph ID already exist.) [COMP][DEFAULT][Get][Attr] _graph_has_been_added from graph:1 fail.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.036.369 [graph_manager.cc:538]7924 AddGraph: ErrorNo: 4294967295(failed) [COMP][DEFAULT][Check][GraphAdded] failed, graph id:1.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.036.425 [ge_generator.cc:1404]7924 BuildModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Add][Graph] fail, graph id: 1
[ERROR] GE(7924,demo):2023-04-05-07:55:45.060.061 [ge_generator.cc:821]7924 GenerateModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Build][Model] failed, ret:1343266818.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.060.131 [ge_ir_build.cc:733]7924 BuildModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Generate][OnlineModel] failed!
[ERROR] GE(7924,demo):2023-04-05-07:55:45.063.208 [graph_manager.cc:586]7924 CheckGraphAdded: ErrorNo: 1343242243(Graph ID already exist.) [COMP][DEFAULT][Get][Attr] _graph_has_been_added from graph:2 fail.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.063.256 [graph_manager.cc:538]7924 AddGraph: ErrorNo: 4294967295(failed) [COMP][DEFAULT][Check][GraphAdded] failed, graph id:2.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.063.291 [ge_generator.cc:1404]7924 BuildModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Add][Graph] fail, graph id: 2
[ERROR] GE(7924,demo):2023-04-05-07:55:45.084.045 [ge_generator.cc:821]7924 GenerateModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Build][Model] failed, ret:1343266818.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.084.114 [ge_ir_build.cc:733]7924 BuildModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Generate][OnlineModel] failed!
[ERROR] GE(7924,demo):2023-04-05-07:55:45.087.008 [graph_manager.cc:586]7924 CheckGraphAdded: ErrorNo: 1343242243(Graph ID already exist.) [COMP][DEFAULT][Get][Attr] _graph_has_been_added from graph:3 fail.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.087.053 [graph_manager.cc:538]7924 AddGraph: ErrorNo: 4294967295(failed) [COMP][DEFAULT][Check][GraphAdded] failed, graph id:3.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.087.090 [ge_generator.cc:1404]7924 BuildModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Add][Graph] fail, graph id: 3
[ERROR] GE(7924,demo):2023-04-05-07:55:45.108.023 [ge_generator.cc:821]7924 GenerateModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Build][Model] failed, ret:1343266818.
[ERROR] GE(7924,demo):2023-04-05-07:55:45.108.090 [ge_ir_build.cc:733]7924 BuildModel: ErrorNo: 1343266818(Graph manager add graph failed.) [COMP][DEFAULT][Generate][OnlineModel] failed!
[F 4/ 5 7:55:45. 33 .../src/driver/huawei_ascend_npu/utility.cc:307 BuildOMModelToBuffer] Check failed: (reinterpret_castge::graphStatus(aclgrphBuildModel(ir_graph, options, om_buffer)) == ge::GRAPH_SUCCESS): 1343266818!==0 1343266818 Unknown ATC error code(1343266818)
[F 4/ 5 7:55:45. 33 .../src/driver/huawei_ascend_npu/utility.cc:307 BuildOMModelToBuffer] Check failed: (reinterpret_castge::graphStatus(aclgrphBuildModel(ir_graph, options, om_buffer)) == ge::GRAPH_SUCCESS): 1343266818!==0 1343266818 Unknown ATC error code(1343266818)

[F 4/ 5 7:55:45.114 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.
[F 4/ 5 7:55:45.114 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.

terminate called after throwing an instance of 'nnadapter::logging::Exception'
what(): NNAdapter C++ Exception:
[F 4/ 5 7:55:45.114 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.

run.sh: line 178: 7924 Aborted ./$BUILD_DIR/demo ../assets/models/$MODEL_NAME ../assets/configs/$CONFIG_NAME ../assets/datasets/$DATASET_NAME $NNADAPTER_DEVICE_NAMES "$NNADAPTER_CONTEXT_PROPERTIES" $NNADAPTER_MODEL_CACHE_DIR $NNADAPTER_MODEL_CACHE_TOKEN $NNADAPTER_SUBGRAPH_PARTITION_CONFIG_PATH $NNADAPTER_MIXED_PRECISION_QUANTIZATION_CONFIG_PATH
/usr/local/python3.7.0/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 91 leaked semaphores to clean up at shutdown
len(cache))
@alixiu alixiu changed the title 华为 Ascend 310部署问题 华为 Ascend 310部署yolov3_darknet53_270e_coco_fp32_608问题 Nov 16, 2023
@qili93
Copy link
Collaborator

qili93 commented Feb 5, 2024

您好，根据报错信息“ Check failed: device_context: No device found.”应该是设备没有被正确识别，您看下其他模型在这个设备和PaddleLite版本上可以正常运行吗？

@hong19860320
Copy link
Collaborator

从错误上看，生成 NPU 模型失败了。
文档 https://www.paddlepaddle.org.cn/lite/v2.12/demo_guides/huawei_ascend_npu.html 自带的 paddlelite-generic-demo 中的 paddlelite 库是基于 CANN 5.1.RC1.alpha001 编译的，你需要严格按照文档的步骤安装对应版本的驱动和 CANN 版本。
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hong19860320 @qili93 @alixiu