C++ CreatePaddlePredictor加载模型出core

[HrBlack]

• 版本、环境信息：
     1）PaddlePaddle版本：paddle@paddle_with_lite_1-7-2-22-xpu-turing_PD_BL@git_tag
     2）GPU：Tesla T4, CUDA：V10.2.89

• 问题描述：
  ernie模型训练完后，利用io.save_inference_model保存了预测模型，然后用c++ CreatePaddlePredictor加载时出core；用python加载模型正常。
  训练的paddle版本尝试过1.6.3与1.8.1，两种模型均会出core。

-报错信息：
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0812 01:19:46.660524 29227 analysis_predictor.cc:140] Profiler is deactivated, and no profiling report will be generated.
I0812 01:19:46.666958 29227 analysis_predictor.cc:908] MODEL VERSION: 1.8.1
I0812 01:19:46.667003 29227 analysis_predictor.cc:910] PREDICTOR VERSION: 0.0.0
W0812 01:19:46.667055 29227 analysis_predictor.cc:923] - Version incompatible (1) concat
W0812 01:19:46.667075 29227 analysis_predictor.cc:923] - Version incompatible (1) cos_sim
W0812 01:19:46.667081 29227 analysis_predictor.cc:923] - Version incompatible (1) dropout
W0812 01:19:46.667088 29227 analysis_predictor.cc:923] - Version incompatible (1) elementwise_add
W0812 01:19:46.667093 29227 analysis_predictor.cc:923] - Version incompatible (1) elementwise_sub
W0812 01:19:46.667109 29227 analysis_predictor.cc:923] - Version incompatible (1) feed
W0812 01:19:46.667114 29227 analysis_predictor.cc:923] - Version incompatible (1) fetch
W0812 01:19:46.667122 29227 analysis_predictor.cc:923] - Version incompatible (1) fill_constant_batch_size_like
W0812 01:19:46.667129 29227 analysis_predictor.cc:923] - Version incompatible (3) layer_norm
W0812 01:19:46.667135 29227 analysis_predictor.cc:923] - Version incompatible (1) lookup_table
W0812 01:19:46.667141 29227 analysis_predictor.cc:923] - Version incompatible (2) matmul
W0812 01:19:46.667147 29227 analysis_predictor.cc:923] - Version incompatible (2) mul
W0812 01:19:46.667153 29227 analysis_predictor.cc:923] - Version incompatible (1) relu
W0812 01:19:46.667160 29227 analysis_predictor.cc:923] - Version incompatible (2) reshape2
W0812 01:19:46.667166 29227 analysis_predictor.cc:923] - Version incompatible (1) scale
W0812 01:19:46.667172 29227 analysis_predictor.cc:923] - Version incompatible (2) slice
W0812 01:19:46.667179 29227 analysis_predictor.cc:923] - Version incompatible (1) softmax
W0812 01:19:46.667186 29227 analysis_predictor.cc:923] - Version incompatible (1) stack
W0812 01:19:46.667191 29227 analysis_predictor.cc:923] - Version incompatible (1) tanh
W0812 01:19:46.667197 29227 analysis_predictor.cc:923] - Version incompatible (1) transpose2
W0812 01:19:46.667204 29227 analysis_predictor.cc:196] WARNING: Results may be DIFF! Please use the corresponding version of the model and prediction library, and do not use the develop branch.
--- Running analysis [ir_graph_build_pass]
--- Running analysis [ir_graph_clean_pass]
--- Running analysis [ir_analysis_pass]
--- Running IR pass [is_test_pass]
--- Running IR pass [simplify_with_basic_ops_pass]
--- Running IR pass [conv_affine_channel_fuse_pass]
--- Running IR pass [conv_eltwiseadd_affine_channel_fuse_pass]
--- Running IR pass [conv_bn_fuse_pass]
--- Running IR pass [conv_eltwiseadd_bn_fuse_pass]
--- Running IR pass [embedding_eltwise_layernorm_fuse_pass]
I0812 01:19:46.781428 29227 graph_pattern_detector.cc:101] --- detected 2 subgraphs
I0812 01:19:46.781893 29227 graph_pattern_detector.cc:101] --- detected 2 subgraphs
I0812 01:19:46.782658 29227 graph_pattern_detector.cc:101] --- detected 12 subgraphs
--- Running IR pass [multihead_matmul_fuse_pass_v2]
I0812 01:19:46.795693 29227 graph_pattern_detector.cc:101] --- detected 6 subgraphs
Segmentation fault (core dumped)

-core堆栈信息：
(gdb) bt
#0 0x00007f2c08903104 in paddle::framework::LoDTensor* paddle::framework::Variable::GetMutablepaddle::framework::LoDTensor()
() from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#1 0x00007f2c08a60595 in paddle::framework::ir::patterns::BuildFusionV2(paddle::framework::ir::Graph*, std::string const&, paddle::framework::Scope*)::{lambda(std::unordered_map<paddle::framework::ir::PDNode*, paddle::framework::ir::Node*, std::hashpaddle::framework::ir::PDNode*, std::equal_topaddle::framework::ir::PDNode*, std::allocator<std::pair<paddle::framework::ir::PDNode* const, paddle::framework::ir::Node*> > > const&, paddle::framework::ir::Graph*)#2}::operator()(std::unordered_map<paddle::framework::ir::PDNode*, paddle::framework::ir::Node*, std::hashpaddle::framework::ir::PDNode*, std::equal_topaddle::framework::ir::PDNode*, std::allocator<std::pair<paddle::framework::ir::PDNode* const, paddle::framework::ir::Node*> > > const&, paddle::framework::ir::Graph*) const [clone .isra.781] () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#2 0x00007f2c0b63a135 in paddle::framework::ir::GraphPatternDetector::operator()(paddle::framework::ir::Graph*, std::function<void (std::unordered_map<paddle::framework::ir::PDNode*, paddle::framework::ir::Node*, std::hashpaddle::framework::ir::PDNode*, std::equal_topaddle::framework::ir::PDNode*, std::allocator<std::pair<paddle::framework::ir::PDNode* const, paddle::framework::ir::Node*> > > const&, paddle::framework::ir::Graph*)>) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#3 0x00007f2c08a598ef in paddle::framework::ir::MultiHeadMatmulV2FusePass::ApplyImpl(paddle::framework::ir::Graph*) const ()
from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#4 0x00007f2c0b64b325 in paddle::framework::ir::Pass::Apply(paddle::framework::ir::Graph*) const ()
from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#5 0x00007f2c0b599ba6 in paddle::inference::analysis::IRPassManager::Apply(std::unique_ptr<paddle::framework::ir::Graph, std::default_deletepaddle::framework::ir::Graph >) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#6 0x00007f2c0b580fa9 in paddle::inference::analysis::IrAnalysisPass::RunImpl(paddle::inference::analysis::Argument*) ()
from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#7 0x00007f2c0b4fc4f5 in paddle::inference::analysis::Analyzer::RunAnalysis(paddle::inference::analysis::Argument*) ()
from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#8 0x00007f2c08926394 in paddle::AnalysisPredictor::OptimizeInferenceProgram() ()
from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#9 0x00007f2c08926cef in paddle::AnalysisPredictor::PrepareProgram(std::shared_ptrpaddle::framework::ProgramDesc const&) ()
from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#10 0x00007f2c08926e6f in paddle::AnalysisPredictor::Init(std::shared_ptrpaddle::framework::Scope const&, std::shared_ptrpaddle::framework::ProgramDesc const&) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#11 0x00007f2c08927075 in std::unique_ptr<paddle::PaddlePredictor, std::default_deletepaddle::PaddlePredictor > paddle::CreatePaddlePredictor<paddle::AnalysisConfig, (paddle::PaddleEngineKind)2>(paddle::AnalysisConfig const&) ()
from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#12 0x00007f2c08927231 in std::unique_ptr<paddle::PaddlePredictor, std::default_deletepaddle::PaddlePredictor > paddle::CreatePa--Type for more, q to quit, c to continue without paging--
ddlePredictorpaddle::AnalysisConfig(paddle::AnalysisConfig const&) ()
from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so
#13 0x0000000000696874 in tarot::agent::PaddleEngine::init (this=, root_path=..., config=...)
at /home/opt/compiler/gcc-8.2/gcc-8.2/include/c++/8.2.0/bits/unique_ptr.h:342
#14 0x0000000000686979 in tarot::agent::Model::init (this=this@entry=0x7ffdc0ea7b70, root_path=...)
at bc_out/baidu/model-service/tarot/src/proto/tarot_config.pb.h:5248
#15 0x0000000000481a75 in tarot::run_infer (batch_size=32) at baidu/model-service/tarot/src/tools/local_inference_tool.cpp:355
#16 0x0000000000463378 in main (argc=, argv=)
at baidu/model-service/tarot/src/tools/local_inference_tool.cpp:508

[chenjiaoAngel]

这个看起来可能是模型版本不兼容引起的

[paddle-bot-old]

Since you haven't replied for more than a year, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您超过一年未回复，我们将关闭这个issue/pr。
若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。