Skip to content

c++ onnxruntime run onnxruntime_global_thread_pools_test with use_dnnl core dump #12232

@zhu-dq

Description

@zhu-dq

Describe the bug
build onnxruntime v1.11.1 with use_dnnl,run onnxruntime_global_thread_pools_test core dump,when std::string destroy,program received signal SIGABRT, Aborted.

c++ stack

Program received signal SIGABRT, Aborted.
0x00007ffff53e73a7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.tl2.x86_64 libgcc-7.3.1-6.tl2.x86_64 libgomp-7.3.1-6.tl2.x86_64 libstdc++-7.3.1-6.tl2.x86_64
(gdb) bt
#0  0x00007ffff53e73a7 in raise () from /lib64/libc.so.6
#1  0x00007ffff53e8a98 in abort () from /lib64/libc.so.6
#2  0x00007ffff5429ef7 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff54326e9 in _int_free () from /lib64/libc.so.6
#4  0x00007fffec528536 in __gnu_cxx::new_allocator<char>::deallocate (this=0x7fffffffd917, __p=0x7ffff7dd6100 <std::string::_Rep::_S_empty_rep_storage> "") at /usr/include/c++/7/ext/new_allocator.h:125
#5  0x00007fffec522c58 in std::string::_Rep::_M_destroy (this=0x7ffff7dd6100 <std::string::_Rep::_S_empty_rep_storage>, __a=...) at /usr/include/c++/7/bits/basic_string.tcc:899
#6  0x00007fffec51e0c0 in std::string::_Rep::_M_dispose (this=0x7ffff7dd6100 <std::string::_Rep::_S_empty_rep_storage>, __a=...) at /usr/include/c++/7/bits/basic_string.h:3251
#7  0x00007fffec51aebc in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x7fffffffda28, __in_chrg=<optimized out>) at /usr/include/c++/7/bits/basic_string.h:3621
#8  0x00007fffec51714f in onnxruntime::DNNLExecutionProvider::DNNLExecutionProvider (this=0xbe2c40, info=...) at /data/home/zdq/onnx/v1.11.1/onnxruntime/onnxruntime/core/providers/dnnl/dnnl_execution_provider.cc:55
#9  0x00007fffec54a41f in std::make_unique<onnxruntime::DNNLExecutionProvider, onnxruntime::DNNLExecutionProviderInfo&> (__args#0=...) at /usr/include/c++/7/bits/unique_ptr.h:825
#10 0x00007fffec54a2cc in onnxruntime::DnnlProviderFactory::CreateProvider (this=0xa1dd70) at /data/home/zdq/onnx/v1.11.1/onnxruntime/onnxruntime/core/providers/dnnl/dnnl_provider_factory.cc:27
#11 0x00007ffff66b3026 in (anonymous namespace)::InitializeSession (options=0xa0ddc0, sess=std::unique_ptr<onnxruntime::InferenceSession> containing 0xa15d10, prepacked_weights_container=0x0)
    at /data/home/zdq/onnx/v1.11.1/onnxruntime/onnxruntime/core/session/onnxruntime_c_api.cc:675
#12 0x00007ffff66b3477 in OrtApis::CreateSession (env=0x91af30, model_path=0x466d00 "testdata/squeezenet/model.onnx", options=0xa0ddc0, out=0x7fffffffdde0) at /data/home/zdq/onnx/v1.11.1/onnxruntime/onnxruntime/core/session/onnxruntime_c_api.cc:708
#13 0x000000000040aba5 in Ort::Session::Session (this=0x7fffffffdde0, env=..., model_path=0x466d00 "testdata/squeezenet/model.onnx", options=...) at /data/home/zdq/onnx/v1.11.1/onnxruntime/include/onnxruntime/core/session/onnxruntime_cxx_inline.h:552
#14 0x000000000040d634 in GetSessionObj<char const*, float> (env=..., model_uri=0x466d00 "testdata/squeezenet/model.onnx", provider_type=2) at /data/home/zdq/onnx/v1.11.1/onnxruntime/onnxruntime/test/global_thread_pools/test_inference.cc:94
#15 0x000000000040b81c in CApiTestGlobalThreadPoolsWithProvider_simple_Test::TestBody (this=0x9197f0) at /data/home/zdq/onnx/v1.11.1/onnxruntime/onnxruntime/test/global_thread_pools/test_inference.cc:151
#16 0x000000000045486c in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x9197f0, method=&virtual testing::Test::TestBody(), location=0x4692eb "the test body")
    at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:2607
#17 0x000000000044cdac in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x9197f0, method=&virtual testing::Test::TestBody(), location=0x4692eb "the test body")
    at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:2643
#18 0x000000000042c130 in testing::Test::Run (this=0x9197f0) at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:2682
#19 0x000000000042c9f0 in testing::TestInfo::Run (this=0x6ba090) at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:2861
#20 0x000000000042d222 in testing::TestSuite::Run (this=0x6b9c20) at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:3015
#21 0x000000000043c089 in testing::internal::UnitTestImpl::RunAllTests (this=0x6b8780) at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:5855
#22 0x0000000000455953 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x6b8780, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x43bcac <testing::internal::UnitTestImpl::RunAllTests()>, location=0x469ca0 "auxiliary test code (environments or event listeners)")
    at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:2607
#23 0x000000000044ddd8 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x6b8780, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x43bcac <testing::internal::UnitTestImpl::RunAllTests()>, location=0x469ca0 "auxiliary test code (environments or event listeners)")
    at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:2643
#24 0x000000000043a892 in testing::UnitTest::Run (this=0x6a09a0 <testing::UnitTest::GetInstance()::instance>) at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/src/gtest.cc:5438
#25 0x00000000004078d0 in RUN_ALL_TESTS () at /data/home/zdq/onnx/v1.11.1/onnxruntime/cmake/external/googletest/googletest/include/gtest/gtest.h:2490
#26 0x0000000000407e59 in main (argc=1, argv=0x7fffffffe378) at /data/home/zdq/onnx/v1.11.1/onnxruntime/onnxruntime/test/global_thread_pools/test_main.cc:92

Urgency
The project uses onnx, a large number of matrix calculations, and hopes to introduce onednn to improve service performance. With reference to onnxruntime_global_thread_pools_test, it is found that there are core dump, and I hope to give some suggestions.

System information

  • OS Platform : CentOS Linux release 7.2 (Final)
  • ONNX Runtime installed from (source or binary): source
  • ONNX Runtime version: v1.11.1
  • Python version: 2.7.5 and 3.6.8
  • Visual Studio version (if applicable):
  • GCC/Compiler version (if compiling from source): gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-6)
  • CUDA/cuDNN version: no
  • GPU model and memory: no
  • cmake : cmake version 3.22.5

To Reproduce

1. git clone --depth=1 --recursive --branch v1.11.1 https://github.com/microsoft/onnxruntime.git
2. cd onnxruntime/
3. ./build.sh --use_dnnl --build_shared_lib --build_dir build/Linux.arm
4. cd build/Linux.arm/Debug/
5. gdb onnxruntime_global_thread_pools_test
6. (gdb) r
7. (gdb) bt

Expected behavior
use in cpu scenarios,hope onnxruntime_global_thread_pools_test pass.

Other info

  1. build is ok (--skip_tests).
  2. I did a test, and when the std::string is released elsewhere in the dynamic library, it will also core dump. Not sure if it has something to do with c++ abi, hope to get some advice.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions