"terminate called without an active exception" in default Pipeline constructor

[ejaybennett]

I have tried reinstalling depthai-core (2.15.4 ), reconfiguring and rebuilding, as well as powercycling my computer and all attatched devices. Code fails when calling default constructor of dai::Pipeline, but I haven't found anyone else encountering this error. Also not a consistent error, this code ran fine then stopped working. Ran into this in the past, and it randomly stopped working. Our application needs high reliability, so this isn't an acceptable problem.
Error:
terminate called without an active exception
terminate called recursively

Thread 1 "Inference" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

Code Snippet:
// Create pipeline
dai::Pipeline pipeline;
std::string mxID = config->get_camera_order()[cam.idx];
std::string nnPath = config->get_engine_path();
// Print which blob we are using
printf("Using blob at path: %s\n", nnPath.c_str());

// Define sources and outputs
auto camRgb = pipeline.create<dai::node::ColorCamera>();
auto detectionNetwork = pipeline.create<dai::node::YoloDetectionNetwork>();
std::shared_ptr<dai::node::XLinkOut> xoutRgb;
if(get_frames){
    xoutRgb = pipeline.create<dai::node::XLinkOut>();
    xoutRgb->setStreamName("rgb");
}
//auto 
auto nnOut = pipeline.create<dai::node::XLinkOut>();

Backtrace:

(gdb) bt
#0  0x0000007fa2ece598 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x0000007fa2ecf974 in __GI_abort () at abort.c:79
#2  0x0000007fa316507c in __gnu_cxx::__verbose_terminate_handler() () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#3  0x0000007fa3162c54 in  () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x0000007fa3162ca0 in  () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x00000000004fe8b8 in std::thread::~thread() (this=0x7fffffcef0, __in_chrg=<optimized out>) at /usr/include/c++/7/thread:135
#6  0x00000000006bd4d8 in dai::Resources::Resources() (this=0x10ad638 <dai::Resources::getInstance()::instance>)
    at /home/sprayer/depthai-core/src/utility/Resources.cpp:355
#7  0x00000000006bd340 in dai::Resources::getInstance() () at /home/sprayer/depthai-core/src/utility/Resources.cpp:248
#8  0x00000000006b1748 in dai::<lambda()>::operator()(void) const (__closure=0x7fffffd038)
    at /home/sprayer/depthai-core/src/utility/Initialization.cpp:83
#9  0x00000000006b1914 in dai::initialize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) (additionalInfo="", in---Type <return> to continue, or q <return> to quit---
stallSignalHandler=true) at /home/sprayer/depthai-core/src/utility/Initialization.cpp:98
#10 0x00000000005c9a54 in dai::Pipeline::Pipeline() (this=0x7fffffdd40) at /home/sprayer/depthai-core/src/pipeline/Pipeline.cpp:21
#11 0x000000000043544c in CameraRunner::run() (this=0x1364be0) at /home/sprayer/Downloads/ComputeUnit/src/depthai_wrapper.cpp:123
#12 0x0000000000452e44 in main(int, char**) (argc=4, argv=0x7fffffecc8) at /home/sprayer/Downloads/ComputeUnit/src/main.cpp:243

Architecture/OS
Jetpack 4.5.1
Jetson TX2

[themarpe]

@ejaybennett

Thanks for the report. We haven't observed this issue on 2.15.4 yet.

By the looks of the backtrace, it seems like the lazyThreadDevice was already existent, causing a destructor to fail... But not sure how it could get into this state. Just to verify, can you add the following to that snippet and retest (Resources.cpp:355):

    // Device resources
    // Create a thread which lazy-loads firmware resources package
    if(lazyThreadDevice.joinable()) {
          printf("joinable thread in constructor\n");
          lazyThreadDevice.join();
    }
    lazyThreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMapDevice));

Also, can you post a log with DEPTHAI_LEVEL=debug env var set, when this happens?

[ejaybennett]

I got the following error while trying this:
~/depthai-core$ cmake --build build
[ 9%] Built target depthai-resources
[ 11%] Building CXX object CMakeFiles/depthai-core.dir/src/utility/Resources.cpp.o
/home/sprayer/depthai-core/src/utility/Resources.cpp: In lambda function:
/home/sprayer/depthai-core/src/utility/Resources.cpp:337:8: error: ‘lazyThreadDevice’ was not declared in this scope
if(lazyThreadDevice.joinable()) {
^~~~~~~~~~~~~~~~
/home/sprayer/depthai-core/src/utility/Resources.cpp:341:5: error: ‘lazyThreadDevice’ was not declared in this scope
lazyThreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMap));
^~~~~~~~~~~~~~~~
/home/sprayer/depthai-core/src/utility/Resources.cpp:341:57: error: ‘mtxDevice’ was not declared in this scope
lazyThreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMap));
^~~~~~~~~
/home/sprayer/depthai-core/src/utility/Resources.cpp:341:57: note: suggested alternative: ‘Device’
lazyThreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMap));
^~~~~~~~~
Device
/home/sprayer/depthai-core/src/utility/Resources.cpp:341:68: error: ‘cvDevice’ was not declared in this scope
lazyThreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMap));
^~~~~~~~
/home/sprayer/depthai-core/src/utility/Resources.cpp:341:68: note: suggested alternative: ‘Device’
lazyThreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMap));
^~~~~~~~
Device
/home/sprayer/depthai-core/src/utility/Resources.cpp:341:78: error: ‘readyDevice’ was not declared in this scope
hreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMap));
^~~~~~~~~~~
/home/sprayer/depthai-core/src/utility/Resources.cpp:341:78: note: suggested alternative: ‘Device’
hreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMap));
^~~~~~~~~~~
Device
CMakeFiles/depthai-core.dir/build.make:803: recipe for target 'CMakeFiles/depthai-core.dir/src/utility/Resources.cpp.o' failed
make[2]: *** [CMakeFiles/depthai-core.dir/src/utility/Resources.cpp.o] Error 1
CMakeFiles/Makefile2:114: recipe for target 'CMakeFiles/depthai-core.dir/all' failed
make[1]: *** [CMakeFiles/depthai-core.dir/all] Error 2
Makefile:145: recipe for target 'all' failed
make: *** [all] Error 2

[ejaybennett]

Sorry: Realized I added it to the wrong places. I added it to the file in the right place, and now get the following error when running:

// Device resources
// Create a thread which lazy-loads firmware resources package
if(lazyThreadDevice.joinable()) {
      printf("joinable thread in constructor\n");
      lazyThreadDevice.join();
}
lazyThreadDevice = std::thread(getLazyTarXzFunction(mtxDevice, cvDevice, readyDevice, CMRC_DEPTHAI_DEVICE_TAR_XZ, RESOURCE_LIST_DEVICE, resourceMapDevice));

joinable thread in constructor
terminate called after throwing an instance of 'std::system_error'
what(): Invalid argument

Thread 1 "Inference" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x0000007fa2ece598 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x0000007fa2ecf974 in __GI_abort () at abort.c:79
#2 0x0000007fa316510c in __gnu_cxx::__verbose_terminate_handler() () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#3 0x0000007fa3162c54 in () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#4 0x0000007fa3162ca0 in () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#5 0x0000007fa3162f68 in __cxa_rethrow () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#6 0x0000007fa318de2c in std::__throw_system_error(int) () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#7 0x0000007fa318e144 in () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#8 0x00000000006bd494 in dai::Resources::Resources() (this=0x10ad638 dai::Resources::getInstance()::instance)
at /home/sprayer/depthai-core/src/utility/Resources.cpp:357
#9 0x00000000006bd340 in dai::Resources::getInstance() () at /home/sprayer/depthai-core/src/utility/Resources.cpp:248
#10 0x00000000006b1748 in dai::<lambda()>::operator()(void) const (__closure=0x7fffffd038)
---Type to continue, or q to quit---
at /home/sprayer/depthai-core/src/utility/Initialization.cpp:83
#11 0x00000000006b1914 in dai::initialize(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool) (additionalInfo="", installSignalHandler=true) at /home/sprayer/depthai-core/src/utility/Initialization.cpp:98
#12 0x00000000005c9a54 in dai::Pipeline::Pipeline() (this=0x7fffffdd40) at /home/sprayer/depthai-core/src/pipeline/Pipeline.cpp:21
#13 0x000000000043544c in CameraRunner::run() (this=0x1364be0) at /home/sprayer/Downloads/ComputeUnit/src/depthai_wrapper.cpp:123
#14 0x0000000000452e44 in main(int, char**) (argc=4, argv=0x7fffffecc8) at /home/sprayer/Downloads/ComputeUnit/src/main.cpp:243

[themarpe]

@ejaybennett

That is odd - can you share a MRE, so I can try reproducing over at my end?
And a run with DEPTHAI_LEVEL=debug env var set, just if there is anything obvious there..

[ejaybennett]

I have been able to fix this error: When I recompiled my project with GCC 8.4 instead of Clang 6.0, the error went away

[themarpe]

Thanks for the data point @ejaybennett . Did you happen to compile the library and your project with different compilers?