Program terminates during error handling

[chinz07]

While debugging the following code snippet

#include <hpx/hpx_main.hpp>

#include <hpx/include/thread_executors.hpp>

int main()
{
    hpx::threads::executors::local_priority_queue_os_executor exec(1);

    return 0;
}

I encountered what seems to be a bug in the error handling path. The program aborts with the error

terminate called after throwing an instance of 'hpx::detail::exception_with_info<hpx::exception>'
  what():  suspending thread while at least one lock is being held, stack backtrace: 25 frames:

during unwinding.
The emitted stack trace is:

0x7ffff6cf92a4  : hpx::detail::backtrace_direct[abi:cxx11](unsigned long) + 0xb4 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff741caf1  : hpx::util::verify_no_locks() + 0xf1 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff7299096  : hpx::this_thread::suspend(hpx::threads::thread_state_enum, boost::intrusive_ptr<hpx::threads::thread_data> const&, hpx::util::thread_description const&, hpx::error_code&) + 0xa6 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff7290daf  : hpx::this_thread::yield_to(hpx::thread::id) + 0x3f in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff73ab541  : hpx::util::backtrace::trace_on_new_stack[abi:cxx11]() const + 0x1e1 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff6cf90fa  : hpx::detail::backtrace[abi:cxx11](unsigned long) + 0x5a in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff6d0e56d  : std::__exception_ptr::exception_ptr hpx::detail::get_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xbd in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff6d0eafc  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) + 0x6c in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff6d67340  : hpx::detail::throw_exception(hpx::error, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) + 0x60 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff71e64a9  : ??? + 0x7ffff71e64a9 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff71e7061  : hpx::resource::detail::partitioner::get_pool_data(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x221 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff71e77a1  : hpx::resource::detail::partitioner::assign_pu(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long) + 0x101 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff72127d5  : hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::add_processing_unit(unsigned long, unsigned long, std::shared_ptr<hpx::compat::barrier>, hpx::error_code&) + 0x55 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff72270f0  : hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::run(std::unique_lock<std::mutex>&, unsigned long) + 0x4d0 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff7276036  : ??? + 0x7ffff7276036 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff7272d6f  : hpx::threads::executors::local_priority_queue_os_executor::local_priority_queue_os_executor(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x2f in /opt/hpx-debug/lib/libhpxd.so.1
0x555555557fb1  : ??? + 0x555555557fb1 in /home/christopher/test/hpx/thread_pool_executor_test/cmake-build-debug/thread_pool_executor_test
0x55555555b850  : ??? + 0x55555555b850 in /home/christopher/test/hpx/thread_pool_executor_test/cmake-build-debug/thread_pool_executor_test
0x55555555b6b7  : ??? + 0x55555555b6b7 in /home/christopher/test/hpx/thread_pool_executor_test/cmake-build-debug/thread_pool_executor_test
0x55555555b168  : ??? + 0x55555555b168 in /home/christopher/test/hpx/thread_pool_executor_test/cmake-build-debug/thread_pool_executor_test
0x7ffff6d5234f  : hpx::runtime_impl::run_helper(hpx::util::function<int (), false>, int&) + 0x3af in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff6d531cc  : ??? + 0x7ffff6d531cc in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff71f3d28  : hpx::threads::coroutines::detail::coroutine_impl::operator()() + 0xc8 in /opt/hpx-debug/lib/libhpxd.so.1
0x7ffff71f6c7a  : ??? + 0x7ffff71f6c7a in /opt/hpx-debug/lib/libhpxd.so.1: HPX(invalid_status)

Since the original issue does not seem to be related to this bug, I will report it separately.

[hkaiser]

What would you expect to happen if an exception is thrown that is not handled?

[chinz07]

Even if I handle the exception in main, the issue persists since the exception handler is never reached during unwinding. At least the debugger never stops at the break point I have placed within the handler. If I continue the execution after the exception has been thrown the debugger only stops within the destructor of boost::intrusive_ptr<hpx::threads::thread_data> due to receiving SIGABRT, prints the stack trace and aborts the program immediately. Since throwing an exception manually within the same try block triggers the handler, it would surprise me if the code is working as intended.

Moreover, the error message is not the same as the one provided by the original exception. Since no other exceptions are thrown during the unwinding, I would except that the program displays the original error message even if I would not handle the exception.

I have also recompiled both HPX and the example in a fresh Docker container to avoid issues with the ABI and get the same results.

[hkaiser]

Could you please give us a small self-contained test that exposes the issue you're reporting on? The code in the original description does not expose any problems.

[chinz07]

That's weird. I can reproduce the issue with the provided code on my system every time. I use the following CMakeLists.txt to compile the code:

cmake_minimum_required(VERSION 3.9)
project(thread_pool_executor_test)

find_package(HPX 1.1.0 REQUIRED)

set(SOURCE_FILES main.cpp)
add_executable(thread_pool_executor_test ${SOURCE_FILES})
hpx_setup_target(thread_pool_executor_test)

Which compiler have you used to test the code? Maybe the issue is exposed by a specific combination of compiler, flags and libraries.

I will investigate, try to come up with a reduced test case and report back ASAP.

[hkaiser]

What I meant is not that I don't see the exception, what I meant is that the code above behaves as it should (if we abstract ourselves from the underlying issue in the executor). The code throws an exception and the program terminates as the exception is not handled. Could you please give us some code which actually demonstrates the issue you're reporting?

[chinz07]

I misunderstood your comment. Sorry.

The following code crashes while I would expect it to terminate normally after printing "Failure" since the exception should be caught.

#include <hpx/hpx_main.hpp>

#include <hpx/include/thread_executors.hpp>

#include <iostream>

int main()
{
    try
    {
        hpx::threads::executors::local_priority_queue_os_executor exec(1);
    }
    catch (...)
    {
        std::cerr << "Failure" << std::endl;
    }

    return 0;
}