Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ bug ] signal handler fails to gracefully shutdown svc when worker_limit() called #354

Closed
whatdoineed2do opened this issue Nov 30, 2018 · 6 comments
Closed

[ bug ] signal handler fails to gracefully shutdown svc when worker_limit() called #354

whatdoineed2do opened this issue Nov 30, 2018 · 6 comments
Assignees
@ben-crowhurst
Labels
bug
Projects
Restbed
Milestone
4.7

Comments

@whatdoineed2do
Copy link

whatdoineed2do commented Nov 30, 2018 •
edited
Loading

TLDR:

calling Service::stop() within a signal handler registered via Service::set_signal_handler() works ONLY if the service was started with a Settings object that has NOT had Settings::set_worker_limit() applied

I'm using what appears to be latest (a1e9c97) code but I observe that I am unable to gracefully stop the service if settings objects has a call to set_worker_limit(). See sample code.

Running with ./a.out 0 (no call to set_worker_limit()) we can hit control-C and the service is shutdown.
Running wtih ./a.out <anything even 0> produces this bug

linux, 4.18.14, g++ 8.2.1

#include <exception>
#include <thread>
#include <memory>
#include <cstdlib>
#include <restbed>
#include <csignal>
#include <sys/types.h>

#include <unistd.h>

using namespace std;
using namespace restbed;


int main(int argc, const char* argv[])
{
    int  workers = 1;
    if (argc == 2) {
        string  tmp = argv[1];
        if (tmp == "max") {
            workers = thread::hardware_concurrency();
        }
        else {
            try {
                workers = std::stoi(tmp);
            }
            catch (...) {
                fprintf( stderr, "invalid #worker `%s`\n", tmp.c_str());
                return -1;
            }
        }
    }

    auto settings = make_shared< Settings >( );
    settings->set_port( 1984 );
    if (workers > 0) {
        settings->set_worker_limit(workers);
    }

    Service service;
    service.set_ready_handler([&settings](Service& svc) {
        fprintf( stderr, "Service %x (workers=%u) PID='%i'.\n", &svc, settings->get_worker_limit(), getpid());
    });
    service.set_signal_handler( SIGINT, [&service](const int signo) {
        fprintf( stderr, "Received SIGINT signal number '%i'.\n", signo);
        service.stop();
    });
    try
    {
        service.start( settings );
        fprintf( stderr, "svc stopped\n");
    }
    catch (const std::exception& ex)
    {
        fprintf( stderr, "%s\n", ex.what());
    }

    return EXIT_SUCCESS;
}

code modelled after: #223

pstack of process after control-c received

Thread 2 (Thread 0x7f463a981700 (LWP 5157)):
#0  0x00007f463ac95879 in syscall () from /lib64/libc.so.6
#1  0x00007f463b5c91e1 in std::__atomic_futex_unsigned_base::_M_futex_wait_until(unsigned int*, unsigned int, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /lib64/libstdc++.so.6
#2  0x00007f463bb22f44 in std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until(unsigned int, unsigned int, bool, std::memory_order, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /usr/local/lib64/librestbed.so.4
#3  0x00007f463bb21fb5 in std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test(unsigned int, unsigned int, bool, std::memory_order) () from /usr/local/lib64/librestbed.so.4
#4  0x00007f463bb20aab in std::__future_base::_State_baseV2::wait() () from /usr/local/lib64/librestbed.so.4
#5  0x00007f463bb38707 in std::__basic_future<void>::_M_get_result() const () from /usr/local/lib64/librestbed.so.4
#6  0x00007f463bb38210 in std::future<void>::get() () from /usr/local/lib64/librestbed.so.4
#7  0x00007f463bb31486 in restbed::Service::stop() () from /usr/local/lib64/librestbed.so.4
#8  0x0000000000404ac0 in main::{lambda(int)#2}::operator()(int) const ()
#9  0x00000000004050fe in std::_Function_handler<void (int), main::{lambda(int)#2}>::_M_invoke(std::_Any_data const&, int&&) ()
#10 0x00007f463bad0da6 in std::function<void (int)>::operator()(int) const () from /usr/local/lib64/librestbed.so.4
#11 0x00007f463bac32eb in restbed::detail::ServiceImpl::signal_handler(std::error_code const&, int) const () from /usr/local/lib64/librestbed.so.4
#12 0x00007f463bae1979 in void std::__invoke_impl<void, void (restbed::detail::ServiceImpl::*&)(std::error_code const&, int) const, restbed::detail::ServiceImpl*&, std::error_code const&, int const&>(std::__invoke_memfun_deref, void (restbed::detail::ServiceImpl::*&)(std::error_code const&, int) const, restbed::detail::ServiceImpl*&, std::error_code const&, int const&) () from /usr/local/lib64/librestbed.so.4
#13 0x00007f463bae15cf in std::__invoke_result<void (restbed::detail::ServiceImpl::*&)(std::error_code const&, int) const, restbed::detail::ServiceImpl*&, std::error_code const&, int const&>::type std::__invoke<void (restbed::detail::ServiceImpl::*&)(std::error_code const&, int) const, restbed::detail::ServiceImpl*&, std::error_code const&, int const&>(void (restbed::detail::ServiceImpl::*&)(std::error_code const&, int) const, restbed::detail::ServiceImpl*&, std::error_code const&, int const&) () from /usr/local/lib64/librestbed.so.4
#14 0x00007f463bae1246 in void std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>::__call<void, std::error_code const&, int const&, 0ul, 1ul, 2ul>(std::tuple<std::error_code const&, int const&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) () from /usr/local/lib64/librestbed.so.4
#15 0x00007f463bae0ea0 in void std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>::operator()<std::error_code const&, int const&, void>(std::error_code const&, int const&) () from /usr/local/lib64/librestbed.so.4
#16 0x00007f463bae046f in asio::detail::binder2<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>, std::error_code, int>::operator()() () from /usr/local/lib64/librestbed.so.4
#17 0x00007f463badf8ab in void asio::asio_handler_invoke<asio::detail::binder2<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>, std::error_code, int> >(asio::detail::binder2<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>, std::error_code, int>&, ...) () from /usr/local/lib64/librestbed.so.4
#18 0x00007f463bade566 in void asio_handler_invoke_helpers::invoke<asio::detail::binder2<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>, std::error_code, int>, std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const> >(asio::detail::binder2<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>, std::error_code, int>&, std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>&) () from /usr/local/lib64/librestbed.so.4
#19 0x00007f463badcc7b in void asio::detail::handler_work<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>, asio::system_executor>::complete<asio::detail::binder2<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>, std::error_code, int> >(asio::detail::binder2<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>, std::error_code, int>&, std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const>&) () from /usr/local/lib64/librestbed.so.4
#20 0x00007f463bada221 in asio::detail::signal_handler<std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, int) const> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) () from /usr/local/lib64/librestbed.so.4
#21 0x00007f463baca116 in asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) () from /usr/local/lib64/librestbed.so.4
#22 0x00007f463bafd939 in asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&) () from /usr/local/lib64/librestbed.so.4
#23 0x00007f463bafd584 in asio::detail::scheduler::run(std::error_code&) () from /usr/local/lib64/librestbed.so.4
#24 0x00007f463bafdc67 in asio::io_context::run() () from /usr/local/lib64/librestbed.so.4
#25 0x00007f463bb3160f in restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}::operator()() const () from /usr/local/lib64/librestbed.so.4
#26 0x00007f463bb336e1 in void std::__invoke_impl<void, restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>(std::__invoke_other, restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}&&) () from /usr/local/lib64/librestbed.so.4
#27 0x00007f463bb3327f in std::__invoke_result<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>::type std::__invoke<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>(std::__invoke_result&&, (restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}&&)...) () from /usr/local/lib64/librestbed.so.4
#28 0x00007f463bb3770c in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) () from /usr/local/lib64/librestbed.so.4
#29 0x00007f463bb376c8 in std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >::operator()() () from /usr/local/lib64/librestbed.so.4
#30 0x00007f463bb3747f in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::operator()() const () from /usr/local/lib64/librestbed.so.4
#31 0x00007f463bb371a9 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void> >::_M_invoke(std::_Any_data const&) () from /usr/local/lib64/librestbed.so.4
#32 0x00007f463bb21477 in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const () from /usr/local/lib64/librestbed.so.4
#33 0x00007f463bb20d81 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) () from /usr/local/lib64/librestbed.so.4
#34 0x00007f463bb23061 in void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) () from /usr/local/lib64/librestbed.so.4
#35 0x00007f463bb22075 in std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) () from /usr/local/lib64/librestbed.so.4
#36 0x00007f463bb211e8 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const () from /usr/local/lib64/librestbed.so.4
#37 0x00007f463bb2121b in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const () from /usr/local/lib64/librestbed.so.4
#38 0x00007f463bb2122c in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () from /usr/local/lib64/librestbed.so.4
#39 0x00007f463a990ea7 in __pthread_once_slow () from /lib64/libpthread.so.0
#40 0x00007f463bb1bcaf in __gthread_once(int*, void (*)()) () from /usr/local/lib64/librestbed.so.4
#41 0x00007f463bb212d6 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) () from /usr/local/lib64/librestbed.so.4
#42 0x00007f463bb20b5f in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) () from /usr/local/lib64/librestbed.so.4
#43 0x00007f463bb36cb3 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}::operator()() const () from /usr/local/lib64/librestbed.so.4
#44 0x00007f463bb3737b in void std::__invoke_impl<void, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}>(std::__invoke_other, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}&&) () from /usr/local/lib64/librestbed.so.4
#45 0x00007f463bb37098 in std::__invoke_result<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}>::type std::__invoke<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}>(std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}&&, (std::__invoke_result&&)...) () from /usr/local/lib64/librestbed.so.4
#46 0x00007f463bb37ebc in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) () from /usr/local/lib64/librestbed.so.4
#47 0x00007f463bb37e12 in std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}> >::operator()() () from /usr/local/lib64/librestbed.so.4
#48 0x00007f463bb37a42 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}> >, void>::_Async_state_impl(std::tuple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}>&&)::{lambda()#1}> > >::_M_run() () from /usr/local/lib64/librestbed.so.4
#49 0x00007f463b5cb543 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#50 0x00007f463a989594 in start_thread () from /lib64/libpthread.so.0
#51 0x00007f463ac9ae6f in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f463bfebb00 (LWP 5156)):
#0  0x00007f463ac95879 in syscall () from /lib64/libc.so.6
#1  0x00007f463b5c917d in std::__atomic_futex_unsigned_base::_M_futex_wait_until(unsigned int*, unsigned int, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /lib64/libstdc++.so.6
#2  0x00007f463bb22f44 in std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until(unsigned int, unsigned int, bool, std::memory_order, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /usr/local/lib64/librestbed.so.4
#3  0x00007f463bb3f25c in unsigned int std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(unsigned int, unsigned int, bool, std::memory_order, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () from /usr/local/lib64/librestbed.so.4
#4  0x00007f463bb3a5df in std::future_status std::__future_base::_State_baseV2::wait_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) () from /usr/local/lib64/librestbed.so.4
#5  0x00007f463bb39088 in std::future_status std::__basic_future<void>::wait_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) const () from /usr/local/lib64/librestbed.so.4
#6  0x00007f463bb31650 in restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}::operator()(std::future<void>&) const () from /usr/local/lib64/librestbed.so.4
#7  0x00007f463bb33ef6 in bool __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>::operator()<__gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > > >(__gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > >) () from /usr/local/lib64/librestbed.so.4
#8  0x00007f463bb349f0 in __gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > > std::__find_if<__gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > >, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}> >(__gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>, std::random_access_iterator_tag) () from /usr/local/lib64/librestbed.so.4
#9  0x00007f463bb33ec7 in __gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > > std::__find_if<__gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > >, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}> >(__gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>) () from /usr/local/lib64/librestbed.so.4
#10 0x00007f463bb3386a in __gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > > std::__remove_if<__gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > >, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}> >(__gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>, __gnu_cxx::__ops::_Iter_pred<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>) () from /usr/local/lib64/librestbed.so.4
#11 0x00007f463bb3347a in __gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > > std::remove_if<__gnu_cxx::__normal_iterator<std::future<void>*, std::vector<std::future<void>, std::allocator<std::future<void> > > >, restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}>(restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}, restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}, restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda(std::future<void>&)#3}) () from /usr/local/lib64/librestbed.so.4
#12 0x00007f463bb31e9a in restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) () from /usr/local/lib64/librestbed.so.4
#13 0x0000000000404c8c in main ()
@katemamba
Copy link
Contributor

From little I know of restbed, Firstly for worker limit, It is the count of initial threadpool that your hardware system can support. It simply means that many threads will do scheduling in case of threads running simultaneously . For best results, I use

        unsigned int n = std::thread::hardware_concurrency();
        settings->set_worker_limit(500);

For me this is 8 on my machine, and 2 or 4 on other machines. Setting a value higher than this, will cause the application to make more threads ready and handle service accordingly, but may also consume time. Setting a value less than concurrency limit may also slow down or not use system to the fullest.
Same is the case when exiting, that many open threads need to close down efficiently/graciously.

You can try to use std::async(and encapsulate the service stop here in separate thread other than main with try catch). To see which all threads it tries to close. This is the same exact behaviour when no workers are set. See whether all of the worker threads are being closed down properly. Most probably this might narrow down problem to pthread or thread library which might cause the issue in your case(from stacktrace), else the settings need to fetch workers and close down one by one.

For referrence, [https://github.com/Corvusoft/restbed/blob/master/source/corvusoft/restbed/service.cpp#L113]
check whether all workers are closed.
For signal handler , write your own signal handler which takes in integer and not just SIGINT(SIGINT, SIGHUP)(2,3,8,9) 18 to suspend and 20 to continue

@whatdoineed2do
Copy link
Author

whatdoineed2do commented Dec 2, 2018 •
edited
Loading

@katemamba I'm not sure how your reply relates to the bug report. For clarity, let me restate:

calling Service::stop() within a signal handler registered via Service::set_signal_handler() works ONLY if the service was started with a Settings object that has NOT had Settings::set_worker_limit() applied

@whatdoineed2do whatdoineed2do changed the title [ bug? ] signal handler shutdown when worker_limit() called [ bug ] signal handler shutdown when worker_limit() called Dec 2, 2018
@whatdoineed2do whatdoineed2do changed the title [ bug ] signal handler shutdown when worker_limit() called [ bug ] signal handler fails to gracefully shutdown svc when worker_limit() called Dec 2, 2018
@ben-crowhurst
Copy link
Member

Sound like a fun bug 💃. Thanks for reporting will take a look.

@ben-crowhurst ben-crowhurst self-assigned this Dec 3, 2018
@KiNgMaR
Copy link

KiNgMaR commented Dec 28, 2018

The same thing also happens when stop() is called from a scheduled job. It looks like the worker threads are either not shut down at all, or some object instance is destroyed while the workers still need it.

@mipac
Copy link

mipac commented Jan 18, 2020

I have the same trouble
can't stop from sighandler->spawn thread->call service stop

if I don't set worker_limit it works

@ben-crowhurst ben-crowhurst added this to the 4.7 milestone Feb 8, 2020
@ben-crowhurst ben-crowhurst added this to To do in Restbed via automation Feb 8, 2020
ben-crowhurst added a commit that referenced this issue Feb 23, 2020
@ben-crowhurst
Resolved bug #354.
2549441
@ben-crowhurst
Copy link
Member

This was caused by the service->stop() method waiting indefinitely on all workers to halt.

        if ( m_pimpl->m_workers_stopped )
        {
            m_pimpl->m_workers_stopped->get( );
            m_pimpl->m_workers_stopped.reset( );
        }

We've now implemented a timeout of 1 second if the other workers don't respond in a timely fashion.

        if ( m_pimpl->m_workers_stopped )
        {
            m_pimpl->m_workers_stopped->wait_for( seconds( 1 ) );
            m_pimpl->m_workers_stopped.reset( );
        }

Restbed automation moved this from To do to Done Feb 23, 2020
maggu2810 pushed a commit to maggu2810/restbed that referenced this issue Mar 14, 2022
@ben-crowhurst
Resolved bug Corvusoft#354.
54e0774
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ben-crowhurst ben-crowhurst

Labels
bug
Projects
Restbed
  
Done
Milestone
4.7
Development

No branches or pull requests
5 participants
@KiNgMaR @ben-crowhurst @mipac @katemamba @whatdoineed2do