storaged cannot be exited gracefully

[HarrisChu]

#3475

After this, cannot be exited gracefully, not sure if it depends on heartbeat thread.

pstrack log as below:

Thread 3 (Thread 0x7fc024fdd700 (LWP 32381)):
#0  0x00007fc034d760e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000021fc345 in ?? ()
#2  0x00000000021f30e8 in event_base_loop ()
#3  0x000000000212a466 in folly::EventBase::loopBody(int, bool) ()
#4  0x000000000212a8ff in folly::EventBase::loop() ()
#5  0x000000000212ca04 in folly::EventBase::loopForever() ()
#6  0x00000000020be8e9 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x00000000020cae27 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000026c2570 in execute_native_thread_routine ()
#9  0x00007fc03504cea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fc034d75b0d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fc0247dc700 (LWP 32390)):
#0  0x00007fc034d760e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000021fc345 in ?? ()
#2  0x00000000021f30e8 in event_base_loop ()
#3  0x000000000212a466 in folly::EventBase::loopBody(int, bool) ()
#4  0x000000000212a8ff in folly::EventBase::loop() ()
#5  0x000000000212ca04 in folly::EventBase::loopForever() ()
#6  0x00000000020be8e9 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x00000000020cae27 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000026c2570 in execute_native_thread_routine ()
#9  0x00007fc03504cea5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fc034d75b0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fc035b969c0 (LWP 32145)):
#0  0x00007fc034d3c9fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fc034d3c894 in sleep () from /lib64/libc.so.6
#2  0x000000000122bf0b in nebula::meta::MetaClient::waitForMetadReady(int, int) ()
#3  0x0000000000f6e18a in nebula::storage::StorageServer::start() ()
#4  0x0000000000f22a59 in main ()

log.log

[darionyaphet]

it seems interrupted ::sleep(retryIntervalSecs)?

[HarrisChu]

if the storage is not registered, isMetadReady() is always false, and then could not be exited gracefully

I20211231 15:31:17.548487 21989 MetaClient.cpp:654] Send request to meta "127.0.0.1":19663
I20211231 15:31:17.549737 21989 AsyncSocket.cpp:2711] this=0x7f8c03026000, AsyncSocket::handleRead() got 14 bytes
I20211231 15:31:17.550246 21989 AsyncSocket.cpp:2711] this=0x7f8c03026000, AsyncSocket::handleRead() got 41 bytes
E20211231 15:31:17.550576 21513 MetaClient.cpp:75] Heartbeat failed, status:Machine not existed!
I20211231 15:31:17.550635 21513 MetaClient.cpp:105] Waiting for the metad to be ready!

[critical27]

Not related to #3475, bug confirmed, #3437