Skip to content
This repository has been archived by the owner on Jan 7, 2022. It is now read-only.

IsLogEmptyTest.SimpleGracePeriod flaking on LogDevice-OSS #30

Closed
calebmarchent opened this issue Oct 4, 2018 · 1 comment
Closed

IsLogEmptyTest.SimpleGracePeriod flaking on LogDevice-OSS #30

calebmarchent opened this issue Oct 4, 2018 · 1 comment
Labels
bug Something isn't working

Comments

@calebmarchent
Copy link

calebmarchent commented Oct 4, 2018 •
edited

Circle CI has reported 2 incidents of IsLogEmptyTest.SimpleGracePeriod failing recently. Need to debug.

Incidents:
https://circleci.com/gh/facebookincubator/LogDevice/852
https://circleci.com/gh/facebookincubator/LogDevice/809

Running 100 cycles of the test:

rm -f passed failed
for i in {1..100}; do \
    test/integration_test --gtest_filter=IsLogEmptyTest.SimpleGracePeriod \
        && echo "${i}" >> passed || echo "${i}" >> failed; \
done

Now run 1000 cycles without reproducing; will probably need to look at the specific failures.
@calebmarchent calebmarchent added the bug Something isn't working label Oct 4, 2018
@calebmarchent
Copy link
Author

We've been running this test in unit test in builds regularly for a while now; it is no longer failing.
Closing.

facebook-github-bot pushed a commit that referenced this issue Jun 21, 2019
@al13n321 @facebook-github-bot
Fix use-after-free in AllServerReadStreams::sendShardStatusToClient()
8215d91 
Summary:
Got a segfault with this stack trace:

  (gdb) bt
  #0  0x00007ff56c2919cf in raise () from /usr/local/fbcode/platform007/lib/libpthread.so.0
  #1  0x0000000002a9656a in facebook::logdevice::handle_fatal_signal (sig=<optimized out>) at logdevice/server/fatalsignal.cpp:58
  #2  <signal handler called>
  #3  facebook::logdevice::configuration::Node::getNumShards (this=0x7fe60c71c610) at logdevice/common/configuration/Node.h:258
  #4  facebook::logdevice::AllServerReadStreams::sendShardStatusToClient (this=<optimized out>, cid=...) at logdevice/server/read_path/AllServerReadStreams.cpp:769
  #5  0x0000000002a9ec3b in facebook::logdevice::AllServerReadStreams::onShardStatusChanged (this=0x7ff4fcbd5c00) at logdevice/server/read_path/AllServerReadStreams.cpp:740
  #6  0x0000000002f9b2d1 in facebook::logdevice::ShardAuthoritativeStatusManager::notifySubscribers (this=<optimized out>) at logdevice/common/ShardAuthoritativeStatusMap.cpp:197
  #7  0x0000000002f9b21e in facebook::logdevice::UpdateShardAuthoritativeMapRequest::execute (this=<optimized out>) at logdevice/common/ShardAuthoritativeStatusMap.cpp:138
  #8  0x0000000002fd4a0e in facebook::logdevice::Worker::processRequest (this=0x7ff56953d600, rq=std::unique_ptr<facebook::logdevice::Request> containing 0x7ff480557e90) at logdevice/common/Worker.cpp:1282
  #9  0x0000000002fdb424 in facebook::logdevice::Worker::forcePost(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >&, signed char)::$_9::operator()() (this=<optimized out>) at logdevice/common/Worker.cpp:1405
  #10 folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::Worker::forcePost(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >&, signed char)::$_9>(folly::detail::function::Data&) (p=...) at folly/Function.h:360
  #11 0x0000000002fd581b in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x0) at folly/Function.h:376
  #12 facebook::logdevice::Worker::pickAndExecuteTask (this=<optimized out>, priority_hint=<optimized out>) at logdevice/common/Worker.cpp:1342
  #13 0x0000000002fdaa62 in facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_8::operator()() const (this=<optimized out>) at logdevice/common/Worker.cpp:1388
  #14 folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_8>(folly::detail::function::Data&) (p=...) at folly/Function.h:360
  #15 0x00000000032de717 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=<optimized out>) at folly/Function.h:376
  #16 facebook::logdevice::EventLoop::add(folly::Function<void ()>)::$_0::operator()() (this=0x7fee0ba685a0) at logdevice/common/EventLoop.cpp:159
  #17 folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::EventLoop::add(folly::Function<void ()>)::$_0>(folly::detail::function::Data&) (p=...) at folly/Function.h:367
  #18 0x0000000000606428 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fee0ba685a0) at folly/Function.h:376
  #19 facebook::logdevice::EventLoopTaskQueue::executeTasks (this=<optimized out>, num_tasks_to_dequeue=1) at logdevice/common/EventLoopTaskQueue.cpp:151
  #20 0x0000000000605801 in facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::$_0::operator()(unsigned long) const (n=1, this=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:119
  #21 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::$_0&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::$_0&, unsigned long) (this=0x7ff56986a4f0, func=..., maxBatchSize=<optimized out>) at logdevice/common/LifoEventSem.h:205
  #22 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler (arg=<optimized out>, what=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:124
  #23 0x0000000000603f52 in facebook::logdevice::EventHandler<&facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler, &facebook::logdevice::(anonymous namespace)::preflight_noop, &facebook::logdevice::(anonymous namespace)::postflight_noop> (what=<optimized out>, arg=0x20) at logdevice/common/EventHandler.h:92
  #24 0x0000000000603e08 in event_process_active_single_queue (base=0x7ff561bddd00, activeq=0x7ff56983c230, max_to_process=2147483647, endtime=0x0) at logdevice/external/libevent-2.1.3-alpha/event.c:1449
  #25 0x000000000548641c in event_process_active (base=<optimized out>) at logdevice/external/libevent-2.1.3-alpha/event.c:1596
  #26 ld_event_base_loop (base=0x7ff561bddd00, flags=<optimized out>) at logdevice/external/libevent-2.1.3-alpha/event.c:1819
  #27 0x00000000032dda80 in facebook::logdevice::EventLoop::run (this=0x7ff56953d600) at logdevice/common/EventLoop.cpp:222
  #28 0x00000000032dd279 in facebook::logdevice::EventLoop::enter (self=0x20) at logdevice/common/EventLoop.cpp:173
  #29 0x00007ff56c2866b6 in start_thread () from /usr/local/fbcode/platform007/lib/libpthread.so.0
  #30 0x00007ff56b967ebf in clone () from /usr/local/fbcode/platform007/lib/libc.so.6

This diff fixes a bug that could cause this.

Reviewed By: gdrane

Differential Revision: D15887025

fbshipit-source-id: eb6b214ae33a0f36506d0d65d95f33469a6cbae1
facebook-github-bot pushed a commit that referenced this issue Oct 9, 2019
@al13n321 @facebook-github-bot
Fix use-after-free in AdminCommandClient
4d72536 
Summary:
ASAN error:

  ==373949==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700007d938 at pc 0x0000005be44b bp 0x7f71060ab440 sp 0x7f71060ab438
  READ of size 2 at 0x60700007d938 thread T7 (IOThreadPool2)
  SCARINESS: 42 (2-byte-read-heap-use-after-free)
  I1008 20:44:07.669364  373949 [not-set] TestUtil.cpp:292] wait_until() Still waiting until: node 0 starts (157.375s)
  I1008 20:44:07.811503  373949 [not-set] TestUtil.cpp:292] wait_until() Still waiting until: node 0 starts (187.389s)
  I1008 20:44:38.524135  373949 [not-set] TestUtil.cpp:292] wait_until() Still waiting until: node 0 starts (218.206s)
      #0 0x5be44a in folly::SocketAddress::SocketAddress(folly::SocketAddress const&) folly/SocketAddress.h:83
      #1 0x2ba49a0 in facebook::logdevice::AdminCommandClient::Request::Request(facebook::logdevice::AdminCommandClient::Request const&) logdevice/ops/admin_command_client/AdminCommandClient.h:32
      #2 0x2ba49a0 in facebook::logdevice::AdminClientConnection::AdminClientConnection(folly::EventBase*, facebook::logdevice::AdminCommandClient::Request const&, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::shared_ptr<folly::SSLContext>) logdevice/ops/admin_command_client/AdminCommandClient.cpp:39
      #3 0x2ba40c1 in std::_MakeUniq<facebook::logdevice::AdminClientConnection>::__single_object std::make_unique<facebook::logdevice::AdminClientConnection, folly::EventBase*, facebook::logdevice::AdminCommandClient::Request const&, std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::shared_ptr<folly::SSLContext> >(folly::EventBase*&&, facebook::logdevice::AdminCommandClient::Request const&, std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::shared_ptr<folly::SSLContext>&&) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:825
      #4 0x2ba20b7  (/data/users/kolmike/fbsource/fbcode/buck-out/dbgo/gen/logdevice/test/tests+0x2ba20b7)
      #5 0x13e1204 in folly::futures::detail::Core<folly::Unit>::doCallback(folly::Executor::KeepAlive<folly::Executor>&&, folly::futures::detail::State)::'lambda'(folly::Executor::KeepAlive<folly::Executor>&&)::operator()(folly::Executor::KeepAlive<folly::Executor>&&) folly/futures/detail/Core.h:940
      #6 0x2d05a3b in folly::detail::function::FunctionTraits<void ()>::operator()() folly/Function.h:376
      #7 0x2d05a3b in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) folly/executors/ThreadPoolExecutor.cpp:84
      #8 0x2caaec7 in folly::IOThreadPoolExecutor::add(folly::Function<void ()>, std::chrono::duration<long, std::ratio<1l, 1000l> >, folly::Function<void ()>)::$_0::operator()() folly/executors/IOThreadPoolExecutor.cpp:106
      #9 0x2caaec7 in void folly::detail::function::FunctionTraits<void ()>::callBig<folly::IOThreadPoolExecutor::add(folly::Function<void ()>, std::chrono::duration<long, std::ratio<1l, 1000l> >, folly::Function<void ()>)::$_0>(folly::detail::function::Data&) folly/Function.h:367
      #10 0x2cfb023 in folly::detail::function::FunctionTraits<void ()>::operator()() folly/Function.h:376
      #11 0x2cfb023 in folly::EventBase::FunctionRunner::messageAvailable(folly::Function<void ()>&&) folly/io/async/EventBase.cpp:59
      #12 0x2cf4b37 in folly::NotificationQueue<folly::Function<void ()> >::Consumer::consumeMessages(bool, unsigned long*) folly/io/async/NotificationQueue.h:729
      #13 0x2cfb9fc in folly::EventHandler::libeventCallback(int, short, void*) folly/io/async/EventHandler.cpp:161
      #14 0x3772f98 in event_process_active /home/engshare/third-party2/libevent/1.4.14b_hphp/src/libevent-1.4.14b-stable/event.c:390
      #15 0x3773237 in event_base_loop /home/engshare/third-party2/libevent/1.4.14b_hphp/src/libevent-1.4.14b-stable/event.c:532
      #16 0x2cec440 in folly::EventBase::loopBody(int, bool) folly/io/async/EventBase.cpp:338
      #17 0x2ceec9b in folly::EventBase::loop() folly/io/async/EventBase.cpp:261
      #18 0x2ceec9b in folly::EventBase::loopForever() folly/io/async/EventBase.cpp:477
      #19 0x2ca7590 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) folly/executors/IOThreadPoolExecutor.cpp:179
      #20 0x2d0fd70 in void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(std::__invoke_memfun_deref, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/invoke.h:73
      #21 0x7f711342467f in execute_native_thread_routine /home/engshare/third-party2/libgcc/7.x/src/gcc-7.x/libstdc++-v3/src/c++11/thread.cc:83:15
      #22 0x7f71129026b5 in start_thread /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/nptl/pthread_create.c:465:7
      #23 0x7f711221bebe in __GI___clone /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/sysdeps/unix/sysv/linux/x86_64/clone.S:95

  0x60700007d938 is located 24 bytes inside of 72-byte region [0x60700007d920,0x60700007d968)
  freed by thread T0 here:
      #0 0x34b2258 in operator delete(void*) (/data/users/kolmike/fbsource/fbcode/buck-out/dbgo/gen/logdevice/test/tests+0x34b2258)
      #1 0x2b07f64 in facebook::logdevice::test::nc(std::shared_ptr<facebook::logdevice::AdminCommandClient> const&, folly::SocketAddress const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >) logdevice/test/utils/nc.cpp:36
      #2 0xe1d5b1 in facebook::logdevice::IntegrationTestUtils::Node::sendCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::chrono::duration<long, std::ratio<1l, 1000l> >) const logdevice/test/utils/IntegrationTestUtils.cpp:1538
      #3 0xe20a03 in facebook::logdevice::IntegrationTestUtils::Node::getServerInfo() const logdevice/test/utils/IntegrationTestUtils.cpp:1619
      #4 0xe72c70 in facebook::logdevice::IntegrationTestUtils::Node::waitUntilStarted(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_5::operator()() const logdevice/test/utils/IntegrationTestUtils.cpp:1648
      #5 0xe72c70 in std::_Function_handler<bool (), facebook::logdevice::IntegrationTestUtils::Node::waitUntilStarted(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_5>::_M_invoke(std::_Any_data const&) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:301
      #6 0x177c3f5 in std::function<bool ()>::operator()() const third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706
      #7 0x177c3f5 in facebook::logdevice::wait_until(char const*, std::function<bool ()>, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >) logdevice/common/test/TestUtil.cpp:277
      #8 0xe0f70d in facebook::logdevice::IntegrationTestUtils::Node::waitUntilStarted(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >) logdevice/test/utils/IntegrationTestUtils.cpp:1676
      #9 0xe04345 in facebook::logdevice::IntegrationTestUtils::Cluster::start(std::vector<short, std::allocator<short> >) logdevice/test/utils/IntegrationTestUtils.cpp:1122
      #10 0xdfd2dd in facebook::logdevice::IntegrationTestUtils::ClusterFactory::createOneTry(facebook::logdevice::Configuration const&) logdevice/test/utils/IntegrationTestUtils.cpp:887
      #11 0xdf9ece in facebook::logdevice::IntegrationTestUtils::ClusterFactory::create(facebook::logdevice::Configuration const&) logdevice/test/utils/IntegrationTestUtils.cpp:1423
      #12 0xdf68e3 in facebook::logdevice::IntegrationTestUtils::ClusterFactory::create(int) logdevice/test/utils/IntegrationTestUtils.cpp:694
      #13 0x6bbe76 in (anonymous namespace)::DataSizeTest::init() logdevice/test/DataSizeTest.cpp:132
      #14 0x6cab7d in (anonymous namespace)::DataSizeTest_RestartNode_Test::TestBody() logdevice/test/DataSizeTest.cpp:336
      #15 0x366771c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417:27
      #16 0x366771c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2453:52
      #17 0x3658ad1 in testing::Test::Run() (.part.595) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2490:50
      #18 0x3658f51 in testing::Test::Run() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2481:3
      #19 0x3658f51 in testing::TestInfo::Run() (.part.596) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2677:14
      #20 0x36590e4 in testing::TestInfo::Run() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2801:21
      #21 0x36590e4 in testing::TestCase::Run() (.part.597) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2802:31
      #22 0x365a664 in testing::TestCase::Run() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:4737:45
      #23 0x365a664 in testing::internal::UnitTestImpl::RunAllTests() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:4739:46
      #24 0x365a904 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417:27
      #25 0x365a904 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2453:52
      #26 0x365a904 in testing::UnitTest::Run() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:4333:55
      #27 0x2c74173 in RUN_ALL_TESTS() third-party-buck/platform007/build/googletest/include/gtest/gtest.h:2255
      #28 0x2c74173 in main common/gtest/LightMain.cpp:19
      #29 0x7f711212b1a5 in __libc_start_main /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/csu/libc-start.c:308:16
      #30 0x4f0c99 in _start /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/sysdeps/x86_64/start.S:120

Reviewed By: MohamedBassem

Differential Revision: D17825971

fbshipit-source-id: f68ee49d5b9a079e55c5bed6e2e6b28dbb161c9b
facebook-github-bot pushed a commit that referenced this issue Oct 23, 2019
@al13n321 @facebook-github-bot
Fix use-after-move in FileBasedVersionedConfigStore
76ddda5 
Summary:
I feel bad for such a hacky fix, but (a) I don't have better simple ideas, (b) it's only used in tests.

Stack trace:
  (gdb) bt
  #0  raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50
  #1  0x000000000096052a in facebook::logdevice::handle_fatal_signal (sig=<optimized out>) at logdevice/server/fatalsignal.cpp:58
  #2  <signal handler called>
  #3  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
  #4  0x00007f0b92e3b935 in __GI_abort () at abort.c:90
  #5  0x00007f0b940fc395 in __gnu_cxx::__verbose_terminate_handler () at ../../.././libstdc++-v3/libsupc++/vterminate.cc:95
  #6  0x00007f0b940fa016 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:47
  #7  0x00007f0b940fa061 in std::terminate () at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:57
  #8  0x00007f0b940fa33b in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x26e5800 <typeinfo for std::bad_function_call>, dest=0x415cf0 <std::bad_function_call::~bad_function_call()plt>) at ../../.././libstdc++-v3/libsupc++/eh_throw.cc:93
  #9  0x00000000004b5f92 in folly::throw_exception<std::bad_function_call> (ex=...) at folly/lang/Exception.h:36
  #10 0x00000000004b5f57 in folly::detail::throw_exception_<std::bad_function_call> () at folly/lang/Exception.h:64
  #11 0x0000000000d70019 in folly::throw_exception<std::bad_function_call> () at folly/lang/Exception.h:82
  #12 folly::detail::function::FunctionTraits<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>::uninitCall(folly::detail::function::Data&, facebook::logdevice::E&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) () at folly/Function.h:372
  #13 0x0000000000f023f8 in folly::detail::function::FunctionTraits<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>::operator()(facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (this=<optimized out>, args=..., args=...) at folly/Function.h:377
  #14 facebook::logdevice::FileBasedVersionedConfigStore::getConfig(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, folly::Function<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>, folly::Optional<facebook::logdevice::vcs_config_version_t>) const (
      this=<optimized out>, key=..., cb=..., base_version=...) at logdevice/common/FileBasedVersionedConfigStore.cpp:258
  #15 0x0000000000d76019 in facebook::logdevice::configuration::nodes::VersionedNodesConfigurationStore<facebook::logdevice::FileBasedVersionedConfigStore>::getConfig(folly::Function<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>, folly::Optional<facebook::logdevice::vcs_config_version_t>) const (
      this=<optimized out>, cb=..., base_version=...) at logdevice/common/configuration/nodes/NodesConfigurationStore.h:88
  #16 0x0000000000d6de2f in facebook::logdevice::configuration::nodes::ncm::Dependencies::readFromStore (this=0x7f0b91967fa0, should_do_consistent_config_fetch=false) at logdevice/common/configuration/nodes/NodesConfigurationManagerDependencies.cpp:342
  #17 0x0000000000d5e64e in facebook::logdevice::configuration::nodes::NodesConfigurationManager::onHeartBeat (this=0x7f0b90d0df10) at logdevice/common/configuration/nodes/NodesConfigurationManager.cpp:538
  #18 0x0000000000d73073 in facebook::logdevice::configuration::nodes::ncm::Dependencies::scheduleHeartBeat()::$_4::operator()() const (this=<optimized out>) at logdevice/common/configuration/nodes/NodesConfigurationManagerDependencies.cpp:365
  #19 std::_Function_handler<void (), facebook::logdevice::configuration::nodes::ncm::Dependencies::scheduleHeartBeat()::$_4>::_M_invoke(std::_Any_data const&) (__functor=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:316
  #20 0x0000000000e250a6 in std::function<void ()>::operator()() const (this=<optimized out>) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706
  #21 facebook::logdevice::(anonymous namespace)::WheelTimerDispatchImpl::makeWheelTimerInternalExecutor(facebook::logdevice::Worker*)::$_0::operator()()::{lambda()#1}::operator()() const (this=0x7f0b8bf34070) at logdevice/common/Timer.cpp:123
  #22 folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::(anonymous namespace)::WheelTimerDispatchImpl::makeWheelTimerInternalExecutor(facebook::logdevice::Worker*)::$_0::operator()()::{lambda()#1}>(folly::detail::function::Data&) (p=...) at folly/Function.h:361
  #23 0x0000000000e63602 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=<optimized out>) at folly/Function.h:377
  #24 facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9::operator()() (this=0x7f0b8bf34060) at logdevice/common/Worker.cpp:1373
  #25 folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9>(folly::detail::function::Data&) (p=...) at folly/Function.h:368
  #26 0x00000000010d7e75 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7f0b8bf34060) at folly/Function.h:377
  #27 facebook::logdevice::EventLoopTaskQueue::executeTasks (this=0x7f0b74023000, tokens=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:154
  #28 0x00000000010d7699 in facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1::operator()(unsigned int) const (this=<optimized out>, n=1) at logdevice/common/EventLoopTaskQueue.cpp:101
  #29 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&, unsigned int) (this=<optimized out>, func=..., maxBatchSize=<optimized out>) at logdevice/common/LifoEventSem.h:351
  #30 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler (this=0x7f0b74023000) at logdevice/common/EventLoopTaskQueue.cpp:106
  #31 0x0000000001a907f2 in event_process_active_single_queue (base=<optimized out>, activeq=0x7f0b74020010, max_to_process=2147483647, endtime=0x0) at logdevice/external/libevent-2.1.3-alpha/event.c:1449
  #32 0x0000000001a8efae in event_process_active (base=<optimized out>) at logdevice/external/libevent-2.1.3-alpha/event.c:1599
  #33 ld_event_base_loop (base=0x7f0b7401b000, flags=<optimized out>) at logdevice/external/libevent-2.1.3-alpha/event.c:1819
  #34 0x000000000176b5f1 in facebook::logdevice::EvBaseLegacy::loop (this=0x7f0b74000018) at logdevice/common/libevent/EvBaseLegacy.cpp:58
  #35 0x00000000010d5a79 in facebook::logdevice::EventLoop::run (this=<optimized out>) at logdevice/common/EventLoop.cpp:140
  #36 0x00000000016fa223 in std::function<void ()>::operator()() const (this=0x7f0b90d0c520) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706
  #37 facebook::logdevice::thread_func (arg=0x7f0b90d0c520) at logdevice/common/PThread.cpp:9
  #38 0x00007f0b936136b6 in start_thread (arg=0x7f0b74dff700) at pthread_create.c:465
  #39 0x00007f0b92f2cebf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Reviewed By: gdrane

Differential Revision: D17975422

fbshipit-source-id: 499c6aceb577f859e9d4e71006b86f73537221ab
facebook-github-bot pushed a commit that referenced this issue Oct 25, 2019
@al13n321 @facebook-github-bot
Fix TSAN error in LifoEventSem shutdown
729de55 
Summary:
TSAN report:

  ==================
  WARNING: ThreadSanitizer: data race (pid=3276373)
    Write of size 8 at 0x7ba000000330 by thread T10:
      #0 close at ??:?
      #1 facebook::logdevice::detail::SynchronizationFd::~SynchronizationFd() at ./logdevice/common/LifoEventSem.cpp:59
      #2 facebook::logdevice::detail::FdBaton<std::atomic>::~FdBaton() at ./logdevice/common/LifoEventSem.h:68
      #3 std::default_delete<facebook::logdevice::detail::FdBaton<std::atomic> >::operator()(facebook::logdevice::detail::FdBaton<std::atomic>*) const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:78
      #4 std::unique_ptr<facebook::logdevice::detail::FdBaton<std::atomic>, std::default_delete<facebook::logdevice::detail::FdBaton<std::atomic> > >::~unique_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:268
      #5 facebook::logdevice::detail::LifoFdBaton<std::atomic>::~LifoFdBaton() at ./logdevice/common/LifoEventSem.h:155
      #6 folly::detail::LifoSemNode<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic>::destroy() at ./folly/synchronization/LifoSem.h:195
      #7 folly::detail::LifoSemNodeRecycler<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic>::operator()(folly::detail::LifoSemNode<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic>*) const at ./folly/synchronization/LifoSem.h:213
      #8 std::unique_ptr<folly::detail::LifoSemNode<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic>, folly::detail::LifoSemNodeRecycler<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic> >::~unique_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:268
      #9 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::~AsyncWaiter() at ./logdevice/common/LifoEventSem.h:380
      #10 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::~AsyncWaiter() at ./logdevice/common/LifoEventSem.h:358
      #11 folly::DelayedDestruction::onDelayedDestroy(bool) at ./folly/io/async/DelayedDestruction.h:115
      #12 folly::DelayedDestruction::destroy() at ./folly/io/async/DelayedDestruction.h:55
      #13 folly::DelayedDestruction::Destructor::operator()(folly::DelayedDestruction*) const at ./folly/io/async/DelayedDestruction.h:70
      #14 std::unique_ptr<facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter, folly::DelayedDestruction::Destructor>::reset(facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:376
      #15 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler() at ./logdevice/common/EventLoopTaskQueue.cpp:113
      #16 facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0::operator()() const at ./logdevice/common/EventLoopTaskQueue.cpp:36
      #17 void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0>(folly::detail::function::Data&) at ./folly/Function.h:361
      #18 folly::detail::function::FunctionTraits<void ()>::operator()() at ./folly/Function.h:377
      #19 facebook::logdevice::EventLegacy::evCallback(int, short, void*) at ./logdevice/common/libevent/EventLegacy.cpp:41
      #20 event_persist_closure at ./logdevice/external/libevent-2.1.3-alpha/event.c:1452
      #21 event_process_active_single_queue at ./logdevice/external/libevent-2.1.3-alpha/event.c:1508
      #22 event_process_active at ./logdevice/external/libevent-2.1.3-alpha/event.c:1596
      #23 ld_event_base_loop at ./logdevice/external/libevent-2.1.3-alpha/event.c:1819
      #24 facebook::logdevice::EvBaseLegacy::loop() at ./logdevice/common/libevent/EvBaseLegacy.cpp:58
      #25 facebook::logdevice::EvBase::loop() at ./logdevice/common/libevent/LibEventCompatibility.cpp:52
      #26 facebook::logdevice::EventLoop::run() at ./logdevice/common/EventLoop.cpp:140
      #27 facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType)::$_0::operator()() const at ./logdevice/common/EventLoop.cpp:81
      #28 std::_Function_handler<void (), facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType)::$_0>::_M_invoke(std::_Any_data const&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:316
      #29 std::function<void ()>::operator()() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706
      #30 facebook::logdevice::thread_func(void*) at ./logdevice/common/PThread.cpp:9
      #31 __tsan_thread_start_func at tsan.c:?

    Previous read of size 8 at 0x7ba000000330 by main thread:
      #0 write at ??:?
      #1 facebook::logdevice::detail::SynchronizationFd::write() at ./logdevice/common/LifoEventSem.cpp:75
      #2 post at ./logdevice/common/LifoEventSem.h:72
      #3 facebook::logdevice::EventLoopTaskQueue::shutdown() at ./logdevice/common/EventLoopTaskQueue.cpp:68
      #4 facebook::logdevice::Processor::shutdown() at ./logdevice/common/Processor.cpp:585
      #5 facebook::logdevice::ClientImpl::~ClientImpl() at ./logdevice/lib/ClientImpl.cpp:314
      #6 void __gnu_cxx::new_allocator<facebook::logdevice::ClientImpl>::destroy<facebook::logdevice::ClientImpl>(facebook::logdevice::ClientImpl*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/ext/new_allocator.h:157
      #7 void std::allocator_traits<std::allocator<facebook::logdevice::ClientImpl> >::destroy<facebook::logdevice::ClientImpl>(std::allocator<facebook::logdevice::ClientImpl>&, facebook::logdevice::ClientImpl*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/alloc_traits.h:488
      #8 std::_Sp_counted_ptr_inplace<facebook::logdevice::ClientImpl, std::allocator<facebook::logdevice::ClientImpl>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:537
      #9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:156
      #10 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:686
      #11 std::__shared_ptr<facebook::logdevice::Client, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:1125
      #12 std::shared_ptr<facebook::logdevice::Client>::~shared_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr.h:93
      #13 SequencerIntegrationTest_ExpandShrinkReactivationDelayTest_Test::TestBody() at ./logdevice/test/SequencerIntegrationTest.cpp:580
      #14 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417
      #15 main at ./common/gtest/LightMain.cpp:19
      #16 ?? ??:0

During shutdown, a write() to fd triggers the worker thread to wake up and destroy the same fd. So, technically, fd might be destroyed before the write() returns, while write() is still accessing the fd. This diff adds some synchronization to make sure fd close() starts only after write() returns.

#thanks Nick Sukhanov for figuring that out.

Reviewed By: mcrnic, tau0

Differential Revision: D18018223

fbshipit-source-id: ddbff95257dd39c835ad6e8733ba0bf2ed432fa0
facebook-github-bot pushed a commit that referenced this issue Nov 7, 2019
@al13n321 @facebook-github-bot
Don't start worker threads until Processor initialization is complete
016132b 
Summary:
When creating various components in Processor::init() (watchdog thread, health monitor, failure detector, etc), at first create all of them in a dormant state, then start all of them. In particular, pause all worker threads until later in the startup sequence. This prevents race conditions when the parts initialized early may try to access parts of the Processor that are not initialized yet.

In particular, fixes this race when assigning Processor::failure_detector_ in a unit test (it's a test-only code path, but afaict the same thing can happen in server):

  ==================
  WARNING: ThreadSanitizer: data race (pid=886149)
    Write of size 8 at 0x7b88001dfe10 by main thread:
      #0 _ZSt4swapIPN8facebook9logdevice15FailureDetectorEENSt9enable_ifIXsr6__and_ISt6__not_ISt15__is_tuple_likeIT_EESt21is_move_constructibleIS7_ESt18is_move_assignableIS7_EEE5valueEvE4typeERS7_SG_ at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/move.h:199
      #1 std::unique_ptr<facebook::logdevice::FailureDetector, std::default_delete<facebook::logdevice::FailureDetector> >::reset(facebook::logdevice::FailureDetector*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:374
      #2 _ZNSt10unique_ptrIN8facebook9logdevice15FailureDetectorESt14default_deleteIS2_EEaSINS1_19MockFailureDetectorES3_IS7_EEENSt9enable_ifIXsr6__and_ISt6__and_IJSt14is_convertibleINS_IT_T0_E7pointerEPS2_ESt6__not_ISt8is_arrayISC_EESt5__or_IJSA_IJSt12is_referenceIS4_ESt7is_sameIS4_SD_EEESA_IJSI_ISO_ESB_ISD_S4_EEEEEEESt13is_assignableIRS4_OSD_EEE5valueERS5_E4typeEOSE_ at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:303
      #3 (anonymous namespace)::make_processor_with_detector(short, unsigned long, facebook::logdevice::GossipSettings const&) at ./logdevice/server/test/FailureDetectorTest.cpp:185
      #4 (anonymous namespace)::create_processors_and_detectors(unsigned long, facebook::logdevice::GossipSettings const&) at ./logdevice/server/test/FailureDetectorTest.cpp:266
      #5 (anonymous namespace)::simulate(unsigned long, facebook::logdevice::GossipSettings const&) at ./logdevice/server/test/FailureDetectorTest.cpp:296
      #6 FailureDetectorTest_RandomGossip_Test::TestBody() at ./logdevice/server/test/FailureDetectorTest.cpp:313
      #7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417
      #8 main at ./common/gtest/LightMain.cpp:19
      #9 ?? ??:0

    Previous read of size 8 at 0x7b88001dfe10 by thread T79:
      #0 std::__uniq_ptr_impl<facebook::logdevice::FailureDetector, std::default_delete<facebook::logdevice::FailureDetector> >::_M_ptr() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:147
      #1 std::unique_ptr<facebook::logdevice::FailureDetector, std::default_delete<facebook::logdevice::FailureDetector> >::get() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:337
      #2 facebook::logdevice::ServerWorker::onServerConfigUpdated() at ./logdevice/server/ServerWorker.cpp:190
      #3 facebook::logdevice::Worker::onNodesConfigurationUpdated() at ./logdevice/common/Worker.cpp:393
      #4 facebook::logdevice::Worker::initializeSubscriptions() at ./logdevice/common/Worker.cpp:541
      #5 facebook::logdevice::Worker::setupWorker() at ./logdevice/common/Worker.cpp:582
      #6 facebook::logdevice::ServerWorker::setupWorker() at ./logdevice/server/ServerWorker.cpp:201
      #7 facebook::logdevice::ServerProcessor::createWorker(folly::Executor::KeepAlive<folly::Executor>, facebook::logdevice::worker_id_t, facebook::logdevice::WorkerType)::$_0::operator()() const at ./logdevice/server/ServerProcessor.cpp:26
      #8 void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::ServerProcessor::createWorker(folly::Executor::KeepAlive<folly::Executor>, facebook::logdevice::worker_id_t, facebook::logdevice::WorkerType)::$_0>(folly::detail::function::Data&) at ./folly/Function.h:376
      #9 folly::detail::function::FunctionTraits<void ()>::operator()() at ./folly/Function.h:392
      #10 facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9::operator()() at ./logdevice/common/Worker.cpp:1413
      #11 void folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9>(folly::detail::function::Data&) at ./folly/Function.h:383
      #12 folly::detail::function::FunctionTraits<void ()>::operator()() at ./folly/Function.h:392
      #13 facebook::logdevice::EventLoopTaskQueue::executeTasks(unsigned int) at ./logdevice/common/EventLoopTaskQueue.cpp:154
      #14 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1::operator()(unsigned int) const at ./logdevice/common/EventLoopTaskQueue.cpp:101
      #15 void facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&, unsigned int) at ./logdevice/common/LifoEventSem.h:368
      #16 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler() at ./logdevice/common/EventLoopTaskQueue.cpp:106
      #17 facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0::operator()() const at ./logdevice/common/EventLoopTaskQueue.cpp:36
      #18 void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0>(folly::detail::function::Data&) at ./folly/Function.h:376
      #19 folly::detail::function::FunctionTraits<void ()>::operator()() at ./folly/Function.h:392
      #20 facebook::logdevice::EventLegacy::evCallback(int, short, void*) at ./logdevice/common/libevent/EventLegacy.cpp:41
      #21 event_persist_closure at ./logdevice/external/libevent-2.1.3-alpha/event.c:1452
      #22 event_process_active_single_queue at ./logdevice/external/libevent-2.1.3-alpha/event.c:1508
      #23 event_process_active at ./logdevice/external/libevent-2.1.3-alpha/event.c:1596
      #24 ld_event_base_loop at ./logdevice/external/libevent-2.1.3-alpha/event.c:1819
      #25 facebook::logdevice::EvBaseLegacy::loop() at ./logdevice/common/libevent/EvBaseLegacy.cpp:58
      #26 facebook::logdevice::EvBase::loop() at ./logdevice/common/libevent/LibEventCompatibility.cpp:52
  W1101 14:22:40.569391  907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 433 msec: NODE_STATE_UPDATED (id: 4294979886), p :LO_PRI
      #27 facebook::logdevice::EventLoop::run() at ./logdevice/common/EventLoop.cpp:140
  W1101 14:22:40.584447  907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 446 msec: NODE_STATE_UPDATED (id: 4294979887), p :LO_PRI
      #28 facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType)::$_0::operator()() const at ./logdevice/common/EventLoop.cpp:81
      #29 std::_Function_handler<void (), facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType)::$_0>::_M_invoke(std::_Any_data const&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:316
  W1101 14:22:40.603377  907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 468 msec: NODE_STATE_UPDATED (id: 4294979888), p :LO_PRI
      #30 std::function<void ()>::operator()() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706
  W1101 14:22:40.621593  907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 491 msec: NODE_STATE_UPDATED (id: 4294979892), p :LO_PRI
      #31 facebook::logdevice::thread_func(void*) at ./logdevice/common/PThread.cpp:18
  W1101 14:22:40.645430  907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 505 msec: NODE_STATE_UPDATED (id: 4294979893), p :LO_PRI
      #32 __tsan_thread_start_func at tsan.c:?

Reviewed By: mcrnic

Differential Revision: D18286463

fbshipit-source-id: 1ea365415f1e9dc8e907eabcbdd8a8249652a16e
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@calebmarchent