Check failed: scheduled_tasks_.empty() in Messenger::Shutdown()

[spolitov]

No description provided.

[spolitov]

Happens infrequently in various tests, for instance QLTransactionTest.ReadRestart in TSAN mode.

[mbautin]

Another occurrence:

1 | F20190115 07:07:21 ../../src/yb/rpc/messenger.cc:223] Scheduled tasks is not empty after messenger shutdown: [{47, 0x00000000071649d0 -> { id: 47 source: ../../src/yb/master/async_rpc_tasks.cc:264 }}]
-- | --
2 | @ 0x7f494f9426b8 yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long) (src/yb/util/logging.cc:474)
3 | @ 0x7f494e3cfc05
4 | @ 0x7f494e3cd439
5 | @ 0x7f494e3cd670
6 | @ 0x7f4952035df3 yb::rpc::Messenger::Shutdown() (src/yb/rpc/messenger.cc:223)
7 | @ 0x7f4955e5634f yb::server::RpcServerBase::Shutdown() (src/yb/server/server_base.cc:341)
8 | @ 0x7f4955e5899d yb::server::RpcAndWebServerBase::Shutdown() (src/yb/server/server_base.cc:515)
9 | @ 0x7f495bdc1b62 yb::master::Master::Shutdown() (src/yb/master/master.cc:267)
10 | @ 0x7f495bdfbfa8 yb::master::MiniMaster::Shutdown() (src/yb/master/mini_master.cc:91)
11 | @ 0x7f495e3480d0 yb::MiniCluster::Shutdown() (src/yb/integration-tests/mini_cluster.cc:317)
12 | @ 0x430fd4 yb::ql::QLTestBase::TearDown() (src/yb/yql/cql/ql/test/ql-test-base.h:176)
13 | @ 0x7f494fdbd6a7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (src/googletest-release-1.8.0/googletest/src/gtest.cc:2402)
14 | @ 0x7f494fdbd6a7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (src/googletest-release-1.8.0/googletest/src/gtest.cc:2438)
15 | @ 0x7f494fdb4a48 testing::Test::Run() (src/googletest-release-1.8.0/googletest/src/gtest.cc:2483)
16 | @ 0x7f494fdb4bc7 testing::TestInfo::Run() (src/googletest-release-1.8.0/googletest/src/gtest.cc:2656)
17 | @ 0x7f494fdb4ca4 testing::TestCase::Run() (src/googletest-release-1.8.0/googletest/src/gtest.cc:2774)
18 | @ 0x7f494fdb55f7 testing::internal::UnitTestImpl::RunAllTests() (src/googletest-release-1.8.0/googletest/src/gtest.cc:4649)
19 | @ 0x7f494fdbdbb7 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (src/googletest-release-1.8.0/googletest/src/gtest.cc:2402)
20 | @ 0x7f494fdbdbb7 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (src/googletest-release-1.8.0/googletest/src/gtest.cc:2438)
21 | @ 0x7f494fdb4d80 testing::UnitTest::Run() (src/googletest-release-1.8.0/googletest/src/gtest.cc:4257)
22 | @ 0x7f495e608e57 RUN_ALL_TESTS() (<skipped>/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2233)
23 | @ 0x7f495e6087f5 main (src/yb/util/test_main.cc:104)
24 | @ 0x7f494b6ea824 __libc_start_main (csu/libc-start.c:289)
25 | @ 0x41b428 (unknown) (sysdeps/x86_64/start.S:118)
26 | @ 0xffffffffffffffff

Repro:

yb_build.sh --cxx-test ql_ql-create-table-test --gtest_filter TestQLCreateTable.TestQLCreateTableWithPartitionScemeOf