Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docdb] TSAN race on MasterFailoverTestIndexCreation_TestPauseAfterCreateIndexIssued #3912

Closed
bmatican opened this issue Mar 10, 2020 · 2 comments
Closed

[docdb] TSAN race on MasterFailoverTestIndexCreation_TestPauseAfterCreateIndexIssued #3912

bmatican opened this issue Mar 10, 2020 · 2 comments
Assignees
@amitanandaiyer
Labels
area/docdb YugabyteDB core features kind/failing-test Tests and testing infra
Projects
YBase features Index backfill

Comments

@bmatican
Copy link
Contributor

/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-tsan/builds/692/archive/build/tsan-clang-dynamic-ninja/yb-test-logs/tests-integration-tests__master_failover-itest/MasterFailoverTestIndexCreation_TestPauseAfterCreateIndexIssued.log

Stack

[ts-1] WARNING: ThreadSanitizer: data race (pid=6184)
[ts-1]   Write of size 8 at 0x7b340004db08 by thread T92:
[ts-1]     #0 yb::tablet::ChangeMetadataOperationState::Finish() /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/change_metadata_operation.h:115:14 (libtablet.so+0x24cc0d)
[ts-1]     #1 yb::tablet::ChangeMetadataOperation::DoReplicated(long, yb::Status*) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/change_metadata_operation.cc:235:12 (libtablet.so+0x24c190)
[ts-1]     #2 yb::tablet::Operation::Replicated(long) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation.cc:63:3 (libtablet.so+0x249187)
[ts-1]     #3 yb::tablet::OperationDriver::ApplyTask(long, std::__1::vector<yb::OpId, std::__1::allocator<yb::OpId> >*) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation_driver.cc:442:31 (libtablet.so+0x251f07)
[ts-1]     #4 yb::tablet::OperationDriver::ApplyOperation(long, std::__1::vector<yb::OpId, std::__1::allocator<yb::OpId> >*) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation_driver.cc:421:3 (libtablet.so+0x25189c)
[ts-1]     #5 yb::tablet::OperationDriver::PrepareAndStart() /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation_driver.cc:287:14 (libtablet.so+0x25144c)
[ts-1]     #6 yb::tablet::OperationDriver::PrepareAndStartTask() /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation_driver.cc:195:27 (libtablet.so+0x251027)
[ts-1]     #7 yb::tablet::PreparerImpl::ProcessItem(yb::tablet::OperationDriver*) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/preparer.cc:212:11 (libtablet.so+0x27ec9e)
[ts-1]     #8 yb::tablet::PreparerImpl::Run() /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/preparer.cc:160:7 (libtablet.so+0x27e874)
[ts-1]     #9 decltype(*(std::__1::forward<yb::tablet::PreparerImpl*&>(fp0)).*fp()) std::__1::__invoke<void (yb::tablet::PreparerImpl::*&)(), yb::tablet::PreparerImpl*&, void>(void (yb::tablet::PreparerImpl::*&)(), yb::tablet::PreparerImpl*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/type_traits:4286:1 (libtablet.so+0x280553)
[ts-1]     #10 std::__1::__bind_return<void (yb::tablet::PreparerImpl::*)(), std::__1::tuple<yb::tablet::PreparerImpl*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::tablet::PreparerImpl::*)(), std::__1::tuple<yb::tablet::PreparerImpl*>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (yb::tablet::PreparerImpl::*)(), std::__1::tuple<yb::tablet::PreparerImpl*>, 0ul, std::__1::tuple<> >(void (yb::tablet::PreparerImpl::*&)(), std::__1::tuple<yb::tablet::PreparerImpl*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:2216 (libtablet.so+0x280553)
[ts-1]     #11 std::__1::__bind_return<void (yb::tablet::PreparerImpl::*)(), std::__1::tuple<yb::tablet::PreparerImpl*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::tablet::PreparerImpl::*)(), std::__1::tuple<yb::tablet::PreparerImpl*>, std::__1::tuple<> >::value>::type std::__1::__bind<void (yb::tablet::PreparerImpl::*)(), yb::tablet::PreparerImpl*>::operator()<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:2249 (libtablet.so+0x280553)
[ts-1]     #12 decltype(std::__1::forward<std::__1::__bind<void (yb::tablet::PreparerImpl::*)(), yb::tablet::PreparerImpl*>&>(fp)()) std::__1::__invoke<std::__1::__bind<void (yb::tablet::PreparerImpl::*)(), yb::tablet::PreparerImpl*>&>(std::__1::__bind<void (yb::tablet::PreparerImpl::*)(), yb::tablet::PreparerImpl*>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/type_traits:4345 (libtablet.so+0x280553)
[ts-1]     #13 void std::__1::__invoke_void_return_wrapper<void>::__call<std::__1::__bind<void (yb::tablet::PreparerImpl::*)(), yb::tablet::PreparerImpl*>&>(std::__1::__bind<void (yb::tablet::PreparerImpl::*)(), yb::tablet::PreparerImpl*>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/__functional_base:349 (libtablet.so+0x280553)
[ts-1]     #14 std::__1::__function::__func<std::__1::__bind<void (yb::tablet::PreparerImpl::*)(), yb::tablet::PreparerImpl*>, std::__1::allocator<std::__1::__bind<void (yb::tablet::PreparerImpl::*)(), yb::tablet::PreparerImpl*> >, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:1562:12 (libtablet.so+0x28045d)
[ts-1]     #15 std::__1::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:1913:12 (libyb-redis.so+0xe92e4)
[ts-1]     #16 yb::FunctionRunnable::Run() /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/threadpool.h:479:5 (libyb_client.so+0x1d57fd)
[ts-1]     #17 yb::ThreadPool::DispatchThread(bool) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/threadpool.cc:608:22 (libyb_util.so+0x3992a9)
[ts-1]     #18 decltype(*(std::__1::forward<yb::ThreadPool*&>(fp0)).*fp(std::__1::forward<bool&>(fp1))) std::__1::__invoke<void (yb::ThreadPool::*&)(bool), yb::ThreadPool*&, bool&, void>(void (yb::ThreadPool::*&)(bool), yb::ThreadPool*&, bool&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/type_traits:4286:1 (libyb_util.so+0x3a2384)
[ts-1]     #19 std::__1::__bind_return<void (yb::ThreadPool::*)(bool), std::__1::tuple<yb::ThreadPool*, bool>, std::__1::tuple<>, __is_valid_bind_return<void (yb::ThreadPool::*)(bool), std::__1::tuple<yb::ThreadPool*, bool>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (yb::ThreadPool::*)(bool), std::__1::tuple<yb::ThreadPool*, bool>, 0ul, 1ul, std::__1::tuple<> >(void (yb::ThreadPool::*&)(bool), std::__1::tuple<yb::ThreadPool*, bool>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:2216 (libyb_util.so+0x3a2384)
[ts-1]     #20 std::__1::__bind_return<void (yb::ThreadPool::*)(bool), std::__1::tuple<yb::ThreadPool*, bool>, std::__1::tuple<>, __is_valid_bind_return<void (yb::ThreadPool::*)(bool), std::__1::tuple<yb::ThreadPool*, bool>, std::__1::tuple<> >::value>::type std::__1::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>::operator()<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:2249 (libyb_util.so+0x3a2384)
[ts-1]     #21 decltype(std::__1::forward<std::__1::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&>(fp)()) std::__1::__invoke<std::__1::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&>(std::__1::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/type_traits:4345 (libyb_util.so+0x3a2384)
[ts-1]     #22 void std::__1::__invoke_void_return_wrapper<void>::__call<std::__1::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&>(std::__1::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/__functional_base:349 (libyb_util.so+0x3a2384)
[ts-1]     #23 std::__1::__function::__func<std::__1::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>, std::__1::allocator<std::__1::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&> >, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:1562:12 (libyb_util.so+0x3a227d)
[ts-1]     #24 std::__1::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:1913:12 (libyb-redis.so+0xe92e4)
[ts-1]     #25 yb::Thread::SuperviseThread(void*) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/thread.cc:744:3 (libyb_util.so+0x38bb09)
[ts-1]
[ts-1]   Previous read of size 8 at 0x7b340004db08 by thread T17 (mutexes: write M932380534594905992):
[ts-1]     #0 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > yb::PointerToString<yb::tserver::ChangeMetadataRequestPB const*>::Apply<yb::tserver::ChangeMetadataRequestPB const* const&>(yb::tserver::ChangeMetadataRequestPB const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/tostring.h:199:7 (libtablet.so+0x24d691)
[ts-1]     #1 std::__1::enable_if<IsPointerLike<yb::tserver::ChangeMetadataRequestPB const* const&>::value, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::type yb::ToString<yb::tserver::ChangeMetadataRequestPB const* const&>(yb::tserver::ChangeMetadataRequestPB const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/tostring.h:155:10 (libtablet.so+0x24d650)
[ts-1]     #2 yb::internal::FormatValue<yb::tserver::ChangeMetadataRequestPB const*, void>::FormatValue(yb::tserver::ChangeMetadataRequestPB const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:28:52 (libtablet.so+0x24d610)
[ts-1]     #3 yb::internal::FormatTuple<yb::tserver::ChangeMetadataRequestPB const*>::FormatTuple(yb::tserver::ChangeMetadataRequestPB const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:119:9 (libtablet.so+0x24d440)
[ts-1]     #4 yb::internal::FormatTuple<yb::Schema const*, yb::tserver::ChangeMetadataRequestPB const*>::FormatTuple(yb::Schema const* const&, yb::tserver::ChangeMetadataRequestPB const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:139:26 (libtablet.so+0x24d3b2)
[ts-1]     #5 yb::internal::FormatTuple<yb::HybridTime, yb::Schema const*, yb::tserver::ChangeMetadataRequestPB const*>::FormatTuple(yb::HybridTime const&, yb::Schema const* const&, yb::tserver::ChangeMetadataRequestPB const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:139:26 (libtablet.so+0x24d0fa)
[ts-1]     #6 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > yb::internal::FormatImpl<yb::HybridTime, yb::Schema const*, yb::tserver::ChangeMetadataRequestPB const*>(char const*, yb::HybridTime const&, yb::Schema const* const&, yb::tserver::ChangeMetadataRequestPB const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:200:24 (libtablet.so+0x24d01a)
[ts-1]     #7 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > yb::Format<yb::HybridTime, yb::Schema const*, yb::tserver::ChangeMetadataRequestPB const*>(char const*, yb::HybridTime const&, yb::Schema const* const&, yb::tserver::ChangeMetadataRequestPB const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:216:10 (libtablet.so+0x24c8b9)
[ts-1]     #8 yb::tablet::ChangeMetadataOperationState::ToString() const /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/change_metadata_operation.cc:65:10 (libtablet.so+0x24b0f9)
[ts-1]     #9 std::__1::enable_if<HasMemberFunction_ToString<yb::tablet::ChangeMetadataOperationState>::value, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::type yb::ToString<yb::tablet::ChangeMetadataOperationState>(yb::tablet::ChangeMetadataOperationState const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/tostring.h:69:16 (libtablet.so+0x24eb79)
[ts-1]     #10 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > yb::PointerToString<yb::tablet::ChangeMetadataOperationState const*>::Apply<yb::tablet::ChangeMetadataOperationState const* const&>(yb::tablet::ChangeMetadataOperationState const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/tostring.h:206:21 (libtablet.so+0x24eab4)
[ts-1]     #11 std::__1::enable_if<IsPointerLike<yb::tablet::ChangeMetadataOperationState const* const&>::value, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::type yb::ToString<yb::tablet::ChangeMetadataOperationState const* const&>(yb::tablet::ChangeMetadataOperationState const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/tostring.h:155:10 (libtablet.so+0x24ea10)
[ts-1]     #12 yb::internal::FormatValue<yb::tablet::ChangeMetadataOperationState const*, void>::FormatValue(yb::tablet::ChangeMetadataOperationState const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:28:52 (libtablet.so+0x24e9d0)
[ts-1]     #13 yb::internal::FormatTuple<yb::tablet::ChangeMetadataOperationState const*>::FormatTuple(yb::tablet::ChangeMetadataOperationState const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:119:9 (libtablet.so+0x24e740)
[ts-1]     #14 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > yb::internal::FormatImpl<yb::tablet::ChangeMetadataOperationState const*>(char const*, yb::tablet::ChangeMetadataOperationState const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:200:24 (libtablet.so+0x24e67a)
[ts-1]     #15 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > yb::Format<yb::tablet::ChangeMetadataOperationState const*>(char const*, yb::tablet::ChangeMetadataOperationState const* const&) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/format.h:216:10 (libtablet.so+0x24cc59)
[ts-1]     #16 yb::tablet::ChangeMetadataOperation::ToString() const /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/change_metadata_operation.cc:247:10 (libtablet.so+0x24c60e)
[ts-1]     #17 yb::tablet::OperationDriver::ToStringUnlocked() const /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation_driver.cc:152:30 (libtablet.so+0x24fa76)
[ts-1]     #18 yb::tablet::OperationDriver::ToString() const /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation_driver.cc:146:10 (libtablet.so+0x24f9bf)
[ts-1]     #19 yb::tablet::OperationTracker::WaitForAllToFinish(yb::MonoDelta const&) const /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation_tracker.cc:268:40 (libtablet.so+0x25549b)
[ts-1]     #20 yb::tablet::OperationTracker::WaitForAllToFinish() const /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/operation_tracker.cc:236:3 (libtablet.so+0x2550e7)
[ts-1]     #21 yb::tablet::TabletPeer::CompleteShutdown(yb::StronglyTypedBool<yb::tablet::IsDropTable_Tag>) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/tablet_peer.cc:429:24 (libtablet.so+0x1f9f57)
[ts-1]     #22 yb::tablet::TabletPeer::Shutdown(yb::StronglyTypedBool<yb::tablet::IsDropTable_Tag>) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/tablet_peer.cc:497:5 (libtablet.so+0x1fab9d)
[ts-1]     #23 yb::tserver::TSTabletManager::DeleteTablet(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, yb::tablet::TabletDataState, boost::optional<long> const&, boost::optional<yb::tserver::TabletServerErrorPB_Code>*) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tserver/ts_tablet_manager.cc:995:16 (libtserver.so+0x1f39a3)
[ts-1]     #24 yb::tserver::TabletServiceAdminImpl::DeleteTablet(yb::tserver::DeleteTabletRequestPB const*, yb::tserver::DeleteTabletResponsePB*, yb::rpc::RpcContext) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tserver/tablet_service.cc:1028:41 (libtserver.so+0x1b58c7)
[ts-1]     #25 yb::tserver::TabletServerAdminServiceIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/src/yb/tserver/tserver_admin.service.cc:124:7 (libtserver_admin_proto.so+0x69620)
[ts-1]     #26 yb::rpc::ServicePoolImpl::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/service_pool.cc:262:19 (libyrpc.so+0x20b2f7)
[ts-1]     #27 yb::rpc::InboundCall::InboundCallTask::Run() /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/inbound_call.cc:212:13 (libyrpc.so+0x172cbe)
[ts-1]     #28 yb::rpc::(anonymous namespace)::Worker::Execute() /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/thread_pool.cc:99:15 (libyrpc.so+0x21d960)
[ts-1]     #29 decltype(*(std::__1::forward<yb::rpc::(anonymous namespace)::Worker*&>(fp0)).*fp()) std::__1::__invoke<void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&, void>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/type_traits:4286:1 (libyrpc.so+0x21dcc3)
[ts-1]     #30 std::__1::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, 0ul, std::__1::tuple<> >(void (yb::rpc::(anonymous namespace)::Worker::*&)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:2216 (libyrpc.so+0x21dcc3)
[ts-1]     #31 std::__1::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::__1::tuple<> >::value>::type std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>::operator()<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:2249 (libyrpc.so+0x21dcc3)
[ts-1]     #32 decltype(std::__1::forward<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(fp)()) std::__1::__invoke<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/type_traits:4345 (libyrpc.so+0x21dcc3)
[ts-1]     #33 void std::__1::__invoke_void_return_wrapper<void>::__call<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/__functional_base:349 (libyrpc.so+0x21dcc3)
[ts-1]     #34 std::__1::__function::__func<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::__1::allocator<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&> >, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:1562:12 (libyrpc.so+0x21dbcd)
[ts-1]     #35 std::__1::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos/installed/tsan/libcxx/include/c++/v1/functional:1913:12 (libyb-redis.so+0xe92e4)
[ts-1]     #36 yb::Thread::SuperviseThread(void*) /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/util/thread.cc:744:3 (libyb_util.so+0x38bb09)

Data race at:

[ts-1] SUMMARY: ThreadSanitizer: data race /nfusr/centos-gcp-cloud/jenkins-slave-bf3/jenkins/jenkins-github-yugabyte-db-centos-master-clang-tsan-692/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/operations/change_metadata_operation.h:115:14 in yb::tablet::ChangeMetadataOperationState::Finish()
@bmatican bmatican added kind/failing-test Tests and testing infra area/docdb YugabyteDB core features labels Mar 10, 2020
@bmatican bmatican added this to To do/Current-Sprint in Index backfill via automation Mar 10, 2020
@bmatican bmatican added this to To Do in YBase features via automation Mar 10, 2020
@amitanandaiyer amitanandaiyer moved this from To do/Current-Sprint to In progress in Index backfill Apr 9, 2020
@amitanandaiyer
Copy link
Contributor

looks like this is a day 0 issue with change metadata operation.

will change the ptr to atomic to fix this.

@amitanandaiyer
Copy link
Contributor

amitanandaiyer commented Apr 9, 2020 •
edited

Also seeing


Fatal failure details written to /nfusr/centos-gcp-cloud/jenkins-slave-930/jenkins/jenkins-github-yugabyte-db-centos-master-gcc-release-784/build/release-gcc-dynamic-ninja/yb-test-logs/tests-integration-tests__master_failover-itest/MasterFailoverTestIndexCreation_TestPauseAfterCreateIndexIssued.fatal_failure_details.m-1.2020-04-08T05_10_59.pid16767.txt
--
21450 | [m-1] | F20200408 05:10:59 ../../src/yb/master/async_rpc_tasks.cc:595] Check failed Aborted (yb/consensus/consensus.cc:79): Operation submitted in term 6 cannot be replicated in term 8 failed for TABLET5 (table testPauseAfterCreateTableIssued [id=be1c9349e679472bb5c4d8611e805cfc]) description Alter Table RPC for TABLET5 (table testPauseAfterCreateTableIssued [id=be1c9349e679472bb5c4d8611e805cfc]) while running AsyncAlterTable::HandleResponse with response
21451 | [m-1] | @ 0x7fcb0e936d9c yb::LogFatalHandlerSink::send()
21452 | [m-1] | @ 0x7fcb0db41d26 google::LogMessage::SendToLog()
21453 | [m-1] | @ 0x7fcb0db3f7aa google::LogMessage::Flush()
21454 | [m-1] | @ 0x7fcb0db42879 google::LogMessageFatal::~LogMessageFatal()
21455 | [m-1] | @ 0x7fcb18117316 yb::master::AsyncAlterTable::HandleResponse()
21456 | [m-1] | @ 0x7fcb1811a25f yb::master::RetryingTSRpcTask::DoRpcCallback()
21457 | [m-1] | @ 0x7fcb0e9c8fd4 yb::ThreadPool::DispatchThread()
21458 | [m-1] | @ 0x7fcb0e9c56df yb::Thread::SuperviseThread()
21459 | [m-1] | @ 0x7fcb0a2b4694 start_thread
21460 | [m-1] | @ 0x7fcb09ff641d __clone

https://detective-gcp.dev.yugabyte.com/job/github-yugabyte-db-centos-master-gcc-release%2F784%2Fartifact%2Fbuild%2Frelease-gcc-dynamic-ninja%2Fyb-test-logs%2Ftests-integration-tests__master_failover-itest%2FMasterFailoverTestIndexCreation_TestPauseAfterCreateIndexIssued.log?max_lines=3000&start_line=27001

@amitanandaiyer amitanandaiyer moved this from In progress to In Review in Index backfill May 1, 2020
amitanandaiyer added a commit that referenced this issue May 8, 2020
@amitanandaiyer
[#3912] Avoid race in change_metadata_operation
93c8e2e 
Summary:
Use atomic<P*> to avoid race between
Finish() and ToString from updating/accessing request_

Test Plan: ./yb_build.sh tsan --cxx-test integration-tests_master_failover-itest --gtest_filter MasterFailoverTestIndexCreation.TestPauseAfterCreateIndexIssued --no-remote -n 10 --tp 1

Reviewers: bogdan, sergei

Reviewed By: bogdan, sergei

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D8265
YBase features automation moved this from To Do to Done May 9, 2020
Index backfill automation moved this from In Review to Done May 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amitanandaiyer amitanandaiyer

Labels
area/docdb YugabyteDB core features kind/failing-test Tests and testing infra
Projects
Index backfill
  
Done
YBase features
  
Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bmatican @amitanandaiyer