Data race between TwoDCOutputClient::WriteCDCRecordDone and TwoDCOutputClient destructor #6876
Assignees
Comments
tedyu
added a commit
that referenced
this issue
Jan 16, 2021
…one and TwoDCOutputClient destructor Summary: From https://detective-gcp.dev.yugabyte.com/job/github-yugabyte-db-phabricator%2F71843%2Fartifact%2Fbuild%2Ftsan-clang-dynamic-ninja%2Fyb-test-logs%2Ftests-integration-tests__twodc-test%2FTwoDCTestParams__TwoDCTest_TestInsertDeleteWorkloadWithRestart__0.log?class=TwoDCTestParams%2FTwoDCTest&max_lines=3000&name=TestInsertDeleteWorkloadWithRestart%2F0&start_line=3001 : ``` WARNING: ThreadSanitizer: data race (pid=3176) 3018 Read of size 8 at 0x7b48004ccfd8 by main thread (mutexes: write M46578954689579608, read M46860429666290280, write M544226541712483464): 3019 #0 boost::container::operator!=(boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false> const&, boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/include/boost/container/stable_vector.hpp:404:16 (libtserver.so+0x28694d) 3020 #1 yb::rpc::Rpcs::Abort(std::initializer_list<boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false>*>) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/rpc.cc:401:22 (libyrpc.so+0x1cf44c) 3021 #2 yb::tserver::enterprise::TwoDCOutputClient::~TwoDCOutputClient() /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:65:12 (libtserver.so+0x286e04) 3022 #3 yb::tserver::enterprise::TwoDCOutputClient::~TwoDCOutputClient() /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:64:24 (libtserver.so+0x286f79) 3023 #4 std::__1::default_delete<yb::cdc::CDCOutputClient>::operator()(yb::cdc::CDCOutputClient*) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/libcxx/include/c++/v1/memory:2321:5 (libtserver.so+0x2890bf) ``` ``` Previous write of size 8 at 0x7b48004ccfd8 by thread T199 (mutexes: write M214619532964243632): 3056 #0 yb::rpc::Rpcs::Unregister(boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false>*) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/rpc.cc:365:11 (libyrpc.so+0x1cf069) 3057 #1 yb::tserver::enterprise::TwoDCOutputClient::WriteCDCRecordDone(yb::Status const&, yb::tserver::WriteResponsePB const&) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:268:26 (libtserver.so+0x285e4d) 3058 #2 decltype(*(std::__1::forward<yb::tserver::enterprise::TwoDCOutputClient*&>(fp0)).*fp(std::__1::forward<yb::Status const&>(fp1), std::__1::forward<yb::tserver::WriteResponsePB const&>(fp1))) std::__1::__invoke<void (yb::tserver::enterprise::TwoDCOutputClient::*&)(yb::Status const&, yb::tserver::WriteResponsePB const&), yb::tserver::enterprise::TwoDCOutputClient*&, yb::Status const&, yb::tserver::WriteResponsePB const&, void>(void (yb::tserver::enterprise::TwoDCOutputClient::*&)(yb::Status const&, yb::tserver::WriteResponsePB const&), yb::tserver::enterprise::TwoDCOutputClient*&, yb::Status const&, yb::tserver::WriteResponsePB const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/libcxx/include/c++/v1/type_traits:4286:1 (libtserver.so+0x288684) ``` The data race is for write_handle_. This revision adds locking when accessing the write_handle_ Test Plan: Ran test suite via Jenkins Reviewers: nicolas, rahuldesirazu Reviewed By: rahuldesirazu Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D10354
tedyu
added a commit
that referenced
this issue
Jan 16, 2021
…riteCDCRecordDone and TwoDCOutputClient destructor Summary: From https://detective-gcp.dev.yugabyte.com/job/github-yugabyte-db-phabricator%2F71843%2Fartifact%2Fbuild%2Ftsan-clang-dynamic-ninja%2Fyb-test-logs%2Ftests-integration-tests__twodc-test%2FTwoDCTestParams__TwoDCTest_TestInsertDeleteWorkloadWithRestart__0.log?class=TwoDCTestParams%2FTwoDCTest&max_lines=3000&name=TestInsertDeleteWorkloadWithRestart%2F0&start_line=3001 : ``` WARNING: ThreadSanitizer: data race (pid=3176) 3018 Read of size 8 at 0x7b48004ccfd8 by main thread (mutexes: write M46578954689579608, read M46860429666290280, write M544226541712483464): 3019 #0 boost::container::operator!=(boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false> const&, boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/include/boost/container/stable_vector.hpp:404:16 (libtserver.so+0x28694d) 3020 #1 yb::rpc::Rpcs::Abort(std::initializer_list<boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false>*>) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/rpc.cc:401:22 (libyrpc.so+0x1cf44c) 3021 #2 yb::tserver::enterprise::TwoDCOutputClient::~TwoDCOutputClient() /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:65:12 (libtserver.so+0x286e04) 3022 #3 yb::tserver::enterprise::TwoDCOutputClient::~TwoDCOutputClient() /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:64:24 (libtserver.so+0x286f79) 3023 #4 std::__1::default_delete<yb::cdc::CDCOutputClient>::operator()(yb::cdc::CDCOutputClient*) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/libcxx/include/c++/v1/memory:2321:5 (libtserver.so+0x2890bf) ``` ``` Previous write of size 8 at 0x7b48004ccfd8 by thread T199 (mutexes: write M214619532964243632): 3056 #0 yb::rpc::Rpcs::Unregister(boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false>*) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/rpc.cc:365:11 (libyrpc.so+0x1cf069) 3057 #1 yb::tserver::enterprise::TwoDCOutputClient::WriteCDCRecordDone(yb::Status const&, yb::tserver::WriteResponsePB const&) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:268:26 (libtserver.so+0x285e4d) 3058 #2 decltype(*(std::__1::forward<yb::tserver::enterprise::TwoDCOutputClient*&>(fp0)).*fp(std::__1::forward<yb::Status const&>(fp1), std::__1::forward<yb::tserver::WriteResponsePB const&>(fp1))) std::__1::__invoke<void (yb::tserver::enterprise::TwoDCOutputClient::*&)(yb::Status const&, yb::tserver::WriteResponsePB const&), yb::tserver::enterprise::TwoDCOutputClient*&, yb::Status const&, yb::tserver::WriteResponsePB const&, void>(void (yb::tserver::enterprise::TwoDCOutputClient::*&)(yb::Status const&, yb::tserver::WriteResponsePB const&), yb::tserver::enterprise::TwoDCOutputClient*&, yb::Status const&, yb::tserver::WriteResponsePB const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/libcxx/include/c++/v1/type_traits:4286:1 (libtserver.so+0x288684) ``` The data race is for write_handle_. This revision adds locking when accessing the write_handle_ Test Plan: Jenkins: rebase: 2.4 Reviewers: nicolas, rahuldesirazu Reviewed By: rahuldesirazu Subscribers: bogdan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D10382
polarweasel
pushed a commit
to lizayugabyte/yugabyte-db
that referenced
this issue
Mar 9, 2021
…CRecordDone and TwoDCOutputClient destructor Summary: From https://detective-gcp.dev.yugabyte.com/job/github-yugabyte-db-phabricator%2F71843%2Fartifact%2Fbuild%2Ftsan-clang-dynamic-ninja%2Fyb-test-logs%2Ftests-integration-tests__twodc-test%2FTwoDCTestParams__TwoDCTest_TestInsertDeleteWorkloadWithRestart__0.log?class=TwoDCTestParams%2FTwoDCTest&max_lines=3000&name=TestInsertDeleteWorkloadWithRestart%2F0&start_line=3001 : ``` WARNING: ThreadSanitizer: data race (pid=3176) 3018 Read of size 8 at 0x7b48004ccfd8 by main thread (mutexes: write M46578954689579608, read M46860429666290280, write M544226541712483464): 3019 #0 boost::container::operator!=(boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false> const&, boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/include/boost/container/stable_vector.hpp:404:16 (libtserver.so+0x28694d) 3020 yugabyte#1 yb::rpc::Rpcs::Abort(std::initializer_list<boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false>*>) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/rpc.cc:401:22 (libyrpc.so+0x1cf44c) 3021 yugabyte#2 yb::tserver::enterprise::TwoDCOutputClient::~TwoDCOutputClient() /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:65:12 (libtserver.so+0x286e04) 3022 yugabyte#3 yb::tserver::enterprise::TwoDCOutputClient::~TwoDCOutputClient() /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:64:24 (libtserver.so+0x286f79) 3023 yugabyte#4 std::__1::default_delete<yb::cdc::CDCOutputClient>::operator()(yb::cdc::CDCOutputClient*) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/libcxx/include/c++/v1/memory:2321:5 (libtserver.so+0x2890bf) ``` ``` Previous write of size 8 at 0x7b48004ccfd8 by thread T199 (mutexes: write M214619532964243632): 3056 #0 yb::rpc::Rpcs::Unregister(boost::container::stable_vector_iterator<std::__1::shared_ptr<yb::rpc::RpcCommand>*, false>*) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../src/yb/rpc/rpc.cc:365:11 (libyrpc.so+0x1cf069) 3057 yugabyte#1 yb::tserver::enterprise::TwoDCOutputClient::WriteCDCRecordDone(yb::Status const&, yb::tserver::WriteResponsePB const&) /nfusr/centos-gcp-cloud/jenkins-worker-939/jenkins/jenkins-github-yugabyte-db-phabricator-71843/build/tsan-clang-dynamic-ninja/../../ent/src/yb/tserver/twodc_output_client.cc:268:26 (libtserver.so+0x285e4d) 3058 yugabyte#2 decltype(*(std::__1::forward<yb::tserver::enterprise::TwoDCOutputClient*&>(fp0)).*fp(std::__1::forward<yb::Status const&>(fp1), std::__1::forward<yb::tserver::WriteResponsePB const&>(fp1))) std::__1::__invoke<void (yb::tserver::enterprise::TwoDCOutputClient::*&)(yb::Status const&, yb::tserver::WriteResponsePB const&), yb::tserver::enterprise::TwoDCOutputClient*&, yb::Status const&, yb::tserver::WriteResponsePB const&, void>(void (yb::tserver::enterprise::TwoDCOutputClient::*&)(yb::Status const&, yb::tserver::WriteResponsePB const&), yb::tserver::enterprise::TwoDCOutputClient*&, yb::Status const&, yb::tserver::WriteResponsePB const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201111062958-7047513803-centos7-linuxbrew/installed/tsan/libcxx/include/c++/v1/type_traits:4286:1 (libtserver.so+0x288684) ``` The data race is for write_handle_. This revision adds locking when accessing the write_handle_ Test Plan: Ran test suite via Jenkins Reviewers: nicolas, rahuldesirazu Reviewed By: rahuldesirazu Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D10354
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From https://detective-gcp.dev.yugabyte.com/job/github-yugabyte-db-phabricator%2F71843%2Fartifact%2Fbuild%2Ftsan-clang-dynamic-ninja%2Fyb-test-logs%2Ftests-integration-tests__twodc-test%2FTwoDCTestParams__TwoDCTest_TestInsertDeleteWorkloadWithRestart__0.log?class=TwoDCTestParams%2FTwoDCTest&max_lines=3000&name=TestInsertDeleteWorkloadWithRestart%2F0&start_line=3001 :
The data race is for write_handle_
The text was updated successfully, but these errors were encountered: