静态链接libbrpc.a Clang TSAN校验报错

[lday0321]

Describe the bug (描述bug)
使用Clang，开TSAN编译，生成静态库libbrpc.a，同样开TSAN编译，生成可执行程序，运行程序，TSAN校验报错

To Reproduce (复现方法)
Clang开TSAN编译libbrpc.a

CC=clang CXX=clang++ \
cmake  \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_STANDARD=11 \
-DWITH_GLOG=ON \
-DWITH_THRIFT=OFF \
-DBUILD_UNIT_TESTS=OFF \
-DDOWNLOAD_GTEST=OFF \
-DGFLAGS_LIBRARY=/opt/gflagsRelease/lib/libgflags.so -DGFLAGS_INCLUDE_PATH=/opt/gflagsRelease/include \
-DLEVELDB_LIB=/opt/leveldbRelease/lib64/libleveldb.so -DLEVELDB_INCLUDE_PATH=/opt/leveldbRelease/include \
-DProtobuf_LIBRARY=/opt/protobuf3Release/lib/libprotobuf.so -DProtobuf_INCLUDE_DIR=/opt/protobuf3Release/include \
-DPROTOC_LIB=/opt/protobuf3Release/lib/libprotoc.so \
-DGLOG_INCLUDE_PATH=/opt/glogRelease/include -DGLOG_LIB=/opt/glogRelease/lib64/libglog.so \
-DCMAKE_INSTALL_PREFIX=/install_libs/brpc \
-DCMAKE_CPP_FLAGS="-fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -DTHREAD_SANITIZER -DDYNAMIC_ANNOTATIONS_ENABLED=1 -fsanitize=thread" \
..

简单的TSAN Check UT:

// /v/tsan_demo/tsan_test.cc
#include <chrono>
#include <string>
#include <thread>

#include "brpc/server.h"
#include "butil/endpoint.h"
#include "glog/logging.h"
#include "gmock/gmock.h"
#include "gtest/gtest.h"

namespace my {
namespace {

class MyTest : public ::testing::Test {
 public:
  MyTest() = default;
};

TEST_F(MyTest, TSAN_Check) {
  LOG(INFO) << "Hello, TSAN!";
}

TEST_F(MyTest, TSAN_Check_brpc) {
  LOG(INFO) << "Hello, TSAN brpc!";
  butil::ip_t ip = butil::my_ip();
  uint16_t port = 12345;
  brpc::Server server;
  butil::EndPoint end_point = butil::EndPoint(ip, port);
  int rc = server.Start(end_point, NULL);
  ASSERT_EQ(0, rc);
  std::this_thread::sleep_for(std::chrono::seconds(20));
  server.Stop(0);
  server.Join();
}

}  // namespace
}  // namespace my

int main(int argc, char** argv) {
  ::testing::InitGoogleTest(&argc, argv);
  ::testing::InitGoogleMock(&argc, argv);
  return RUN_ALL_TESTS();
}

同样开TSAN，编译连接tsan_test:

# 编译
/usr/bin/clang++ -I/v/tsan_demo -I/install_libs/brpc/include -Wno-inconsistent-missing-override -DTHREAD_SANITIZER -DDYNAMIC_ANNOTATIONS_ENABLED=1 -fsanitize=thread -fPIC -g -std=gnu++11 -MD -MT /v/tsan_demo/build_tsan/tsan_test.cc.o -MF /v/tsan_demo/build_tsan/tsan_test.cc.o.d -o /v/tsan_demo/build_tsan/tsan_test.cc.o -c /v/tsan_demo/tsan_test.cc

# 连接
/usr/bin/clang++  -DTHREAD_SANITIZER -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DCOMPILER_CLANG -fsanitize=thread -fPIC -g -fsanitize=thread -pie /v/tsan_demo/build_tsan/tsan_test.cc.o -o /v/tsan_demo/build_tsan/tsan_test  -lgtest -lgmock /install_libs/brpc/lib64/libbrpc.a -lprotobuf -lleveldb -lglog -lrt -lssl -lcrypto -ldl -lz -lsnappy -lzstd -llz4 -lbz2 /usr/lib64/libunwind.so -lgflags -lpthread

Expected behavior (期望行为)
期望行为是UT正常运行， 但是实际会保很多TSAN校验错误：

WARNING: ThreadSanitizer: data race (pid=73930)
  Read of size 8 at 0x7b2c00000120 by main thread:
    #0 butil::LinkNode<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>::InsertBefore(butil::LinkNode<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>*) /v/brpc/incubator-brpc/src/butil/containers/linked_list.h:96:26 (tsan_test+0x2d7c77)
    #1 butil::LinkedList<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>::Append(butil::LinkNode<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>*) /v/brpc/incubator-brpc/src/butil/containers/linked_list.h:171 (tsan_test+0x2d7c77)
    #2 bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::get_or_create_tls_agent() /v/brpc/incubator-brpc/src/bvar/detail/combiner.h:310 (tsan_test+0x2d7c77)
    #3 bvar::Reducer<bvar::detail::Sampler*, bvar::detail::CombineSampler, bvar::detail::VoidOp>::operator<<(bvar::detail::Sampler* const&) /v/brpc/incubator-brpc/src/bvar/reducer.h:194:35 (tsan_test+0x2d2d88)
    #4 bvar::detail::Sampler::schedule() /v/brpc/incubator-brpc/src/bvar/detail/sampler.cpp:201:53 (tsan_test+0x2d2c9a)
    #5 bvar::PassiveStatus<int>::expose_impl(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, bvar::DisplayFilter) /v/brpc/incubator-brpc/src/bvar/passive_status.h:179:30 (tsan_test+0x28b998)
    #6 bvar::Variable::expose(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, bvar::DisplayFilter) /v/brpc/incubator-brpc/src/bvar/variable.h:132:16 (tsan_test+0x178c66)
    #7 bvar::PassiveStatus<int>::PassiveStatus(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, int (*)(void*), void*) /v/brpc/incubator-brpc/src/bvar/passive_status.h:85 (tsan_test+0x178c66)
    #8 __cxx_global_var_init.1 /v/brpc/incubator-brpc/src/bthread/key.cpp:266 (tsan_test+0x178c66)
    #9 _GLOBAL__sub_I_key.cpp /v/brpc/incubator-brpc/src/bthread/key.cpp (tsan_test+0x178c66)
    #10 __libc_csu_init <null> (tsan_test+0x596e2c)

  Previous write of size 8 at 0x7b2c00000120 by thread T1:
    #0 butil::LinkNode<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>::InsertBefore(butil::LinkNode<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>*) /v/brpc/incubator-brpc/src/butil/containers/linked_list.h:98:18 (tsan_test+0x2d7c9f)
    #1 butil::LinkedList<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>::Append(butil::LinkNode<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>*) /v/brpc/incubator-brpc/src/butil/containers/linked_list.h:171 (tsan_test+0x2d7c9f)
    #2 bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::get_or_create_tls_agent() /v/brpc/incubator-brpc/src/bvar/detail/combiner.h:310 (tsan_test+0x2d7c9f)
    #3 bvar::Reducer<bvar::detail::Sampler*, bvar::detail::CombineSampler, bvar::detail::VoidOp>::operator<<(bvar::detail::Sampler* const&) /v/brpc/incubator-brpc/src/bvar/reducer.h:194:35 (tsan_test+0x2d2d88)
    #4 bvar::detail::Sampler::schedule() /v/brpc/incubator-brpc/src/bvar/detail/sampler.cpp:201:53 (tsan_test+0x2d5deb)
    #5 bvar::PassiveStatus<double>::get_sampler() /v/brpc/incubator-brpc/src/bvar/passive_status.h:146 (tsan_test+0x2d5deb)
    #6 bvar::detail::WindowBase<bvar::PassiveStatus<double>, (bvar::SeriesFrequency)1>::WindowBase(bvar::PassiveStatus<double>*, long) /v/brpc/incubator-brpc/src/bvar/window.h:82:25 (tsan_test+0x2d5797)
    #7 bvar::PerSecond<bvar::PassiveStatus<double> >::PerSecond(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, bvar::PassiveStatus<double>*, long) /v/brpc/incubator-brpc/src/bvar/window.h:209:11 (tsan_test+0x2d2480)
    #8 bvar::detail::SamplerCollector::run() /v/brpc/incubator-brpc/src/bvar/detail/sampler.cpp:143 (tsan_test+0x2d2480)
    #9 bvar::detail::SamplerCollector::sampling_thread(void*) /v/brpc/incubator-brpc/src/bvar/detail/sampler.cpp:110:46 (tsan_test+0x2d3839)

  Location is heap block of size 168 at 0x7b2c000000b0 allocated by main thread:
    #0 operator new(unsigned long) <null> (tsan_test+0x226f57)
    #1 butil::GetLeakySingleton<bvar::detail::SamplerCollector>::create_leaky_singleton() /v/brpc/incubator-brpc/src/butil/memory/singleton_on_pthread_once.h:42:14 (tsan_test+0x2d2ed9)
    #2 pthread_once <null> (tsan_test+0x1b77df)
    #3 bvar::detail::SamplerCollector* butil::get_leaky_singleton<bvar::detail::SamplerCollector>() /v/brpc/incubator-brpc/src/butil/memory/singleton_on_pthread_once.h:60:5 (tsan_test+0x2d2c74)
    #4 bvar::detail::Sampler::schedule() /v/brpc/incubator-brpc/src/bvar/detail/sampler.cpp:201 (tsan_test+0x2d2c74)
    #5 bvar::PassiveStatus<int>::expose_impl(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, bvar::DisplayFilter) /v/brpc/incubator-brpc/src/bvar/passive_status.h:179:30 (tsan_test+0x28b998)
    #6 bvar::Variable::expose(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, bvar::DisplayFilter) /v/brpc/incubator-brpc/src/bvar/variable.h:132:16 (tsan_test+0x178c66)
    #7 bvar::PassiveStatus<int>::PassiveStatus(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, int (*)(void*), void*) /v/brpc/incubator-brpc/src/bvar/passive_status.h:85 (tsan_test+0x178c66)
    #8 __cxx_global_var_init.1 /v/brpc/incubator-brpc/src/bthread/key.cpp:266 (tsan_test+0x178c66)
    #9 _GLOBAL__sub_I_key.cpp /v/brpc/incubator-brpc/src/bthread/key.cpp (tsan_test+0x178c66)
    #10 __libc_csu_init <null> (tsan_test+0x596e2c)

  Thread T1 (tid=73932, running) created by main thread at:
    #0 pthread_create <null> (tsan_test+0x1b6fd6)
    #1 bvar::detail::SamplerCollector::create_sampling_thread() /v/brpc/incubator-brpc/src/bvar/detail/sampler.cpp:90:24 (tsan_test+0x2d30a0)
    #2 bvar::detail::SamplerCollector::SamplerCollector() /v/brpc/incubator-brpc/src/bvar/detail/sampler.cpp:67:9 (tsan_test+0x2d3037)
    #3 butil::GetLeakySingleton<bvar::detail::SamplerCollector>::create_leaky_singleton() /v/brpc/incubator-brpc/src/butil/memory/singleton_on_pthread_once.h:42:18 (tsan_test+0x2d2ee4)
    #4 pthread_once <null> (tsan_test+0x1b77df)
    #5 bvar::detail::SamplerCollector* butil::get_leaky_singleton<bvar::detail::SamplerCollector>() /v/brpc/incubator-brpc/src/butil/memory/singleton_on_pthread_once.h:60:5 (tsan_test+0x2d2c74)
    #6 bvar::detail::Sampler::schedule() /v/brpc/incubator-brpc/src/bvar/detail/sampler.cpp:201 (tsan_test+0x2d2c74)
    #7 bvar::PassiveStatus<int>::expose_impl(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, bvar::DisplayFilter) /v/brpc/incubator-brpc/src/bvar/passive_status.h:179:30 (tsan_test+0x28b998)
    #8 bvar::Variable::expose(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, bvar::DisplayFilter) /v/brpc/incubator-brpc/src/bvar/variable.h:132:16 (tsan_test+0x178c66)
    #9 bvar::PassiveStatus<int>::PassiveStatus(butil::BasicStringPiece<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, int (*)(void*), void*) /v/brpc/incubator-brpc/src/bvar/passive_status.h:85 (tsan_test+0x178c66)
    #10 __cxx_global_var_init.1 /v/brpc/incubator-brpc/src/bthread/key.cpp:266 (tsan_test+0x178c66)
    #11 _GLOBAL__sub_I_key.cpp /v/brpc/incubator-brpc/src/bthread/key.cpp (tsan_test+0x178c66)
    #12 __libc_csu_init <null> (tsan_test+0x596e2c)

SUMMARY: ThreadSanitizer: data race /v/brpc/incubator-brpc/src/butil/containers/linked_list.h:96:26 in butil::LinkNode<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>::InsertBefore(butil::LinkNode<bvar::detail::AgentCombiner<bvar::detail::Sampler*, bvar::detail::Sampler*, bvar::detail::CombineSampler>::Agent>*)

Versions (各种版本)
OS: Linux 3.10.0-327
Compiler: clang-8
brpc: master分支( 2c98d0a, "Update oncall.md")
protobuf: protobuf-3.17.3

Additional context/screenshots (更多上下文/截图)

[wwbmmm]

注释掉这里的pthread_mutex_lock/unlock试试
https://github.com/apache/incubator-brpc/blob/master/src/bthread/mutex.cpp#L811

[lday0321]

感谢回复! 移除这里的pthread_mutex_lock/pthread_mutex_unlock，combiner.h:310这里的TSAN warning的确没了。但还是些其他的报错，例如：

WARNING: ThreadSanitizer: data race (pid=12180)
  Write of size 4 at 0x7b4800020080 by main thread (mutexes: write M924):
    #0 butil::BoundedQueue<unsigned long>::push(unsigned long const&) /v/brpc/incubator-brpc/src/butil/containers/bounded_queue.h:103:13 (tsan_test+0x31aee7)
    #1 bthread::RemoteTaskQueue::push_locked(unsigned long) /v/brpc/incubator-brpc/src/bthread/remote_task_queue.h:69 (tsan_test+0x31aee7)
    #2 bthread::TaskGroup::ready_to_run_remote(unsigned long, bool) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:669 (tsan_test+0x31aee7)
    #3 int bthread::TaskGroup::start_background<true>(unsigned long*, bthread_attr_t const*, void* (*)(void*), void*) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:446:9 (tsan_test+0x31fa09)
    #4 bthread::start_from_non_worker(unsigned long*, bthread_attr_t const*, void* (*)(void*), void*) /v/brpc/incubator-brpc/src/bthread/bthread.cpp:144:35 (tsan_test+0x2f0411)
    #5 bthread_start_background /v/brpc/incubator-brpc/src/bthread/bthread.cpp:193 (tsan_test+0x2f0411)
    #6 brpc::GlobalInitializeOrDieImpl() /v/brpc/incubator-brpc/src/brpc/global.cpp:612:5 (tsan_test+0x393922)
    #7 pthread_once <null> (tsan_test+0x1bf41f)
    #8 brpc::GlobalInitializeOrDie() /v/brpc/incubator-brpc/src/brpc/global.cpp:617:9 (tsan_test+0x391b44)
    #9 brpc::Server::InitializeOnce() /v/brpc/incubator-brpc/src/brpc/server.cpp:610:5 (tsan_test+0x279436)
    #10 brpc::Server::StartInternal(in_addr const&, brpc::PortRange const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:712:9 (tsan_test+0x2796bc)
    #11 brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:1062:12 (tsan_test+0x27deef)
    #12 my::(anonymous namespace)::MyTest_TSAN_Check_brpc_Test::TestBody() /v/tsan_demo/tsan_test.cc:29:19 (tsan_test+0x23067c)
    #13 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (tsan_test+0x264c1c)
    #14 main /v/tsan_demo/tsan_test.cc:46:10 (tsan_test+0x22fea8)

  Previous read of size 4 at 0x7b4800020080 by thread T5:
    #0 butil::BoundedQueue<unsigned long>::empty() const /v/brpc/incubator-brpc/src/butil/containers/bounded_queue.h:260:34 (tsan_test+0x30f002)
    #1 bthread::RemoteTaskQueue::pop(unsigned long*) /v/brpc/incubator-brpc/src/bthread/remote_task_queue.h:52 (tsan_test+0x30f002)
    #2 bthread::TaskControl::steal_task(unsigned long*, unsigned long*, unsigned long) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:354 (tsan_test+0x30f002)
    #3 bthread::TaskGroup::steal_task(unsigned long*) /v/brpc/incubator-brpc/src/bthread/task_group.h:221:26 (tsan_test+0x32017f)
    #4 bthread::TaskGroup::wait_task(unsigned long*) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:125:13 (tsan_test+0x31b5cd)
    #5 bthread::TaskGroup::run_main_task() /v/brpc/incubator-brpc/src/bthread/task_group.cpp:152 (tsan_test+0x31b5cd)
    #6 bthread::TaskControl::worker_thread(void*) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:76:8 (tsan_test+0x30c7f3)

  Location is heap block of size 384 at 0x7b480001ff80 allocated by thread T4:
    #0 operator new(unsigned long, std::nothrow_t const&) <null> (tsan_test+0x22ee53)
    #1 bthread::TaskControl::create_group() /v/brpc/incubator-brpc/src/bthread/task_control.cpp:89:20 (tsan_test+0x30cb8d)
    #2 bthread::TaskControl::worker_thread(void*) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:65:23 (tsan_test+0x30c66d)

  Mutex M924 (0x7b4800020098) created at:
    #0 pthread_mutex_init <null> (tsan_test+0x1bf03a)
    #1 butil::Mutex::Mutex() /v/brpc/incubator-brpc/src/butil/synchronization/lock.h:51:9 (tsan_test+0x31c007)
    #2 bthread::RemoteTaskQueue::RemoteTaskQueue() /v/brpc/incubator-brpc/src/bthread/remote_task_queue.h:38 (tsan_test+0x31c007)
    #3 bthread::TaskGroup::TaskGroup(bthread::TaskControl*) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:176 (tsan_test+0x31c007)
    #4 bthread::TaskControl::create_group() /v/brpc/incubator-brpc/src/bthread/task_control.cpp:89:39 (tsan_test+0x30cba4)
    #5 bthread::TaskControl::worker_thread(void*) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:65:23 (tsan_test+0x30c66d)

  Thread T5 (tid=12186, running) created by main thread at:
    #0 pthread_create <null> (tsan_test+0x1bec16)
    #1 bthread::TaskControl::init(int) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:165:24 (tsan_test+0x30dc40)
    #2 bthread::get_or_new_task_control() /v/brpc/incubator-brpc/src/bthread/bthread.cpp:91:12 (tsan_test+0x2f1753)
    #3 bthread::start_from_non_worker(unsigned long*, bthread_attr_t const*, void* (*)(void*), void*) /v/brpc/incubator-brpc/src/bthread/bthread.cpp:128:22 (tsan_test+0x2f03d7)
    #4 bthread_start_background /v/brpc/incubator-brpc/src/bthread/bthread.cpp:193 (tsan_test+0x2f03d7)
    #5 brpc::GlobalInitializeOrDieImpl() /v/brpc/incubator-brpc/src/brpc/global.cpp:612:5 (tsan_test+0x393922)
    #6 pthread_once <null> (tsan_test+0x1bf41f)
    #7 brpc::GlobalInitializeOrDie() /v/brpc/incubator-brpc/src/brpc/global.cpp:617:9 (tsan_test+0x391b44)
    #8 brpc::Server::InitializeOnce() /v/brpc/incubator-brpc/src/brpc/server.cpp:610:5 (tsan_test+0x279436)
    #9 brpc::Server::StartInternal(in_addr const&, brpc::PortRange const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:712:9 (tsan_test+0x2796bc)
    #10 brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:1062:12 (tsan_test+0x27deef)
    #11 my::(anonymous namespace)::MyTest_TSAN_Check_brpc_Test::TestBody() /v/tsan_demo/tsan_test.cc:29:19 (tsan_test+0x23067c)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (tsan_test+0x264c1c)
    #13 main /v/tsan_demo/tsan_test.cc:46:10 (tsan_test+0x22fea8)

  Thread T4 (tid=12185, running) created by main thread at:
    #0 pthread_create <null> (tsan_test+0x1bec16)
    #1 bthread::TaskControl::init(int) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:165:24 (tsan_test+0x30dc40)
    #2 bthread::get_or_new_task_control() /v/brpc/incubator-brpc/src/bthread/bthread.cpp:91:12 (tsan_test+0x2f1753)
    #3 bthread::start_from_non_worker(unsigned long*, bthread_attr_t const*, void* (*)(void*), void*) /v/brpc/incubator-brpc/src/bthread/bthread.cpp:128:22 (tsan_test+0x2f03d7)
    #4 bthread_start_background /v/brpc/incubator-brpc/src/bthread/bthread.cpp:193 (tsan_test+0x2f03d7)
    #5 brpc::GlobalInitializeOrDieImpl() /v/brpc/incubator-brpc/src/brpc/global.cpp:612:5 (tsan_test+0x393922)
    #6 pthread_once <null> (tsan_test+0x1bf41f)
    #7 brpc::GlobalInitializeOrDie() /v/brpc/incubator-brpc/src/brpc/global.cpp:617:9 (tsan_test+0x391b44)
    #8 brpc::Server::InitializeOnce() /v/brpc/incubator-brpc/src/brpc/server.cpp:610:5 (tsan_test+0x279436)
    #9 brpc::Server::StartInternal(in_addr const&, brpc::PortRange const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:712:9 (tsan_test+0x2796bc)
    #10 brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:1062:12 (tsan_test+0x27deef)
    #11 my::(anonymous namespace)::MyTest_TSAN_Check_brpc_Test::TestBody() /v/tsan_demo/tsan_test.cc:29:19 (tsan_test+0x23067c)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (tsan_test+0x264c1c)
    #13 main /v/tsan_demo/tsan_test.cc:46:10 (tsan_test+0x22fea8)

SUMMARY: ThreadSanitizer: data race /v/brpc/incubator-brpc/src/butil/containers/bounded_queue.h:103:13 in butil::BoundedQueue<unsigned long>::push(unsigned long const&)

这里我感觉像是brpc有意为之?

我在另一个报在timer_thread.cpp的data race:

WARNING: ThreadSanitizer: data race (pid=12180)
  Write of size 8 at 0x7b5c00001650 by thread T7:
    #0 bthread::TimerThread::Bucket::schedule(void (*)(void*), void*, timespec const&) /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:207:20 (tsan_test+0x323497)
    #1 bthread::TimerThread::schedule(void (*)(void*), void*, timespec const&) /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:226:10 (tsan_test+0x3235fa)
    #2 bthread::TaskGroup::_add_sleep_event(void*) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:745:43 (tsan_test+0x31e62a)
    #3 bthread::TaskGroup::sched_to(bthread::TaskGroup**, bthread::TaskMeta*) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:623:9 (tsan_test+0x31def5)
    #4 bthread::TaskGroup::sched_to(bthread::TaskGroup**, unsigned long) /v/brpc/incubator-brpc/src/bthread/task_group_inl.h:78:5 (tsan_test+0x2f4954)
    #5 bthread::TaskGroup::sched(bthread::TaskGroup**) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:561:5 (tsan_test+0x31ea8c)
    #6 bthread::TaskGroup::usleep(bthread::TaskGroup**, unsigned long) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:790 (tsan_test+0x31ea8c)
    #7 bthread_usleep /v/brpc/incubator-brpc/src/bthread/bthread.cpp:364:16 (tsan_test+0x2f0e98)
    #8 brpc::GlobalUpdate(void*) /v/brpc/incubator-brpc/src/brpc/global.cpp:234:17 (tsan_test+0x3944ed)
    #9 bthread::TaskGroup::task_runner(long) /v/brpc/incubator-brpc/src/bthread/task_group.cpp:295:29 (tsan_test+0x31bb3e)
    #10 bthread_make_fcontext <null> (tsan_test+0x2f74c0)
    #11 bthread::TaskGroup::sched_to(bthread::TaskGroup**, unsigned long) /v/brpc/incubator-brpc/src/bthread/task_group_inl.h:78:5 (tsan_test+0x2f4954)
    #12 bthread::TaskGroup::run_main_task() /v/brpc/incubator-brpc/src/bthread/task_group.cpp:153:9 (tsan_test+0x31b5e6)
    #13 bthread::TaskControl::worker_thread(void*) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:76:8 (tsan_test+0x30c7f3)

  Previous read of size 8 at 0x7b5c00001650 by thread T2:
    #0 bthread::TimerThread::Bucket::consume_tasks() /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:169:9 (tsan_test+0x321e1c)
    #1 bthread::TimerThread::run() /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:354 (tsan_test+0x321e1c)
    #2 bthread::TimerThread::run_this(void*) /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:120:37 (tsan_test+0x321949)

  Location is heap block of size 896 at 0x7b5c00001500 allocated by main thread:
    #0 operator new[](unsigned long, std::nothrow_t const&) <null> (tsan_test+0x22eef3)
    #1 bthread::TimerThread::start(bthread::TimerThreadOptions const*) /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:154:16 (tsan_test+0x322e84)
    #2 bthread::init_global_timer_thread() /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:465:36 (tsan_test+0x323d1d)
    #3 pthread_once <null> (tsan_test+0x1bf41f)
    #4 bthread::get_or_create_global_timer_thread() /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:474:5 (tsan_test+0x323b81)
    #5 bthread::TaskControl::init(int) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:158:9 (tsan_test+0x30dae6)
    #6 bthread::get_or_new_task_control() /v/brpc/incubator-brpc/src/bthread/bthread.cpp:91:12 (tsan_test+0x2f1753)
    #7 bthread::start_from_non_worker(unsigned long*, bthread_attr_t const*, void* (*)(void*), void*) /v/brpc/incubator-brpc/src/bthread/bthread.cpp:128:22 (tsan_test+0x2f03d7)
    #8 bthread_start_background /v/brpc/incubator-brpc/src/bthread/bthread.cpp:193 (tsan_test+0x2f03d7)
    #9 brpc::GlobalInitializeOrDieImpl() /v/brpc/incubator-brpc/src/brpc/global.cpp:612:5 (tsan_test+0x393922)
    #10 pthread_once <null> (tsan_test+0x1bf41f)
    #11 brpc::GlobalInitializeOrDie() /v/brpc/incubator-brpc/src/brpc/global.cpp:617:9 (tsan_test+0x391b44)
    #12 brpc::Server::InitializeOnce() /v/brpc/incubator-brpc/src/brpc/server.cpp:610:5 (tsan_test+0x279436)
    #13 brpc::Server::StartInternal(in_addr const&, brpc::PortRange const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:712:9 (tsan_test+0x2796bc)
    #14 brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:1062:12 (tsan_test+0x27deef)
    #15 my::(anonymous namespace)::MyTest_TSAN_Check_brpc_Test::TestBody() /v/tsan_demo/tsan_test.cc:29:19 (tsan_test+0x23067c)
    #16 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (tsan_test+0x264c1c)
    #17 main /v/tsan_demo/tsan_test.cc:46:10 (tsan_test+0x22fea8)

  Thread T7 (tid=12188, running) created by main thread at:
    #0 pthread_create <null> (tsan_test+0x1bec16)
    #1 bthread::TaskControl::init(int) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:165:24 (tsan_test+0x30dc40)
    #2 bthread::get_or_new_task_control() /v/brpc/incubator-brpc/src/bthread/bthread.cpp:91:12 (tsan_test+0x2f1753)
    #3 bthread::start_from_non_worker(unsigned long*, bthread_attr_t const*, void* (*)(void*), void*) /v/brpc/incubator-brpc/src/bthread/bthread.cpp:128:22 (tsan_test+0x2f03d7)
    #4 bthread_start_background /v/brpc/incubator-brpc/src/bthread/bthread.cpp:193 (tsan_test+0x2f03d7)
    #5 brpc::GlobalInitializeOrDieImpl() /v/brpc/incubator-brpc/src/brpc/global.cpp:612:5 (tsan_test+0x393922)
    #6 pthread_once <null> (tsan_test+0x1bf41f)
    #7 brpc::GlobalInitializeOrDie() /v/brpc/incubator-brpc/src/brpc/global.cpp:617:9 (tsan_test+0x391b44)
    #8 brpc::Server::InitializeOnce() /v/brpc/incubator-brpc/src/brpc/server.cpp:610:5 (tsan_test+0x279436)
    #9 brpc::Server::StartInternal(in_addr const&, brpc::PortRange const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:712:9 (tsan_test+0x2796bc)
    #10 brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:1062:12 (tsan_test+0x27deef)
    #11 my::(anonymous namespace)::MyTest_TSAN_Check_brpc_Test::TestBody() /v/tsan_demo/tsan_test.cc:29:19 (tsan_test+0x23067c)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (tsan_test+0x264c1c)
    #13 main /v/tsan_demo/tsan_test.cc:46:10 (tsan_test+0x22fea8)

  Thread T2 (tid=12183, running) created by main thread at:
    #0 pthread_create <null> (tsan_test+0x1bec16)
    #1 bthread::TimerThread::start(bthread::TimerThreadOptions const*) /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:159:21 (tsan_test+0x32313f)
    #2 bthread::init_global_timer_thread() /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:465:36 (tsan_test+0x323d1d)
    #3 pthread_once <null> (tsan_test+0x1bf41f)
    #4 bthread::get_or_create_global_timer_thread() /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:474:5 (tsan_test+0x323b81)
    #5 bthread::TaskControl::init(int) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:158:9 (tsan_test+0x30dae6)
    #6 bthread::get_or_new_task_control() /v/brpc/incubator-brpc/src/bthread/bthread.cpp:91:12 (tsan_test+0x2f1753)
    #7 bthread::start_from_non_worker(unsigned long*, bthread_attr_t const*, void* (*)(void*), void*) /v/brpc/incubator-brpc/src/bthread/bthread.cpp:128:22 (tsan_test+0x2f03d7)
    #8 bthread_start_background /v/brpc/incubator-brpc/src/bthread/bthread.cpp:193 (tsan_test+0x2f03d7)
    #9 brpc::GlobalInitializeOrDieImpl() /v/brpc/incubator-brpc/src/brpc/global.cpp:612:5 (tsan_test+0x393922)
    #10 pthread_once <null> (tsan_test+0x1bf41f)
    #11 brpc::GlobalInitializeOrDie() /v/brpc/incubator-brpc/src/brpc/global.cpp:617:9 (tsan_test+0x391b44)
    #12 brpc::Server::InitializeOnce() /v/brpc/incubator-brpc/src/brpc/server.cpp:610:5 (tsan_test+0x279436)
    #13 brpc::Server::StartInternal(in_addr const&, brpc::PortRange const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:712:9 (tsan_test+0x2796bc)
    #14 brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) /v/brpc/incubator-brpc/src/brpc/server.cpp:1062:12 (tsan_test+0x27deef)
    #15 my::(anonymous namespace)::MyTest_TSAN_Check_brpc_Test::TestBody() /v/tsan_demo/tsan_test.cc:29:19 (tsan_test+0x23067c)
    #16 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (tsan_test+0x264c1c)
    #17 main /v/tsan_demo/tsan_test.cc:46:10 (tsan_test+0x22fea8)

SUMMARY: ThreadSanitizer: data race /v/brpc/incubator-brpc/src/bthread/timer_thread.cpp:207:20 in bthread::TimerThread::Bucket::schedule(void (*)(void*), void*, timespec const&)

中看到timer_thread.cpp:169，有一段注释说是为了减少一次mutex访问：

类似这类场景，似乎TSAN也无法正确识别。

完整的error msg见附件:
tsan_error.log

[lday0321]

另外，这个UT程序，跑到server.Join的时候就挂掉了, 非TSAN时，能够正常结束。

TEST_F(MyTest, TSAN_Check_brpc) {
  LOG(INFO) << "Hello, TSAN brpc!";
  butil::ip_t ip = butil::my_ip();
  uint16_t port = 12345;
  brpc::Server server;
  butil::EndPoint end_point = butil::EndPoint(ip, port);
  int rc = server.Start(end_point, NULL);
  ASSERT_EQ(0, rc);
  LOG(INFO) << "Hello, before sleep";
  std::this_thread::sleep_for(std::chrono::seconds(10));
  LOG(INFO) << "Hello, before stop";
  server.Stop(0);
  LOG(INFO) << "Hello, before join";
  server.Join();
  LOG(INFO) << "Hello, before exit";
}

出错信息：

I0128 20:44:08.095814 12180 tsan_test.cc:33] Hello, before stop
I0128 20:44:08.095880 12180 server.cpp:1101] Server[] is going to quit
I0128 20:44:08.097417 12180 tsan_test.cc:35] Hello, before join
ThreadSanitizer:DEADLYSIGNAL
==12180==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f25c05052be bp 0x000000000000 sp 0x7f25961fe9e8 T12233)
==12180==The signal is caused by a WRITE memory access.
==12180==Hint: address points to the zero page.
    #0 __sanitizer::internal_memcpy(void*, void const*, unsigned long) <null> (tsan_test+0x1952bd)
    #1 __tsan::VarSizeStackTrace::Init(unsigned long const*, unsigned long, unsigned long) <null> (tsan_test+0x229813)
    #2 __tsan::ReportRace(__tsan::ThreadState*) <null> (tsan_test+0x2278db)
    #3 <null> <null> (tsan_test+0x22c6dd)
    #4 __tsan_write4 <null> (tsan_test+0x21c41c)
    #5 bthread::TaskGroup::task_runner(long) /v/brpc/incubator-brpc/src/bthread/task_group.cpp (tsan_test+0x31bcc7)
    #6 bthread_make_fcontext <null> (tsan_test+0x2f74c0)

ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV (/v/tsan_demo/build_tsan/tsan_test+0x1952bd) in __sanitizer::internal_memcpy(void*, void const*, unsigned long)
==12180==ABORTING

[wwbmmm]

对，这两个地方的原理都是类似的，就是先判断一下非空，是的话再加锁。这个实际上是安全的，但确实会引发检查器报错。
最后的crash，看起来是TSAN和bthread不兼容导致的，bthread的堆栈是从0这个地址开始的，估计TSAN的遍历堆栈的时候没有处理这种情况

[lday0321]

我把整个brpc库加到tsan suppression file(.tsan_suppression_file)里面，其他warning没有了，但是最后的那个ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000还是会报错。

# .tsan_suppression_file
race:brpc

执行UT结果：

#TSAN_OPTIONS="suppressions=/v/tsan_demo/.tsan_suppression_file" ./tsan_test --gtest_filter=MyTest.TSAN_Check_brpc
Note: Google Test filter = MyTest.TSAN_Check_brpc
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MyTest
[ RUN      ] MyTest.TSAN_Check_brpc
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0129 17:52:53.699350  9388 tsan_test.cc:24] Hello, TSAN brpc!
I0129 17:52:53.865265  9388 server.cpp:1046] Server[] is serving on port=12345.
I0129 17:52:53.865690  9388 server.cpp:1049] Check out http://i39f13438.sqa.eu95:12345 in web browser.
I0129 17:52:53.865866  9388 tsan_test.cc:31] Hello, before sleep
LLVMSymbolizer: error reading file: No such file or directory
I0129 17:53:03.865993  9388 tsan_test.cc:33] Hello, before stop
I0129 17:53:03.866055  9388 server.cpp:1101] Server[] is going to quit
I0129 17:53:03.867576  9388 tsan_test.cc:35] Hello, before join
ThreadSanitizer:DEADLYSIGNAL
==9388==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f31a05d23de bp 0x000000000000 sp 0x7f31761789e8 T9444)
==9388==The signal is caused by a WRITE memory access.
==9388==Hint: address points to the zero page.
    #0 __sanitizer::internal_memcpy(void*, void const*, unsigned long) <null> (tsan_test+0x1933dd)
    #1 __tsan::VarSizeStackTrace::Init(unsigned long const*, unsigned long, unsigned long) <null> (tsan_test+0x227933)
    #2 __tsan::ReportRace(__tsan::ThreadState*) <null> (tsan_test+0x2259fb)
    #3 <null> <null> (tsan_test+0x22a7fd)
    #4 __tsan_write4 <null> (tsan_test+0x21a53c)
    #5 bthread::TaskGroup::task_runner(long) /v/brpc/incubator-brpc/src/bthread/task_group.cpp (tsan_test+0x313e99)
    #6 bthread_make_fcontext <null> (tsan_test+0x2efe50)

ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV (/v/tsan_demo/build_tsan/tsan_test+0x1933dd) in __sanitizer::internal_memcpy(void*, void const*, unsigned long)
==9388==ABORTING

[lday0321]

@wwbmmm 绕开brpc TSAN warning的时候， 把mutex.cpp的pthread_mutex_lock/pthread_mutex_unlock移除是不是还不够？ 我移掉之后，跑简单UT TSAN check（简单的braft case)，报很多mutex相关的错误：
Mutex M93303508761559560 is already destroyed.
Mutex M92177797832769712 is already destroyed.

WARNING: ThreadSanitizer: data race (pid=17071)
  Read of size 8 at 0x7b080006f080 by thread T69 (mutexes: write M93303508761559560):
    #0 memcmp <null> (braft_tsan_test+0x23169f)
    #1 std::char_traits<char>::compare(char const*, char const*, unsigned long) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/char_traits.h:325:9 (braft_tsan_test+0x4cdb2e)
    #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.h:2859 (braft_tsan_test+0x4cdb2e)
    #3 bool std::operator<<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.h:6229 (braft_tsan_test+0x4cdb2e)
    #4 std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_function.h:386 (braft_tsan_test+0x4cdb2e)
    #5 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> > > >::_M_lower_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> > > const*, std::_Rb_tree_node_base const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:1945 (braft_tsan_test+0x4cdb2e)
    #6 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> > > >::find(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:2570 (braft_tsan_test+0x4cdb2e)
    #7 void std::__invoke_impl<void, my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>(std::__invoke_other, my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0&&, braft::Node*&&, braft::NodeOptions*&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/invoke.h:60:14 (braft_tsan_test+0x27be5d)
    #8 std::__invoke_result<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>::type std::__invoke<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>(my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0&&, braft::Node*&&, braft::NodeOptions*&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/invoke.h:95:14 (braft_tsan_test+0x27bc6a)
    #9 void std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:244:13 (braft_tsan_test+0x27bbce)
    #10 std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> >::operator()() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:251:11 (braft_tsan_test+0x27bb38)
    #11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> > >::_M_run() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:195:13 (braft_tsan_test+0x27b8ac)
    #12 <null> <null> (libstdc++.so.6+0xd2bd3)

  Previous write of size 8 at 0x7b080006f080 by thread T68 (mutexes: write M92177797832769712):
    #0 operator new(unsigned long) <null> (braft_tsan_test+0x2777e7)
    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.tcc:219:14 (braft_tsan_test+0x4cd74f)
    #2 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char*>(char*, char*, std::__false_type) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.h:247 (braft_tsan_test+0x4cd74f)
    #3 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.h:266 (braft_tsan_test+0x4cd74f)
    #4 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.h:451 (braft_tsan_test+0x4cd74f)
    #5 std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> >::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, scoped_refptr<braft::KVBasedMergedMetaStorageImpl>, true>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_pair.h:261 (braft_tsan_test+0x4cd74f)
    #6 std::pair<std::__decay_and_strip<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::__type, std::__decay_and_strip<scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&>::__type> std::make_pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_pair.h:529 (braft_tsan_test+0x4cd74f)
    #7 braft::MetaStorageManager::_add(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, scoped_refptr<braft::KVBasedMergedMetaStorageImpl>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&) /v/braft/braft/src/braft/raft_meta.cpp:111 (braft_tsan_test+0x4cd74f)
    #8 void std::__invoke_impl<void, my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>(std::__invoke_other, my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0&&, braft::Node*&&, braft::NodeOptions*&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/invoke.h:60:14 (braft_tsan_test+0x27be5d)
    #9 std::__invoke_result<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>::type std::__invoke<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>(my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0&&, braft::Node*&&, braft::NodeOptions*&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/invoke.h:95:14 (braft_tsan_test+0x27bc6a)
    #10 void std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:244:13 (braft_tsan_test+0x27bbce)
    #11 std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> >::operator()() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:251:11 (braft_tsan_test+0x27bb38)
    #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> > >::_M_run() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:195:13 (braft_tsan_test+0x27b8ac)
    #13 <null> <null> (libstdc++.so.6+0xd2bd3)

  Location is heap block of size 29 at 0x7b080006f080 allocated by thread T68:
    #0 operator new(unsigned long) <null> (braft_tsan_test+0x2777e7)
    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.tcc:219:14 (braft_tsan_test+0x4cd74f)
    #2 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char*>(char*, char*, std::__false_type) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.h:247 (braft_tsan_test+0x4cd74f)
    #3 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.h:266 (braft_tsan_test+0x4cd74f)
    #4 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/basic_string.h:451 (braft_tsan_test+0x4cd74f)
    #5 std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> >::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, scoped_refptr<braft::KVBasedMergedMetaStorageImpl>, true>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_pair.h:261 (braft_tsan_test+0x4cd74f)
    #6 std::pair<std::__decay_and_strip<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::__type, std::__decay_and_strip<scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&>::__type> std::make_pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_pair.h:529 (braft_tsan_test+0x4cd74f)
    #7 braft::MetaStorageManager::_add(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, scoped_refptr<braft::KVBasedMergedMetaStorageImpl>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<braft::KVBasedMergedMetaStorageImpl> const&) /v/braft/braft/src/braft/raft_meta.cpp:111 (braft_tsan_test+0x4cd74f)
    #8 void std::__invoke_impl<void, my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>(std::__invoke_other, my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0&&, braft::Node*&&, braft::NodeOptions*&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/invoke.h:60:14 (braft_tsan_test+0x27be5d)
    #9 std::__invoke_result<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>::type std::__invoke<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*>(my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0&&, braft::Node*&&, braft::NodeOptions*&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/invoke.h:95:14 (braft_tsan_test+0x27bc6a)
    #10 void std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:244:13 (braft_tsan_test+0x27bbce)
    #11 std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> >::operator()() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:251:11 (braft_tsan_test+0x27bb38)
    #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody()::$_0, braft::Node*, braft::NodeOptions*> > >::_M_run() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/thread:195:13 (braft_tsan_test+0x27b8ac)
    #13 <null> <null> (libstdc++.so.6+0xd2bd3)

  Mutex M93303508761559560 is already destroyed.

  Mutex M92177797832769712 is already destroyed.

  Thread T69 (tid=17142, running) created by main thread at:
    #0 pthread_create <null> (braft_tsan_test+0x207866)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xd2e98)
    #2 my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody() /v/tsan_demo/braft_tsan_test.cc:113:15 (braft_tsan_test+0x27a216)
    #3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (braft_tsan_test+0x2b5abc)
    #4 main /v/tsan_demo/braft_tsan_test.cc:157:10 (braft_tsan_test+0x278a88)

  Thread T68 (tid=17141, running) created by main thread at:
    #0 pthread_create <null> (braft_tsan_test+0x207866)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xd2e98)
    #2 my::(anonymous namespace)::MyTest_TSAN_Check_braft_Test::TestBody() /v/tsan_demo/braft_tsan_test.cc:111:15 (braft_tsan_test+0x27a19c)
    #3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (braft_tsan_test+0x2b5abc)
    #4 main /v/tsan_demo/braft_tsan_test.cc:157:10 (braft_tsan_test+0x278a88)

SUMMARY: ThreadSanitizer: data race (/v/tsan_demo/build_tsan/braft_tsan_test+0x23169f) in memcmp

以及：

WARNING: ThreadSanitizer: data race (pid=17071)
  Read of size 8 at 0x7b0c00063108 by thread T38:
    #0 std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_S_right(std::_Rb_tree_node_base*) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:798:45 (braft_tsan_test+0x27f006)
    #1 std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_erase(std::_Rb_tree_node<braft::PeerId>*) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:1913:13 (braft_tsan_test+0x2815d7)
    #2 std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::~_Rb_tree() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:995:9 (braft_tsan_test+0x506d86)
    #3 std::set<braft::PeerId, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::~set() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_set.h:281 (braft_tsan_test+0x506d86)
    #4 braft::NodeImpl::elect_self(std::unique_lock<butil::Mutex>*, bool) /v/braft/braft/src/braft/node.cpp:1707 (braft_tsan_test+0x506d86)

  Previous write of size 8 at 0x7b0c00063108 by thread T54 (mutexes: write M3894):
    #0 operator new(unsigned long) <null> (braft_tsan_test+0x2777e7)
    #1 __gnu_cxx::new_allocator<std::_Rb_tree_node<braft::PeerId> >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/ext/new_allocator.h:114:27 (braft_tsan_test+0x51b679)
    #2 std::allocator_traits<std::allocator<std::_Rb_tree_node<braft::PeerId> > >::allocate(std::allocator<std::_Rb_tree_node<braft::PeerId> >&, unsigned long) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/alloc_traits.h:444 (braft_tsan_test+0x51b679)
    #3 std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_get_node() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:580 (braft_tsan_test+0x51b679)
    #4 std::_Rb_tree_node<braft::PeerId>* std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_create_node<braft::PeerId const&>(braft::PeerId const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:630 (braft_tsan_test+0x51b679)
    #5 std::_Rb_tree_node<braft::PeerId>* std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node::operator()<braft::PeerId const&>(braft::PeerId const&) const /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:548 (braft_tsan_test+0x51b679)
    #6 std::_Rb_tree_iterator<braft::PeerId> std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_insert_<braft::PeerId const&, std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, braft::PeerId const&, std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:1810 (braft_tsan_test+0x51b679)
    #7 std::_Rb_tree_iterator<braft::PeerId> std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_insert_unique_<braft::PeerId const&, std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node>(std::_Rb_tree_const_iterator<braft::PeerId>, braft::PeerId const&, std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:2261 (braft_tsan_test+0x51b679)
    #8 std::enable_if<__same_value_type<std::_Rb_tree_const_iterator<braft::PeerId> >::value, void>::type std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_insert_range_unique<std::_Rb_tree_const_iterator<braft::PeerId> >(std::_Rb_tree_const_iterator<braft::PeerId>, std::_Rb_tree_const_iterator<braft::PeerId>) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:1113 (braft_tsan_test+0x51b679)

  Location is heap block of size 48 at 0x7b0c000630f0 allocated by thread T54:
    #0 operator new(unsigned long) <null> (braft_tsan_test+0x2777e7)
    #1 __gnu_cxx::new_allocator<std::_Rb_tree_node<braft::PeerId> >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/ext/new_allocator.h:114:27 (braft_tsan_test+0x51b679)
    #2 std::allocator_traits<std::allocator<std::_Rb_tree_node<braft::PeerId> > >::allocate(std::allocator<std::_Rb_tree_node<braft::PeerId> >&, unsigned long) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/alloc_traits.h:444 (braft_tsan_test+0x51b679)
    #3 std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_get_node() /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:580 (braft_tsan_test+0x51b679)
    #4 std::_Rb_tree_node<braft::PeerId>* std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_create_node<braft::PeerId const&>(braft::PeerId const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:630 (braft_tsan_test+0x51b679)
    #5 std::_Rb_tree_node<braft::PeerId>* std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node::operator()<braft::PeerId const&>(braft::PeerId const&) const /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:548 (braft_tsan_test+0x51b679)
    #6 std::_Rb_tree_iterator<braft::PeerId> std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_insert_<braft::PeerId const&, std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, braft::PeerId const&, std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:1810 (braft_tsan_test+0x51b679)
    #7 std::_Rb_tree_iterator<braft::PeerId> std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_insert_unique_<braft::PeerId const&, std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node>(std::_Rb_tree_const_iterator<braft::PeerId>, braft::PeerId const&, std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_Alloc_node&) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:2261 (braft_tsan_test+0x51b679)
    #8 std::enable_if<__same_value_type<std::_Rb_tree_const_iterator<braft::PeerId> >::value, void>::type std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_M_insert_range_unique<std::_Rb_tree_const_iterator<braft::PeerId> >(std::_Rb_tree_const_iterator<braft::PeerId>, std::_Rb_tree_const_iterator<braft::PeerId>) /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:1113 (braft_tsan_test+0x51b679)

  Mutex M3894 (0x7b7000002b58) created at:
    #0 pthread_mutex_init <null> (braft_tsan_test+0x207c8a)
    #1 butil::Mutex::Mutex() /install_libs/brpc/include/butil/synchronization/lock.h:51:9 (braft_tsan_test+0x5014e0)
    #2 braft::NodeImpl::NodeImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, braft::PeerId const&) /v/braft/braft/src/braft/node.cpp:136 (braft_tsan_test+0x5014e0)
    #3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (braft_tsan_test+0x2b5abc)
    #4 main /v/tsan_demo/braft_tsan_test.cc:157:10 (braft_tsan_test+0x278a88)

  Thread T38 (tid=17110, running) created by main thread at:
    #0 pthread_create <null> (braft_tsan_test+0x207866)
    #1 bthread::TaskControl::add_workers(int) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:199:24 (braft_tsan_test+0x31f90d)
    #2 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (braft_tsan_test+0x2b5abc)
    #3 main /v/tsan_demo/braft_tsan_test.cc:157:10 (braft_tsan_test+0x278a88)

  Thread T54 (tid=17126, running) created by main thread at:
    #0 pthread_create <null> (braft_tsan_test+0x207866)
    #1 bthread::TaskControl::add_workers(int) /v/brpc/incubator-brpc/src/bthread/task_control.cpp:199:24 (braft_tsan_test+0x31f90d)
    #2 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (braft_tsan_test+0x2b5abc)
    #3 main /v/tsan_demo/braft_tsan_test.cc:157:10 (braft_tsan_test+0x278a88)

SUMMARY: ThreadSanitizer: data race /usr/bin/../lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/bits/stl_tree.h:798:45 in std::_Rb_tree<braft::PeerId, braft::PeerId, std::_Identity<braft::PeerId>, std::less<braft::PeerId>, std::allocator<braft::PeerId> >::_S_right(std::_Rb_tree_node_base*)

[wwbmmm]

这个可能和braft的代码实现有关，可以给braft提个issue?

[lday0321]

好的， 我想先确认下， 把mutex.cpp的pthread_mutex_lock/pthread_mutex_unlock移除是不是，pthread这两个方法的劫持就彻底无效了？因为我看mutex.cpp里，还有一部分和劫持这两个方法相关的逻辑：https://github.com/apache/incubator-brpc/blob/master/src/bthread/mutex.cpp#L407-L430

[wwbmmm]

好的， 我想先确认下， 把mutex.cpp的pthread_mutex_lock/pthread_mutex_unlock移除是不是，pthread这两个方法的劫持就彻底无效了？因为我看mutex.cpp里，还有一部分和劫持这两个方法相关的逻辑：https://github.com/apache/incubator-brpc/blob/master/src/bthread/mutex.cpp#L407-L430

是的。那个逻辑不影响的。