Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Second attempt: client channel: don't hold mutexes while calling the ConfigSelector or the LB picker #32326

Merged
merged 11 commits into from
Feb 16, 2023
Merged

Second attempt: client channel: don't hold mutexes while calling the ConfigSelector or the LB picker #32326

merged 11 commits into from
Feb 16, 2023

Conversation

markdroth
Copy link
Member

@markdroth markdroth commented Feb 8, 2023 •
edited
Loading

Original attempt was #31973, reverted in #32324 due to test flakiness.

There were two problems causing test flakiness here.

The first problem was that, upon resolver error, we were dispatching an async callback to re-process each of the queued picks before we updated the channel's connectivity state, which meant that the queued picks might be re-processed in another thread before the new connectivity state was set, so tests that expected the state to be TRANSIENT_FAILURE once RPCs failed might not see the expected state. This is fixed in 1aee2fb.

The second problem affected the xDS ring hash tests, and it's a bit more involved to explain.

We have an e2e test that simulates an aggregate cluster failover from a primary cluster using ring_hash at startup. The primary cluster has two addresses, both of which are unreachable when the client starts up, so the client should immediately fail over to the secondary cluster, which does have reachable endpoints. The test requires that no RPCs are failed while this failover occurs. The original PR made this test flaky.

The problem here was caused by a combination of two factors:

  1. Prior to the original PR, when the picker was updated (which happens inside the WorkSerializer), we re-processed previously queued picks synchronously, so it was not possible for another subchannel connectivity state update (which also happens in the WorkSerializer) to be processed between the time that we updated the picker and the time that we re-processed the previously queued picks. The original PR changed this such that the queued picks are re-processed asynchronously (outside of the WorkSerializer), so it is now possible for a subchannel connectivity state update to be processed between when the picker is updated and when we re-process the previously queued picks.

  2. Unlike most LB policies, where the picker does not see updated subchannel connectivity states until a new picker is created, the ring_hash picker gets the subchannel connectivity states from the LB policy via a lock, so it can wind up seeing the new states before it gets updated. This means that when a subchannel connectivity state update is processed by the ring_hash policy in the WorkSerializer, it will immediately be seen by the existing picker, even without a picker update.

With those two points in mind, the sequence of events in the failing test were as follows:

  1. The pick is attempted in the ring_hash picker for the primary cluster. This causes the first subchannel to attempt to connect.
  2. The subchannel transitions from IDLE to CONNECTING. A new picker is returned due to the subchannel connectivity state change, and the channel retries the queued pick. The retried pick is done asynchronously, but in this case it does not matter: the call will be re-queued.
  3. The connection attempt fails, and the subchannel reports TRANSIENT_FAILURE. A new picker is again returned, and the channel retries the queued pick. The retried pick is done asynchronously, but in this case it does not matter: this causes the picker to trigger a connection attempt for the second subchannel.
  4. The second subchannel transitions from IDLE to CONNECTING. A new picker is again returned, and the channel retries the queued pick. The retried pick is done asynchronously, and in this case it does matter.
  5. The second subchannel now transitions to TRANSIENT_FAILURE. The ring_hash policy will now report TRANSIENT_FAILURE, but before it can finish that...
  6. ...In another thread, the channel now tries to re-process the queued pick using the CONNECTING picker from step 4. However, because the ring_hash policy has already seen the TRANSIENT_FAILURE report from the second subchannel, that picker will now fail the pick instead of queuing it.

After discussion with @ejona86 and @dfawley (since this bug actually exists in Java and Go as well), we agreed that the right solution is to change the ring_hash picker to contain its own copy of the subchannel connectivity state information, rather than sharing that information with the LB policy using synchronization. I've made that change in 98cdac3.

@markdroth markdroth added the release notes: no Indicates if PR should not be in release notes label Feb 8, 2023
@grpc-checks grpc-checks bot added bloat/none and removed bloat/low labels Feb 9, 2023
@markdroth markdroth marked this pull request as ready for review February 9, 2023 23:36
@markdroth markdroth merged commit 8249fc1 into grpc:master Feb 16, 2023
@copybara-service copybara-service bot added the imported Specifies if the PR has been imported to the internal repository label Feb 16, 2023
@markdroth markdroth mentioned this pull request Feb 17, 2023
Merged
markdroth added a commit that referenced this pull request Feb 17, 2023
@markdroth
xDS fault injection e2e test: fix flakes caused by processing queued …
e022a3d 
…calls in parallel (#32429)

The `XdsFaultInjectionMaxFault` test has seen a few flakes since #32326
was merged. I believe the flakiness is caused by the fact that when a
large number of RPCs are queued up before the resolver result comes in,
those RPCs are now re-processed in parallel instead of sequentially,
which can cause us to delay more RPCs than we should due to the
`max_faults` setting. To fix this, we change the test to ensure that the
channel is connected (i.e., the resolver result has already been
returned) before we start sending a large number of concurrent RPCs.

Although this is the only test that I've seen flakes in, I've made this
same change consistently to all fault injection tests that are creating
a large number of concurrent RPCs, since the same flake could affect any
of them.
@markdroth markdroth mentioned this pull request Feb 27, 2023
Merged
markdroth added a commit that referenced this pull request Feb 28, 2023
@markdroth
client channel: fix TSAN flake in FilterBasedLoadBalancedCall (#32491)
9e61e72 
This fixes a very strange TSAN flake seen in an internal test
(b/268292646), which seems to have been introduced in
#32326.

I've included an example of the TSAN failure below, for future
reference. I don't fully understand what is causing this failure. It
looks like the async callback from a previous picker update caused the
pick to be re-queued but then got stuck draining queued callbacks in its
`ExecCtx` instance, while the async callback from the next picker update
had already failed the call. But the part I don't understand is why the
call combiner cancellation callback wound up scheduled on the `ExecCtx`
instance from the first thread -- that seems likely to be caused by
either the `WorkSerializer` or maybe `ExecCtx` work-stealing, but the
exact details are likely to be hard to nail down.

Switching from `EventEngine::Run()` back to `ExecCtx::Run()` seems to
fix the flake, so that's what I'm doing in this PR. I find this
work-around deeply dissatisfying, especially since I do not fully
understand the root cause of the problem. However, given that we are
working toward eliminating the `FilterBasedLoadBalancedCall` code
entirely as part of the promise conversion, this problem is probably not
worth further investigation.

```
WARNING: ThreadSanitizer: data race (pid=7406)
  Read of size 8 at 0x7b8800037558 by thread T47:
    #0 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::~FilterBasedLoadBalancedCall() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2780:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x530ed) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #1 Delete third_party/grpc/src/core/lib/gprpp/ref_counted.h:248:31 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #2 Unref third_party/grpc/src/core/lib/gprpp/orphanable.h:102:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #3 ~RefCountedPtr third_party/grpc/src/core/lib/gprpp/ref_counted_ptr.h:103:36 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #4 ~LbQueuedCallCanceller third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3132:51 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #5 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::LbQueuedCallCanceller::CancelLocked(void*, absl::Status) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3168:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #6 exec_ctx_run third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:45:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x4199) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #7 grpc_core::ExecCtx::Flush() third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:72:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x4199)
    #8 ~ExecCtx third_party/grpc/src/core/lib/iomgr/exec_ctx.h:117:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #9 operator() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3201:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #10 __invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #11 invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #12 InvokeR<void, (lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &, void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #13 void absl::internal_any_invocable::LocalInvoker<false, void, grpc_core::ClientChannel::FilterBasedLoadBalancedCall::RetryPickLocked()::$_0&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #14 operator() third_party/absl/functional/internal/any_invocable.h:855:1 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74) (BuildId: 7802a789e61dddac5a33230a2f294046)
    #15 util::functional::internal::FunctorCallback<Closure, false, absl::AnyInvocable<void ()>, void ()>::Run() util/functional/to_callback_internal.h:87:27 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74)
    #16 void eventmanager::EventManager::RunTask<false>(eventmanager::EventManager::WorkerThread*, eventmanager::TaskInfo*) net/eventmanager/em1/eventmanager.cc:2464:20 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2ae1b) (BuildId: b3d97518d96e160722658c4aa0795013)
    #17 void eventmanager::EventManager::RunWorkerLoop<false, false>(eventmanager::EventManager::WorkerThread*) net/eventmanager/em1/eventmanager.cc (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2a45d) (BuildId: b3d97518d96e160722658c4aa0795013)
    #18 eventmanager::EventManager::WorkerThread::WorkerMain() net/eventmanager/em1/eventmanager.cc:430:19 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1d841) (BuildId: b3d97518d96e160722658c4aa0795013)
    #19 operator() net/eventmanager/em1/eventmanager.cc:396:64 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351) (BuildId: b3d97518d96e160722658c4aa0795013)
    #20 __invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #21 invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #22 InvokeR<void, (lambda at net/eventmanager/em1/eventmanager.cc:396:55), void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #23 void absl::internal_any_invocable::LocalInvoker<false, void, eventmanager::EventManager::WorkerThread::StartThread(int, int, thread::SchedPolicy, std::__tsan::basic_string_view<char, std::__tsan::char_traits<char>>)::'lambda'()&&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #24 operator() third_party/absl/functional/internal/any_invocable.h:863:1 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e) (BuildId: a86eb59aee0be7b5882a1b532922e3ca)
    #25 ClosureThread::Run() thread/thread.h:459:25 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e)
    #26 Thread::ThreadBody(void*) thread/thread.cc:1284:16 (libST-a5842b3303c3_thread_Slibthread.so+0x251e9) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)

  Previous write of size 8 at 0x7b8800037558 by thread T40:
    #0 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::PendingBatchesFail(absl::Status, bool (*)(grpc_core::CallCombinerClosureList const&)) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2863:13 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x537c7) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #1 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::TryPick(bool) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3179:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x5537f) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #2 operator() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3200:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #3 __invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #4 invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #5 InvokeR<void, (lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &, void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #6 void absl::internal_any_invocable::LocalInvoker<false, void, grpc_core::ClientChannel::FilterBasedLoadBalancedCall::RetryPickLocked()::$_0&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #7 operator() third_party/absl/functional/internal/any_invocable.h:855:1 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74) (BuildId: 7802a789e61dddac5a33230a2f294046)
    #8 util::functional::internal::FunctorCallback<Closure, false, absl::AnyInvocable<void ()>, void ()>::Run() util/functional/to_callback_internal.h:87:27 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74)
    #9 void eventmanager::EventManager::RunTask<false>(eventmanager::EventManager::WorkerThread*, eventmanager::TaskInfo*) net/eventmanager/em1/eventmanager.cc:2464:20 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2ae1b) (BuildId: b3d97518d96e160722658c4aa0795013)
    #10 void eventmanager::EventManager::RunWorkerLoop<false, false>(eventmanager::EventManager::WorkerThread*) net/eventmanager/em1/eventmanager.cc (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2a45d) (BuildId: b3d97518d96e160722658c4aa0795013)
    #11 eventmanager::EventManager::WorkerThread::WorkerMain() net/eventmanager/em1/eventmanager.cc:430:19 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1d841) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 operator() net/eventmanager/em1/eventmanager.cc:396:64 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351) (BuildId: b3d97518d96e160722658c4aa0795013)
    #13 __invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #14 invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #15 InvokeR<void, (lambda at net/eventmanager/em1/eventmanager.cc:396:55), void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #16 void absl::internal_any_invocable::LocalInvoker<false, void, eventmanager::EventManager::WorkerThread::StartThread(int, int, thread::SchedPolicy, std::__tsan::basic_string_view<char, std::__tsan::char_traits<char>>)::'lambda'()&&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #17 operator() third_party/absl/functional/internal/any_invocable.h:863:1 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e) (BuildId: a86eb59aee0be7b5882a1b532922e3ca)
    #18 ClosureThread::Run() thread/thread.h:459:25 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e)
    #19 Thread::ThreadBody(void*) thread/thread.cc:1284:16 (libST-a5842b3303c3_thread_Slibthread.so+0x251e9) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)

  Thread T47 'EventManager_Default' (tid=7454, running) created by main thread at:
    #0 pthread_create third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1048:3 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x1887b9) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #1 Thread::CreatePthread(pthread_attr_t&) thread/thread.cc:485:13 (libST-a5842b3303c3_thread_Slibthread.so+0x24b1d) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #2 Thread::Start() thread/thread.cc:667:3 (libST-a5842b3303c3_thread_Slibthread.so+0x2574b) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #3 StartThread net/eventmanager/em1/eventmanager.cc:397:14 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed) (BuildId: b3d97518d96e160722658c4aa0795013)
    #4 eventmanager::EventManager::EventManager(eventmanager::EventManager::Options const&) net/eventmanager/em1/eventmanager.cc:1274:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed)
    #5 InitializeDefaultEventManager() net/eventmanager/em1/eventmanager.cc:3042:39 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25c06) (BuildId: b3d97518d96e160722658c4aa0795013)
    #6 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #7 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #8 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #9 GoogleOnceInternalInit base/once.cc:23:5 (libST-a5842b3303c3_base_Slibonce.so+0xeee) (BuildId: 6c436371e89d7962c2b544d27465d6a0)
    #10 GoogleOnceInternalInitSchedCoopAndKernel(std::__tsan::atomic<unsigned int>*, void (*)()) base/once.cc:15:3 (libST-a5842b3303c3_base_Slibonce.so+0xeee)
    #11 GoogleOnceInit base/once.h:72:5 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 eventmanager::EventManager::DefaultEventManager() net/eventmanager/em1/eventmanager.cc:3059:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9)
    #13 Default net/eventmanager/em1/eventmanager.h:780:43 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5) (BuildId: 2e1dde05e8bce5319856279d896c4405)
    #14 eventmanager::Default() net/eventmanager/eventmanager_default.cc:22:10 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5)
    #15 grpc::(anonymous namespace)::InitGlobalEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:38:23 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x1c28) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #16 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #17 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #18 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #19 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #20 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #21 grpc::GetGrpcEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:51:3 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x18c1) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #22 check_engine_available(bool) net/grpc/internal/src/core/ext/poller/event_manager_poller/ev_event_manager_linux.cc:1271:21 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Spoller_Sevent_Umanager_Upoller_Slibev_Uevent_Umanager_Ulinux.so+0x4fcb) (BuildId: 620adb83259aed32440614774678b3a1)
    #23 try_engine third_party/grpc/src/core/lib/iomgr/ev_posix.cc:141:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #24 operator() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:184:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #25 grpc_event_engine_init()::$_0::__invoke() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #26 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #27 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #28 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #29 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #30 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #31 grpc_event_engine_init() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xacab1) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #32 iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_posix.cc:43:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xaf1d0) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #33 grpc_iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_internal.cc:35:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x688b) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #34 grpc_iomgr_init() third_party/grpc/src/core/lib/iomgr/iomgr.cc:66:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xae777) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #35 grpc_init third_party/grpc/src/core/lib/surface/init.cc:144:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc.so+0x4518) (BuildId: 778b43e2e8f8c132a7acf9b3da117fff)
    #36 k3::(anonymous namespace)::GrpcInitHackEnvironment::SetUp() storage/k3/grpc/grpc_server_test.cc:144:27 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x212d21) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #37 SetUpEnvironment third_party/googletest/googletest/src/gtest.cc:5763:55 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #38 for_each<std::__tsan::__wrap_iter<testing::Environment *const *>, void (*)(testing::Environment *)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__algorithm/for_each.h:26:5 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #39 ForEach<std::__tsan::vector<testing::Environment *, std::__tsan::allocator<testing::Environment *> >, void (*)(testing::Environment *)> third_party/googletest/googletest/src/gtest-internal-inl.h:288:3 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #40 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5876:9 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #41 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/googletest/src/gtest.cc (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #42 testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5464:10 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41)
    #43 RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2329:73 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #44 main testing/base/internal/gunit_main.cc:86:10 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a)

  Thread T40 'EventManager_Default' (tid=7447, running) created by main thread at:
    #0 pthread_create third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1048:3 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x1887b9) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #1 Thread::CreatePthread(pthread_attr_t&) thread/thread.cc:485:13 (libST-a5842b3303c3_thread_Slibthread.so+0x24b1d) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #2 Thread::Start() thread/thread.cc:667:3 (libST-a5842b3303c3_thread_Slibthread.so+0x2574b) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #3 StartThread net/eventmanager/em1/eventmanager.cc:397:14 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed) (BuildId: b3d97518d96e160722658c4aa0795013)
    #4 eventmanager::EventManager::EventManager(eventmanager::EventManager::Options const&) net/eventmanager/em1/eventmanager.cc:1274:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed)
    #5 InitializeDefaultEventManager() net/eventmanager/em1/eventmanager.cc:3042:39 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25c06) (BuildId: b3d97518d96e160722658c4aa0795013)
    #6 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #7 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #8 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #9 GoogleOnceInternalInit base/once.cc:23:5 (libST-a5842b3303c3_base_Slibonce.so+0xeee) (BuildId: 6c436371e89d7962c2b544d27465d6a0)
    #10 GoogleOnceInternalInitSchedCoopAndKernel(std::__tsan::atomic<unsigned int>*, void (*)()) base/once.cc:15:3 (libST-a5842b3303c3_base_Slibonce.so+0xeee)
    #11 GoogleOnceInit base/once.h:72:5 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 eventmanager::EventManager::DefaultEventManager() net/eventmanager/em1/eventmanager.cc:3059:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9)
    #13 Default net/eventmanager/em1/eventmanager.h:780:43 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5) (BuildId: 2e1dde05e8bce5319856279d896c4405)
    #14 eventmanager::Default() net/eventmanager/eventmanager_default.cc:22:10 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5)
    #15 grpc::(anonymous namespace)::InitGlobalEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:38:23 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x1c28) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #16 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #17 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #18 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #19 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #20 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #21 grpc::GetGrpcEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:51:3 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x18c1) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #22 check_engine_available(bool) net/grpc/internal/src/core/ext/poller/event_manager_poller/ev_event_manager_linux.cc:1271:21 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Spoller_Sevent_Umanager_Upoller_Slibev_Uevent_Umanager_Ulinux.so+0x4fcb) (BuildId: 620adb83259aed32440614774678b3a1)
    #23 try_engine third_party/grpc/src/core/lib/iomgr/ev_posix.cc:141:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #24 operator() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:184:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #25 grpc_event_engine_init()::$_0::__invoke() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #26 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #27 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #28 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #29 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #30 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #31 grpc_event_engine_init() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xacab1) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #32 iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_posix.cc:43:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xaf1d0) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #33 grpc_iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_internal.cc:35:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x688b) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #34 grpc_iomgr_init() third_party/grpc/src/core/lib/iomgr/iomgr.cc:66:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xae777) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #35 grpc_init third_party/grpc/src/core/lib/surface/init.cc:144:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc.so+0x4518) (BuildId: 778b43e2e8f8c132a7acf9b3da117fff)
    #36 k3::(anonymous namespace)::GrpcInitHackEnvironment::SetUp() storage/k3/grpc/grpc_server_test.cc:144:27 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x212d21) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #37 SetUpEnvironment third_party/googletest/googletest/src/gtest.cc:5763:55 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #38 for_each<std::__tsan::__wrap_iter<testing::Environment *const *>, void (*)(testing::Environment *)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__algorithm/for_each.h:26:5 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #39 ForEach<std::__tsan::vector<testing::Environment *, std::__tsan::allocator<testing::Environment *> >, void (*)(testing::Environment *)> third_party/googletest/googletest/src/gtest-internal-inl.h:288:3 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #40 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5876:9 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #41 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/googletest/src/gtest.cc (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #42 testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5464:10 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41)
    #43 RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2329:73 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #44 main testing/base/internal/gunit_main.cc:86:10 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a)

SUMMARY: ThreadSanitizer: data race third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2780:5 in grpc_core::ClientChannel::FilterBasedLoadBalancedCall::~FilterBasedLoadBalancedCall()
```
XuanWang-Amos pushed a commit to XuanWang-Amos/grpc that referenced this pull request May 1, 2023
@markdroth @XuanWang-Amos
Second attempt: client channel: don't hold mutexes while calling the …
f82236c 
…ConfigSelector or the LB picker (grpc#32326)

Original attempt was grpc#31973, reverted in grpc#32324 due to test flakiness.

There were two problems causing test flakiness here.

The first problem was that, upon resolver error, we were dispatching an
async callback to re-process each of the queued picks *before* we
updated the channel's connectivity state, which meant that the queued
picks might be re-processed in another thread before the new
connectivity state was set, so tests that expected the state to be
TRANSIENT_FAILURE once RPCs failed might not see the expected state.

The second problem affected the xDS ring hash tests, and it's a bit more
involved to explain.

We have an e2e test that simulates an aggregate cluster failover from a
primary cluster using ring_hash at startup. The primary cluster has two
addresses, both of which are unreachable when the client starts up, so
the client should immediately fail over to the secondary cluster, which
does have reachable endpoints. The test requires that no RPCs are failed
while this failover occurs. The original PR made this test flaky.

The problem here was caused by a combination of two factors:

1. Prior to the original PR, when the picker was updated (which happens
inside the WorkSerializer), we re-processed previously queued picks
synchronously, so it was not possible for another subchannel
connectivity state update (which also happens in the WorkSerializer) to
be processed between the time that we updated the picker and the time
that we re-processed the previously queued picks. The original PR
changed this such that the queued picks are re-processed asynchronously
(outside of the WorkSerializer), so it is now possible for a subchannel
connectivity state update to be processed between when the picker is
updated and when we re-process the previously queued picks.

2. Unlike most LB policies, where the picker does not see updated
subchannel connectivity states until a new picker is created, the
ring_hash picker gets the subchannel connectivity states from the LB
policy via a lock, so it can wind up seeing the new states before it
gets updated. This means that when a subchannel connectivity state
update is processed by the ring_hash policy in the WorkSerializer, it
will immediately be seen by the existing picker, even without a picker
update.

With those two points in mind, the sequence of events in the failing
test were as follows:

1. The pick is attempted in the ring_hash picker for the primary
cluster. This causes the first subchannel to attempt to connect.
2. The subchannel transitions from IDLE to CONNECTING. A new picker is
returned due to the subchannel connectivity state change, and the
channel retries the queued pick. The retried pick is done
asynchronously, but in this case it does not matter: the call will be
re-queued.
3. The connection attempt fails, and the subchannel reports
TRANSIENT_FAILURE. A new picker is again returned, and the channel
retries the queued pick. The retried pick is done asynchronously, but in
this case it does not matter: this causes the picker to trigger a
connection attempt for the second subchannel.
4. The second subchannel transitions from IDLE to CONNECTING. A new
picker is again returned, and the channel retries the queued pick. The
retried pick is done asynchronously, and in this case it *does* matter.
5. The second subchannel now transitions to TRANSIENT_FAILURE. The
ring_hash policy will now report TRANSIENT_FAILURE, but before it can
finish that...
6. ...In another thread, the channel now tries to re-process the queued
pick using the CONNECTING picker from step 4. However, because the
ring_hash policy has already seen the TRANSIENT_FAILURE report from the
second subchannel, that picker will now fail the pick instead of queuing
it.

After discussion with @ejona86 and @dfawley (since this bug actually
exists in Java and Go as well), we agreed that the right solution is to
change the ring_hash picker to contain its own copy of the subchannel
connectivity state information, rather than sharing that information
with the LB policy using synchronization.
XuanWang-Amos pushed a commit to XuanWang-Amos/grpc that referenced this pull request May 1, 2023
@markdroth @XuanWang-Amos
xDS fault injection e2e test: fix flakes caused by processing queued …
f9670a9 
…calls in parallel (grpc#32429)

The `XdsFaultInjectionMaxFault` test has seen a few flakes since grpc#32326
was merged. I believe the flakiness is caused by the fact that when a
large number of RPCs are queued up before the resolver result comes in,
those RPCs are now re-processed in parallel instead of sequentially,
which can cause us to delay more RPCs than we should due to the
`max_faults` setting. To fix this, we change the test to ensure that the
channel is connected (i.e., the resolver result has already been
returned) before we start sending a large number of concurrent RPCs.

Although this is the only test that I've seen flakes in, I've made this
same change consistently to all fault injection tests that are creating
a large number of concurrent RPCs, since the same flake could affect any
of them.
XuanWang-Amos pushed a commit to XuanWang-Amos/grpc that referenced this pull request May 1, 2023
@markdroth @XuanWang-Amos
client channel: fix TSAN flake in FilterBasedLoadBalancedCall (grpc#3…
f0438d3 
…2491)

This fixes a very strange TSAN flake seen in an internal test
(b/268292646), which seems to have been introduced in
grpc#32326.

I've included an example of the TSAN failure below, for future
reference. I don't fully understand what is causing this failure. It
looks like the async callback from a previous picker update caused the
pick to be re-queued but then got stuck draining queued callbacks in its
`ExecCtx` instance, while the async callback from the next picker update
had already failed the call. But the part I don't understand is why the
call combiner cancellation callback wound up scheduled on the `ExecCtx`
instance from the first thread -- that seems likely to be caused by
either the `WorkSerializer` or maybe `ExecCtx` work-stealing, but the
exact details are likely to be hard to nail down.

Switching from `EventEngine::Run()` back to `ExecCtx::Run()` seems to
fix the flake, so that's what I'm doing in this PR. I find this
work-around deeply dissatisfying, especially since I do not fully
understand the root cause of the problem. However, given that we are
working toward eliminating the `FilterBasedLoadBalancedCall` code
entirely as part of the promise conversion, this problem is probably not
worth further investigation.

```
WARNING: ThreadSanitizer: data race (pid=7406)
  Read of size 8 at 0x7b8800037558 by thread T47:
    #0 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::~FilterBasedLoadBalancedCall() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2780:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x530ed) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #1 Delete third_party/grpc/src/core/lib/gprpp/ref_counted.h:248:31 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #2 Unref third_party/grpc/src/core/lib/gprpp/orphanable.h:102:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #3 ~RefCountedPtr third_party/grpc/src/core/lib/gprpp/ref_counted_ptr.h:103:36 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #4 ~LbQueuedCallCanceller third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3132:51 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #5 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::LbQueuedCallCanceller::CancelLocked(void*, absl::Status) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3168:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #6 exec_ctx_run third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:45:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x4199) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #7 grpc_core::ExecCtx::Flush() third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:72:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x4199)
    #8 ~ExecCtx third_party/grpc/src/core/lib/iomgr/exec_ctx.h:117:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #9 operator() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3201:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #10 __invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #11 invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #12 InvokeR<void, (lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &, void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #13 void absl::internal_any_invocable::LocalInvoker<false, void, grpc_core::ClientChannel::FilterBasedLoadBalancedCall::RetryPickLocked()::$_0&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #14 operator() third_party/absl/functional/internal/any_invocable.h:855:1 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74) (BuildId: 7802a789e61dddac5a33230a2f294046)
    #15 util::functional::internal::FunctorCallback<Closure, false, absl::AnyInvocable<void ()>, void ()>::Run() util/functional/to_callback_internal.h:87:27 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74)
    #16 void eventmanager::EventManager::RunTask<false>(eventmanager::EventManager::WorkerThread*, eventmanager::TaskInfo*) net/eventmanager/em1/eventmanager.cc:2464:20 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2ae1b) (BuildId: b3d97518d96e160722658c4aa0795013)
    #17 void eventmanager::EventManager::RunWorkerLoop<false, false>(eventmanager::EventManager::WorkerThread*) net/eventmanager/em1/eventmanager.cc (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2a45d) (BuildId: b3d97518d96e160722658c4aa0795013)
    #18 eventmanager::EventManager::WorkerThread::WorkerMain() net/eventmanager/em1/eventmanager.cc:430:19 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1d841) (BuildId: b3d97518d96e160722658c4aa0795013)
    #19 operator() net/eventmanager/em1/eventmanager.cc:396:64 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351) (BuildId: b3d97518d96e160722658c4aa0795013)
    #20 __invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #21 invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #22 InvokeR<void, (lambda at net/eventmanager/em1/eventmanager.cc:396:55), void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #23 void absl::internal_any_invocable::LocalInvoker<false, void, eventmanager::EventManager::WorkerThread::StartThread(int, int, thread::SchedPolicy, std::__tsan::basic_string_view<char, std::__tsan::char_traits<char>>)::'lambda'()&&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #24 operator() third_party/absl/functional/internal/any_invocable.h:863:1 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e) (BuildId: a86eb59aee0be7b5882a1b532922e3ca)
    #25 ClosureThread::Run() thread/thread.h:459:25 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e)
    #26 Thread::ThreadBody(void*) thread/thread.cc:1284:16 (libST-a5842b3303c3_thread_Slibthread.so+0x251e9) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)

  Previous write of size 8 at 0x7b8800037558 by thread T40:
    #0 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::PendingBatchesFail(absl::Status, bool (*)(grpc_core::CallCombinerClosureList const&)) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2863:13 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x537c7) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #1 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::TryPick(bool) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3179:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x5537f) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #2 operator() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3200:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #3 __invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #4 invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #5 InvokeR<void, (lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &, void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #6 void absl::internal_any_invocable::LocalInvoker<false, void, grpc_core::ClientChannel::FilterBasedLoadBalancedCall::RetryPickLocked()::$_0&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #7 operator() third_party/absl/functional/internal/any_invocable.h:855:1 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74) (BuildId: 7802a789e61dddac5a33230a2f294046)
    #8 util::functional::internal::FunctorCallback<Closure, false, absl::AnyInvocable<void ()>, void ()>::Run() util/functional/to_callback_internal.h:87:27 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74)
    #9 void eventmanager::EventManager::RunTask<false>(eventmanager::EventManager::WorkerThread*, eventmanager::TaskInfo*) net/eventmanager/em1/eventmanager.cc:2464:20 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2ae1b) (BuildId: b3d97518d96e160722658c4aa0795013)
    #10 void eventmanager::EventManager::RunWorkerLoop<false, false>(eventmanager::EventManager::WorkerThread*) net/eventmanager/em1/eventmanager.cc (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2a45d) (BuildId: b3d97518d96e160722658c4aa0795013)
    #11 eventmanager::EventManager::WorkerThread::WorkerMain() net/eventmanager/em1/eventmanager.cc:430:19 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1d841) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 operator() net/eventmanager/em1/eventmanager.cc:396:64 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351) (BuildId: b3d97518d96e160722658c4aa0795013)
    #13 __invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #14 invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #15 InvokeR<void, (lambda at net/eventmanager/em1/eventmanager.cc:396:55), void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #16 void absl::internal_any_invocable::LocalInvoker<false, void, eventmanager::EventManager::WorkerThread::StartThread(int, int, thread::SchedPolicy, std::__tsan::basic_string_view<char, std::__tsan::char_traits<char>>)::'lambda'()&&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #17 operator() third_party/absl/functional/internal/any_invocable.h:863:1 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e) (BuildId: a86eb59aee0be7b5882a1b532922e3ca)
    #18 ClosureThread::Run() thread/thread.h:459:25 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e)
    #19 Thread::ThreadBody(void*) thread/thread.cc:1284:16 (libST-a5842b3303c3_thread_Slibthread.so+0x251e9) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)

  Thread T47 'EventManager_Default' (tid=7454, running) created by main thread at:
    #0 pthread_create third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1048:3 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x1887b9) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #1 Thread::CreatePthread(pthread_attr_t&) thread/thread.cc:485:13 (libST-a5842b3303c3_thread_Slibthread.so+0x24b1d) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #2 Thread::Start() thread/thread.cc:667:3 (libST-a5842b3303c3_thread_Slibthread.so+0x2574b) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #3 StartThread net/eventmanager/em1/eventmanager.cc:397:14 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed) (BuildId: b3d97518d96e160722658c4aa0795013)
    #4 eventmanager::EventManager::EventManager(eventmanager::EventManager::Options const&) net/eventmanager/em1/eventmanager.cc:1274:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed)
    #5 InitializeDefaultEventManager() net/eventmanager/em1/eventmanager.cc:3042:39 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25c06) (BuildId: b3d97518d96e160722658c4aa0795013)
    #6 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #7 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #8 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #9 GoogleOnceInternalInit base/once.cc:23:5 (libST-a5842b3303c3_base_Slibonce.so+0xeee) (BuildId: 6c436371e89d7962c2b544d27465d6a0)
    #10 GoogleOnceInternalInitSchedCoopAndKernel(std::__tsan::atomic<unsigned int>*, void (*)()) base/once.cc:15:3 (libST-a5842b3303c3_base_Slibonce.so+0xeee)
    #11 GoogleOnceInit base/once.h:72:5 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 eventmanager::EventManager::DefaultEventManager() net/eventmanager/em1/eventmanager.cc:3059:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9)
    #13 Default net/eventmanager/em1/eventmanager.h:780:43 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5) (BuildId: 2e1dde05e8bce5319856279d896c4405)
    #14 eventmanager::Default() net/eventmanager/eventmanager_default.cc:22:10 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5)
    #15 grpc::(anonymous namespace)::InitGlobalEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:38:23 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x1c28) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #16 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #17 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #18 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #19 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #20 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #21 grpc::GetGrpcEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:51:3 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x18c1) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #22 check_engine_available(bool) net/grpc/internal/src/core/ext/poller/event_manager_poller/ev_event_manager_linux.cc:1271:21 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Spoller_Sevent_Umanager_Upoller_Slibev_Uevent_Umanager_Ulinux.so+0x4fcb) (BuildId: 620adb83259aed32440614774678b3a1)
    #23 try_engine third_party/grpc/src/core/lib/iomgr/ev_posix.cc:141:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #24 operator() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:184:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #25 grpc_event_engine_init()::$_0::__invoke() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #26 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #27 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #28 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #29 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #30 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #31 grpc_event_engine_init() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xacab1) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #32 iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_posix.cc:43:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xaf1d0) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #33 grpc_iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_internal.cc:35:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x688b) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #34 grpc_iomgr_init() third_party/grpc/src/core/lib/iomgr/iomgr.cc:66:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xae777) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #35 grpc_init third_party/grpc/src/core/lib/surface/init.cc:144:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc.so+0x4518) (BuildId: 778b43e2e8f8c132a7acf9b3da117fff)
    #36 k3::(anonymous namespace)::GrpcInitHackEnvironment::SetUp() storage/k3/grpc/grpc_server_test.cc:144:27 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x212d21) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #37 SetUpEnvironment third_party/googletest/googletest/src/gtest.cc:5763:55 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #38 for_each<std::__tsan::__wrap_iter<testing::Environment *const *>, void (*)(testing::Environment *)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__algorithm/for_each.h:26:5 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #39 ForEach<std::__tsan::vector<testing::Environment *, std::__tsan::allocator<testing::Environment *> >, void (*)(testing::Environment *)> third_party/googletest/googletest/src/gtest-internal-inl.h:288:3 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #40 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5876:9 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #41 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/googletest/src/gtest.cc (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #42 testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5464:10 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41)
    #43 RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2329:73 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #44 main testing/base/internal/gunit_main.cc:86:10 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a)

  Thread T40 'EventManager_Default' (tid=7447, running) created by main thread at:
    #0 pthread_create third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1048:3 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x1887b9) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #1 Thread::CreatePthread(pthread_attr_t&) thread/thread.cc:485:13 (libST-a5842b3303c3_thread_Slibthread.so+0x24b1d) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #2 Thread::Start() thread/thread.cc:667:3 (libST-a5842b3303c3_thread_Slibthread.so+0x2574b) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #3 StartThread net/eventmanager/em1/eventmanager.cc:397:14 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed) (BuildId: b3d97518d96e160722658c4aa0795013)
    #4 eventmanager::EventManager::EventManager(eventmanager::EventManager::Options const&) net/eventmanager/em1/eventmanager.cc:1274:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed)
    #5 InitializeDefaultEventManager() net/eventmanager/em1/eventmanager.cc:3042:39 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25c06) (BuildId: b3d97518d96e160722658c4aa0795013)
    #6 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #7 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #8 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #9 GoogleOnceInternalInit base/once.cc:23:5 (libST-a5842b3303c3_base_Slibonce.so+0xeee) (BuildId: 6c436371e89d7962c2b544d27465d6a0)
    #10 GoogleOnceInternalInitSchedCoopAndKernel(std::__tsan::atomic<unsigned int>*, void (*)()) base/once.cc:15:3 (libST-a5842b3303c3_base_Slibonce.so+0xeee)
    #11 GoogleOnceInit base/once.h:72:5 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 eventmanager::EventManager::DefaultEventManager() net/eventmanager/em1/eventmanager.cc:3059:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9)
    #13 Default net/eventmanager/em1/eventmanager.h:780:43 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5) (BuildId: 2e1dde05e8bce5319856279d896c4405)
    #14 eventmanager::Default() net/eventmanager/eventmanager_default.cc:22:10 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5)
    #15 grpc::(anonymous namespace)::InitGlobalEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:38:23 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x1c28) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #16 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #17 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #18 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #19 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #20 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #21 grpc::GetGrpcEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:51:3 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x18c1) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #22 check_engine_available(bool) net/grpc/internal/src/core/ext/poller/event_manager_poller/ev_event_manager_linux.cc:1271:21 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Spoller_Sevent_Umanager_Upoller_Slibev_Uevent_Umanager_Ulinux.so+0x4fcb) (BuildId: 620adb83259aed32440614774678b3a1)
    #23 try_engine third_party/grpc/src/core/lib/iomgr/ev_posix.cc:141:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #24 operator() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:184:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #25 grpc_event_engine_init()::$_0::__invoke() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #26 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #27 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #28 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #29 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #30 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #31 grpc_event_engine_init() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xacab1) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #32 iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_posix.cc:43:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xaf1d0) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #33 grpc_iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_internal.cc:35:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x688b) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #34 grpc_iomgr_init() third_party/grpc/src/core/lib/iomgr/iomgr.cc:66:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xae777) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #35 grpc_init third_party/grpc/src/core/lib/surface/init.cc:144:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc.so+0x4518) (BuildId: 778b43e2e8f8c132a7acf9b3da117fff)
    #36 k3::(anonymous namespace)::GrpcInitHackEnvironment::SetUp() storage/k3/grpc/grpc_server_test.cc:144:27 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x212d21) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #37 SetUpEnvironment third_party/googletest/googletest/src/gtest.cc:5763:55 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #38 for_each<std::__tsan::__wrap_iter<testing::Environment *const *>, void (*)(testing::Environment *)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__algorithm/for_each.h:26:5 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #39 ForEach<std::__tsan::vector<testing::Environment *, std::__tsan::allocator<testing::Environment *> >, void (*)(testing::Environment *)> third_party/googletest/googletest/src/gtest-internal-inl.h:288:3 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #40 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5876:9 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #41 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/googletest/src/gtest.cc (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #42 testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5464:10 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41)
    #43 RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2329:73 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #44 main testing/base/internal/gunit_main.cc:86:10 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a)

SUMMARY: ThreadSanitizer: data race third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2780:5 in grpc_core::ClientChannel::FilterBasedLoadBalancedCall::~FilterBasedLoadBalancedCall()
```
wanlin31 pushed a commit that referenced this pull request May 18, 2023
@markdroth @wanlin31
Second attempt: client channel: don't hold mutexes while calling the …
c5a8574 
…ConfigSelector or the LB picker (#32326)

Original attempt was #31973, reverted in #32324 due to test flakiness.

There were two problems causing test flakiness here.

The first problem was that, upon resolver error, we were dispatching an
async callback to re-process each of the queued picks *before* we
updated the channel's connectivity state, which meant that the queued
picks might be re-processed in another thread before the new
connectivity state was set, so tests that expected the state to be
TRANSIENT_FAILURE once RPCs failed might not see the expected state.

The second problem affected the xDS ring hash tests, and it's a bit more
involved to explain.

We have an e2e test that simulates an aggregate cluster failover from a
primary cluster using ring_hash at startup. The primary cluster has two
addresses, both of which are unreachable when the client starts up, so
the client should immediately fail over to the secondary cluster, which
does have reachable endpoints. The test requires that no RPCs are failed
while this failover occurs. The original PR made this test flaky.

The problem here was caused by a combination of two factors:

1. Prior to the original PR, when the picker was updated (which happens
inside the WorkSerializer), we re-processed previously queued picks
synchronously, so it was not possible for another subchannel
connectivity state update (which also happens in the WorkSerializer) to
be processed between the time that we updated the picker and the time
that we re-processed the previously queued picks. The original PR
changed this such that the queued picks are re-processed asynchronously
(outside of the WorkSerializer), so it is now possible for a subchannel
connectivity state update to be processed between when the picker is
updated and when we re-process the previously queued picks.

2. Unlike most LB policies, where the picker does not see updated
subchannel connectivity states until a new picker is created, the
ring_hash picker gets the subchannel connectivity states from the LB
policy via a lock, so it can wind up seeing the new states before it
gets updated. This means that when a subchannel connectivity state
update is processed by the ring_hash policy in the WorkSerializer, it
will immediately be seen by the existing picker, even without a picker
update.

With those two points in mind, the sequence of events in the failing
test were as follows:

1. The pick is attempted in the ring_hash picker for the primary
cluster. This causes the first subchannel to attempt to connect.
2. The subchannel transitions from IDLE to CONNECTING. A new picker is
returned due to the subchannel connectivity state change, and the
channel retries the queued pick. The retried pick is done
asynchronously, but in this case it does not matter: the call will be
re-queued.
3. The connection attempt fails, and the subchannel reports
TRANSIENT_FAILURE. A new picker is again returned, and the channel
retries the queued pick. The retried pick is done asynchronously, but in
this case it does not matter: this causes the picker to trigger a
connection attempt for the second subchannel.
4. The second subchannel transitions from IDLE to CONNECTING. A new
picker is again returned, and the channel retries the queued pick. The
retried pick is done asynchronously, and in this case it *does* matter.
5. The second subchannel now transitions to TRANSIENT_FAILURE. The
ring_hash policy will now report TRANSIENT_FAILURE, but before it can
finish that...
6. ...In another thread, the channel now tries to re-process the queued
pick using the CONNECTING picker from step 4. However, because the
ring_hash policy has already seen the TRANSIENT_FAILURE report from the
second subchannel, that picker will now fail the pick instead of queuing
it.

After discussion with @ejona86 and @dfawley (since this bug actually
exists in Java and Go as well), we agreed that the right solution is to
change the ring_hash picker to contain its own copy of the subchannel
connectivity state information, rather than sharing that information
with the LB policy using synchronization.
wanlin31 pushed a commit that referenced this pull request May 18, 2023
@markdroth @wanlin31
xDS fault injection e2e test: fix flakes caused by processing queued …
1fd411d 
…calls in parallel (#32429)

The `XdsFaultInjectionMaxFault` test has seen a few flakes since #32326
was merged. I believe the flakiness is caused by the fact that when a
large number of RPCs are queued up before the resolver result comes in,
those RPCs are now re-processed in parallel instead of sequentially,
which can cause us to delay more RPCs than we should due to the
`max_faults` setting. To fix this, we change the test to ensure that the
channel is connected (i.e., the resolver result has already been
returned) before we start sending a large number of concurrent RPCs.

Although this is the only test that I've seen flakes in, I've made this
same change consistently to all fault injection tests that are creating
a large number of concurrent RPCs, since the same flake could affect any
of them.
wanlin31 pushed a commit that referenced this pull request May 18, 2023
@markdroth @wanlin31
client channel: fix TSAN flake in FilterBasedLoadBalancedCall (#32491)
4f472ec 
This fixes a very strange TSAN flake seen in an internal test
(b/268292646), which seems to have been introduced in
#32326.

I've included an example of the TSAN failure below, for future
reference. I don't fully understand what is causing this failure. It
looks like the async callback from a previous picker update caused the
pick to be re-queued but then got stuck draining queued callbacks in its
`ExecCtx` instance, while the async callback from the next picker update
had already failed the call. But the part I don't understand is why the
call combiner cancellation callback wound up scheduled on the `ExecCtx`
instance from the first thread -- that seems likely to be caused by
either the `WorkSerializer` or maybe `ExecCtx` work-stealing, but the
exact details are likely to be hard to nail down.

Switching from `EventEngine::Run()` back to `ExecCtx::Run()` seems to
fix the flake, so that's what I'm doing in this PR. I find this
work-around deeply dissatisfying, especially since I do not fully
understand the root cause of the problem. However, given that we are
working toward eliminating the `FilterBasedLoadBalancedCall` code
entirely as part of the promise conversion, this problem is probably not
worth further investigation.

```
WARNING: ThreadSanitizer: data race (pid=7406)
  Read of size 8 at 0x7b8800037558 by thread T47:
    #0 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::~FilterBasedLoadBalancedCall() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2780:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x530ed) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #1 Delete third_party/grpc/src/core/lib/gprpp/ref_counted.h:248:31 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #2 Unref third_party/grpc/src/core/lib/gprpp/orphanable.h:102:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #3 ~RefCountedPtr third_party/grpc/src/core/lib/gprpp/ref_counted_ptr.h:103:36 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #4 ~LbQueuedCallCanceller third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3132:51 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #5 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::LbQueuedCallCanceller::CancelLocked(void*, absl::Status) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3168:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x64a73)
    #6 exec_ctx_run third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:45:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x4199) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #7 grpc_core::ExecCtx::Flush() third_party/grpc/src/core/lib/iomgr/exec_ctx.cc:72:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x4199)
    #8 ~ExecCtx third_party/grpc/src/core/lib/iomgr/exec_ctx.h:117:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #9 operator() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3201:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #10 __invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #11 invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #12 InvokeR<void, (lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &, void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #13 void absl::internal_any_invocable::LocalInvoker<false, void, grpc_core::ClientChannel::FilterBasedLoadBalancedCall::RetryPickLocked()::$_0&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c095)
    #14 operator() third_party/absl/functional/internal/any_invocable.h:855:1 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74) (BuildId: 7802a789e61dddac5a33230a2f294046)
    #15 util::functional::internal::FunctorCallback<Closure, false, absl::AnyInvocable<void ()>, void ()>::Run() util/functional/to_callback_internal.h:87:27 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74)
    #16 void eventmanager::EventManager::RunTask<false>(eventmanager::EventManager::WorkerThread*, eventmanager::TaskInfo*) net/eventmanager/em1/eventmanager.cc:2464:20 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2ae1b) (BuildId: b3d97518d96e160722658c4aa0795013)
    #17 void eventmanager::EventManager::RunWorkerLoop<false, false>(eventmanager::EventManager::WorkerThread*) net/eventmanager/em1/eventmanager.cc (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2a45d) (BuildId: b3d97518d96e160722658c4aa0795013)
    #18 eventmanager::EventManager::WorkerThread::WorkerMain() net/eventmanager/em1/eventmanager.cc:430:19 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1d841) (BuildId: b3d97518d96e160722658c4aa0795013)
    #19 operator() net/eventmanager/em1/eventmanager.cc:396:64 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351) (BuildId: b3d97518d96e160722658c4aa0795013)
    #20 __invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #21 invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #22 InvokeR<void, (lambda at net/eventmanager/em1/eventmanager.cc:396:55), void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #23 void absl::internal_any_invocable::LocalInvoker<false, void, eventmanager::EventManager::WorkerThread::StartThread(int, int, thread::SchedPolicy, std::__tsan::basic_string_view<char, std::__tsan::char_traits<char>>)::'lambda'()&&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #24 operator() third_party/absl/functional/internal/any_invocable.h:863:1 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e) (BuildId: a86eb59aee0be7b5882a1b532922e3ca)
    #25 ClosureThread::Run() thread/thread.h:459:25 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e)
    #26 Thread::ThreadBody(void*) thread/thread.cc:1284:16 (libST-a5842b3303c3_thread_Slibthread.so+0x251e9) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)

  Previous write of size 8 at 0x7b8800037558 by thread T40:
    #0 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::PendingBatchesFail(absl::Status, bool (*)(grpc_core::CallCombinerClosureList const&)) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2863:13 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x537c7) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #1 grpc_core::ClientChannel::FilterBasedLoadBalancedCall::TryPick(bool) third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3179:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x5537f) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #2 operator() third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3200:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077) (BuildId: a8369742d3c435f7f1a244a535f4907c)
    #3 __invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #4 invoke<(lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #5 InvokeR<void, (lambda at third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:3196:46) &, void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #6 void absl::internal_any_invocable::LocalInvoker<false, void, grpc_core::ClientChannel::FilterBasedLoadBalancedCall::RetryPickLocked()::$_0&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Uclient_Uchannel.so+0x6c077)
    #7 operator() third_party/absl/functional/internal/any_invocable.h:855:1 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74) (BuildId: 7802a789e61dddac5a33230a2f294046)
    #8 util::functional::internal::FunctorCallback<Closure, false, absl::AnyInvocable<void ()>, void ()>::Run() util/functional/to_callback_internal.h:87:27 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibevent_Uengine_Uem_Uimpl.so+0x16a74)
    #9 void eventmanager::EventManager::RunTask<false>(eventmanager::EventManager::WorkerThread*, eventmanager::TaskInfo*) net/eventmanager/em1/eventmanager.cc:2464:20 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2ae1b) (BuildId: b3d97518d96e160722658c4aa0795013)
    #10 void eventmanager::EventManager::RunWorkerLoop<false, false>(eventmanager::EventManager::WorkerThread*) net/eventmanager/em1/eventmanager.cc (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x2a45d) (BuildId: b3d97518d96e160722658c4aa0795013)
    #11 eventmanager::EventManager::WorkerThread::WorkerMain() net/eventmanager/em1/eventmanager.cc:430:19 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1d841) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 operator() net/eventmanager/em1/eventmanager.cc:396:64 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351) (BuildId: b3d97518d96e160722658c4aa0795013)
    #13 __invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #14 invoke<(lambda at net/eventmanager/em1/eventmanager.cc:396:55)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #15 InvokeR<void, (lambda at net/eventmanager/em1/eventmanager.cc:396:55), void> third_party/absl/functional/internal/any_invocable.h:131:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #16 void absl::internal_any_invocable::LocalInvoker<false, void, eventmanager::EventManager::WorkerThread::StartThread(int, int, thread::SchedPolicy, std::__tsan::basic_string_view<char, std::__tsan::char_traits<char>>)::'lambda'()&&>(absl::internal_any_invocable::TypeErasedState*) third_party/absl/functional/internal/any_invocable.h:301:10 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x28351)
    #17 operator() third_party/absl/functional/internal/any_invocable.h:863:1 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e) (BuildId: a86eb59aee0be7b5882a1b532922e3ca)
    #18 ClosureThread::Run() thread/thread.h:459:25 (libST-a5842b3303c3_security_Sganpati_Ssharded_Uclient_Slibacl_Ucache.so+0x24c1e)
    #19 Thread::ThreadBody(void*) thread/thread.cc:1284:16 (libST-a5842b3303c3_thread_Slibthread.so+0x251e9) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)

  Thread T47 'EventManager_Default' (tid=7454, running) created by main thread at:
    #0 pthread_create third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1048:3 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x1887b9) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #1 Thread::CreatePthread(pthread_attr_t&) thread/thread.cc:485:13 (libST-a5842b3303c3_thread_Slibthread.so+0x24b1d) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #2 Thread::Start() thread/thread.cc:667:3 (libST-a5842b3303c3_thread_Slibthread.so+0x2574b) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #3 StartThread net/eventmanager/em1/eventmanager.cc:397:14 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed) (BuildId: b3d97518d96e160722658c4aa0795013)
    #4 eventmanager::EventManager::EventManager(eventmanager::EventManager::Options const&) net/eventmanager/em1/eventmanager.cc:1274:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed)
    #5 InitializeDefaultEventManager() net/eventmanager/em1/eventmanager.cc:3042:39 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25c06) (BuildId: b3d97518d96e160722658c4aa0795013)
    #6 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #7 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #8 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #9 GoogleOnceInternalInit base/once.cc:23:5 (libST-a5842b3303c3_base_Slibonce.so+0xeee) (BuildId: 6c436371e89d7962c2b544d27465d6a0)
    #10 GoogleOnceInternalInitSchedCoopAndKernel(std::__tsan::atomic<unsigned int>*, void (*)()) base/once.cc:15:3 (libST-a5842b3303c3_base_Slibonce.so+0xeee)
    #11 GoogleOnceInit base/once.h:72:5 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 eventmanager::EventManager::DefaultEventManager() net/eventmanager/em1/eventmanager.cc:3059:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9)
    #13 Default net/eventmanager/em1/eventmanager.h:780:43 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5) (BuildId: 2e1dde05e8bce5319856279d896c4405)
    #14 eventmanager::Default() net/eventmanager/eventmanager_default.cc:22:10 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5)
    #15 grpc::(anonymous namespace)::InitGlobalEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:38:23 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x1c28) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #16 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #17 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #18 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #19 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #20 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #21 grpc::GetGrpcEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:51:3 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x18c1) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #22 check_engine_available(bool) net/grpc/internal/src/core/ext/poller/event_manager_poller/ev_event_manager_linux.cc:1271:21 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Spoller_Sevent_Umanager_Upoller_Slibev_Uevent_Umanager_Ulinux.so+0x4fcb) (BuildId: 620adb83259aed32440614774678b3a1)
    #23 try_engine third_party/grpc/src/core/lib/iomgr/ev_posix.cc:141:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #24 operator() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:184:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #25 grpc_event_engine_init()::$_0::__invoke() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #26 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #27 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #28 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #29 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #30 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #31 grpc_event_engine_init() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xacab1) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #32 iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_posix.cc:43:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xaf1d0) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #33 grpc_iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_internal.cc:35:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x688b) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #34 grpc_iomgr_init() third_party/grpc/src/core/lib/iomgr/iomgr.cc:66:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xae777) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #35 grpc_init third_party/grpc/src/core/lib/surface/init.cc:144:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc.so+0x4518) (BuildId: 778b43e2e8f8c132a7acf9b3da117fff)
    #36 k3::(anonymous namespace)::GrpcInitHackEnvironment::SetUp() storage/k3/grpc/grpc_server_test.cc:144:27 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x212d21) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #37 SetUpEnvironment third_party/googletest/googletest/src/gtest.cc:5763:55 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #38 for_each<std::__tsan::__wrap_iter<testing::Environment *const *>, void (*)(testing::Environment *)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__algorithm/for_each.h:26:5 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #39 ForEach<std::__tsan::vector<testing::Environment *, std::__tsan::allocator<testing::Environment *> >, void (*)(testing::Environment *)> third_party/googletest/googletest/src/gtest-internal-inl.h:288:3 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #40 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5876:9 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #41 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/googletest/src/gtest.cc (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #42 testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5464:10 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41)
    #43 RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2329:73 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #44 main testing/base/internal/gunit_main.cc:86:10 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a)

  Thread T40 'EventManager_Default' (tid=7447, running) created by main thread at:
    #0 pthread_create third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1048:3 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x1887b9) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #1 Thread::CreatePthread(pthread_attr_t&) thread/thread.cc:485:13 (libST-a5842b3303c3_thread_Slibthread.so+0x24b1d) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #2 Thread::Start() thread/thread.cc:667:3 (libST-a5842b3303c3_thread_Slibthread.so+0x2574b) (BuildId: a1762ef5c5732b3ad8a3b3d798292e27)
    #3 StartThread net/eventmanager/em1/eventmanager.cc:397:14 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed) (BuildId: b3d97518d96e160722658c4aa0795013)
    #4 eventmanager::EventManager::EventManager(eventmanager::EventManager::Options const&) net/eventmanager/em1/eventmanager.cc:1274:23 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x1bbed)
    #5 InitializeDefaultEventManager() net/eventmanager/em1/eventmanager.cc:3042:39 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25c06) (BuildId: b3d97518d96e160722658c4aa0795013)
    #6 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #7 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #8 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #9 GoogleOnceInternalInit base/once.cc:23:5 (libST-a5842b3303c3_base_Slibonce.so+0xeee) (BuildId: 6c436371e89d7962c2b544d27465d6a0)
    #10 GoogleOnceInternalInitSchedCoopAndKernel(std::__tsan::atomic<unsigned int>*, void (*)()) base/once.cc:15:3 (libST-a5842b3303c3_base_Slibonce.so+0xeee)
    #11 GoogleOnceInit base/once.h:72:5 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9) (BuildId: b3d97518d96e160722658c4aa0795013)
    #12 eventmanager::EventManager::DefaultEventManager() net/eventmanager/em1/eventmanager.cc:3059:3 (libST-a5842b3303c3_net_Seventmanager_Sem1_Slibem1.so+0x25ad9)
    #13 Default net/eventmanager/em1/eventmanager.h:780:43 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5) (BuildId: 2e1dde05e8bce5319856279d896c4405)
    #14 eventmanager::Default() net/eventmanager/eventmanager_default.cc:22:10 (libST-a5842b3303c3_net_Seventmanager_Slibeventmanager_Udefault.so+0x14c5)
    #15 grpc::(anonymous namespace)::InitGlobalEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:38:23 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x1c28) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #16 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #17 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #18 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #19 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #20 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #21 grpc::GetGrpcEventManager() net/grpc/internal/src/core/ext/event_engine/grpc_event_manager.cc:51:3 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Sevent_Uengine_Slibgrpc_Uevent_Umanager.so+0x18c1) (BuildId: 8bd3c1826fd11ffa128e7839a85c05a0)
    #22 check_engine_available(bool) net/grpc/internal/src/core/ext/poller/event_manager_poller/ev_event_manager_linux.cc:1271:21 (libST-a5842b3303c3_net_Sgrpc_Sinternal_Ssrc_Score_Sext_Spoller_Sevent_Umanager_Upoller_Slibev_Uevent_Umanager_Ulinux.so+0x4fcb) (BuildId: 620adb83259aed32440614774678b3a1)
    #23 try_engine third_party/grpc/src/core/lib/iomgr/ev_posix.cc:141:9 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #24 operator() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:184:7 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #25 grpc_event_engine_init()::$_0::__invoke() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xad888)
    #26 __invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #27 invoke<void (*&)()> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:539:12 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #28 void absl::base_internal::CallOnceImpl<void (*&)()>(std::__tsan::atomic<unsigned int>*, absl::base_internal::SchedulingMode, void (*&)()) third_party/absl/base/call_once.h:184:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaab9)
    #29 call_once<void (*&)()> third_party/absl/base/call_once.h:216:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b) (BuildId: fe43c5a995ecc22efa243d05ef607fb9)
    #30 gpr_once_init third_party/grpc/src/core/lib/gpr/sync_abseil.cc:107:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgpr.so+0xaa0b)
    #31 grpc_event_engine_init() third_party/grpc/src/core/lib/iomgr/ev_posix.cc:175:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xacab1) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #32 iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_posix.cc:43:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xaf1d0) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #33 grpc_iomgr_platform_init() third_party/grpc/src/core/lib/iomgr/iomgr_internal.cc:35:35 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibexec_Uctx.so+0x688b) (BuildId: cafb035b95f6c781e93706a71dce95f3)
    #34 grpc_iomgr_init() third_party/grpc/src/core/lib/iomgr/iomgr.cc:66:3 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc_Ubase.so+0xae777) (BuildId: 3cb6b043bd10222927301e33a3760646)
    #35 grpc_init third_party/grpc/src/core/lib/surface/init.cc:144:5 (libST-a5842b3303c3_third_Uparty_Sgrpc_Sgoogle_Slibgrpc.so+0x4518) (BuildId: 778b43e2e8f8c132a7acf9b3da117fff)
    #36 k3::(anonymous namespace)::GrpcInitHackEnvironment::SetUp() storage/k3/grpc/grpc_server_test.cc:144:27 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x212d21) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #37 SetUpEnvironment third_party/googletest/googletest/src/gtest.cc:5763:55 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #38 for_each<std::__tsan::__wrap_iter<testing::Environment *const *>, void (*)(testing::Environment *)> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__algorithm/for_each.h:26:5 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #39 ForEach<std::__tsan::vector<testing::Environment *, std::__tsan::allocator<testing::Environment *> >, void (*)(testing::Environment *)> third_party/googletest/googletest/src/gtest-internal-inl.h:288:3 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #40 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5876:9 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6f58b)
    #41 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/googletest/src/gtest.cc (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41) (BuildId: ec51e8d56b1390e9939f663d4700c7da)
    #42 testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5464:10 (libST-a5842b3303c3_third_Uparty_Sgoogletest_Slibgtest.so+0x6ee41)
    #43 RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2329:73 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a) (BuildId: e468d94e410a7e13e19b26c4f7277674)
    #44 main testing/base/internal/gunit_main.cc:86:10 (c0da98875dfb74bd2cfcbd76306f382f2d05e1326f778b3a4cefdef1a42a75e7_020000b94930+0x2aa83a)

SUMMARY: ThreadSanitizer: data race third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc:2780:5 in grpc_core::ClientChannel::FilterBasedLoadBalancedCall::~FilterBasedLoadBalancedCall()
```
@markdroth markdroth mentioned this pull request May 19, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ctiller ctiller ctiller approved these changes

Assignees
No one assigned
Labels
bloat/low imported Specifies if the PR has been imported to the internal repository lang/core per-call-memory/neutral per-channel-memory/neutral release notes: no Indicates if PR should not be in release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@markdroth @ctiller