syncContext.execute(exitIdleMode) blocks business thread when reprocessing many pending streams

[jammyMarse]

Description:

We encountered a performance issue related to syncContext.execute(() -> exitIdleMode()) when creating a gRPC stream on a not-yet-ready TCP connection.

Context:

Our business thread executes:
streamObserver = connect(); // -> internally calls ManagedChannel.newCall(...)
Under the hood, this triggers:

ManagedChannel.newCall(...)
  → syncContext.execute(() -> exitIdleMode())
      → starts NameResolver / LoadBalancer
      → LoadBalancerHelper.updateBalancingState(...) triggers updateSubchannelPicker(...)
      → DelayedClientTransport.reprocess()

If the TCP connection is not yet ready, the stream is placed in a pendingStreams list. Once the connection becomes ready, DelayedClientTransport.reprocess() will process all pending streams.

Problem:

When the number of pending streams is large, DelayedClientTransport.reprocess() becomes time-consuming and is executed on the same thread as syncContext.execute(...), i.e., the business thread that originally called connect().

This causes noticeable delays and performance degradation on our business logic path.

Question:

Is it possible to specify or offload the execution of syncContext.execute(() -> exitIdleMode()) to a dedicated thread pool or event loop, to avoid blocking the business thread?

If not, would it make sense to provide an option for decoupling such internal reprocessing logic from application threads?

[jammyMarse]

[kannanjgithub]

For the pending streams being reprocessed, the more intensive work of creating the real streams on the transport doesn't happen on the same thread but using the executor, here.

The code you pointed for removing pending streams from the set is supposed to be a simple book keeping operation, and it did have the inefficiency you mentioned (it takes O(n2) complexity) and it was only recently fixed and is available on 1.72.0.

As an observation about your question, in the stack trace you showed it is originating from the Netty epoll transport thread and not from the business thread. It makes sense since you mentioned you are invoking newCall from an observer callback.

[jammyMarse]

For the pending streams being reprocessed, the more intensive work of creating the real streams on the transport doesn't happen on the same thread but using the executor, here.

The code you pointed for removing pending streams from the set is supposed to be a simple book keeping operation, and it did have the inefficiency you mentioned (it takes O(n2) complexity) and it was only recently fixed and is available on 1.72.0.

As an observation about your question, in the stack trace you showed it is originating from the Netty epoll transport thread and not from the business thread. It makes sense since you mentioned you are invoking newCall from an observer callback.

Thanks， the epolleEventLoop thread is our own business thread~

[kannanjgithub]

I hope your question is answered. Feel free to comment to reopen in the future if needed.