-
Notifications
You must be signed in to change notification settings - Fork 4.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SubChannels stuck in Idle when using Weighted Round Robin #8173
Comments
I tried to debug this issue today and the only thing that I found so far is that it looks like balancer_wrapper serializer is the bottleneck. acBalancerWrapper.updateState is responsible for triggering reconnection by exiting idle mode, but by the time this method is called serializer is already filled with callbacks registred by the RegisterHealthListener method, and for some reason it takes a really long time to process all those callbacks. This is reproducible with both short and large maxConnectionAge. |
It looks like the callback_serializer in balancer_wrapper is getting blocked. So the subchannel notification for IDLE is not getting delivered to the LB policy. There is a O(n^2) loop in wrr here every time an endpoint reports ready: grpc-go/balancer/weightedroundrobin/balancer.go Lines 260 to 264 in 775150f
When all n children become ready the loop is run n times, so total time spent is O(n^3). For n = 2*10^3, this will probably take minutes to complete. |
I can confirm this theory by profiles, as Do you have a sense of what the right fix is? It's a bit confusing that something called "map" has O(n) lookup. There is good context in the PR that introduces EndpointMap: #6679. |
The simplest fix I can think of is to make the EndpointMap operation complexities match a Go map instead of a slice. I've thought of a way to make lookups O(1). The key of an endpoint map is a slice of strings. We can create a canonical representation of a slice of strings by sorting it and removing duplicates. We can then encode the string slice (using json or using base64) to convert []string->string that can be used as a map key. This should allow using a Go map to store and query endpoints instead of a slice. |
What version of gRPC are you using?
What version of Go are you using (
go version
)?go version go1.24.0 darwin/arm64
What operating system (Linux, Windows, …) and version?
Discovered on Linux, reproduced locally on MacOS.
What did you do?
We use the
weighted_round_robin
balancer withMaxConnectionAge
set to 5 minutes. Most services worked fine. However, one service with a large number of instances (500+) experienced a surge of errors waiting for subchannels to transition to ready:What did you expect to see?
RPC succeed.
What did you see instead?
Additional context
Turning on the logs, we found that after a period of the max connection age causing servers to close connections, subchannels would transition to Idle and never go back to Connecting. Since weighted round robin is not a lazy balancer, transitions to
Idle
happening due to the server closing the connection should immediately trigger a transition toConnecting
. The logs for a single subchannel look like this:No subsequent update happen for this subchannel. Since all subchannel follow the same patterns, RPCs start failing.
Note that:
endpointsharding
and uses the new load balancer infrastructure.round_robin
is not affected.Local reproduction steps
We were able to reproduce the problem in the repo by modifying the load balancing examples. The commit to reproduce is here: atollena@ec3b1c7
The changes included in the commit:
When running the server and client examples, you'll end up with logs like this (debug logs enabled):
The text was updated successfully, but these errors were encountered: