Introduce new config to randomly pick the xDS server host

[PapaCharlie]

Instead of using the default "pick_first" policy, which always picks the first IP address returned by DNS, "round_robin" will pick a random starting point in the list of IPs. This will fix the very lopsided load seen on the xDS servers.

Here is the current traffic distribution with using pick_first:

[shivamgupta1]

Please share local testing results.

[logstashbugreporter]

can you add your ingraph link showing how skewed this problem is right now? ie., why it deserve a fix right now instead of waiting for a more generic solution?
please add background information on the root cause of the issue, i.e., why we need client side round-robin given that DNS return a random shuffle each time?

[shivamgupta1]

Here's the actual implementation of Round robin load balancer that does seem to use a random for the first index: https://github.com/grpc/grpc-java/blob/3e8e56feea2b675c4238ce5e5cab93dc5c601fd5/util/src/main/java/io/grpc/util/RoundRobinLoadBalancer.java#L47

Here's a relevant test: https://github.com/grpc/grpc-java/blob/3e8e56feea2b675c4238ce5e5cab93dc5c601fd5/util/src/test/java/io/grpc/util/RoundRobinLoadBalancerTest.java#L383

[PapaCharlie]

@logstashbugreporter updated the PR description

[bohhyang]

lgtm.
Only that I couldn't think of a good way to test locally, since the client can't get info about the sub-channel picked in the grpc/xds channel.

[shivamgupta1]

Only that I couldn't think of a good way to test locally, since the client can't get info about the sub-channel picked in the grpc/xds channel.

Should be pretty simple to test locally. The observer host a client connects to is in the lsof output for the client process. getaddrinfo results can be hardcoded in the /etc/hosts/ file: https://jameshfisher.com/2018/02/03/what-does-getaddrinfo-do/

[shivamgupta1]

@PapaCharlie suggest to test locally and update results.

[PapaCharlie]

Yeah that's what I was doing today, will post the results soon

…

On Fri, Jan 26, 2024 at 16:06 Shivam Gupta ***@***.***> wrote: @PapaCharlie <https://github.com/PapaCharlie> suggest to test locally and update results. — Reply to this email directly, view it on GitHub <#972 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHDA2POKBNFP6EV6CGR2BLYQRAKHAVCNFSM6AAAAABCHT7U76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJSHA2TCNRWGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[PapaCharlie]

Tested this locally by adding the following to my /etc/hosts:

IP1 main.indis-registry-observer.ei-ltx1.atd.disco.linkedin.com
IP2 main.indis-registry-observer.ei-ltx1.atd.disco.linkedin.com
IP3 main.indis-registry-observer.ei-ltx1.atd.disco.linkedin.com

Without the round_robin config, it will always pick IP1 when starting up. I tested this by repeatedly redeploying the application and checking the logs on the observer. After applying the change, it picks a random IP every time it starts!

[shivamgupta1]

Thanks for testing it locally too.

[whutwhu]

lgtm!

[logstashbugreporter]

LGTM,
just to make sure I understand this, without setting round-robin as default LB policy, NettyChannelBuilder will use the first one of the sorted list of addresses?

[PapaCharlie]

@logstashbugreporter yeah exactly. The sorting is based on IPv6 proximity, so it picks whichever observer it thinks is closest. The problem is that some observers seem to be "closer" than most, which is why we see this very skewed distribution. The round_robin policy ignores proximity and picks a random IP from the set of returned IPs