Enable grpclb for java implementation

[wicharypawel]

Hello,
I am a .NET developer trying to communicate java client with custom C# implementation of lookaside server. The question is if somebody may share working configuration of how to enable grpclb in Java? I have tried examples and read release notest but without success.

Steps I have tried before:

• connect C# client with server using my custom lookaside server (IT WORKS OK)
• connect Go client with server using my custom lookaside server (IT WORKS OK)
• when working with JAVA I have faced few pages over the internet with some puzzels but I can not get them working together.

My current setup:

Java Client -------port 8000 Service replica 1 written in C# (name: grpc-server)
| |
| |----------port 8000 Service replica 1 written in C# (name: grpc-server)
|
port 9000
Lookaside Loadbalancer written in C# (name: grpc-server-balancer)

Answering questions below would help: (assume I am using io.grpc version 1.28.0)

• is it required to configure dns resolver like this .nameResolverFactory(new DnsNameResolverProvider()) ?
• how to force grpclb policy for ManagedChannel?
• it is required for lookaside server to listen to port 80 or 443 or port can be variable when setting target?
• is it required to support TXT records? (I see a log like this FINE: No TXT records found for grpc-server-balancer)
• is it required to support grpc service config?

So sorry for "newbie" questions bear in mind I'm not a full blast Java engineer

Regards

[ejona86]

Quick answers:

• No, do not use nameResolverFactory. Nothing should be necessary here. Prefer using (in order) defaults, dns:/// target syntax, or mcb.defaultNameResolver("dns")
• mcb.defaultLoadBalancer("grpc") mcb.defaultLoadBalancer("grpclb") (easiest) or mcb.defaultltServiceConfig(...). Dns TXT is an option here as well, but TXT lookup is off by default (I don't know why you see the log entry)
• port can be anything, but it will need to match what the grpclb server expects (backend ports provided by grpclb server may be different)
• TXT is not required
• Service config is not required

Make sure to have a runtime dependency on grpc-grpclb

[wicharypawel]

@ejona86 are you sure that mcb.defaultLoadBalancer("grpc") is typo free? It gives me a compilation error.

--- update ---

I saw a log message:
FINE: Service loader found GrpclbLoadBalancerProvider{policy=grpclb, priority=5, available=true}

Above compilation error and log message pushed me to use something like mcb.defaultLoadBalancingPolicy("grpclb") instead of mcb.defaultLoadBalancer("grpc")

Success. Thank you for your help.

[wicharypawel]

Extra workaround for people using grpc-java lookaside load balancing in kubernetes (k8s) like me.

Steps:

0. Make sure to have a runtime dependency on grpc-grpclb as @ejona86 said
1. Crete&deploy target service to K8s (we assume that it's service is named grpc-server)
2. Create SRV record of lookaside load balancer

In order to create SRV records in K8s we have to use Headless service as described here https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

Example of such service:

apiVersion: v1
kind: Service
metadata:
  name: grpc-server-balancer
spec:
  clusterIP: None
  ports:
  - name: grpclb
    port: 9000
  selector:
    app: grpc-server-balancer
---
grpc-server-balancer pod or deployment here
this lookaside load balancer should be compatible with grpclb protocol
this lookaside load balancer should return IP address list of grpc-server

This will create SRV record like this _grpclb._tcp.grpc-server-balancer.default.svc.cluster.local

3. Adjust java client:

• set target property to dns:///grpc-server-balancer.default.svc.cluster.local:9000
• configure mcb.defaultLoadBalancingPolicy("grpclb") as described by @ejona86

4. Deploy lookaside loadbalancer created in step 1 to K8s

5. Deploy client

How it is working internally:

• client is picking load balancer policy called grpclb, but this only happends if package grpc-grpclb is specified as runtime dependency
• DNS resolver is trying to find out loadbalancer address. It takes target value and prepend it with _grpclb._tcp. Which match our SRV record
• client is calling loadbalancer for IP address list
• load balancer returns a full list
• client create subchannels for every ip from list
• client round-robin subchannels per call (actual load balancing takes place)

Working example available here https://github.com/wicharypawel/net-core-grpc-load-balance

❗️ NOTE ❗️
According to documentation target should have server value instead of loadbalancer. In This case dns:///grpc-server, instead of dns:///grpc-server-balancer.default.svc.cluster.local:9000

This workaround is only because of K8s

[ejona86]

Oops, sorry for the "grpc" vs "grpclb" typo. I was actually typing from a phone so am happy that is the only problem :)

Glad you got it working. Since you are using the grpc-server-balancer instead of grpc-server, that will mean that each service will need its own grpclb balancer. Depending on your environment that may not be an issue. Note that the target string is used for TLS, so I don't think that setup will work properly for TLS since the backends won't have grpc-server-balancer certificates.

(It's actually possible to add on another hacktrick: specify mcb.overrideAuthority("grpc-server"). That will at least fix TLS. I don't remember if the overridden authority or the original target authority is sent to grpclb. In any case, it is probably better to change the name passed to the balancer with the serviceName grpclb config in service config. It was added in #6563, with slightly more information in grpc/grpc-proto#73 .)

[wicharypawel]

Thanks for the trick. Moreover, I agree that specifing serviceName as target is best approach because of convention _grpclb._tcp.serviceName. 👍

I was forced by kubernetes, and agree that it looks ugly. 💩