grpc service behind k8-headless service is not loadbalancing.

[vkc12uec]

Hi, I have this setup/request flow:

service1 -----> k8-headless-service -------> grpcservice2

(followed this blog)
That is service1 wants to talk to grpcservice2 via a k8-service (headless). The way I make a grpc connection object is like this:

	svcurl := "lenses-sharded-worker-10-0-us-east-1a:9090"
	dnsSvcUrl := "dns:///" + svcurl

	pconn1, err := grpc.Dial(dnsSvcUrl, grpc.WithInsecure(), //nolint:staticcheck
		grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy": "round_robin"}`)) //nolint:staticcheck

	cliUsingSvc := pbsearch.NewProductServiceClient(pconn1) // passed the connection object to grpc generated stub for grpcservice2

I have 8 server pods for grpcservice2. When i do k describe <service> I can also see 8 IPs being exposed in the output hence I am assuming that grpc client internally should continue load-balance traffic equally on all of the 8 pods. But when check the cpu-utilzation of those 8 pods, I see only 3 of them heavy utilized and not the other 5. (see screenshot)

I researched a bit before and hence was using "dns:///" prefix to enable load-balancing but seems like its not working 100% correctly. Sometime back it was working but not anymore. Even the nslookup of the k8-service gives back 8 ip-address.

Can someone suggest what to try next or solve it?

[vkc12uec]

@easwars any idea on this? (sorry first time posting, i m not sure how these doubts are answered on github forums)

[easwars]

Do you have any grpc logs from the client?

You could set these environment variables GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=2 on the client to see grpc logs.

[github-actions]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.