grpc server get stuck but clientHealthCheck do not report server unhealthy

[heartlock]

What version of gRPC are you using?

version: v1.38.0

What version of Go are you using (go version)?

golang: 1.17.4

What operating system (Linux, Windows, …) and version?

linux

What did you do?

I use kill -19 to stop grpc server, but clientHealthCheck do not report server unhealthy.

What did you expect to see?

report server unhealthy

What did you see instead?

grpc server get stuck, but clientHealthCheck do not report server unhealthy

[heartlock]

@dfawley Would you like to have a look?

grpc-go/health/client.go

Line 94 in 6b15b13

err = s.RecvMsg(resp)

watch stream receives nothing when GRPC server gets stuck. GRPC balancer may send requests to this stuck endpoint.

[dfawley]

The client-side health checks (which must be enabled via the service config) interact with the server's health check service (go impl). The client also needs to import the health package.

Are you sure you have both sides configured properly?

[dfawley]

Please also see the example: https://github.com/grpc/grpc-go/tree/master/examples/features/health

[heartlock]

The client-side health checks (which must be enabled via the service config) interact with the server's health check service (go impl). The client also needs to import the health package.

Are you sure you have both sides configured properly?

@dfawley I am sure that healthcheck service works on both sides. Client-side got first HealthCheckResponse, but got nothing after I kill -19 server. clientHealthCheck function will always be stuck in err = s.RecvMsg(resp). The server becomes a zombie process, will do not send anything to the client.

root@luke:~/workspace/mod-pro/test# ./client 
2022/03/02 09:28:46 Greeting: Hello world
2022/03/02 09:28:46 health respnse: status:SERVING

[dfawley]

Can you try enabling keepalives on your client? https://github.com/grpc/grpc-go/tree/master/examples/features/keepalive for an example.

[heartlock]

GRPC keepalive works for me. Thanks for your help. @dfawley I will close this issue.