How to add health check to service and expose it via the gateway

[Jthomas54]

📚 Documentation

There are many issues around checking the health of a service and it still isn't quite clear to me how this should be done. It seems that the gateway and health checks assume you are running in a different process. I'm currently hosting the two on the same port and serve the traffic based on headers (Go).

The most obvious way to me is to implement the check/watch in the service definition so the gateway creates the needed proxy configurations. This seems to make an awkward health check from something like an AWS application load balancer. You wouldn't be able to expose the same interface of HTTP 1.1 as HTTP 2. Even if using the HealthServer, this doesn't handle the actual health response from the service that is being checked.

I know this sounds like a general grpc question, but a lot of it stems from creating a endpoint through the gateway and
it being hosted on the same port.

[johanbrandhorst]

Hi, thanks for your issue. I am a little confused, could you clarify what information you need? There are essentially two well supported paths for doing health checking in the grpc gateway today:

1. Define your own HealthService and add a HTTP mapping as with any other gRPC service. Benefits of this is that it works like any other gRPC-Gateway service and you can use protoc-gen-openapiv2 to get an OpenAPI spec.
2. Use the new WithHealthEndpointAt which adds an automatic translation from the gRPC health API to a HTTP healthz style API. This is simpler and lets your gRPC service remain agnostic to HTTP standards (it just implements the gRPC health check API). This endpoint does not get generated when using protoc-gen-openapiv2.

Which one of these are you interested in pursuing? Or are you looking for something else?

[Jthomas54]

The first option is the solution I was settling for. I want to use the WithHeathEndpointAt solution, it isn't very clear how I can implement this when the client and server are the same thing. The automatic status resolution and supporting tools that can't use grpc is what I want. HealthService and MyService are listening on the same port locally. I don't see a obvious way to make this happen. Can you provide guidance.

Adding what I had to add to accomplish this:

type InProcessHealthClient struct {
	Server grpc_health_v1.HealthServer

	grpc_health_v1.HealthClient
}

func (client *InProcessHealthClient) Check(ctx context.Context, req *grpc_health_v1.HealthCheckRequest, opts ...grpc.CallOption) (*grpc_health_v1.HealthCheckResponse, error) {
	// we ignore call options since it is in-process
	return client.Server.Check(ctx, req)
}

func (svr *Server) Serve(ctx context.Context, addr string) error {
	v1.RegisterStreamServer(svr.grpc, svr)
	grpc_health_v1.RegisterHealthServer(svr.grpc, svr.health)

	svr.health.SetServingStatus("", grpc_health_v1.HealthCheckResponse_NOT_SERVING)
	svr.gateway = runtime.NewServeMux(runtime.WithHealthzEndpoint(&InProcessHealthClient{Server: svr.health}))
	v1.RegisterResourceHandlerServer(ctx, svr.gateway, svr)

	return http.ListenAndServe(addr, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if r.ProtoMajor == 2 && strings.HasPrefix(r.Header.Get("Content-Type"), "application/grpc") {
			svr.grpc.ServeHTTP(w, r)
		} else {
			svr.gateway.ServeHTTP(w, r)
		}
	}))
}

I could be misunderstanding the usage here but it just isn't clicking with me.

[johanbrandhorst]

With option 1 you just define a custom health check endpoint, you don't concern yourself with the grpc_health_v1.HealthServer definitions. Just make a service definition as usual and define a mapping to do what you want.

With option 2, you implement the grpc_health_v1.HealthServer on your server, and then dial that and hand it over to the WithHealthServiceAt option. It doesn't matter that your client and server are in the same application, the distinction is still there.

Option 1 is probably easiest if you don't want to bother with grpc_health_v1.HealthServer, which I think is maybe unnecessarily complicated. Does that make sense? I don't have any practical examples available for you.

[Jthomas54]

The "then dial that and hand it over to the" is the problem here, since they are listening on the same port, and is hasn't happened yet, it seems to cause some issues with the actually calls. The call ends with connection closed before server preface received. I was hoping to use grpc_health_v1.HealthServer since it is the recommended approach, and I didn't want to have to define it over and over for each of the services that would be published.

I will consider this closed, but it would be nice if there was a way we could provide a way to communicate with the grpc service without having to proxy it over the loopback address and avoid the serialization. It would provide benifits and I think it would allow a more elegant solution to my problem.

[yogeshlonkar]

This might be late entry but I was able implement grpc_health_v1.HealthServer and not use connection between Rest server and gRPC server.

Instead of using client/ connection I created a stub of grpc ServerStream HealthReporter which watches health method using instance of ServiceServicer

health_watcher.go

var _ grpc_health_v1.Health_WatchServer = &HealthReporter{}

// HealthReporter reports the status of the RestAPI gateway by watching gRPC health service method
type HealthReporter struct {
	grpc.ServerStream // stub
        // this context should live as long as rest API is up
        // instead of doing this you could return context background from Context()
	ctx    context.Context 
	status grpc_health_v1.HealthCheckResponse_ServingStatus // current status
}

// Send called by the gRPC health watch method.
func (h *HealthReporter) Send(response *grpc_health_v1.HealthCheckResponse) error {
	h.status = response.Status
	return nil
}

// Report status to RestAPI.
func (h *HealthReporter) Report(c *gin.Context) {
	switch h.status {
	case grpc_health_v1.HealthCheckResponse_SERVING:
		c.String(http.StatusOK, "OK")
	case grpc_health_v1.HealthCheckResponse_NOT_SERVING:
		c.String(http.StatusServiceUnavailable, "NOT_SERVING")
	default:
		c.String(http.StatusInternalServerError, "UNKNOWN")
	}
}

// Watch the gRPC health service. Should be called as a goroutine.
func (h *HealthReporter) Watch(service myServiceServer) {
	req := &grpc_health_v1.HealthCheckRequest{
		Service: "",
	}
	if err := service.Watch(req, h); err != nil {
		panic(err) // handle better
	}
}

// Context need to implement required method from grpc.ServerStream interface.
func (h *HealthReporter) Context() context.Context {
	return h.ctx
}

rest_api.go

// StartRestApi setup api gateway and additional services
func StartRestApi(ctx context.Context, s myServiceServer, h HealthReporter) {
        go h.Watch(service)
	mux := runtime.NewServeMux()
        // register services with mux
	// _ := pb.RegisterXYZServiceHandlerServer(ctx, mux, s)
	g := gin.New()
	grpcProxy := gin.WrapF(mux.ServeHTTP)
	s.gin.Any("/*any", func(c *gin.Context) {
		path := c.Param("any")
		switch {
		case strings.HasPrefix(path, "/liveness"):
			c.String(http.StatusOK, "OK")
		case strings.HasPrefix(path, "/readiness"):
			h.Report(c)
		default:
			grpcProxy(c)
		}
	})
}