Health and Ready checks library

Summary

This library helps to implement ready and health checks in Go driven projects. It includes following functionality:

• GRPC health implementation according to the https://github.com/grpc/grpc/blob/master/doc/health-checking.md
• GRPC ready implementation based on the internal protos
• REST health and ready server as a standalone/sidecar implementation
• REST health and ready handlers which can be added to your running REST servers
• Cli GRPC client for the K8s integrations

Health implementation

Health checking logic is based on the assumption, that if a running service sending too many critical errors per time unit, it's considered to be unhealthy.

This approach allows to create flexible implementations with variadic sensitivity thresholds and error types. Your implementation might decide to send only critical or caught fatal errors combined with a quite low threshold (e.g. 3 errors per minute) or you can send all possible errors with a threshold (5 errors per second). However we recommend to send only the errors which possibly can be resolved after the restart of the whole service.

You can implement health functionality with following steps:

//create a buffered error channel
errStream := errs.NewErrStream(10)

//create health checker with the threshold parameteers
healthChecker := health.NewErrsListener(maxErrorsCount, time.Minute, errStream)

//create context to indicate programm exit to the running go routine
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

//start errors consumption from the stream
go healthChecker.Start(ctx)

//start health server as standalone/sidecar http instance
srv := WithHealth(Server{}, healthChecker)   
if err := srv.Start(ctx, targetPort); err != nil {
    log.Fatal(err)
}

//or add health handler to your own http server
handler := NewHealthHandler(healthChecker)
router := mux.NewRouter() #see https://github.com/gorilla/mux
router.Handle("/healthz", handler)

//inject errStream into your logic to indicate possible health failures
//... inside your logic ...
//errStream.Send(errors.New("some critical error"))

If you are unhappy with this health implementation, you can provide another implementation of health.Checker interface to both GRPC and REST servers.

Ready implementation

Ready server is equipped with a collection of test functions which are expected to call external or internal APIs which are crucial for your service which you want to test for readiness. As an example you can imagine a products microservice which allows to update, read and delete products over HTTP. We assume that it completely depends on the database service, so when both are simultaneously started, API should not accept traffic when db service is not ready to accept new requests. Similarly if the db is temporary unavailable or is being restarted, API should stop accept incoming requests rather than fail immediately. You can create readiness check as following:

//create ready tests for your crucial services
readyChecks := []ready.Test{
	{
		TestFunc: func() error {
		    isHealthy := db.IsAlive()
		    if !isHealthy {
		        return errors.New("db is not ready yet")
	        }
		    return nil
	    },
		Name: "Db Ready Check",
	}
}

//create ready checker which will repeat all test functions, if any fails, ready checker will retry 2 times and sleep one second between attempts
readyChecker := ready.NewTestChecker(readyChecks, 2, time.Second, sleep.RuntimeSleeper{})

//start standalone/sidecar ready server, timeout is shared among all ready checks
srv := WithReady(Server{}, readyChecker, time.Second)   
if err := srv.Start(ctx, targetPort); err != nil {
    log.Fatal(err)
}

//or register ready handler as part of your server, timeout is shared among all ready checks
handler := NewReadyHandler(time.Second, readyChecker)
router := mux.NewRouter() #see https://github.com/gorilla/mux
router.Handle("/readyz", handler)

//now you can trigger rediness checks against /readyz url

For more examples see example_Server_test.go

Kubernetes integration

For REST APIs you can use following k8s manifest:

        ...
        containers:
              ...
              livenessProbe:
                  httpGet:
                    path: /healthz
                    port: 8099
                  initialDelaySeconds: 10
                  periodSeconds: 10
              readinessProbe:
                httpGet:
                  path: /readyz
                  port: 8100
                initialDelaySeconds: 15
                periodSeconds: 20

For GRPC APIs you first need to create a cli interface:

        import "github.com/breathbath/healthReadyChecks"
        
        func main() {
            err := grpc.CheckHealth(address, "My microservice health")
            if err != nil {
                panic(err)
            }
        }

Add to k8s manifest

        ...
        containers:
              ...
              livenessProbe:
                  exec:
                      command:
                          - "/main"
                          - "health-grpc"
                  initialDelaySeconds: 5
                  periodSeconds: 10
              readinessProbe:
                  exec:
                      command:
                          - "/main"
                          - "ready-grpc"
                  initialDelaySeconds: 8
                  periodSeconds: 20            

Running tests

make test