registry/health: adding healthcheck package

[diogomonica]

Summary

Package health provides a generic health checking framework. The health package works expvar style. By importing the package the debug server is getting a /debug/health endpoint that returns the current status of the application. If there are no errors, /debug/health will return a HTTP 200 status, together with an empty JSON reply {}. If there are any checks with errors, the JSON reply will include all the failed checks, and the response will be have a HTTP 500 status.

A Check can either be run synchronously, or asynchronously. We recommend that most checks are registered as an asynchronous check, so a call to the /debug/health endpoint always returns
immediately. This pattern is particularly useful for checks that verify upstream connectivity or database status, since they might take a long time to return/timeout.

Installing

To install health, just import it in your application:

import "github.com/docker/distribution/health"

You can also (optionally) import health/api that will add two convenience endpoints: /debug/health/down and /debug/health/up. These endpoints add "manual" checks that allow the service to quickly be brought in/out of rotation.

import _ "github.com/docker/distribution/registry/health/api"

# curl localhost:5001/debug/health
{}
# curl -X POST localhost:5001/debug/health/down
# curl localhost:5001/debug/health
{"manual_http_status":"Manual Check"}

After importing these packages to your main application, you can start registering checks.

Registering Checks

The recommended way of registering checks is using a periodic Check. PeriodicChecks run on a certain schedule and asynchronously update the status of the check. This allows CheckStatus() to return without blocking on an expensive check.

A trivial example of a check that runs every 5 seconds and shuts down our server if the current minute is even, could be added as follows:

 func currentMinuteEvenCheck() error {
   m := time.Now().Minute()
   if m%2 == 0 {
     return errors.New("Current minute is even!")
   }
   return nil
 }

 health.RegisterPeriodicFunc("minute_even", currentMinuteEvenCheck, time.Second*5)

Alternatively, you can also make use of RegisterPeriodicThresholdFunc to implement the exact same check, but add a threshold of failures after which the check will be unhealthy. This is particularly useful for flaky Checks, ensuring some stability of the service when handling them.

health.RegisterPeriodicThresholdFunc("minute_even", currentMinuteEvenCheck, time.Second*5, 4)

The lowest-level way to interact with the health package is calling Register directly. Register allows you to pass in an arbitrary string and something that implements Checker and runs your check. If your method returns an error with nil, it is considered a healthy check, otherwise it will make the health check endpoint /debug/health start returning a 500 and list the specific check that failed.

Assuming you wish to register a method called currentMinuteEvenCheck() error you could do that by doing:

health.Register("even_minute", health.CheckFunc(currentMinuteEvenCheck))

CheckFunc is a convenience type that implements Checker.

Another way of registering a check could be by using an anonymous function and the convenience method RegisterFunc. An example that makes the status endpoint always return an error:

 health.RegisterFunc("my_check", func() error {
  return Errors.new("This is an error!")
}))

Examples

You could also use the health checker mechanism to ensure your application only comes up if certain conditions are met, or to allow the developer to take the service out of rotation immediately. An example that checks database connectivity and immediately takes the server out of rotation on err:

 updater = health.NewStatusUpdater()
  health.RegisterFunc("database_check", func() error {
   return updater.Check()
 }))

 conn, err := Connect(...) // database call here
 if err != nil {
   updater.Update(errors.New("Error connecting to the database: " + err.Error()))
 }

You can also use the predefined Checkers that come included with the health package. First, import the checks:

import "github.com/docker/distribution/health/checks

After that you can make use of any of the provided checks. An example of using a FileChecker to take the application out of rotation if a certain file exists can be done as follows:

health.Register("fileChecker", health.PeriodicChecker(checks.FileChecker("/tmp/disable"), time.Second*5))

After registering the check, it is trivial to take an application out of rotation from the console:

# curl localhost:5001/debug/health
{}
# touch /tmp/disable
# curl localhost:5001/debug/health
{"fileChecker":"file exists"}

You could also test the connectivity to a downstream service by using a HTTPChecker, but ensure that you only mark the test unhealthy if there are a minimum of two failures in a row:

health.Register("httpChecker", health.PeriodicThresholdChecker(checks.HTTPChecker("https://www.google.pt"), time.Second*5, 2))

[diogomonica]

cc/ @NathanMcCauley @endophage @stevvooe

[stevvooe]

These string values can just be "ok", "warning" and "error".

[stevvooe]

Let's move this package into distribution/health. I want to avoid having registry-specific functionality in this package.

[stevvooe]

Was this structure going to have a "meta" field, as proposed by #133? Or, did we find that information should be a part of expvar?

[diogomonica]

@stevvooe @NathanMcCauley would love another review.

[dmp42]

cc @icecrime @aluzzardi @mavenugo @endophage because this kind of stuff has a broader interest.

[NathanMcCauley]

nit: s/HTTTP/HTTP/

[icecrime]

Got a few comments, but overall I think it's cool 👍

[stevvooe]

if record.Code != 404 {
// report error
}

[stevvooe]

LGTM!

Nice work on this one.

[noxiouz]

You have to use mutex.Lock & mutex.Unlock as there is a write access 5 lines below.

[diogomonica]

Good find. Fixed.

[noxiouz]

503 =)

[endophage]

LGTM. Might be nice to be able to configure the checkers but that can be future work.

...
health:
  checkers:
    filechecker: [/path/one/, /path/two/]
    httpchecker: [http://foo.com, http://bar.com]
...

[diogomonica]

@endophage white-belt changes?

[endophage]

Sounds good to me

[aluzzardi]

ping @vieux @abronan

Is this something we could use?

[stevvooe]

@aluzzardi Please! Let us know if you need help!