Proposal: health monitors and probes

[Zaba505]

Note

The Summary, Proposed API, design decisions, testing, migration, open questions and out of scope are amended again by revision 2. This proposal is narrowed to health's core: Monitor, Probes, and a health Service that owns readiness and serves no protocol. Exposing the probes over HTTP and gRPC moves to Proposal: exposing health probes, in wave 3, as the charter's revision 3 plans. health has no settings.

Note

The Summary, Proposed API, design decisions, testing, migration and open questions in this post are superseded by the revision below. It applies the decisions of the lifecycle proposal. Health is a Service that owns readiness, instead of being part of the lifecycle, and open questions 1 and 6 are answered. The rest of this post is unchanged and kept for context.

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model. It depends on Proposal: the humus App lifecycle, which owns readiness. Proposal: HTTP server foundation and Proposal: gRPC services use it.

Summary

• Keep health.Monitor and its combinators.
• Give every App two probes, liveness and readiness, owned by the lifecycle. Readiness turns true once the App is serving, and false as soon as shutdown starts. Users add their own monitors to either probe from Init.
• Serve both probes from one model: over HTTP on a port of their own, and over gRPC health v1 on a gRPC server.

Background

• The health package is pure (health/health.go):

  • Monitor is Healthy(ctx) (bool, error);
  • Binary is an atomic flag whose zero value is unhealthy;
  • And stops at the first failure, and Or joins the errors.

  It has no globals and no I/O, and it probably survives as it is.

• HTTP. The README promises /health/liveness and /health/readiness, but rest has served no health endpoints since story(issue-484): rest migrate to bedrock v0210 #485.

• gRPC. grpc.Api.RegisterService registers a monitor under a service's name when the implementation happens to be a health.Monitor (grpc/api.go#L66-L67). internal/grpchealth serves it, with these gaps:

  • No overall status. Nothing is registered under the empty service name "", which Kubernetes' gRPC probe and grpc-health-probe check by default, so the overall check answers NotFound (internal/grpchealth/server.go#L56).
  • The wrong status for errors. A monitor's error maps to SERVICE_UNKNOWN (L70), which the protocol reserves for a Watch on an unknown service.
  • Slow Watch. It polls every 10s (L36).
  • A global logger. It logs through humus.Logger, the global (L34).

• Kubernetes. Readiness should fail as soon as shutdown starts, and load balancers need time to stop routing to the pod. That is the lifecycle's HUMUS_SHUTDOWN_DRAIN_DELAY.

• Consumers and jobs. Queue consumers serve no traffic but still need liveness in Kubernetes. Jobs need neither probe.

Proposed API

package health

type Monitor interface {
	Healthy(ctx context.Context) (bool, error)
}

// MonitorFunc adapts a function to Monitor. New.
type MonitorFunc func(ctx context.Context) (bool, error)

func (f MonitorFunc) Healthy(ctx context.Context) (bool, error)

// Unchanged: Binary, And, Or, AndMonitor, OrMonitor.

// Probes is an App's health as an orchestrator sees it.
type Probes struct {
	Liveness  Monitor
	Readiness Monitor
}

// HTTPHandler serves GET /health/liveness and GET /health/readiness. It
// answers 200 when the probe is healthy and 503 when it is not, and logs a
// monitor's error rather than returning it to the caller.
func HTTPHandler(p Probes, log *slog.Logger) http.Handler

// Service serves HTTPHandler on its own listener. It satisfies
// humus.Service structurally, so health does not import humus.
type Service struct{ /* unexported */ }

func NewService(ln net.Listener, p Probes, log *slog.Logger) *Service
func (s *Service) Serve(ctx context.Context) error
func (s *Service) Shutdown(ctx context.Context) error

// Settings has one setting, HUMUS_HEALTH_ADDR, which defaults to ":8081".
type Settings struct {
	Addr string
}

func ReadSettings(ctx context.Context, env bedrock.Environment) (Settings, error)

The root package adds three methods to Setup, as proposed in Proposal: the humus App lifecycle:

func (s *Setup) Liveness(m health.Monitor)  // adds m to the App's liveness
func (s *Setup) Readiness(m health.Monitor) // adds m to the App's readiness
func (s *Setup) Probes() health.Probes      // the App's probes, for Services that expose them

• Readiness is And(lifecycle, user monitors…). The lifecycle's part is:
  • not ready during Init;
  • ready once every Service's Serve has started;
  • not ready from the moment shutdown starts.
• Liveness is And(user monitors…), and is healthy when there are none.

In use:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	// ... pool, api as in the lifecycle proposal ...
	s.Readiness(health.MonitorFunc(func(ctx context.Context) (bool, error) {
		return pool.Ping(ctx) == nil, nil
	}))

	hs, err := health.ReadSettings(ctx, s.Process.Environment)
	if err != nil {
		return nil, err
	}
	ln, err := s.Host.Listen("tcp", hs.Addr)
	if err != nil {
		return nil, err
	}
	probes := health.NewService(ln, s.Probes(), s.Logger)
	return []humus.Service{api, probes}, nil
}

gRPC health v1

Proposed here, and implemented in Proposal: gRPC services:

Request	Answer
service ""	the App's readiness
a registered service name	that service's monitors, combined with And with the App's readiness
an unknown service name	NotFound for Check, SERVICE_UNKNOWN for Watch
a monitor returns an error	NOT_SERVING, and the error is logged

Design decisions

• The lifecycle owns readiness. No user code is needed to fail readiness when shutdown starts, and none can forget to.
• Probes get their own port. This keeps them off the public API, out of its authentication and out of its OpenAPI document. It works for consumers that serve nothing else, and it matches Kubernetes' usual layout.
• Explicit, not automatic. The health Service is added in Init like any other, so jobs don't open a port they don't need.
• Liveness has no dependency checks by default. Restarting a pod doesn't fix a database that is down, and documentation says so.
• No named checks and no registry. Monitor stays one method. Failures are logged; they are not exposed in the response.
• Monitors honour the request context. The handler adds no timeout of its own, because the probe's timeoutSeconds already bounds the request.

Testing

func TestReadinessFailsDuringTheDrainDelay(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost()
		app := humus.App{
			Name:      "probes",
			Host:      host.Host(),
			Telemetry: telemetry.NoopBuilder,
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				ln, err := s.Host.Listen("tcp", ":8081")
				if err != nil {
					return nil, err
				}
				return []humus.Service{health.NewService(ln, s.Probes(), s.Logger)}, nil
			},
		}

		p, status := humustest.Start(t, app, bedrocktest.WithEnv(map[string]string{
			"HUMUS_SHUTDOWN_DRAIN_DELAY": "5s",
		}))
		synctest.Wait()
		// probe GETs a path on :8081 through host.HTTPClient() and returns the status code.
		require.Equal(t, http.StatusOK, probe(t, host, "/health/readiness"))

		p.Signal(syscall.SIGTERM)
		synctest.Wait() // shutdown has started; the drain delay is running
		require.Equal(t, http.StatusServiceUnavailable, probe(t, host, "/health/readiness"))
		require.Equal(t, http.StatusOK, probe(t, host, "/health/liveness"))

		require.Zero(t, (<-status).Code())
	})
}

The Monitor combinators keep their table tests, and health.Service passes humustest.TestService.

Migration from today

• health.Monitor, Binary, And and Or are unchanged.
• grpc.Api.RegisterService detecting a health.Monitor becomes explicit registration (Proposal: gRPC services).
• internal/grpchealth is replaced, and gets an entry for "".
• The REST server's health endpoints, which the README promises but nothing serves, come back as health.Service on its own port.

Open questions

1. Explicit or automatic? A health Service added in Init, or one that humus.App always runs when HUMUS_HEALTH_ADDR is set?
2. Paths. Keep humus's /health/liveness and /health/readiness, or use Kubernetes' /livez and /readyz?
3. Response bodies. None, or a JSON body that lists the failing monitors? That would need names on monitors.
4. Startup probes. Is Kubernetes' startup probe just readiness during Init, or a probe of its own?
5. Caching. Should expensive monitors, such as a database ping on every probe, be cached?
6. Drain delay default. The lifecycle proposal asks the same question.

Out of scope

• Metrics derived from health checks.
• Health of outbound dependencies beyond what users add as monitors.

References

• health/health.go
• internal/grpchealth/server.go
• gRPC health checking protocol
• Kubernetes: liveness, readiness and startup probes

[Zaba505]

Revision: health is a Service, not part of the lifecycle

This revision applies the decisions in the lifecycle proposal's revision. The lifecycle no longer owns readiness: Setup.Liveness, Setup.Readiness and Setup.Probes are gone, and the root package no longer imports health. Readiness belongs to a health Service, which an App runs like any other Service.

This answers open questions 1 and 6. Each section below replaces the section of the same name in the post. Out of scope and References are unchanged and still apply.

Summary

• Keep health.Monitor and its combinators.
• Health is a Service. A health Service reports liveness and readiness. Readiness turns true once its Serve starts, and false as soon as its Serve context is done, which is when shutdown starts. It keeps answering probes until its own Shutdown. Users pass their own monitors when they build it.
• Explicit at the root, a default in the runtime packages. humus.App runs only the Services that Init returns. A runtime package's App includes a health Service where it fits.
• Serve both probes from one model: over HTTP on a port of their own, and over gRPC health v1 on a gRPC server.

Background

Unchanged, except the Kubernetes bullet, which now reads:

• Kubernetes. Readiness should fail as soon as shutdown starts, and load balancers need time to stop routing to the pod. That is the lifecycle's App.DrainDelay, which a runtime package's App may read from HUMUS_SHUTDOWN_DRAIN_DELAY.

Proposed API

package health

type Monitor interface {
	Healthy(ctx context.Context) (bool, error)
}

// MonitorFunc adapts a function to Monitor. New.
type MonitorFunc func(ctx context.Context) (bool, error)

func (f MonitorFunc) Healthy(ctx context.Context) (bool, error)

// Unchanged: Binary, And, Or, AndMonitor, OrMonitor.

// Probes is an App's health as an orchestrator sees it.
type Probes struct {
	Liveness  Monitor
	Readiness Monitor
}

// HTTPHandler serves GET /health/liveness and GET /health/readiness. It
// answers 200 when the probe is healthy and 503 when it is not, and logs a
// monitor's error rather than returning it to the caller.
func HTTPHandler(p Probes, log *slog.Logger) http.Handler

// Service is a humus.Service that serves an App's probes with HTTPHandler,
// on a listener of its own.
type Service struct{ /* unexported */ }

type Option func(*options)

// Liveness adds m to the Service's liveness probe.
func Liveness(m Monitor) Option

// Readiness adds m to the Service's readiness probe.
func Readiness(m Monitor) Option

// NewService listens on settings.Addr through s.Host, and logs through
// s.Telemetry.
func NewService(s *humus.Setup, settings Settings, opts ...Option) (*Service, error)

// Probes returns the Service's probes, its own readiness included, for
// another server to expose, such as gRPC health v1.
func (s *Service) Probes() Probes

func (s *Service) Serve(ctx context.Context) error
func (s *Service) Shutdown(ctx context.Context) error

// Settings has one setting, HUMUS_HEALTH_ADDR, which defaults to ":8081".
type Settings struct {
	Addr string
}

func ReadSettings(ctx context.Context, env bedrock.Environment) (Settings, error)

• Readiness is And(the Service's own state, readiness monitors…). Its own state is:
  • not ready until its Serve starts;
  • ready while Serve runs;
  • not ready from the moment its Serve context is done, which is when shutdown starts, before the drain delay.
• Liveness is And(liveness monitors…), and is healthy when there are none.

In use:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	// ... pool, api as in the lifecycle proposal ...
	hs, err := health.ReadSettings(ctx, s.Process.Environment)
	if err != nil {
		return nil, err
	}
	probes, err := health.NewService(s, hs, health.Readiness(health.MonitorFunc(func(ctx context.Context) (bool, error) {
		return pool.Ping(ctx) == nil, nil
	})))
	if err != nil {
		return nil, err
	}
	return []humus.Service{api, probes}, nil
}

A runtime package's App, such as REST's, does this for the user by default. Each wave-2 proposal says how its users add monitors, and how they turn the health Service off.

gRPC health v1

Proposed here, and implemented in Proposal: gRPC services:

Request	Answer
service ""	readiness, as the health Service's Probes reports it
a registered service name	that service's monitors, combined with And with readiness
an unknown service name	NotFound for Check, SERVICE_UNKNOWN for Watch
a monitor returns an error	NOT_SERVING, and the error is logged

grpc-go's health server already has Shutdown(), which sets every status to NOT_SERVING. The gRPC proposal decides whether its server uses it, and how the server gets the health Service's Probes.

Design decisions

• The health Service owns readiness, not the lifecycle. Readiness steps in the lifecycle would tie every App, jobs included, to a health endpoint. The lifecycle doesn't need them:
  • any Service returning ends the App, so an App is never partly up;
  • listeners are opened in Init, so connections wait in the backlog until Serve runs, and the health Service can report ready once its own Serve starts;
  • the health Service reports not ready as soon as its Serve context is done, so no user code is needed to fail readiness when shutdown starts, and none can forget to.
• health imports the root package. The root package no longer imports health, so health takes a *humus.Setup like any other runtime package.
• Probes get their own port. This keeps them off the public API, out of its authentication and out of its OpenAPI document. It works for consumers that serve nothing else, and it matches Kubernetes' usual layout.
• Explicit at the root, a default in the runtime packages. humus.App runs only the Services Init returns, so a job opens no port it doesn't need. A runtime package's App includes a health Service where it fits: a REST App includes it unless it's turned off, and a job App leaves it out.
• Liveness has no dependency checks by default. Restarting a pod doesn't fix a database that is down, and documentation says so.
• No named checks and no registry. Monitor stays one method. Failures are logged; they are not exposed in the response.
• Monitors honour the request context. The handler adds no timeout of its own, because the probe's timeoutSeconds already bounds the request.

Testing

func TestReadinessFailsDuringTheDrainDelay(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost(t)
		app := humus.App{
			Host:       host.Host(),
			Telemetry:  telemetry.NoopBuilder,
			DrainDelay: 5 * time.Second,
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				svc, err := health.NewService(s, health.Settings{Addr: ":8081"})
				if err != nil {
					return nil, err
				}
				return []humus.Service{svc}, nil
			},
		}

		p := humustest.Start(t, app)
		synctest.Wait()
		// probe GETs a path on :8081 through host.HTTPClient() and returns the status code.
		require.Equal(t, http.StatusOK, probe(t, host, "/health/readiness"))

		p.Signal(syscall.SIGTERM)
		synctest.Wait() // shutdown has started; the drain delay is running
		require.Equal(t, http.StatusServiceUnavailable, probe(t, host, "/health/readiness"))
		require.Equal(t, http.StatusOK, probe(t, host, "/health/liveness"))

		require.Zero(t, p.Wait().Code())
	})
}

The Monitor combinators keep their table tests, and health.Service passes humustest.TestService.

Migration from today

• health.Monitor, Binary, And and Or are unchanged.
• grpc.Api.RegisterService detecting a health.Monitor becomes explicit registration (Proposal: gRPC services).
• internal/grpchealth is replaced, and gets an entry for "".
• The REST server's health endpoints, which the README promises but nothing serves, come back as health.Service on its own port. The REST package's App includes it by default.

Open questions

Question 1 is answered: explicit at the root, a default in the runtime packages. Question 6 is answered by the lifecycle: the root has no drain-delay default, and each runtime package's App picks its own. The rest are still open:

2. Paths. Keep humus's /health/liveness and /health/readiness, or use Kubernetes' /livez and /readyz?
3. Response bodies. None, or a JSON body that lists the failing monitors? That would need names on monitors.
4. Startup probes. Is Kubernetes' startup probe just readiness during Init, or a probe of its own?
5. Caching. Should expensive monitors, such as a database ping on every probe, be cached?

[Zaba505]

Revision 2: health's core, with exposing it split out

This revision narrows this proposal to health's core: the Monitor model, Probes, and a health Service that owns readiness and serves no protocol.

Exposing the probes is split out. That covers HTTP, either on a server of their own or attached to one an App already runs, and gRPC health v1. It needs the HTTP server foundation and gRPC services proposals, so it moves to a proposal of its own in wave 3, Proposal: exposing health probes, as the charter's revision 3 plans.

It also applies where Proposal: configuration for humus apps landed. The core opens nothing and logs nothing, so health has no Settings, no ReadSettings and no telemetry.SDK.

Each section below replaces the section of the same name, as amended by the first revision. Background and References are unchanged and still apply.

Summary

• Keep health.Monitor and its combinators.
• Health is a Service that owns readiness, and serves no protocol. Readiness turns true once its Serve starts, and false as soon as its Serve context is done, which is when shutdown starts. Users pass their own monitors when they build it.
• Exposing the probes is a wave-3 proposal. Proposal: exposing health probes serves the health Service's Probes over HTTP and gRPC, and decides which runtime packages' Apps expose them by default.
• Explicit at the root. humus.App runs only the Services that Init returns, so a job runs no health Service it doesn't need.

Proposed API

package health

type Monitor interface {
	Healthy(ctx context.Context) (bool, error)
}

// MonitorFunc adapts a function to Monitor. New.
type MonitorFunc func(ctx context.Context) (bool, error)

func (f MonitorFunc) Healthy(ctx context.Context) (bool, error)

// Unchanged: Binary, And, Or, AndMonitor, OrMonitor.

// Probes is an App's health as an orchestrator sees it.
type Probes struct {
	Liveness  Monitor
	Readiness Monitor
}

// Service is a humus.Service that owns an App's readiness. It opens no
// listener and serves no protocol.
type Service struct{ /* unexported */ }

type Option func(*options)

// Liveness adds m to the Service's liveness probe.
func Liveness(m Monitor) Option

// Readiness adds m to the Service's readiness probe.
func Readiness(m Monitor) Option

func NewService(opts ...Option) *Service

// Probes returns the Service's probes, its own readiness included.
func (s *Service) Probes() Probes

// Serve marks the Service ready, and returns nil once ctx is done or
// Shutdown is called, marking it not ready for good.
func (s *Service) Serve(ctx context.Context) error
func (s *Service) Shutdown(ctx context.Context) error

• Readiness is And(the Service's own state, readiness monitors…). Its own state is:
  • not ready until its Serve starts;
  • ready while Serve runs;
  • not ready from the moment its Serve context is done, which is when shutdown starts, before the drain delay.
• Liveness is And(liveness monitors…), and is healthy when there are none.
• The health Service stops on its Serve context, as a queue consumer does. Whatever exposes its Probes keeps answering, not ready, during the drain delay. Proposal: exposing health probes specifies that.

In use, in the end user's Init:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	// ... pool, api as in the lifecycle proposal ...
	hs := health.NewService(health.Readiness(health.MonitorFunc(func(ctx context.Context) (bool, error) {
		return pool.Ping(ctx) == nil, nil
	})))
	// hs.Probes() is what Proposal: exposing health probes serves.
	return []humus.Service{api, hs}, nil
}

Design decisions

The decisions of the first revision stand, except for these.

• The health Service serves no protocol. Serving probes means running an HTTP or gRPC server. Its timeouts, instrumentation, shutdown and per-request panic recovery belong to the protocol proposals, and the charter leaves panic recovery to each protocol. A health Service with a server of its own would either decide these before those proposals do, or decide them differently. Without one, the same Probes can be exposed over HTTP, gRPC, or both.
• Exposing health is its own proposal, in wave 3. Whether HTTP probes get a server of their own, as a job or an event processor pod needs, or attach to the server an API already runs, depends on what the HTTP server foundation provides. So the first revision's exposure decisions move there:
  • probes on a port of their own;
  • a handler that adds no timeout to the request context;
  • failures logged, not returned in the response.
• health is a leaf. It imports no other humus package, and Service satisfies humus.Service structurally rather than importing the root package. Whatever exposes probes imports health, never the reverse.
• No settings and no telemetry. The core opens nothing and logs nothing. A monitor's error reaches whatever exposes the probes, which decides how to log it.
• Forgetting to run the health Service fails loudly. Its readiness is never true until its Serve starts. An App that builds the health Service but doesn't return it from Init never turns ready, and the first deployment or test shows it.

Testing

func TestServiceIsNotReadyDuringTheDrainDelay(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		hs := health.NewService()
		readiness := hs.Probes().Readiness
		app := humus.App{
			Host:                humustest.NewHost(t).Host(),
			Telemetry:           telemetry.NoopBuilder,
			ShutdownGracePeriod: 10 * time.Second,
			DrainDelay:          5 * time.Second,
			Init: func(context.Context, *humus.Setup) ([]humus.Service, error) {
				return []humus.Service{hs}, nil
			},
		}

		p := bedrocktest.NewProcess(t)
		status := make(chan bedrock.ExitStatus, 1)
		go func() { status <- app.Run(t.Context(), p.Process()) }()

		<-p.Notified(syscall.SIGTERM)
		synctest.Wait() // Serve has started
		ready, err := readiness.Healthy(t.Context())
		require.NoError(t, err)
		require.True(t, ready)

		p.Signal(syscall.SIGTERM)
		synctest.Wait() // shutdown has started; the drain delay is running
		ready, err = readiness.Healthy(t.Context())
		require.NoError(t, err)
		require.False(t, ready)

		require.Zero(t, (<-status).Code())
	})
}

• health.Service passes humustest.TestService.
• The Monitor combinators keep their table tests.
• Tests that probe over HTTP or gRPC, including the first revision's GET of /health/readiness during the drain delay, belong to Proposal: exposing health probes.

Migration from today

• health.Monitor, Binary, And and Or are unchanged.
• These move to Proposal: exposing health probes:
  • grpc.Api.RegisterService detecting a health.Monitor;
  • internal/grpchealth;
  • the REST health endpoints that the README promises but nothing serves.

Open questions

The first revision's questions 2 (paths) and 3 (response bodies) are about exposing probes, so they move to Proposal: exposing health probes. Question 3 may also need names on monitors, and that proposal adds them if it chooses to. These are still open here:

4. Startup probes. Is Kubernetes' startup probe just readiness during Init, or a probe of its own, and so a field of Probes?
5. Caching. Should expensive monitors, such as a database ping on every probe, be cached?

Consequences for the other proposals

• Proposal: exposing health probes, new in wave 3. It takes over everything about serving Probes, and the first revision's gRPC health v1 answers are a starting point for it.
• The root package. The Service doc comment in app.go and docs/content/lifecycle.md list a health Service among those that keep serving until Shutdown. The health Service now stops on its Serve context, so this proposal's stories change both to list only the HTTP and gRPC Services.

Out of scope

• Exposing the probes, over HTTP or gRPC: Proposal: exposing health probes.
• Metrics derived from health checks.
• Health of outbound dependencies beyond what users add as monitors.

[Zaba505]

Closing: health's core has landed

Revision 2's stories are done:

• story(health): bring back Monitor and its combinators, with MonitorFunc and Probes #673 brought back Monitor and its combinators, with MonitorFunc and Probes (story(health): bring back Monitor and its combinators, with MonitorFunc and Probes #676);
• story(health): a health Service that owns readiness #674 added the health Service that owns readiness (story(health): a health Service that owns readiness #677);
• story(docs): document health #675 documented health (story(docs): document health #678).

Revision 2's open questions 4 (startup probes) and 5 (caching) move to Proposal: exposing health probes, in wave 3 of the charter, with questions 2 and 3. Both depend on how the probes are served and how often they're checked. If that proposal gives startup probes a probe of their own, its stories add the field to Probes.