Proposal: exposing health probes

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 4. It builds on Proposal: health monitors and probes, Proposal: humus.Main, and the four wave-3 proposals: HTTP server foundation, gRPC services, jobs and queue processing and the Kafka runtime. It depends on no other wave-4 proposal. It answers the questions health monitors and probes handed on: the probe paths, the response bodies, the gRPC health v1 answers, startup probes and caching. It also answers the Kafka proposal's open question 5, whether a consumer whose brokers are unreachable is ready.

Summary

health gives an App its probes, health.Probes, and a health Service that owns readiness, and serves no protocol. This proposal serves those probes to an orchestrator, in two new packages and one new method:

• httphealth, in the core module. NewHandler answers GET /health/liveness and GET /health/readiness from an App's Probes, with 200 or 503 and no detail in the body. NewService serves that handler, and nothing else, over plain HTTP on a port of its own (open question 1), for a pod with no HTTP server of its own, such as a job's or a consumer's. The handler can also be attached to an HTTPS server the App already runs.
• grpc/grpchealth, in the grpc module. NewServer implements gRPC health v1 from the same Probes. The service names "" and readiness answer readiness, and liveness answers liveness, as Kubernetes recommends. NewService serves it in plain text on a port of its own, with nothing else registered, as gRPC services planned. It can also be registered on the end user's grpcserver.Service.
• (*kafka.Service).Joined, a method whose signature is health.MonitorFunc's. A consumer is ready once it has joined its group, so a rollout whose new pods can't reach the brokers stops, rather than replacing pods that consume with pods that don't.
• Probes are explicit. The end user's Init returns a health Service and whatever exposes it, like any other Service. No runtime package's Main exposes probes by default: humus imposes no port, so it has nowhere to serve them.
• No startup probe of its own, and no caching. A probes server answers only once Init has returned, so a startup probe on the liveness path already waits for Init. Each probe request checks its monitors with the request's context, and the probe's own timeout bounds the check.

A consumer's pod, for example:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	settings, probeSettings, err := readSettings(ctx, s) // kafka.Settings and httphealth.Settings
	if err != nil {
		return nil, err
	}
	consumer, err := kafka.NewService(s, settings, queue.ProcessorFunc[kafka.Message](h.Handle))
	if err != nil {
		return nil, err
	}
	hs := health.NewService(health.Readiness(health.MonitorFunc(consumer.Joined)))
	probes, err := httphealth.NewService(s, probeSettings, hs.Probes())
	if err != nil {
		return nil, err
	}
	return []humus.Service{consumer, hs, probes}, nil
}

Background

• What health provides (health/service.go). Probes holds a liveness and a readiness Monitor. health.Service.Probes returns them, with the Service's own state first in readiness: not ready until its Serve starts, and not ready for good once its Serve context is done, which is when shutdown starts, before the drain. health is a leaf that imports nothing but the standard library, and it opens nothing and logs nothing. Whatever exposes the probes logs a monitor's error (health's revision 2).
• What health handed on (closing comment): the paths, the response bodies, startup probes (and a field on Probes if they get a probe of their own), and caching. Its first revision's decisions about exposure moved here too: probes on a port of their own, a handler that adds no timeout, failures logged and not returned, and a gRPC health v1 table as a starting point.
• Before the reset, HTTP probes lived on the API. rest.NewApi served GET /health/liveness and GET /health/readiness on the API's own mux, answering 200 unless the end user replaced them with rest.Liveness and rest.Readiness (rest/api.go#L46-L67 at v0.15.0). story(issue-484): rest migrate to bedrock v0210 #485 removed them, and the README went on promising them.
• Before the reset, gRPC health had gaps (internal/grpchealth/server.go). Nothing answered the empty service name, which Kubernetes' gRPC probe sends by default (L56). A monitor's error answered SERVICE_UNKNOWN, which the protocol reserves for a Watch on an unknown service (L70). Watch polled every 10s (L36), and logging went through the global logger (L34).
• What wave 3 hands on.
  • HTTP server foundation: both HTTP shapes work with httpserver.NewService, given a probe handler. TLS is required, so a probes server built on httpserver needs a certificate. Whether a job's or a consumer's pod serves probes that way, whether any Main exposes probes by default, and whether probe requests get spans are decided here.
  • gRPC services: Kubernetes' gRPC probes connect in plain text unless the GRPCContainerProbeTLS gate is on, and never present a client certificate. So gRPC health v1 goes on a second, plaintext gRPC server, on a port of its own, with only grpc.health.v1.Health registered, built on the grpc module's internal/server. *grpcserver.Service is a grpc.ServiceRegistrar, so health v1 can be registered on it too.
  • Jobs: whether a job's pod exposes probes.
  • Queue processing and the Kafka runtime: how a consumer's pod, which has no HTTP server, serves probes, whether a consumer exposes them by default, and open question 5. franz-go keeps retrying unreachable brokers, and the Service keeps serving.
• What Kubernetes does with a probe (probes, configuring them):
  • an httpGet probe succeeds on a status from 200 to 399, decided by the status alone. The kubelet reads at most 10 KiB of the body, and recommends a minimal one. It speaks HTTP/1.1, and over scheme: HTTPS it doesn't verify the certificate;
  • a grpc probe connects in plain text by default. Its service field picks the service name, and Kubernetes suggests liveness and readiness to answer the two probes on one port;
  • a startup probe holds off liveness and readiness until it first succeeds. Kubernetes' advice is to point it at the liveness check, with a failureThreshold × periodSeconds that covers the slowest start.

Proposed API

package httphealth // github.com/z5labs/humus/httphealth

// Settings configures a Service. Its one field has no default, so the end
// user always sources it.
type Settings struct {
	// Addr is the TCP address the Service listens on through the Host, as
	// host:port. It is required: humus imposes no port. An empty host
	// listens on every name, and port 0 picks a free port, which
	// Service.Addr reports.
	//
	// Default: none. Valid: host:port, with a port from 0 to 65535.
	// Check: CheckAddr.
	Addr string
}

// CheckAddr rejects an address that Host.Listen can't listen on, without
// naming the field.
func CheckAddr(addr string) error

// NewHandler returns a handler that answers an orchestrator's probes from p:
//
//   - GET /health/liveness checks p.Liveness;
//   - GET /health/readiness checks p.Readiness.
//
// HEAD is answered as GET is, any other method gets 405, and any other path
// 404. A probe answers 200 with the body "OK" when its monitor is healthy,
// and 503 with the body "Service Unavailable" when it is unhealthy or
// returns an error. The monitor's error is never in the body: it is logged,
// at ERROR, through s.Telemetry's LoggerProvider.
//
// The monitor is checked with the request's context, and the handler adds
// no timeout: the probe's own timeout bounds the check.
//
// Mount it at "/health/" of a mux, without StripPrefix. It panics if
// p.Liveness or p.Readiness is nil.
func NewHandler(s *humus.Setup, p health.Probes) http.Handler

// Service is a humus.Service that serves NewHandler's handler, and nothing
// else, over plain HTTP/1.1 on a port of its own. It is safe for concurrent
// use.
type Service struct{ /* unexported */ }

// NewService checks settings and p, reporting every problem in one error,
// and listens on settings.Addr through s.Host. It registers the listener's
// close with s.OnStop, so a listener whose Service never serves, because
// Init failed after NewService, is still closed.
func NewService(s *humus.Setup, settings Settings, p health.Probes) (*Service, error)

// Addr returns the address the Service listens on, with the port the Host
// picked when settings.Addr has port 0.
func (svc *Service) Addr() net.Addr

// Serve serves until Shutdown. It keeps serving after ctx is done, so the
// probes are answered, not ready, through the App's drain.
func (svc *Service) Serve(ctx context.Context) error

// Shutdown stops accepting connections and waits for the probes in flight.
// If ctx is done first, it closes every connection and returns an error
// that wraps ctx's error. Calling it again returns nil.
func (svc *Service) Shutdown(ctx context.Context) error

package grpchealth // github.com/z5labs/humus/grpc/grpchealth

// Settings configures a Service. Its one field has no default, so the end
// user always sources it.
type Settings struct {
	// Addr is the TCP address the Service listens on through the Host, as
	// host:port. It is required: humus imposes no port.
	//
	// Default: none. Valid: host:port, with a port from 0 to 65535.
	// Check: CheckAddr.
	Addr string
}

// CheckAddr rejects an address that Host.Listen can't listen on, without
// naming the field.
func CheckAddr(addr string) error

// NewServer returns a gRPC health v1 server that answers from p:
// the service names "" and "readiness" from p.Readiness, and "liveness"
// from p.Liveness. Register it on any grpc.ServiceRegistrar, such as a
// *grpcserver.Service:
//
//	healthpb.RegisterHealthServer(svc, grpchealth.NewServer(s, hs.Probes()))
//
// It panics if p.Liveness or p.Readiness is nil.
func NewServer(s *humus.Setup, p health.Probes) healthpb.HealthServer

// Service is a humus.Service that serves NewServer's server, and nothing
// else, over plaintext gRPC on a port of its own.
type Service struct{ /* unexported */ }

// NewService checks settings and p, reporting every problem in one error,
// listens on settings.Addr through s.Host, and registers NewServer's
// server. It is served and shut down as a grpcserver.Service is, without
// TLS and without instrumentation.
func NewService(s *humus.Setup, settings Settings, p health.Probes) (*Service, error)

func (svc *Service) Addr() net.Addr
func (svc *Service) Serve(ctx context.Context) error
func (svc *Service) Shutdown(ctx context.Context) error

package kafka // github.com/z5labs/humus/queue/kafka

// Joined reports whether the Service has joined its consumer group: false
// until franz-go first hands the member an assignment, empty or not, and
// true from then until Serve returns. A broker outage after that doesn't
// make it false. It never returns an error, and its signature is
// health.MonitorFunc's, so a consumer is made ready once it has joined with
//
//	health.Readiness(health.MonitorFunc(consumer.Joined))
func (s *Service) Joined(ctx context.Context) (bool, error)

httphealth is in the core module, next to httpserver, and imports nothing new. grpchealth is in the grpc module, next to grpcserver: it needs grpc-go's grpc_health_v1, which is in the grpc-go module the grpc module already requires, and the module's internal/server. health itself doesn't change and stays a leaf.

Wiring the probes

The end user's Init builds a health Service, hands its Probes to whatever exposes them, and returns them all. Which exposure fits depends on what else the pod serves:

The pod	Its main	The probes	Kubernetes
a consumer, or a long-running job	humus.Main	httphealth.NewService on a port of its own	httpGet on that port
an HTTPS API	httpserver.Main	httphealth.NewHandler attached to the API's mux, or httphealth.NewService (open question 2)	httpGet, scheme: HTTPS on the API's port, or scheme: HTTP on the probes port
a gRPC service	grpcserver.Main	grpchealth.NewService on a plaintext port of its own, or grpchealth.NewServer registered on the end user's Service where probes can use TLS (open question 2)	grpc with service: liveness or readiness
a Kubernetes Job's pod	humus.Main	none	none

A probes server adds no drain: nothing routes traffic to it. So a consumer's or a job's pod with a probes server stays on humus.Main, as revision 6 of the charter has it.

An API with the handler attached:

hs := health.NewService(health.Readiness(health.MonitorFunc(func(ctx context.Context) (bool, error) {
	return pool.Ping(ctx) == nil, nil
})))
mux := http.NewServeMux()
mux.Handle("/", api)
mux.Handle("/health/", httphealth.NewHandler(s, hs.Probes()))
svc, err := httpserver.NewService(s, settings, mux)
if err != nil {
	return nil, err
}
return []humus.Service{svc, hs}, nil

A gRPC service with its health server on a plaintext port:

svc, err := grpcserver.NewService(s, settings)
if err != nil {
	return nil, err
}
cartpb.RegisterCartServiceServer(svc, carts)
hs := health.NewService()
probes, err := grpchealth.NewService(s, probeSettings, hs.Probes())
if err != nil {
	return nil, err
}
return []humus.Service{svc, hs, probes}, nil

livenessProbe:
  grpc:
    port: 8081 # the health server's port; the end user's services are served over TLS on their own
    service: liveness
readinessProbe:
  grpc:
    port: 8081
    service: readiness

One health Service's Probes can be handed to both packages, to answer HTTP and gRPC probes from one state.

HTTP: paths, bodies and status codes

Request	Healthy	Unhealthy, or the monitor returned an error
GET or HEAD /health/liveness	200, OK	503, Service Unavailable
GET or HEAD /health/readiness	200, OK	503, Service Unavailable
another method on either path	405	405
any other path	404	404

• The paths are humus's from before the reset, which the README still names. No path is a standard for workloads: /livez and /readyz are what Kubernetes' own components serve, and other frameworks use their own.
• The body is the status text and a newline, with Content-Type: text/plain; charset=utf-8 and Cache-Control: no-store. The kubelet decides on the status code alone, and a person with curl sees which it was. A monitor's error is never in the body: it can name a host or a credential's user, and a probe port is reachable by anything that can reach the pod.
• No per-monitor detail. Monitor stays one method with no name, as health decided. The failing monitor's error is in the log record.

The handler routes on the full path, with an http.ServeMux of its own, so it works mounted at "/health/" of the end user's mux, and serves 404 for anything else under it.

The probes servers

httphealth.NewService and grpchealth.NewService serve plain text (open question 1). Each answers its probes and nothing else, so nothing of the end user's is reachable on that port, with or without TLS on their own ports.

• They serve through the drain. A probes server keeps serving until Shutdown, like an HTTP or gRPC Service. During an API's drain, a readiness probe is answered 503, not refused, and a liveness probe still 200.
• HTTP is served by code shared with httpserver. The listener, Serve, Shutdown with its Close when the grace period runs out, and net/http's error log written to telemetry move from httpserver into an internal package of the core module, which knows nothing of TLS. httpserver.NewService is that code with TLS required, HTTP/2 and its instrumentation, and httphealth.NewService is that code over plain HTTP/1.1, with the probe handler alone. This is the shape the grpc module already has, and httpserver serves exactly as before.
• gRPC is served by the grpc module's internal/server. That package already listens through the Host, recovers panics, ends streams at Shutdown with Unavailable, and calls GracefulStop and then Stop. grpchealth.NewService passes it no credentials and no stats handler.
• Fixed limits. The HTTP probes server reads a request's headers within 2s and the whole request within 5s, the httpserver defaults, and sets no write timeout, since the probe's own timeout bounds a check. Neither server has a setting besides Addr. A field is added when an end user needs one, which breaks no one.
• Plain HTTP/1.1 only. The kubelet speaks HTTP/1.1 unless the H2CContainerProbe gate is on and the probe asks for h2c, and humus serves no h2c.

grpcserver, httpserver and their Mains are unchanged. Neither Main runs a health Service or serves probes, as their docs already say.

gRPC health v1: the answers

Request	Check	Watch
service ""	readiness	readiness
service readiness	readiness	readiness
service liveness	liveness	liveness
any other service	NotFound	one SERVICE_UNKNOWN, and the stream stays open
the monitor is healthy	SERVING	SERVING
it is unhealthy, or returns an error	NOT_SERVING; an error is logged	NOT_SERVING; an error is logged once

List answers the three names with their current status.

• "" is readiness. It is the overall status, which a grpc probe without service, grpc-health-probe and grpc-go's client-side health checking all ask for.
• liveness and readiness are Kubernetes' names. Its docs suggest them for answering both probes on one port.
• No status per gRPC service. health has no per-service monitors. A registered service's name, such as cart.v1.CartService, answers NotFound, as any unknown name does.
• An error is NOT_SERVING, not SERVICE_UNKNOWN, which the protocol keeps for a Watch on an unknown name.
• The server is humus's own, not grpc-go's health.Server. grpc-go's holds statuses that the application sets. humus's checks the monitors when it is asked, so it needs nothing to keep a status in step.

Watch polls, once a second, once per server and name. Monitors are checked when asked and send no changes, so a Watch checks the monitor every second and sends the status when it changes, starting with the current one. Every Watch on one server for one name shares one poller, which runs while the name has watchers. So a hundred clients watching "" cost one check a second, not a hundred. Each poll checks with a context that is done after a second, and a check that runs out of time is NOT_SERVING. A failing monitor is logged when the status changes, not once a second.

Watch is what grpc-go's client-side health checking uses, against a health server registered on the end user's Service. With a one-second poll, a client learns that a pod is draining within a second of shutdown starting, well within grpcserver.Main's 5s drain, and moves its RPCs to other pods before the GOAWAY. At Shutdown, internal/server cancels every stream, and a Watch ends with Unavailable, as every stream does. The interval is fixed. A setting is added if an end user needs another.

Startup probes

There is no startup probe of its own, and Probes gets no field. A probes server answers only once Serve runs, and the App calls Serve only after Init has returned. While Init runs, a probe's connection waits in the listener's backlog, or is refused if the listener isn't open yet, and the probe times out or fails. So a startup probe on the liveness path succeeds exactly when Init is done, which is Kubernetes' own advice:

startupProbe:
  httpGet:
    path: /health/liveness
    port: probes
  failureThreshold: 30 # 30 × 2s: Init has a minute
  periodSeconds: 2

A service whose Init is slow, such as one that waits for a database or runs migrations, sets a startup probe, so liveness doesn't restart it before it has started. One that starts fast needs none.

Checks: timeouts and caching

• No timeout of the handler's own. Each monitor is checked with the request's context. The kubelet's timeoutSeconds closes the connection, which cancels an HTTP request's context, and sets a gRPC call's deadline. A monitor honours the context it is given, as health's docs already ask.
• No caching. Each probe request checks its monitor. The kubelet checks each probe every periodSeconds, 10s by default, so a database ping on readiness costs one ping per pod every 10 seconds. A monitor that is too expensive to check that often caches its own result, in the end user's MonitorFunc. A cache in humus would need a setting for its age, and would answer a probe with a result older than the probe asked for. The one shared check is Watch's poller, above, because a Watch has no requester to time it.
• A probe port is not for the public. The probes servers answer anyone who can reach the port, and every request runs a check. The port is the container's, for the kubelet, and isn't routed by a Kubernetes Service or an ingress.

Telemetry

• Logs. Each package logs through s.Telemetry.LoggerProvider, under its import path, github.com/z5labs/humus/httphealth and github.com/z5labs/humus/grpc/grpchealth. A monitor's error is one ERROR record, in the request's context, whose body is the error and whose attribute health.probe is liveness or readiness. An unhealthy monitor that returns no error, such as the health Service's own state during the drain, logs nothing. The probes servers log listening at INFO when they start, and net/http's error-log lines at ERROR, as httpserver does. A recovered gRPC panic is logged under grpchealth's scope, so internal/server takes the scope from the package that builds it.
• No spans and no metrics on the probes servers. A probe every 10 seconds per pod per probe would be most of a quiet service's traces, and says nothing a probe failure event doesn't. So the probes servers have no otelhttp and no otelgrpc, and extract no trace context.
• Attached probes are traced like any request. The handler mounted on an httpserver.Service, or the server registered on a grpcserver.Service, sits inside that Service's instrumentation, which records every request. Leaving probes out would need a filter on httpserver and grpcserver that knows the probe paths. A team that doesn't want the spans serves the probes on their own port, or drops them in its sampler.
• A monitor panics: on a probes server, net/http recovers it, aborts the response and logs the stack at ERROR, through the error log, and internal/server fails the RPC with Internal. The kubelet sees a failed probe.

A consumer's readiness

The Kafka proposal's open question 5 asked whether a consumer whose brokers are unreachable is ready. Its answer here is: not until it has joined its group; after that, an outage doesn't make it unready.

Nothing routes traffic to a consumer, so its readiness does one thing: it gates rollouts and disruption budgets. A Deployment's rolling update replaces old pods only as new pods turn ready. Without Joined, a release whose brokers are misconfigured, such as a wrong address, rolls out ready pods that never consume, while franz-go keeps retrying, and the old pods that did consume are gone. With Joined in readiness, the new pods never turn ready, and the rollout stops at its progress deadline with the old pods still consuming.

After the first join, a broker outage hits every pod at once. Turning them all unready would change nothing for traffic, and would block a rollout that might be the fix. So Joined stays true until Serve returns.

franz-go calls OnPartitionsAssigned after every join, "even if nothing new is assigned", so a member with no partitions, in a group with more replicas than partitions, is joined too. The Service records the first call in the hook it already has (service.go#L491). Liveness never checks the brokers: restarting the pod doesn't bring them back.

Jobs

A Kubernetes Job's pod usually has no probes: nothing routes to it, and a job that hangs is ended by its activeDeadlineSeconds, when it sets one. So job gets nothing new. A long-running job that wants a liveness probe returns httphealth.NewService's Service and a health Service next to its job.Service, and stays on humus.Main. When the job finishes, the App ends, and the probes server with it.

Changes to landed packages and pages

This proposal's stories make these, as the proposal that owns health's exposure:

• httpserver: serving and shutdown move into an internal package of the core module, as above. Its API and behaviour don't change.
• grpc/internal/server: New takes the logger scope of the package building it.
• queue/kafka: the Joined method. The doc's "Readiness is not decided here" is replaced by how a consumer turns ready, with an example.
• Package docs: health's doc ("Exposing probes is Proposal: exposing health probes") points to the two packages. The root package's "Being rebuilt" names them.
• The docs site: health.md's "Exposing the probes" section is replaced by how to expose them, with the wiring table, the paths and answers, startup probes and the manifests. grpc.md's "Health probes", kafka.md's readiness paragraph, and http.md and jobs.md link to it. The README's /health/liveness and /health/readiness are served again.
• The one-server guidance. grpc.md already says a probe with mode: TLS reaches the end user's Service on a cluster with GRPCContainerProbeTLS on, unless the Service sets ClientAuth. The new section recommends one server, with grpchealth.NewServer registered on the end user's Service, wherever the cluster's probes can use TLS, and the plaintext port where they can't.

No lint rule, module or .claude/backlog.json change is needed: both packages are in modules that already exist.

What waves 5 and 6 build on this

• Failure injection (wave 5) needs nothing here. The probes servers extract no trace context or baggage, so no scenario reaches a monitor through them. An attached handler is behind its Service's stripping, like every other request.
• Authentication and authorization (wave 5): a probes server has no authentication, and needs none, since it answers nothing but the probes. An attached handler sits behind whatever the end user's mux or interceptors put in front of it, and the kubelet presents no credentials. That proposal says how a probe path is left open.
• Walkthroughs and a humus MIGRATING.md (wave 6) show probes in each production-ready service, and the move from rest.Liveness and rest.Readiness.

Design decisions

• Two packages, one per protocol, named after it. httphealth sits next to httpserver, and grpchealth next to grpcserver in the grpc module, so a pod that serves only HTTP probes doesn't pull in grpc-go. health stays a standard-library leaf, as it decided.
• Probes are explicit Services. The end user's Init returns the health Service and its exposure, as it returns every other Service. An Option that wraps Init and adds them would have to source an address, which humus leaves to the end user, and would build a health Service the end user's monitors can't reach. A runtime Main that exposed probes by default would need a default port, and both server proposals decided that humus imposes none (open question 3).
• Probes, not *health.Service. Both packages take health.Probes, so an end user can expose probes built some other way. The health Service's Probes is what the docs show, because its readiness follows the lifecycle.
• A probes server serves the probes and nothing else. It is what keeps a plaintext port harmless, and it is what Proposal: gRPC services #681 already decided for gRPC.
• A probes server keeps serving through the drain. A readiness probe during the drain is answered not ready, rather than refused.
• The paths are kept from before the reset. See HTTP above.
• Status codes, not bodies. The kubelet decides on the code, and recommends a minimal body. Errors go to the log, where they can be read, and not to the network.
• No startup probe of its own. The probes server's start already marks the end of Init. A third probe would mean a third monitor whose meaning is "Init is done", which the lifecycle already enforces.
• No caching and no handler timeout. The prober's period and timeout already bound both.
• Watch shares one poll per name. Without it, each watching client multiplies the cost of every monitor.
• No spans for probes servers. See Telemetry above.
• A consumer is ready once joined. See A consumer's readiness above. Joined is a method, not a type, because a method value already is a MonitorFunc, and it is API because the end user has no other way to learn that the member joined.

Testing

The HTTP sketches below were prototyped in a scratch module against this repository's code: the plaintext probes Service and the handler, on a humustest Host, through a 5s drain, mounted under /health/, with HEAD and a 405, passed under -race -count=3. The gRPC answers, List, and the shared Watch poller were prototyped on a plain *grpc.Server on a humustest Host, not on internal/server, and passed the same way. Joined was not prototyped. Every test runs inside testing/synctest and asserts with testify/require.

Both Services pass the contract. They keep serving after Serve's context is done, so TestService's subtest for Shutdown after Serve has returned on its own reports SKIP:

func TestService(t *testing.T) {
	humustest.TestService(t, func(t *testing.T, s *humus.Setup) humus.Service {
		svc, err := httphealth.NewService(s, httphealth.Settings{Addr: ":8081"}, health.NewService().Probes())
		require.NoError(t, err)
		return svc
	})
}

Readiness through an API's drain, on a pod with a probes port:

func TestInit_is_not_ready_during_the_drain(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		p, status := humustest.Start(t, apiInit, // returns an API, a health Service and httphealth's Service on :8081
			humustest.WithApp(humus.WithHost(h.Host()), httpserver.Defaults()),
		)
		synctest.Wait()

		c := h.HTTPClient()
		require.Equal(t, http.StatusOK, probe(t, c, "http://localhost:8081/health/readiness"))

		p.Signal(syscall.SIGTERM)
		time.Sleep(time.Second) // inside the 5s drain
		require.Equal(t, http.StatusServiceUnavailable, probe(t, c, "http://localhost:8081/health/readiness"))
		require.Equal(t, http.StatusOK, probe(t, c, "http://localhost:8081/health/liveness"))

		require.Zero(t, (<-status).Code())
	})
}

probe is the test's helper: it makes the request, reads and closes the body, and returns the status code.

gRPC health v1 on a plaintext port, with the client grpcservertest already provides:

conn := grpcservertest.NewClient(t, h, "localhost:8081", nil,
	grpc.WithTransportCredentials(insecure.NewCredentials()),
)
client := healthpb.NewHealthClient(conn)

resp, err := client.Check(t.Context(), &healthpb.HealthCheckRequest{Service: "liveness"})
require.NoError(t, err)
require.Equal(t, healthpb.HealthCheckResponse_SERVING, resp.GetStatus())

_, err = client.Check(t.Context(), &healthpb.HealthCheckRequest{Service: "cart.v1.CartService"})
require.Equal(t, codes.NotFound, status.Code(err))

The tests also show the following:

• Every answer in the two tables, for a healthy monitor, an unhealthy one and one that returns an error, with the body checked byte for byte and never holding the error.
• A monitor's error is logged once per check. With humustest's Telemetry, one ERROR record with the error as its body, health.probe set, and the request's context. An unhealthy monitor without an error logs nothing, and nothing reaches p.Stderr().
• The attached handler. Mounted at /health/ on an httpserver.Service with a certificate from humustest.NewCA, through httpservertest.NewClient, it answers both probes, and other paths reach the end user's handler.
• No answer before Init returns. An Init that sleeps 30s after building the probes Service gets no answer to a probe with a 1s timeout until the 30s are up, and 200 after.
• Watch. It sends SERVING at once, NOT_SERVING within a second of SIGTERM, and ends with Unavailable at Shutdown. Two watchers of "" cause one check a second, counted by a monitor that counts its calls. An unknown name gets one SERVICE_UNKNOWN and nothing else.
• List returns the three names.
• No spans. With humustest's Telemetry, a probes server records no span and no metric.
• Joined. On kfake listening on the test Host, Joined is false before Serve, true once the group is joined, including for a member assigned no partition, and false after Serve returns. With no broker listening, it stays false while franz-go retries.
• The settings convention. CheckAddr rejects "" and a port out of range without naming the field. NewService names each bad field, and rejects a Probes with a nil monitor.
• No globals. With OTel's global providers set to recorders, nothing is recorded in them.

Each package has an Example, and both are held to the 80% coverage gate.

Migration from today

Before the reset	With this proposal
GET /health/liveness and /health/readiness on every rest.Api, 200 by default	httphealth.NewHandler attached to the API's mux, or httphealth.NewService on a port of its own, answering from a health Service
rest.Liveness(h), rest.Readiness(h) with a handler of the end user's	health.Liveness(m), health.Readiness(m) with a Monitor, when building the health Service
grpc.Api.RegisterService health-checking an implementation that is a health.Monitor	nothing implicit: the monitor goes in the health Service's readiness
internal/grpchealth	grpchealth.NewServer, or grpchealth.NewService on a plaintext port
no answer for the service ""	"" is readiness, and liveness and readiness are named
a monitor's error as SERVICE_UNKNOWN	NOT_SERVING, and the error logged
Watch polling every 10s, per stream	every second, once per server and name
the global humus.Logger	s.Telemetry.LoggerProvider

Open questions

1. Plain text or TLS for a probes-only HTTP server? Proposed: httphealth.NewService serves plain HTTP, on code shared with httpserver, and answers nothing but the probes. That is the gRPC decision of Proposal: gRPC services #681. The kubelet verifies no certificate over HTTPS, so TLS on a probe port proves nothing to it, and a consumer's or a job's pod would need a certificate for probes alone. The alternative keeps "TLS is required" for every HTTP port humus serves: httphealth would have only NewHandler, and a probes server would be httpserver.NewService with a certificate. Recommendation: plain text, probes only.
2. Where an HTTPS or gRPC service's probes go. Attached to the service's own server, one port and one certificate, or on a port of their own? Attached, they reach whatever fronts that port, such as an ingress, they pass through the end user's authentication, and every probe is a span. On their own port, the pod has a second port. Kubernetes' httpGet probes can use HTTPS on every cluster, and its grpc probes only with GRPCContainerProbeTLS. Recommendation: the docs recommend one server wherever the kubelet's probes can use TLS (HTTPS always, gRPC where the gate is on), and a probes port otherwise, which is also the only choice for a consumer's or a job's pod.
3. Should any runtime Main expose probes by default? Health's first revision proposed it for REST. Since then, httpserver and grpcserver require their address, and a default would need a probe port, which humus doesn't impose, and a health Service the end user can't add monitors to. Recommendation: no. Every Init that wants probes returns them, as the examples show.
4. A consumer's readiness. Add (*kafka.Service).Joined, true from the first join until Serve returns, as proposed? The alternatives are no monitor, so a consumer is ready as soon as it serves, or a monitor that also turns false while the member is out of its group. Recommendation: Joined as proposed. The first gates a rollout on a working configuration, and the third makes every pod unready during a broker outage, which only blocks rollouts.

Out of scope

• Metrics derived from probes, and probe history.
• Named monitors, and a body that lists them: health keeps Monitor one method.
• /livez, /readyz, ?verbose and other paths.
• Serving probes over h2c, or over a Unix socket.
• A metrics endpoint on the probes port. Telemetry is exported by the SDK.
• A readiness monitor for a RabbitMQ consumer: Proposal: RabbitMQ (wave 4) proposes one in its own package, Consuming, with the semantics Joined has here.
• Per-service gRPC health statuses.
• Health of outbound dependencies beyond the monitors the end user adds. The wave-4 client proposals keep their breakers out of health, as the charter's revision 7 has it.

References

• Proposal: humus on the bedrock process model, revisions 6 and 7
• Proposal: health monitors and probes: its first revision, revision 2 and closing comment
• Proposal: humus.Main
• HTTP server foundation, revision 1, "What waves 4 and 5 build on this"
• gRPC services, revision, "What waves 4 and 5 build on this"
• Jobs, revision
• Queue processing and the Kafka runtime, revision, "What waves 4 and 5 build on this", and its open question 5
• health/service.go, httpserver/service.go, grpc/internal/server/server.go, queue/kafka/service.go
• Before the reset: rest/api.go at v0.15.0, internal/grpchealth/server.go
• gRPC health checking protocol, and grpc-go v1.84.0's grpc_health_v1
• Kubernetes: liveness, readiness and startup probes, and configuring them, including TLS with gRPC probes
• franz-go v1.22.2's OnPartitionsAssigned

[Zaba505]

Revision: a probes server owns its readiness

The first post put a health Service next to every probes server. A consumer's pod returned {consumer, hs, probes}, two Services for one concern. This revision folds the health Service into the probes servers. httphealth.NewService and grpchealth.NewService take the monitors as health.Options and own readiness as a health Service does, so a pod with a probes port returns one Service for its probes. Attached probes are unchanged: NewHandler and NewServer take Probes, and the end user returns a health Service next to the server they're attached to.

The first post's shape had one wiring for every pod, and two costs:

• A second Service for one concern, on every pod with a probes port.
• A pod that never turns ready, silently. A health Service that is built and handed to a probes server, but not returned from Init, is never served. Its readiness stays false, every readiness probe is answered 503, an unhealthy state with no error logs nothing, and the rollout stops at its progress deadline with nothing in the log saying why.

The lifecycles don't get in the way. A health Service stops on its Serve context, and a probes server keeps serving until Shutdown. One Service does both: it turns not ready when its context is done, and answers through the drain.

Open questions 1 to 4 are unchanged, and still open. Under question 2, attaching the probes now also means building and returning a health Service, and a probes port doesn't.

Proposal: RabbitMQ needs nothing: health.Readiness(health.MonitorFunc(consumer.Consuming)), as its Consuming doc shows it, is the Option httphealth.NewService takes, and a probes server's readiness is a health Service's.

Each section below replaces the part of the same name in the post. Background, the HTTP and gRPC answer tables, Startup probes, Checks, Telemetry, A consumer's readiness, What waves 5 and 6 build on this, Open questions, Out of scope and References are unchanged.

Summary

The first, second and fourth bullets become:

• httphealth, in the core module. NewHandler answers GET /health/liveness and GET /health/readiness from an App's Probes, with 200 or 503 and no detail in the body. NewService serves that handler, and nothing else, over plain HTTP on a port of its own (open question 1), for a pod with no HTTP server of its own, such as a job's or a consumer's. It owns readiness as a health Service does, from the monitors the end user passes as health.Options, so the pod returns no health Service next to it. The handler can also be attached to an HTTPS server the App already runs, next to a health Service.
• grpc/grpchealth, in the grpc module. NewServer implements gRPC health v1 from the same Probes. The service names "" and readiness answer readiness, and liveness answers liveness, as Kubernetes recommends. NewService serves it in plain text on a port of its own, with nothing else registered, as gRPC services planned, and owns readiness as httphealth.NewService does. NewServer can also be registered on the end user's grpcserver.Service, next to a health Service.
• Probes are explicit. The end user's Init returns a probes server, or a health Service and the server its probes are attached to, like any other Service. No runtime package's Main exposes probes by default: humus imposes no port, so it has nowhere to serve them.

The consumer's pod becomes:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	settings, probeSettings, err := readSettings(ctx, s) // kafka.Settings and httphealth.Settings
	if err != nil {
		return nil, err
	}
	consumer, err := kafka.NewService(s, settings, queue.ProcessorFunc[kafka.Message](h.Handle))
	if err != nil {
		return nil, err
	}
	probes, err := httphealth.NewService(s, probeSettings, health.Readiness(health.MonitorFunc(consumer.Joined)))
	if err != nil {
		return nil, err
	}
	return []humus.Service{consumer, probes}, nil
}

Proposed API

In httphealth, Service, NewService, Serve and Shutdown become the following. Settings, CheckAddr, NewHandler and Addr are unchanged.

// Service is a humus.Service that serves NewHandler's handler, and nothing
// else, over plain HTTP/1.1 on a port of its own. It owns the App's
// readiness as a health.Service does, so the end user's Init returns no
// health Service next to it. It is safe for concurrent use.
type Service struct{ /* unexported */ }

// NewService checks settings and opts, reporting every problem in one
// error, and listens on settings.Addr through s.Host. It registers the
// listener's close with s.OnStop, so a listener whose Service never serves,
// because Init failed after NewService, is still closed.
//
// The Service answers from the probes of a health.Service built from opts:
//
//   - readiness is false until Serve starts, false for good once Serve's
//     context is done or Shutdown is called, and otherwise checks the
//     monitors health.Readiness adds, in the order they were added;
//   - liveness checks the monitors health.Liveness adds, in the order they
//     were added, and is healthy when there are none.
//
// NewService rejects a nil Option.
func NewService(s *humus.Setup, settings Settings, opts ...health.Option) (*Service, error)

// Serve marks the Service ready, and serves until Shutdown. It keeps
// serving after ctx is done, so the probes are answered, not ready, through
// the App's drain.
func (svc *Service) Serve(ctx context.Context) error

// Shutdown marks the Service not ready for good, stops accepting
// connections and waits for the probes in flight. If ctx is done first, it
// closes every connection and returns an error that wraps ctx's error.
// Calling it again returns nil.
func (svc *Service) Shutdown(ctx context.Context) error

In grpchealth, Service and NewService become the following. Settings, CheckAddr, NewServer, Addr, Serve and Shutdown are unchanged.

// Service is a humus.Service that serves NewServer's server, and nothing
// else, over plaintext gRPC on a port of its own. It owns the App's
// readiness as a health.Service does, so the end user's Init returns no
// health Service next to it.
type Service struct{ /* unexported */ }

// NewService checks settings and opts, reporting every problem in one
// error, listens on settings.Addr through s.Host, and registers NewServer's
// server, answering from the probes of a health.Service built from opts,
// as httphealth.NewService does. It is served and shut down as a
// grpcserver.Service is, without TLS and without instrumentation.
// NewService rejects a nil Option.
func NewService(s *humus.Setup, settings Settings, opts ...health.Option) (*Service, error)

health doesn't change, and stays a leaf. Both packages already import it.

Wiring the probes

The paragraph before the table becomes:

The end user's Init returns a probes server, which owns readiness, or a health Service whose Probes it hands to a handler or server attached to one the App already runs. Which fits depends on what else the pod serves:

The pod	Its main	The probes	Kubernetes
a consumer, or a long-running job	humus.Main	httphealth.NewService on a port of its own	httpGet on that port
an HTTPS API	httpserver.Main	a health Service, and httphealth.NewHandler attached to the API's mux; or httphealth.NewService (open question 2)	httpGet, scheme: HTTPS on the API's port, or scheme: HTTP on the probes port
a gRPC service	grpcserver.Main	grpchealth.NewService on a plaintext port of its own; or a health Service, and grpchealth.NewServer registered on the end user's Service where probes can use TLS (open question 2)	grpc with service: liveness or readiness
a Kubernetes Job's pod	humus.Main	none	none

The API with the handler attached is unchanged. The gRPC service with its health server on a plaintext port becomes:

svc, err := grpcserver.NewService(s, settings)
if err != nil {
	return nil, err
}
cartpb.RegisterCartServiceServer(svc, carts)
probes, err := grpchealth.NewService(s, probeSettings)
if err != nil {
	return nil, err
}
return []humus.Service{svc, probes}, nil

"One health Service's Probes can be handed to both packages, to answer HTTP and gRPC probes from one state" becomes: attached, one health Service's Probes can go to NewHandler and NewServer both, to answer HTTP and gRPC probes from one state.

The probes servers

A bullet is added before "They serve through the drain":

• They own readiness. Each builds a health Service from the Options it is given, and runs it for as long as it serves: ready once its Serve starts, and not ready for good once its Serve context is done, which is when shutdown starts, before the drain. A pod that serves probes in two ways, such as an HTTP probes port and NewServer registered on its gRPC Service for client-side health checking, gives each the same monitors. Both follow the one App's lifecycle, so they agree.

Jobs

The last two sentences become: A long-running job that wants a liveness probe returns httphealth.NewService's Service next to its job.Service, and stays on humus.Main. When the job finishes, the App ends, and the probes server with it.

Changes to landed packages and pages

The bullets on package docs and the docs site become:

• Package docs: health's doc ("Exposing probes is Proposal: exposing health probes") points to the two packages. Its section on the health Service says that a probes server owns readiness itself, so a health Service is built for probes attached to a server the App already runs. The root package's "Being rebuilt" names the two packages.
• The docs site: health.md's "Exposing the probes" section is replaced by how to expose them, with the wiring table, the paths and answers, startup probes and the manifests. Its examples return one probes server for a consumer's pod, and a health Service next to an API whose mux has the handler. grpc.md's "Health probes", kafka.md's readiness paragraph, and http.md and jobs.md link to it. The README's /health/liveness and /health/readiness are served again.

Design decisions

The bullet "Probes are explicit Services" becomes:

• Probes are explicit Services. The end user's Init returns a probes server, or a health Service next to the server its probes are attached to, as it returns every other Service. An Option that wraps Init and adds them would have to source an address, which humus leaves to the end user, and couldn't take the end user's monitors. A runtime Main that exposed probes by default would need a default port, and both server proposals decided that humus imposes none (open question 3).

The bullet "Probes, not *health.Service" becomes:

• A probes server owns readiness, and attached probes take Probes. A probes server runs for the App's whole life, and is the only exposure on its port, so it builds its health Service from the end user's Options and runs it inside its own Serve. A pod with a probes port returns one Service for its probes, and has no health Service to forget to return. An attached handler or server has no Service of its own to run one in, so it takes Probes, and the end user returns the health Service. Monitors built some other way still reach a probes server, through health.Liveness and health.Readiness.
• Rejected: a health Service next to every probes server, the first post's shape. It wires every pod the same way, but returns two Services for one concern on every pod with a probes port. A health Service that is built but not returned is never served, so the pod is never ready, and nothing logs why.

Testing

This revision's change was not prototyped. The first post's prototypes ran a probes server next to a health Service, and the probes server now runs that health Service itself. The health Service's lifecycle is landed and tested.

TestService's constructor becomes:

svc, err := httphealth.NewService(s, httphealth.Settings{Addr: ":8081"})

In the drain test, apiInit returns an API and httphealth's Service on :8081, and no health Service.

The bullet on the settings convention becomes, and a bullet is added after it:

• The settings convention. CheckAddr rejects "" and a port out of range without naming the field. NewService names each bad field, and rejects a nil Option.
• A probes server's readiness, for both packages. With no Options, readiness is 200 while it serves and 503 from the moment Serve's context is done, through the drain, and liveness is 200 throughout. A monitor added with health.Readiness is checked after the Service's own state, and one added with health.Liveness on liveness.

Migration from today

The rest.Liveness row becomes:

Before the reset	With this proposal
rest.Liveness(h), rest.Readiness(h) with a handler of the end user's	health.Liveness(m), health.Readiness(m) with a Monitor, given to the probes server, or to the health Service when the probes are attached

[Zaba505]

Revision 2: the open questions answered

The open questions:

1. TLS is required on every HTTP port, the probes port included. httphealth.Settings gains TLS, with no default, and CheckTLS. The probes server serves HTTP/1.1 over TLS, and a probe on it sets scheme: HTTPS. The kubelet doesn't verify the certificate, so the end user's own certificate serves. httphealth.NewService stays, rather than probes on httpserver.NewService with a certificate: it owns readiness, as revision 1 has it, and records no spans. The gRPC probes port stays plaintext for now. The kubelet's grpc probes use TLS only with the GRPCContainerProbeTLS gate, which is alpha and off by default. TLS is made required there once Kubernetes' grpc probes use it without a gate (see Out of scope).
2. The probes are attached to a server the pod already runs, rather than given another port, as recommended. A probes port is for:
   • a pod with no server, such as a consumer's or a job's;
   • a server that requires client certificates, which the kubelet never presents;
   • a gRPC service on a cluster whose grpc probes can't use TLS.
3. No runtime Main exposes probes by default, as recommended. It goes out of scope, and is reconsidered only when end users report needing it.
4. Joined is added, and returns a health.Monitor. health.Readiness(consumer.Joined()) needs no conversion, and the type says what the method is for. queue/kafka imports health, which imports nothing but the standard library. Joined is for readiness only. A liveness monitor for a member that has lost its group would restart every pod in a broker outage, and a restart fixes nothing that doesn't already end Serve.

Proposal: RabbitMQ's Consuming has Joined's first shape, a method with health.MonitorFunc's signature. The two are kept in step, so RabbitMQ's revision gives Consuming this one: func (svc *Service) Consuming() health.Monitor. Revision 1's note that RabbitMQ needs nothing is replaced by this.

Each section below replaces the part of the same name in the post, as revision 1 left it. Background, the HTTP and gRPC answer tables, Checks, Jobs, What waves 5 and 6 build on this, Migration from today and References are unchanged.

Summary

health gives an App its probes, health.Probes, and a health Service that owns readiness, and serves no protocol. This proposal serves those probes to an orchestrator, in two new packages and one new method:

• httphealth, in the core module. NewHandler answers GET /health/liveness and GET /health/readiness from an App's Probes, with 200 or 503 and no detail in the body. The docs recommend attaching it to the HTTPS server the App already runs, next to a health Service. NewService serves that handler, and nothing else, over HTTPS on a port of its own, for a pod with no server, such as a consumer's or a job's, or whose server requires client certificates. It requires TLS, as every HTTP port humus serves does, and owns readiness as a health Service does, from the monitors the end user passes as health.Options.
• grpc/grpchealth, in the grpc module. NewServer implements gRPC health v1 from the same Probes. The service names "" and readiness answer readiness, and liveness answers liveness, as Kubernetes recommends. It is registered on the end user's grpcserver.Service, next to a health Service, where the cluster's grpc probes can use TLS. NewService serves it in plain text on a port of its own, with nothing else registered, where they can't, which is most clusters today. It owns readiness as httphealth.NewService does.
• (*kafka.Service).Joined, a method that returns a health.Monitor. A consumer is ready once it has joined its group, so a rollout whose new pods can't reach the brokers stops, rather than replacing pods that consume with pods that don't. It is for readiness only.
• Probes are explicit. The end user's Init returns a health Service next to the server its probes are attached to, or a probes server, like any other Service. No runtime package's Main exposes probes by default.
• No startup probe of its own, and no caching. A probes server answers only once Init has returned, so a startup probe on the liveness path already waits for Init. Each probe request checks its monitors with the request's context, and the probe's own timeout bounds the check.

A consumer's pod, for example:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	settings, probeSettings, err := readSettings(ctx, s) // kafka.Settings and httphealth.Settings
	if err != nil {
		return nil, err
	}
	consumer, err := kafka.NewService(s, settings, queue.ProcessorFunc[kafka.Message](h.Handle))
	if err != nil {
		return nil, err
	}
	probes, err := httphealth.NewService(s, probeSettings, health.Readiness(consumer.Joined()))
	if err != nil {
		return nil, err
	}
	return []humus.Service{consumer, probes}, nil
}

Proposed API

In httphealth, Settings gains TLS, and CheckTLS is added. Addr, CheckAddr, NewHandler and the Service's methods are as revision 1 has them, and the Service's doc says "over HTTP/1.1 and TLS" where it said "over plain HTTP/1.1".

// Settings configures a Service. Neither of its fields has a default, so
// the end user always sources both.
type Settings struct {
	// Addr is unchanged.
	Addr string

	// TLS is the config the Service serves HTTPS with. It is required: the
	// Service serves nothing in plain text. The Service serves a clone of
	// it, offering only HTTP/1.1, so the end user's config is never changed.
	// The kubelet doesn't verify the certificate of a probe with
	// scheme: HTTPS, and presents none of its own.
	//
	// Default: none. Valid: a config with a certificate, that doesn't
	// require a client certificate.
	// Check: CheckTLS.
	TLS *tls.Config
}

// CheckTLS rejects a nil config, one with no certificate, where none of
// Certificates, GetCertificate and GetConfigForClient is set, and one whose
// ClientAuth requires a client certificate, which the kubelet never
// presents, without naming the field.
func CheckTLS(c *tls.Config) error

grpchealth is as revision 1 has it.

In queue/kafka, Joined becomes:

// Joined returns a monitor that is healthy once the Service has joined its
// consumer group: unhealthy until franz-go first hands the member an
// assignment, empty or not, and healthy from then until Serve returns. A
// broker outage after that doesn't make it unhealthy. It never returns an
// error. A consumer is made ready once it has joined with
//
//	health.Readiness(consumer.Joined())
//
// It is for readiness only: restarting the pod doesn't bring the brokers
// back, and an error that a restart would fix already ends Serve.
func (s *Service) Joined() health.Monitor

Wiring the probes

The paragraph before the table, and the table, become:

The end user's Init returns a health Service whose Probes it hands to a handler or server attached to one the App already runs, or a probes server, which owns readiness. The docs recommend attaching the probes to a server the pod already runs, rather than opening another port. A probes port is for a pod with no server, a server that requires client certificates, and a gRPC service on a cluster whose grpc probes can't use TLS:

The pod	Its main	The probes	Kubernetes
a consumer, or a long-running job	humus.Main	httphealth.NewService on a port of its own	httpGet, scheme: HTTPS, on that port
an HTTPS API	httpserver.Main	a health Service, and httphealth.NewHandler attached to the API's mux; httphealth.NewService instead when the API requires client certificates	httpGet, scheme: HTTPS, on the API's port or the probes port
a gRPC service, on a cluster whose grpc probes can use TLS	grpcserver.Main	a health Service, and grpchealth.NewServer registered on the end user's Service; grpchealth.NewService instead when the Service requires client certificates	grpc with service: liveness or readiness, over TLS on the Service's port, or on the probes port
a gRPC service, on any other cluster	grpcserver.Main	grpchealth.NewService on a plaintext port of its own	grpc with service: liveness or readiness, on that port
a Kubernetes Job's pod	humus.Main	none	none

The examples are as revision 1 has them.

The probes servers

httphealth.NewService serves HTTPS, as every HTTP port humus serves does. grpchealth.NewService serves plain text, because the kubelet's grpc probes can't use TLS on most clusters. Each answers its probes and nothing else, so nothing of the end user's is reachable on the plaintext port.

• They own readiness, as revision 1 has it.
• They serve through the drain. A probes server keeps serving until Shutdown, like an HTTP or gRPC Service. During an API's drain, a readiness probe is answered 503, not refused, and a liveness probe still 200.
• HTTP is served by code shared with httpserver. The listener, serving over TLS, Shutdown with its Close when the grace period runs out, and net/http's error log written to telemetry move from httpserver into an internal package of the core module, which knows nothing of HTTP/2 or instrumentation. httpserver.NewService is that code with HTTP/2 and its instrumentation, and httphealth.NewService is that code over HTTP/1.1, with the probe handler alone. This is the shape the grpc module already has, and httpserver serves exactly as before.
• gRPC is served by the grpc module's internal/server. That package already listens through the Host, recovers panics, ends streams at Shutdown with Unavailable, and calls GracefulStop and then Stop. grpchealth.NewService passes it no credentials and no stats handler.
• HTTP/1.1 over TLS. The HTTP probes server offers only http/1.1 in ALPN: the kubelet speaks HTTP/1.1, and a probe needs nothing more. A request in plain text fails its TLS handshake, which is logged at ERROR, as httpserver logs it. So a probe that leaves out scheme: HTTPS fails, and the log says why.
• No client certificates. CheckTLS rejects a config whose ClientAuth is RequireAnyClientCert or RequireAndVerifyClientCert. The kubelet presents none, so every probe would fail and the pod would never turn ready.
• Fixed limits. The HTTP probes server reads a request's headers within 2s and the whole request within 5s, the httpserver defaults, and sets no write timeout, since the probe's own timeout bounds a check. Its settings are Addr and TLS, and the gRPC probes server's is Addr. A field is added when an end user needs one, which breaks no one.

grpcserver, httpserver and their Mains are unchanged. Neither Main runs a health Service or serves probes, as their docs already say.

Startup probes

The startup probe in the manifest sets scheme: HTTPS:

startupProbe:
  httpGet:
    path: /health/liveness
    port: probes
    scheme: HTTPS
  failureThreshold: 30 # 30 × 2s: Init has a minute
  periodSeconds: 2

Telemetry

The last sentence of "Attached probes are traced like any request" becomes: A team that doesn't want the spans drops them in its sampler.

A consumer's readiness

The last paragraph's last sentence, "Liveness never checks the brokers: restarting the pod doesn't bring them back", is replaced by a paragraph of its own:

Joined is for readiness only. A restart doesn't help a member that has lost its group. A broker outage hits every pod at once, so a liveness monitor would restart them all. The kubelet's restart backoff, up to five minutes by default, would then hold them down after the brokers come back. franz-go rejoins the group on its own once they do, and a fenced-out member rejoins at once. The errors a restart would fix, an error code franz-go doesn't retry and credentials a broker refuses, already end Serve, so the pod exits and is restarted without a probe.

Changes to landed packages and pages

The bullets on httpserver, queue/kafka and the one-server guidance become:

• httpserver: serving over TLS and shutdown move into an internal package of the core module, as above. Its API and behaviour don't change.
• queue/kafka: the Joined method, and an import of health, which imports nothing but the standard library. The doc's "Readiness is not decided here" is replaced by how a consumer turns ready, and why Joined is for readiness only, with an example.
• The one-server guidance. The new section recommends attaching the probes to a server the pod already runs: the handler on an HTTPS API's mux, and grpchealth.NewServer on the end user's gRPC Service where the cluster's grpc probes can use TLS. A probes port is for a pod with no server, a server that requires client certificates, and a gRPC service where the probes can't use TLS. grpc.md already says a probe with mode: TLS reaches the end user's Service on a cluster with GRPCContainerProbeTLS on, unless the Service sets ClientAuth.

Design decisions

The bullet "A probes server serves the probes and nothing else" becomes:

• A probes server serves the probes and nothing else. It is what keeps the plaintext gRPC port harmless, and it is what Proposal: gRPC services #681 already decided for gRPC.

The bullet "A consumer is ready once joined" becomes:

• A consumer is ready once joined. See A consumer's readiness above. Joined returns a health.Monitor, so the call site needs no conversion and the type says what it is for. It is API because the end user has no other way to learn that the member joined.

These are added:

• TLS on every HTTP port. The kubelet's httpGet probes speak HTTPS on every cluster and don't verify the certificate, so a probes port needs nothing in plain text, and any certificate the end user has serves.
• The gRPC probes port stays plaintext while Kubernetes needs it. The kubelet's grpc probes use TLS only with an alpha gate, off by default. On most clusters, a gRPC service that wants grpc probes has no other way to answer them.
• Attach where a server exists. A pod that already runs an HTTPS or gRPC server answers its probes there, rather than opening another port, unless the kubelet can't reach that server.
• Rejected: a liveness monitor for a consumer out of its group. It can't tell a stuck client from a broker outage, and in an outage it restarts every pod, then holds them in backoff after the brokers return. The errors a restart would fix already end Serve.

Testing

TestService's constructor becomes:

cert := humustest.NewCA(t).Issue(t, "localhost")
svc, err := httphealth.NewService(s, httphealth.Settings{
	Addr: ":8081",
	TLS:  &tls.Config{Certificates: []tls.Certificate{cert.TLS()}},
})

In the drain test, the probes are requested at https://localhost:8081, with h.HTTPClient() given the CA's pool as its roots, over HTTP/1.1 as the kubelet speaks.

The bullets on Joined and on the settings convention become, and two bullets are added after them:

• Joined. On kfake listening on the test Host, Joined's monitor is unhealthy before Serve, healthy once the group is joined, including for a member assigned no partition, and unhealthy after Serve returns. With no broker listening, it stays unhealthy while franz-go retries.
• The settings convention. CheckAddr rejects "" and a port out of range, and CheckTLS rejects nil, a config with no certificate, and one that requires a client certificate, neither naming the field. NewService names each bad field, and rejects a nil Option.
• Plain text is refused. A plain-HTTP request to the HTTP probes server fails its handshake, and one ERROR record says so.
• HTTP/1.1 only. A client that offers h2 and http/1.1 gets HTTP/1.1 from the HTTP probes server.

Open questions

All four are answered above. None are left.

Out of scope

These are added:

• A runtime Main that exposes probes by default. It is reconsidered only when end users report needing it, by a proposal of its own.
• TLS on the gRPC probes port. It is made required once Kubernetes' grpc probes use TLS without a feature gate, by a proposal of its own.

[Zaba505]

Closing: the health probes have landed

The revisions' stories are done:

• story(httpserver): serve and shut down from an internal package a probes server shares #767 moved serving and shutting down into an internal package that httpserver and a probes server share (story(httpserver): serve and shut down from an internal package a probes server shares #782);
• story(httphealth): answer an orchestrator's HTTP probes from an App's Probes #768 started httphealth with a handler that answers an orchestrator's HTTP probes from an App's Probes (story(httphealth): answer an orchestrator's HTTP probes from an App's Probes #785);
• story(httphealth): Settings for a probes server, with a check per field #769 added httphealth.Settings with a check per field (story(httphealth): Settings for a probes server, with a check per field #786);
• story(httphealth): serve the probes over HTTPS on a port of their own, owning readiness #770 served the probes over HTTPS on a port of their own, with the probes server owning readiness (story(httphealth): serve the probes over HTTPS on a port of their own, owning readiness #789);
• story(grpchealth): answer gRPC health v1 from an App's Probes #771 started grpchealth, answering gRPC health v1 from an App's Probes (story(grpchealth): answer gRPC health v1 from an App's Probes #791);
• story(grpchealth): serve gRPC health v1 in plain text on a port of its own, owning readiness #772 served gRPC health v1 in plain text on a port of its own, owning readiness (story(grpchealth): serve gRPC health v1 in plain text on a port of its own, owning readiness #798);
• story(kafka): a consumer turns ready once it has joined its group #773 added kafka.Service.Joined, which turns ready once a consumer has joined its group (story(kafka): a consumer turns ready once it has joined its group #800);
• story(docs): document exposing health probes #774 documented exposing health probes (story(docs): document exposing health probes #821).

Revision 2 left no open questions.