Proposal: HTTP server foundation

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 2. It builds on Proposal: the humus App lifecycle, Proposal: configuration for humus apps, Proposal: OpenTelemetry without globals and Proposal: humustest. It replaces internal/httpserver and the server half of rest. It blocks two wave-3 proposals: Proposal: REST endpoint DSL and OpenAPI, which is built on this package, and Proposal: exposing health probes, which serves probes over HTTP with it.

Summary

A new package, httpserver, serves an http.Handler as a humus.Service. It is the HTTP half of what rest.Run did before the reset, without the REST DSL:

• httpserver.Service serves a handler on a listener from the App's Host. The listener is opened when the Service is built, in Init, so connections wait in the backlog until Serve runs. The Service keeps serving through the drain delay, and Shutdown drains it within the grace period, then closes whatever is left.
• httpserver.Settings holds the address, the TLS config and the net/http limits, as plain Go values with a DefaultSettings() and a check per field (#631).
• Every request gets a server span and the HTTP server metrics, through otelhttp and the App's telemetry.SDK. net/http's error log goes to the App's telemetry, never to the real standard error.
• A panicking handler is recorded on its span and in the logs, and its response is aborted, as net/http already does. The Service writes no 500 for it.
• httpserver.App is a bedrock.App built on humus.App for a service that is one HTTP server. The end user's Init returns the Settings and the handler, and the App builds the Service. Its drain delay defaults to 5s and its grace period to 10s.

func main() {
	bedrock.Main(httpserver.App{
		Host:      humus.OSHost(),
		Telemetry: buildTelemetry, // the end user's telemetry.Builder (#632)
		Init:      app.Init,       // func(ctx, *humus.Setup) (httpserver.Settings, http.Handler, error)
	})
}

The foundation decides nothing about REST or health. It is general enough that REST's Service and App are built on it, and that health probes can be served on a server of their own or attached to the API's server.

Background

• The server half of rest depends on removed bedrock APIs. rest.Run builds a bedrockhttp.Runtime from bedrock/runtime/http and runs it under DefaultRunner, RecoverPanics and NotifyOnSignal (rest/rest.go#L201-L223). All of these are gone from bedrock v0.22.0.
• Its settings are environment variables humus named. HUMUS_REST_PORT (8443), HUMUS_REST_READ_TIMEOUT (5s), HUMUS_REST_READ_HEADER_TIMEOUT (2s), HUMUS_REST_WRITE_TIMEOUT (10s), HUMUS_REST_IDLE_TIMEOUT (120s) and HUMUS_REST_MAX_HEADER_BYTES (1 MiB) are read with bedrockconfig.Env from the real environment (rest/rest.go#L62-L96). Under #631, humus packages read no variables.
• TLS is always on, and falls back to a self-signed certificate. buildTLSConfig reads a PKCS#12 file from HUMUS_REST_TLS_PKCS12_FILE with os.ReadFile, and generates a self-signed certificate for localhost when the variable isn't set (rest/tls.go#L28-L32). A misconfigured service starts with a certificate no client trusts, rather than failing.
• Instrumentation uses OTel's globals. The handler is wrapped with otelhttp.NewHandler(h, "rest") and no options, so it reads the global providers and propagator (rest/rest.go#L287).
• Shutdown is unbounded. internal/httpserver calls Shutdown(context.Background()), which waits for every in-flight request however long it takes (internal/httpserver/server.go#L32-L47).
• net/http's error log goes to the real standard error. Neither rest nor internal/httpserver sets http.Server.ErrorLog, so TLS handshake errors and handler panics are written by the log package's default logger, around the App's Process.
• What wave 1 provides. The lifecycle runs Services and opens their listeners in Init (#630). The Host hands out listeners, which humustest makes in-memory. Setup.Telemetry carries the providers (#632). humustest.TestService checks the Service contract.
• The reference. bedrock's example/httpserver serves on a listener it is handed, sets ErrorLog so nothing reaches the real stderr, starts the grace period from a context the signal doesn't cancel, and calls Close when Shutdown runs out of time. This proposal does the same, split into a Service and the humus lifecycle.

Proposed API

package httpserver // github.com/z5labs/humus/httpserver

// Settings configures a Service. Every field is a plain value, and a field's
// zero value means zero, never "use the default": start from DefaultSettings
// and change the fields you source.
type Settings struct {
	// Addr is the TCP address the Service listens on through the Host, as
	// host:port. An empty host listens on every address. Port 0 picks a free
	// port, which Service.Addr reports.
	//
	// Default: ":8080". Valid: host:port, with a port from 0 to 65535.
	// Check: CheckAddr.
	Addr string

	// TLS serves HTTPS with this config. Nil serves plain HTTP. Over TLS,
	// HTTP/2 is negotiated with ALPN, as net/http does by default.
	//
	// Default: nil. Valid: nil, or a config with Certificates, GetCertificate
	// or GetConfigForClient set. Check: CheckTLS.
	TLS *tls.Config

	// ReadHeaderTimeout bounds reading a request's headers. Zero means
	// ReadTimeout applies, as in net/http.
	//
	// Default: 2s. Valid: not negative. Check: CheckReadHeaderTimeout.
	ReadHeaderTimeout time.Duration

	// ReadTimeout bounds reading a whole request, its body included. Zero
	// means no timeout, as in net/http.
	//
	// Default: 5s. Valid: not negative. Check: CheckReadTimeout.
	ReadTimeout time.Duration

	// WriteTimeout bounds writing a response, from the end of the request's
	// headers. Zero means no timeout, as in net/http. A handler that streams
	// extends its own deadline with http.ResponseController.
	//
	// Default: 10s. Valid: not negative. Check: CheckWriteTimeout.
	WriteTimeout time.Duration

	// IdleTimeout bounds how long a keep-alive connection waits for its next
	// request. Zero means ReadTimeout applies, as in net/http.
	//
	// Default: 120s. Valid: not negative. Check: CheckIdleTimeout.
	IdleTimeout time.Duration

	// MaxHeaderBytes bounds the size of a request's headers.
	//
	// Default: 1 MiB (1 << 20). Valid: positive. Check: CheckMaxHeaderBytes.
	MaxHeaderBytes int
}

// DefaultSettings returns the Service's defaults.
func DefaultSettings() Settings

// The checks, one for each field with a valid range, as #631 sets out. Each
// reports what is wrong, without naming the field.
func CheckAddr(addr string) error
func CheckTLS(c *tls.Config) error
func CheckReadHeaderTimeout(d time.Duration) error
func CheckReadTimeout(d time.Duration) error
func CheckWriteTimeout(d time.Duration) error
func CheckIdleTimeout(d time.Duration) error
func CheckMaxHeaderBytes(n int) error

// Service is a humus.Service that serves an http.Handler.
type Service struct{ /* unexported */ }

// NewService runs every check on settings, reporting every bad field in one
// error that names each one, rejects a nil handler, and listens on
// settings.Addr through s.Host. It registers the listener's close with
// s.OnStop, so a listener whose Service never serves, because Init failed
// after NewService, is still closed.
//
// h is served wrapped in the Service's instrumentation, which takes its
// providers and propagator from s.Telemetry.
func NewService(s *humus.Setup, settings Settings, h http.Handler) (*Service, error)

// Addr returns the address the Service listens on, with the port the Host
// picked when settings.Addr has port 0.
func (svc *Service) Addr() net.Addr

// Serve serves until Shutdown. It keeps serving after ctx is done, so the
// Service answers requests through the App's drain delay. It returns nil
// once Shutdown has stopped it, including a Shutdown called before Serve.
func (svc *Service) Serve(ctx context.Context) error

// Shutdown stops accepting connections, closes idle ones and waits for
// in-flight requests to finish. If ctx is done first, or is done already, it
// closes every connection at once, which cancels the context of each
// in-flight request, and returns an error that wraps ctx's error. Calling it
// again returns nil.
func (svc *Service) Shutdown(ctx context.Context) error

// App is a bedrock.App for a service that is one HTTP server. It is built on
// humus.App: Run builds a humus.App whose Init calls the App's Init and
// returns one Service. Host, Telemetry and Init are required.
//
// Its grace period defaults to 10s and its drain delay to 5s. An operator
// changes either with HUMUS_SHUTDOWN_GRACE_PERIOD and
// HUMUS_SHUTDOWN_DRAIN_DELAY, which humus.App.Run reads.
type App struct {
	// Host is the network and filesystem the App runs on: humus.OSHost() in
	// main, and a humustest Host in tests.
	Host humus.Host

	// Telemetry builds the App's telemetry.SDK, as humus.App.Telemetry does.
	Telemetry telemetry.Builder

	// Init builds the end user's dependencies, sources the Service's
	// Settings, and returns them with the handler to serve. It registers the
	// release of what it builds with s.OnStop.
	Init func(ctx context.Context, s *humus.Setup) (Settings, http.Handler, error)
}

// Run implements bedrock.App.
func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus

The package is in the core module. Its one new dependency is otelhttp (see Telemetry below).

The Service

Listening. NewService calls s.Host.Listen("tcp", settings.Addr), as the lifecycle decided: listeners are opened in Init, so a connection that arrives before Serve waits in the backlog rather than being refused. A failure to listen, such as EADDRINUSE, is NewService's error, and the App exits 1 with it.

Serving. Serve calls http.Server.Serve on the listener, or http.Server.ServeTLS with settings.TLS as the server's TLSConfig when it is set. ServeTLS is what turns on HTTP/2 over TLS: it adds h2 to the config's ALPN protocols. Plain HTTP serves HTTP/1.1 only.

The two shutdown signals (#630):

• Serve's context being done changes nothing. The Service keeps serving through the drain delay, while the load balancer stops sending it traffic.
• Shutdown calls http.Server.Shutdown with the grace-period context. When that context is done before the in-flight requests finish, it calls http.Server.Close, which closes every connection and so cancels each in-flight request's context, and returns an error wrapping the context's error. The lifecycle turns that into a *humus.GracePeriodError.

Request contexts. A request's context does not derive from Serve's context. Serve's context is done as soon as shutdown starts, and a request that arrives during the drain delay must still be served. A request's context is done when its client goes away, or when Shutdown runs out of time and closes its connection.

The listener's release. http.Server.Serve closes the listener when it returns, and it does so even when Shutdown came first. The OnStop function closes it too, for the case where Serve was never called, and ignores net.ErrClosed.

Defaults. DisableGeneralOptionsHandler stays at net/http's default, so OPTIONS * is answered with 200. Nothing else on http.Server is set beyond the Settings fields, ErrorLog and the TLS config.

Settings

Field	Default	Valid	Check	Before the reset
Addr	":8080"	host:port, port 0 to 65535	CheckAddr	HUMUS_REST_PORT, default 8443
TLS	nil, plain HTTP	nil, or a config that has a certificate	CheckTLS	HUMUS_REST_TLS_PKCS12_FILE, or a self-signed certificate
ReadHeaderTimeout	2s	not negative	CheckReadHeaderTimeout	HUMUS_REST_READ_HEADER_TIMEOUT
ReadTimeout	5s	not negative	CheckReadTimeout	HUMUS_REST_READ_TIMEOUT
WriteTimeout	10s	not negative	CheckWriteTimeout	HUMUS_REST_WRITE_TIMEOUT
IdleTimeout	120s	not negative	CheckIdleTimeout	HUMUS_REST_IDLE_TIMEOUT
MaxHeaderBytes	1 << 20	positive	CheckMaxHeaderBytes	HUMUS_REST_MAX_HEADER_BYTES

• The timeouts keep net/http's meaning of zero, which is "no timeout" or "use ReadTimeout". That is the case Proposal: configuration for humus apps #631 gives for "a zero field means zero": DefaultSettings() supplies the defaults, and a zero the end user sets is the net/http zero.
• MaxHeaderBytes must be positive. In net/http, zero means 1 MiB, which is a default applied to a zero value. Proposal: configuration for humus apps #631 rules that out, so zero is rejected instead of silently meaning 1 MiB.
• CheckTLS rejects a config with no certificate. ServeTLS with no Certificates, GetCertificate or GetConfigForClient fails only when Serve runs, after Init, with an error about a file named "". The check fails startup instead, naming the variable that sourced the config.
• Optional settings (#631, question 2). TLS is the only optional field, and nil is its "not set". Every other field always has a value.

In the end user's Init, sourced as #631 and the configuration page describe, with whatever variable names the service uses:

func Init(ctx context.Context, s *humus.Setup) (httpserver.Settings, http.Handler, error) {
	d := httpserver.DefaultSettings()
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }

	settings, err := config.Struct(func(r *config.Resolver) httpserver.Settings {
		return httpserver.Settings{
			Addr:              r.Get(env("ADDR").Default(d.Addr).Validate(httpserver.CheckAddr)),
			TLS:               r.Get(env("TLS_PEM_FILE").Parse(readFile(s.Host.FS)).Parse(serverTLS).Default(d.TLS).Validate(httpserver.CheckTLS)),
			ReadHeaderTimeout: r.Get(env("READ_HEADER_TIMEOUT").Parse(time.ParseDuration).Default(d.ReadHeaderTimeout).Validate(httpserver.CheckReadHeaderTimeout)),
			ReadTimeout:       r.Get(env("READ_TIMEOUT").Parse(time.ParseDuration).Default(d.ReadTimeout).Validate(httpserver.CheckReadTimeout)),
			WriteTimeout:      r.Get(env("WRITE_TIMEOUT").Parse(time.ParseDuration).Default(d.WriteTimeout).Validate(httpserver.CheckWriteTimeout)),
			IdleTimeout:       r.Get(env("IDLE_TIMEOUT").Parse(time.ParseDuration).Default(d.IdleTimeout).Validate(httpserver.CheckIdleTimeout)),
			MaxHeaderBytes:    r.Get(env("MAX_HEADER_BYTES").Parse(strconv.Atoi).Default(d.MaxHeaderBytes).Validate(httpserver.CheckMaxHeaderBytes)),
		}
	}).Read(ctx)
	if err != nil {
		return httpserver.Settings{}, nil, err // exit 78, naming every bad variable
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /hello", hello)
	return settings, mux, nil
}

readFile and serverTLS are the end user's helpers, as in #631. How a certificate is stored and encoded, PEM or PKCS#12, is the end user's decision.

The App

httpserver.App is the convenience App the lifecycle revision calls for: "Services for composition, plus an App per runtime package". Its Run is, in full:

func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus {
	return humus.App{
		Host:                a.Host,
		ShutdownGracePeriod: 10 * time.Second,
		DrainDelay:          5 * time.Second,
		Telemetry:           a.Telemetry,
		Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
			settings, h, err := a.Init(ctx, s)
			if err != nil {
				return nil, err
			}
			svc, err := httpserver.NewService(s, settings, h)
			if err != nil {
				return nil, err
			}
			return []humus.Service{svc}, nil
		},
	}.Run(ctx, p)
}

• Defaults. The drain delay is 5s, because Kubernetes removes a terminating pod from a Service's endpoints asynchronously, and an API that stops accepting connections at once refuses requests still being routed to it. The grace period is 10s, as before the reset. Both together fit within Kubernetes' default terminationGracePeriodSeconds of 30s.
• No duration fields. The defaults are not fields of App. An operator sets HUMUS_SHUTDOWN_GRACE_PERIOD and HUMUS_SHUTDOWN_DRAIN_DELAY in the same manifest that sets terminationGracePeriodSeconds, which is where the three have to agree. A service that wants other defaults in code uses humus.App and NewService directly.
• Telemetry is required, as it is on humus.App. A default of telemetry.New with DefaultSettings() would name every service unknown_service, which only shows up once its logs are in a backend. The end user's Builder is the place that names the service, and a test passes telemetry.NoopBuilder or humustest's recorder.
• Init returns Settings and a handler, not a Service. The App builds the Service, so a later proposal can change what the App builds around the handler, such as attaching probes, without changing Init's shape. The end user who needs more than one Service in a process uses humus.App directly, which the lifecycle documents as the advanced path.
• No health. The App runs no health Service and serves no probes. Proposal: exposing health probes (wave 3) decides whether this App exposes them by default, and adds the fields it needs. Adding a field to App doesn't break the end user's code.

Telemetry

The Service takes everything from s.Telemetry and never touches OTel's globals (#632):

otelhttp.NewHandler(h, "",
	otelhttp.WithTracerProvider(s.Telemetry.TracerProvider),
	otelhttp.WithMeterProvider(s.Telemetry.MeterProvider),
	otelhttp.WithPropagators(s.Telemetry.Propagator),
)

• Spans. One server span per request, a child of the trace context the request brought in, extracted with the App's propagator. It is named after the method and the route pattern, such as GET /carts/{id}, when the handler is an http.ServeMux or anything else that sets Request.Pattern, and after the method alone otherwise. This is otelhttp's default name. It follows the semantic conventions, and it is why the operation name passed above is empty.
• Metrics. The HTTP server metrics of OpenTelemetry's semantic conventions, such as http.server.request.duration, recorded by otelhttp on the App's MeterProvider.
• Logs. The Service logs through s.Telemetry.LoggerProvider.Logger("github.com/z5labs/humus/httpserver"), with the OTel Logs API, as the lifecycle does. It emits:
  • one record at INFO when Serve starts, listening, with the attribute server.address set to Addr();
  • one record at ERROR for each line net/http writes to http.Server.ErrorLog, such as a TLS handshake error or a failure to accept a connection. ErrorLog is a *log.Logger on an unexported io.Writer that emits each line as a record. Without it, net/http writes to the log package's default logger, which is the real standard error;
  • one record at ERROR for each handler panic, in the request's context, as described next.

otelhttp is the one new dependency. It is OpenTelemetry's own instrumentation for net/http, it follows the HTTP semantic conventions as they change, and rest used it before the reset. Writing the same spans and metrics by hand would be a second implementation of the conventions to keep in step. It is pinned to the contrib release that matches humus's go.opentelemetry.io/otel version (v0.72.0 for v1.47.0). otelhttp reads OTel's global propagator and MeterProvider as defaults before applying its options. The Service passes all three providers, so the globals are never used.

Panics

The lifecycle never recovers a panic, and the charter leaves a panic inside one request to each protocol. For HTTP the decision is:

• A handler's panic aborts that request, as net/http does today. net/http recovers a handler's panic, logs its stack to ErrorLog, and closes the connection, or resets the stream over HTTP/2. The client sees an interrupted response. The Service keeps serving other requests.
• The Service adds telemetry to it, and nothing else. Between otelhttp and the end user's handler, an unexported layer recovers the panic, records it on the request's span as an exception with its stack trace and an error status, emits an ERROR record in the request's context, so the record carries the trace and span IDs, and then panics again with http.ErrAbortHandler. net/http aborts the response as before, and doesn't log the panic a second time.
• A handler that panics with http.ErrAbortHandler itself is passed through untouched, which is net/http's documented way to abort a response without logging.
• No 500. Writing a 500 would turn a bug into an ordinary error response, which dashboards then count with every other 5xx. And the handler may already have written headers or part of a body. A handler whose protocol defines an error response for this, such as REST's, recovers in its own handler, before the Service's layer sees the panic.

What wave 3 builds on this

Both wave-3 proposals that need HTTP are expressible with this API, without anything in it that is about REST or health.

Proposal: REST endpoint DSL and OpenAPI. REST produces an http.Handler, and its Service is this Service serving that handler. Its App can be this App with REST's Init in front. For illustration only, since REST's proposal designs its own API:

// REST's App, sketched on httpserver.App. Not a proposal for REST's API.
func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus {
	return httpserver.App{
		Host:      a.Host,
		Telemetry: a.Telemetry,
		Init: func(ctx context.Context, s *humus.Setup) (httpserver.Settings, http.Handler, error) {
			settings, api, err := a.Init(ctx, s) // REST's own Init shape
			if err != nil {
				return httpserver.Settings{}, nil, err
			}
			return settings, api.Handler(), nil // REST's handler, with its OpenAPI endpoint
		},
	}.Run(ctx, p)
}

So REST inherits this App's defaults, its instrumentation and its panic handling, and whatever wave 3 adds to this App for health.

Proposal: exposing health probes. Both ways of serving probes over HTTP work with humus.App and two building blocks of this package, given a probe handler that the wave-3 proposal provides (written probes below):

// On a server of their own, as a job or an event-processor pod needs: a
// second Service on a second address.
probeSvc, err := httpserver.NewService(s, probeSettings, probes)

// Attached to the API's server: one handler that serves both.
mux := http.NewServeMux()
mux.Handle("/", api)
mux.Handle("/health/", probes)
apiSvc, err := httpserver.NewService(s, settings, mux)

Exposing probes by default from httpserver.App is the wave-3 proposal's decision. Since App builds the Service from the handler Init returns, the App can wrap that handler, or add a second Service and a health Service next to it, by adding fields to App. Whether probe requests get spans, and whether that needs a filter on this package's instrumentation, is decided there too, and its stories make the change here.

Design decisions

• The package is httpserver. Proposal: configuration for humus apps #631 and the configuration page already use httpserver.Settings, httpserver.DefaultSettings(), httpserver.CheckReadTimeout and httpserver.CheckAddr as their example, and the name says what it is: the server, not the protocol. http would clash with net/http at every call site.
• A handler in, not a mux or a router. The Service serves one http.Handler. Routing, middleware and composing handlers are the end user's net/http code, or REST's. A foundation that owned routing would decide REST's design.
• The listener is opened in NewService. That is the lifecycle's rule. It also makes Addr() meaningful before Serve, which a test with port 0 needs, and it makes a port that is already taken fail Init rather than the first Serve.
• The Service keeps serving until Shutdown. The drain delay exists for HTTP: during it, a load balancer stops routing new requests to the pod, and requests already routed must still be answered. So Serve's context is not a signal to stop, and a request's context doesn't derive from it.
• Shutdown closes what the grace period didn't drain. http.Server.Shutdown returns when its context is done, but leaves the connections open. Close ends them, so their handlers' contexts are cancelled and the App can exit. That is bedrock's example/httpserver.
• Plain HTTP by default, on :8080. In Kubernetes, TLS is usually terminated by an ingress or a mesh, and a pod serves plain HTTP. Before the reset, TLS was always on, with a self-signed certificate when none was configured. That certificate is one no client trusts, so a service that forgot its certificate started anyway and failed every request. Now TLS is on when the end user sets Settings.TLS, and CheckTLS fails startup when it is set but empty.
• HTTP/2 over TLS, HTTP/1.1 in plain text. That is net/http's default. Unencrypted HTTP/2 (h2c) is open question 2.
• Defaults are DefaultSettings(), not zero values (#631). The timeouts keep net/http's zero, and MaxHeaderBytes rejects the zero that net/http would replace with 1 MiB.
• ErrorLog goes to telemetry. It is the only output of net/http that bypasses the handler, and left unset it reaches the real standard error. The Service writes it through the Logs API, as the lifecycle writes its own records, so humus needs no slog bridge in library code.
• Panics are aborted, recorded and not answered. See Panics above.
• The App takes no duration fields and requires Telemetry. See The App above.
• Addr() is API. It is the only way to learn the port when Addr has port 0, which humustest supports and tests use. It is not a convenience over something the end user can do.
• No options on NewService. Every knob is a field of Settings, so it can be sourced like the rest. A later proposal that needs a knob that isn't a setting, such as a filter on instrumentation for probes, adds what it needs in its own stories.

Testing

These tests are sketches and have not been run. Every test runs inside a testing/synctest bubble, on bedrocktest's process and a humustest Host, and asserts with testify/require.

The Service passes the contract:

func TestService(t *testing.T) {
	humustest.TestService(t, func(t *testing.T, s *humus.Setup) humus.Service {
		svc, err := httpserver.NewService(s, httpserver.DefaultSettings(), http.NotFoundHandler())
		require.NoError(t, err)
		return svc
	})
}

The Service keeps serving after Serve's context is done, so TestService's subtest for Shutdown after Serve has returned on its own reports SKIP, as TestService documents.

A whole App, with its telemetry recorded:

func TestApp_ServesARequestAndRecordsItsSpan(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		tel := humustest.NewTelemetry()
		app := httpserver.App{
			Host:      h.Host(),
			Telemetry: tel.Builder(),
			Init: func(ctx context.Context, s *humus.Setup) (httpserver.Settings, http.Handler, error) {
				mux := http.NewServeMux()
				mux.HandleFunc("GET /hello/{name}", func(w http.ResponseWriter, r *http.Request) {
					_, _ = io.WriteString(w, "hello, "+r.PathValue("name"))
				})
				return httpserver.DefaultSettings(), mux, nil
			},
		}

		p, status := bedrocktest.Start(t, app)
		<-h.Listening("localhost:8080")

		resp, err := h.HTTPClient().Get("http://localhost:8080/hello/humus")
		require.NoError(t, err)
		body, err := io.ReadAll(resp.Body)
		require.NoError(t, err)
		require.NoError(t, resp.Body.Close())
		require.Equal(t, "hello, humus", string(body))

		p.Signal(syscall.SIGTERM)
		require.Zero(t, (<-status).Code())

		spans := tel.Spans()
		require.Len(t, spans, 1)
		require.Equal(t, "GET /hello/{name}", spans[0].Name())
		require.Empty(t, p.Stderr())
	})
}

Over TLS, with a CA generated for the test, so no key, certificate or password is fixed:

func TestApp_ServesHTTPSWithACertificateFromTheTestCA(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		ca := humustest.NewCA(t)
		cert := ca.Issue(t, "localhost")
		app := httpserver.App{
			Host:      h.Host(),
			Telemetry: telemetry.NoopBuilder,
			Init: func(context.Context, *humus.Setup) (httpserver.Settings, http.Handler, error) {
				settings := httpserver.DefaultSettings()
				settings.Addr = ":8443"
				settings.TLS = &tls.Config{Certificates: []tls.Certificate{cert.TLS()}}
				return settings, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
					_, _ = io.WriteString(w, "secure")
				}), nil
			},
		}

		p, status := bedrocktest.Start(t, app)
		<-h.Listening("localhost:8443")

		client := h.HTTPClient()
		transport := client.Transport.(*http.Transport)
		transport.TLSClientConfig = &tls.Config{RootCAs: ca.Pool()}
		transport.ForceAttemptHTTP2 = true

		resp, err := client.Get("https://localhost:8443/")
		require.NoError(t, err)
		require.NoError(t, resp.Body.Close())
		require.Equal(t, 2, resp.ProtoMajor)

		p.Signal(syscall.SIGTERM)
		require.Zero(t, (<-status).Code())
	})
}

The tests also show the following:

• An in-flight request finishes during shutdown. A handler that sleeps 3s, with SIGTERM sent while it sleeps, returns its whole response. Serve's context is done at once, the drain delay is 5s, and the response arrives 3s after the handler started.
• A request that arrives during the drain delay is served. A request sent 1s after SIGTERM gets its response.
• The grace period closes what it couldn't drain. With HUMUS_SHUTDOWN_GRACE_PERIOD=2s, a handler blocked on r.Context() sees its context cancelled exactly 5s + 2s after SIGTERM, and the App exits 1 with a *humus.GracePeriodError.
• A handler that ignores r.Context() is caught. It outlives the App once Close has dropped its connection, and synctest reports the goroutine still blocked when the test ends. That is the lifecycle's leak detector at work (#630, Testing), and the package doc says so, because the end user's tests run into it too.
• A panic is aborted, recorded, and doesn't stop the Service. The client's request fails, the span has an error status and an exception event with the stack, an ERROR record carries the span's trace_id and span_id, nothing is written to p.Stderr(), and a second request succeeds.
• ErrorLog reaches telemetry. A plain-HTTP request to a TLS Service produces an ERROR record with the scope github.com/z5labs/humus/httpserver and the handshake error, and nothing on p.Stderr().
• A taken address fails Init. Two Services on :8080 in one humus.App exit 1 with EADDRINUSE, and the first one's listener is closed by its OnStop function: a dial to :8080 is then refused.
• A listener is closed when Init fails after NewService. An Init that fails after building the Service exits 1, and a dial to its address is refused.
• Port 0. A Service on :0 reports the port the Host picked through Addr(), and a request to it succeeds.
• The settings convention (#631). Each check accepts an in-range value with nil and rejects an out-of-range one without naming the field; NewService names each bad field, and two bad fields are reported together; DefaultSettings() passes NewService; the package never reads a bedrock.Environment and doesn't import bedrock/config.
• No globals. With OTel's global providers set to recorders of their own, a request records nothing in them.

The package's example is a package main with an app_test.go, or an Example function, as the charter requires, and the package is held to the 80% coverage gate. A story documents the package on the docs site, in a new page, "Serving HTTP", next to the lifecycle and configuration pages.

Migration from today

Before the reset	With this proposal
rest.Run(ctx, opts...), its server half	httpserver.App, or humus.App with httpserver.NewService. Its REST half moves to Proposal: REST endpoint DSL and OpenAPI (wave 3).
internal/httpserver.App	httpserver.Service
rest.Port(...), HUMUS_REST_PORT=8443	Settings.Addr, default ":8080", sourced from whatever variable the end user names
rest.ReadTimeout, ReadHeaderTimeout, WriteTimeout, IdleTimeout, MaxHeaderBytes and their HUMUS_REST_* variables	the Settings fields of the same names, with the same defaults
rest.TLSConfig(...), HUMUS_REST_TLS_PKCS12_FILE and _PASSWORD	Settings.TLS, a *tls.Config the end user decodes from a file read through s.Host.FS. PKCS#12 decoding is the end user's, with go-pkcs12 if they like.
a self-signed certificate when no TLS is configured	plain HTTP when Settings.TLS is nil; a config with no certificate fails CheckTLS
otelhttp.NewHandler(h, "rest") on the globals	the Service's instrumentation, with s.Telemetry's providers and propagator, and spans named after the route
Shutdown(context.Background())	Shutdown bounded by the grace period, then Close
RecoverPanics around the runtime	no recovery in the lifecycle; a handler's panic aborts its request, recorded in telemetry
net/http's error log on the real stderr	ERROR records through the App's telemetry

The root package's tests keep their test-only httpService in app_http_test.go. They test the lifecycle, which shouldn't depend on a runtime package.

Open questions

1. The default address. ":8080" with plain HTTP, as proposed, or ":8443" with TLS required, as before the reset and as Proposal: configuration for humus apps #631's illustration had it? Requiring TLS means every service needs a certificate, even behind a mesh that already encrypts. Recommendation: ":8080", plain HTTP.
2. Unencrypted HTTP/2 (h2c). net/http can serve HTTP/1.1 and h2c on one port through http.Server.Protocols. That matters for a mesh sidecar that speaks HTTP/2 to the pod, or for gRPC and HTTP on one port. Add a Settings field now, or wait for a proposal that needs it? Recommendation: wait, so the field is designed against a real use.
3. Panics. Abort and record, as proposed, or recover and write a 500 when no header has been written? Recommendation: abort. A protocol that defines an error response, such as REST's, writes it in its own handler.
4. The severity of ErrorLog lines. Every line at ERROR, as proposed, or TLS handshake errors at WARN? A load balancer that probes the port with TCP checks causes a handshake error on every probe. Telling the lines apart means matching net/http's message text, which isn't API. Recommendation: ERROR for every line; a team that is bothered by the noise filters it in its log pipeline.

Out of scope

• The REST endpoint DSL, OpenAPI and the shoppingcart example: Proposal: REST endpoint DSL and OpenAPI (wave 3).
• Serving health probes, and whether httpserver.App exposes them by default: Proposal: exposing health probes (wave 3).
• Authentication and authorization (the charter's "REST security", later).
• The outbound HTTP client (the charter's "Later" list).
• Middleware beyond instrumentation and panic recording: CORS, compression, request IDs, body size limits, rate limiting. These are handlers the end user composes.
• WebSockets and other hijacked connections. http.Server.Shutdown and Close don't track a hijacked connection, so a handler that hijacks one closes it itself.
• HTTP/3, Unix domain sockets, and serving one Service on several addresses.

References

• Proposal: humus on the bedrock process model, the charter, and its wave plan
• The lifecycle's revision: listeners in Init, the two shutdown signals, Apps per runtime package, panics left to each protocol
• The configuration revision: Settings, DefaultSettings, Check<Field>, and the httpserver example
• Telemetry's revision 2: telemetry.SDK, no globals, and the wave-2 consequence for Builders
• humustest's revision 2: the in-memory network, HTTPClient, NewCA and TestService
• Health's revision 2: exposing probes moves to wave 3
• bedrock's example/httpserver, the reference service
• net/http's Handler documentation, on panics and ErrAbortHandler
• otelhttp v0.72.0, and OpenTelemetry's semantic conventions for HTTP
• Before the reset: rest/rest.go, rest/tls.go, internal/httpserver/server.go

[Zaba505]

Revision: the open questions answered, on humus.Main

The charter's revision 5 moved this proposal to wave 3, behind Proposal: humus.Main, which has landed. This revision answers the four open questions, applies humus.Main, and takes in the charter's revision 7.

The open questions:

1. TLS is required, and the default address is ":8443". Settings.TLS has no default and CheckTLS rejects nil. A service that hasn't sourced a certificate fails Init with exit 78, naming the variable it left unset. Nothing is served in plain text, and there is no self-signed fallback.
2. HTTP/2 is always on and preferred, and h2c isn't supported. The Service sets http.Server.Protocols itself: HTTP/2 and HTTP/1.1, with h2 first in ALPN, so every client that offers HTTP/2 gets it. net/http has been able to serve HTTP/2 alone since Go 1.24, so a new setting, HTTP2Only, turns the HTTP/1.1 fallback off for a service that wants to enforce HTTP/2. It is off by default. With TLS required there is no plain text to serve h2c on.
3. A handler's panic aborts its request, as recommended. Panics is unchanged.
4. Every ErrorLog line is an ERROR record, as recommended. Telemetry is unchanged.

What humus.Main changes:

• httpserver.App is gone. httpserver.Main is humus.Main with the package's defaults, the 5s drain and the 10s grace period, which httpserver.Defaults() exports as one humus.Option. The end user's Init builds the Service with NewService and returns it.
• Telemetry is no longer required. humus.Main's telemetry default replaces it, as humus.Main's revision 2 sets out.
• The package ships a test package, httpservertest. Its Start runs an App with the package's defaults on a test Host. Its NewClient returns an HTTP/2 client that dials that Host and trusts the test's CA.

From the charter's revision 7, REST and exposing health probes are now wave 4, failure injection strips scenarios at this server in wave 5, and the deferrals to "later" have waves.

Each section below replaces the section of the same name in the post. Background and References are unchanged, apart from the wording fixes at the end.

Summary

A new package, httpserver, serves an http.Handler over TLS as a humus.Service. It is the HTTP half of what rest.Run did before the reset, without the REST DSL:

• httpserver.Service serves a handler on a listener from the Setup's Host. The listener is opened when the Service is built, in Init, so connections wait in the backlog until Serve runs. The Service keeps serving through the drain, and Shutdown drains it within the grace period, then closes whatever is left.
• TLS is required. Settings.TLS has no default. The Service listens on ":8443" by default, and serves nothing in plain text.
• HTTP/2 is always on and preferred. Every client that offers HTTP/2 gets it, and the rest fall back to HTTP/1.1. Settings.HTTP2Only turns the fallback off. Unencrypted HTTP/2 (h2c) isn't served.
• httpserver.Settings holds the address, the TLS config, HTTP2Only and the net/http limits, as plain Go values with a DefaultSettings() and a check per field (#631).
• Every request gets a server span and the HTTP server metrics, through otelhttp and the Setup's telemetry.SDK. net/http's error log goes to the App's telemetry, never to the real standard error.
• A panicking handler is recorded on its span and in the logs, and its response is aborted, as net/http already does. The Service writes no 500 for it.
• httpserver.Main is humus.Main with an HTTP service's defaults: a 5s drain and a 10s grace period, exported as one humus.Option, httpserver.Defaults().
• httpservertest starts an App with those defaults on a test Host, and gives a test an HTTP/2 client for that Host.

func main() {
	httpserver.Main(app.Init, humus.WithTelemetry(buildTelemetry))
}

app.Init sources the Settings, builds the handler, and returns the Service that httpserver.NewService(s, settings, h) builds.

The foundation decides nothing about REST, health or failure injection. It is general enough that REST's Service is built on it, that health probes can be served on a server of their own or attached to the API's server, and that failure injection can strip scenarios from a request before the end user's handler sees it.

Proposed API

The checks other than CheckTLS, Service, Addr, Serve and Shutdown are unchanged, and so are the four timeouts and MaxHeaderBytes in Settings. App is gone. These declarations change or are new:

package httpserver // github.com/z5labs/humus/httpserver

// Settings configures a Service. Every field is a plain value, and a field's
// zero value means zero, never "use the default": start from DefaultSettings
// and change the fields you source. TLS has no default, so it is always one
// of them.
type Settings struct {
	// Addr is the TCP address the Service listens on through the Host, as
	// host:port. An empty host listens on every address. Port 0 picks a free
	// port, which Service.Addr reports.
	//
	// Default: ":8443". Valid: host:port, with a port from 0 to 65535.
	// Check: CheckAddr.
	Addr string

	// TLS is the config the Service serves HTTPS with. It is required: the
	// Service serves nothing in plain text.
	//
	// Default: none. Valid: a config with Certificates, GetCertificate or
	// GetConfigForClient set, that HTTP/2 can use. Check: CheckTLS.
	TLS *tls.Config

	// HTTP2Only turns HTTP/1.1 off, so the Service serves HTTP/2 alone. When
	// it is false, the Service serves HTTP/2 to every client that offers it
	// and HTTP/1.1 to the rest. When it is true, a client that offers only
	// HTTP/1.1 completes the TLS handshake and is then disconnected without
	// a response.
	//
	// Default: false. Valid: either.
	HTTP2Only bool

	// ReadHeaderTimeout, ReadTimeout, WriteTimeout, IdleTimeout and
	// MaxHeaderBytes are unchanged.
}

// CheckTLS reports why c can't serve HTTPS, or nil if it can. c must not be
// nil, must have Certificates, GetCertificate or GetConfigForClient set, and
// must allow HTTP/2: a MaxVersion of TLS 1.2 or later, or none, and when it
// sets CipherSuites with a MinVersion below TLS 1.3, one of the two suites
// HTTP/2 requires.
func CheckTLS(c *tls.Config) error

// NewService runs every check on settings, reporting every bad field in one
// error that names each one, rejects a nil handler, and listens on
// settings.Addr through s.Host. It registers the listener's close with
// s.OnStop, so a listener whose Service never serves, because Init failed
// after NewService, is still closed.
//
// The Service serves a clone of settings.TLS, so the end user's config is
// never changed. The clone's ALPN protocols are h2, then http/1.1 unless
// settings.HTTP2Only is set, then any others settings.TLS lists.
//
// h is served wrapped in the Service's instrumentation, which takes its
// providers and propagator from s.Telemetry.
func NewService(s *humus.Setup, settings Settings, h http.Handler) (*Service, error)

// Defaults sets an HTTP service's defaults over humus.Main's: a drain of
// humus.DrainDelay(5 * time.Second) and a 10s grace period. The Host and
// the telemetry Builder stay humus.Main's.
func Defaults() humus.Option

// Main is humus.Main with Defaults applied before opts, so the end user's
// options win over the package's. It never returns.
func Main(init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humus.Option)

package httpservertest // github.com/z5labs/humus/httpserver/httpservertest

// Start is humustest.Start with httpserver.Defaults applied before opts, so
// a test runs its App with the drain and the grace period that
// httpserver.Main gives it in production. Options given with
// humustest.WithApp come after the defaults, and win over them.
func Start(t testing.TB, init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humustest.Option) (*bedrocktest.Process, <-chan bedrock.ExitStatus)

// NewClient returns an http.Client that dials h, trusts only the
// certificates in roots, and speaks only HTTP/2. It is built on
// h.HTTPClient, so it keeps no idle connection. It works with or without
// Start.
func NewClient(h *humustest.Host, roots *x509.CertPool) *http.Client

Both packages are in the core module. httpserver's one new dependency is still otelhttp, and httpservertest adds none.

The Service

Unchanged, except that these paragraphs replace the ones with the same heading, and Protocols is new:

Serving. Serve calls http.Server.ServeTLS on the listener, with the clone of settings.TLS as the server's TLSConfig and no file names, since CheckTLS made sure the config has a certificate. It never calls http.Server.Serve, so nothing is served in plain text.

Protocols. The Service sets http.Server.Protocols itself, rather than leaving net/http's default: HTTP/2 and HTTP/1.1, or HTTP/2 alone when HTTP2Only is set, and never unencrypted HTTP/2. In ALPN the server's order wins, and the clone lists h2 first, so every client that offers HTTP/2 gets it. A client that offers only http/1.1, or offers no ALPN at all:

• by default, is served HTTP/1.1;
• with HTTP2Only, completes the TLS handshake, which crypto/tls allows an http/1.1 client of an h2 server, and is then disconnected by net/http before a request is read. The client sees its connection closed, with no response. net/http writes nothing to ErrorLog for it.

A client that offers only protocols the clone doesn't list fails the handshake, and ErrorLog reports a TLS handshake error.

Defaults. DisableGeneralOptionsHandler stays at net/http's default, so OPTIONS * is answered with 200. Nothing else on http.Server is set beyond the Settings fields, Protocols, ErrorLog and the TLS config.

Settings

Field	Default	Valid	Check	Before the reset
Addr	":8443"	host:port, port 0 to 65535	CheckAddr	HUMUS_REST_PORT, default 8443
TLS	none: required	a config that has a certificate and allows HTTP/2	CheckTLS	HUMUS_REST_TLS_PKCS12_FILE, or a self-signed certificate
HTTP2Only	false	either	none	(none): HTTP/2 or HTTP/1.1, as the client offered
ReadHeaderTimeout	2s	not negative	CheckReadHeaderTimeout	HUMUS_REST_READ_HEADER_TIMEOUT
ReadTimeout	5s	not negative	CheckReadTimeout	HUMUS_REST_READ_TIMEOUT
WriteTimeout	10s	not negative	CheckWriteTimeout	HUMUS_REST_WRITE_TIMEOUT
IdleTimeout	120s	not negative	CheckIdleTimeout	HUMUS_REST_IDLE_TIMEOUT
MaxHeaderBytes	1 << 20	positive	CheckMaxHeaderBytes	HUMUS_REST_MAX_HEADER_BYTES

• The timeouts keep net/http's meaning of zero, which is "no timeout" or "use ReadTimeout". That is the case Proposal: configuration for humus apps #631 gives for "a zero field means zero": DefaultSettings() supplies the defaults, and a zero the end user sets is the net/http zero.
• MaxHeaderBytes must be positive. In net/http, zero means 1 MiB, which is a default applied to a zero value. Proposal: configuration for humus apps #631 rules that out, so zero is rejected instead of silently meaning 1 MiB.
• TLS is required, and has no default. DefaultSettings().TLS is nil, and CheckTLS rejects nil, so DefaultSettings() alone doesn't pass NewService. The end user's chain for it has no .Default, so a variable that isn't set fails as not found, with exit 78.
• CheckTLS rejects what would otherwise fail only once Serve runs, or on every HTTP/2 connection. ServeTLS with no Certificates, GetCertificate or GetConfigForClient fails with an error about a file named "". With HTTP/2 always on, it also fails when CipherSuites is set, MinVersion is below TLS 1.3, and neither TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 nor TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 is listed. A MaxVersion below TLS 1.2 passes ServeTLS, but Go's HTTP/2 server refuses every connection with INADEQUATE_SECURITY, as RFC 9113 requires. The check fails startup instead, naming the variable that sourced the config.
• HTTP2Only has no check. Both values are valid.
• Optional settings (#631, question 2). No field is optional now. TLS was the only one, and it is required.

In the end user's Init, sourced as #631 and the configuration page describe, with whatever variable names the service uses:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	d := httpserver.DefaultSettings()
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }

	settings, err := config.Struct(func(r *config.Resolver) httpserver.Settings {
		return httpserver.Settings{
			Addr: r.Get(env("ADDR").Default(d.Addr).Validate(httpserver.CheckAddr)),
			// No .Default: TLS is required, so an unset TLS_PEM_FILE fails as not found.
			TLS:               r.Get(env("TLS_PEM_FILE").Parse(readFile(s.Host.FS)).Parse(serverTLS).Validate(httpserver.CheckTLS)),
			HTTP2Only:         r.Get(env("HTTP2_ONLY").Parse(strconv.ParseBool).Default(d.HTTP2Only)),
			ReadHeaderTimeout: r.Get(env("READ_HEADER_TIMEOUT").Parse(time.ParseDuration).Default(d.ReadHeaderTimeout).Validate(httpserver.CheckReadHeaderTimeout)),
			ReadTimeout:       r.Get(env("READ_TIMEOUT").Parse(time.ParseDuration).Default(d.ReadTimeout).Validate(httpserver.CheckReadTimeout)),
			WriteTimeout:      r.Get(env("WRITE_TIMEOUT").Parse(time.ParseDuration).Default(d.WriteTimeout).Validate(httpserver.CheckWriteTimeout)),
			IdleTimeout:       r.Get(env("IDLE_TIMEOUT").Parse(time.ParseDuration).Default(d.IdleTimeout).Validate(httpserver.CheckIdleTimeout)),
			MaxHeaderBytes:    r.Get(env("MAX_HEADER_BYTES").Parse(strconv.Atoi).Default(d.MaxHeaderBytes).Validate(httpserver.CheckMaxHeaderBytes)),
		}
	}).Read(ctx)
	if err != nil {
		return nil, err // exit 78, naming every bad variable
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /hello", hello)

	svc, err := httpserver.NewService(s, settings, mux)
	if err != nil {
		return nil, err
	}
	return []humus.Service{svc}, nil
}

readFile and serverTLS are the end user's helpers, as in #631. How a certificate is stored and encoded, PEM or PKCS#12, is the end user's decision.

Main and its defaults

This section replaces The App. humus.Main runs the Services of any runtime, so the package exports only what App added on top: its defaults, as one humus.Option, and a Main that applies them. Main is, in full:

func Defaults() humus.Option {
	return func(a *humus.App) {
		a.Drain = humus.DrainDelay(5 * time.Second)
		a.ShutdownGracePeriod = 10 * time.Second
	}
}

func Main(init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humus.Option) {
	humus.Main(init, append([]humus.Option{Defaults()}, opts...)...)
}

• The defaults are unchanged. The drain is 5s, because Kubernetes removes a terminating pod from a Service's endpoints asynchronously, and an API that stops accepting connections at once refuses requests still being routed to it. The grace period is 10s, as before the reset. With telemetry's 5s shutdown they add up to 20s, inside the 25s budget that humus.Main keeps within Kubernetes' default terminationGracePeriodSeconds of 30s.
• The HTTP Service drains for the whole delay. It keeps serving until Shutdown, so its Serve hasn't returned during the drain, and the Drain's context isn't done early.
• The Host and the telemetry are humus.Main's. Telemetry was required on App, so that a service couldn't start as unknown_service by accident. humus.Main decided otherwise for every runtime: the default Builder calls telemetry.New with DefaultSettings() and writes to stdout, and a service that names itself or exports anything passes humus.WithTelemetry.
• The end user changes a default with an option. In httpserver.Main(Init, humus.WithShutdownGracePeriod(15*time.Second)), the end user's option comes after Defaults(), so it wins. An operator still sets HUMUS_SHUTDOWN_GRACE_PERIOD and HUMUS_SHUTDOWN_DRAIN_DELAY in the same manifest that sets terminationGracePeriodSeconds.
• The end user's Init builds the Service. App's Init took the Settings and the handler from the end user's Init and called NewService. Now that call is in the end user's Init, which returns the Service along with any others the process runs. More than one Service is no longer the advanced path. An API next to a job returns both, and uses httpserver.Main.
• httpserver.Main or humus.Main? As humus.Main's revision 2 decides, it depends on what a load balancer routes to. A process uses httpserver.Main when a load balancer routes to its HTTP server. A job's pod whose only HTTP server, built with NewService, serves probes or metrics isn't routed to, so it stays on humus.Main, with no drain.
• No health. Main runs no health Service and serves no probes. Proposal: exposing health probes (wave 4) decides whether an HTTP service exposes them by default, in terms of humus.Main, this package's Main and Defaults, Options that wrap Init, and the Services the end user's Init returns.

httpservertest

This section is new. httpservertest is the test package that humus.Main's revision 2 lets a runtime package ship. It has two functions:

• Start is humustest.Start with httpserver.Defaults() first. A test of a main that calls httpserver.Main runs with the drain and the grace period production runs with, and can't forget them. The test passes its Host, its telemetry and its environment as it would to humustest.Start.
• NewClient returns an HTTP/2 client for a humustest Host. It is h.HTTPClient() with a TLSClientConfig that trusts only roots, and a Transport whose Protocols is HTTP/2 alone. Every request it makes is HTTP/2, so a Service that stopped offering h2 fails the test rather than falling back unnoticed. A test that needs HTTP/1.1, to check the fallback or HTTP2Only, configures h.HTTPClient() itself.

The client is API because HTTP/2 over a test Host is easy to get wrong. A Go Transport with its own TLS config or dialler negotiates only HTTP/1.1 unless ForceAttemptHTTP2 or Protocols says otherwise, and h.HTTPClient() has its own dialler. Such a client gets HTTP/1.1 from a default Service. Against an HTTP2Only Service, its request fails with the connection closed and no hint why.

Start makes no CA. How the end user's Init sources its certificate, from which variable and in which encoding, is the end user's, so Start couldn't hand it one. The test makes a humustest.CA, issues a certificate, and gives it to the App the way production does: with humustest.WithFiles and humustest.WithEnv, or in the Settings when the test builds them. It passes the CA's Pool() to NewClient.

The package sits beside httpserver, as net/http/httptest sits beside net/http, and its name doesn't clash with httptest. NewClient works without Start, so a test of a process that runs an HTTP and a gRPC server can use one package's Start and both packages' clients.

What waves 4 and 5 build on this

This section replaces What wave 3 builds on this. The three later proposals that build on this server can do so with this API, and nothing in it is about REST, health or failure injection.

Proposal: REST endpoint DSL and OpenAPI (wave 4). REST produces an http.Handler, and its Service is this Service serving that handler. For illustration only, since REST's proposal designs its own API:

// A REST service, sketched on httpserver. Not a proposal for REST's API.
func main() {
	httpserver.Main(Init, humus.WithTelemetry(buildTelemetry))
}

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	settings, err := readSettings(ctx, s) // httpserver.Settings, sourced as under Settings
	if err != nil {
		return nil, err
	}
	api, err := newAPI(ctx, s) // REST's handler, with its OpenAPI endpoint
	if err != nil {
		return nil, err
	}
	svc, err := httpserver.NewService(s, settings, api)
	if err != nil {
		return nil, err
	}
	return []humus.Service{svc}, nil
}

So REST inherits TLS, HTTP/2, the instrumentation and the panic handling. Its services use httpserver.Main and httpservertest, unless REST's proposal gives it a Main of its own.

Proposal: exposing health probes (wave 4). Both ways of serving probes over HTTP still work with NewService, given a probe handler that the wave-4 proposal provides (written probes below):

// On a server of their own, as a job or an event-processor pod needs: a
// second Service on a second address.
probeSvc, err := httpserver.NewService(s, probeSettings, probes)

// Attached to the API's server: one handler that serves both.
mux := http.NewServeMux()
mux.Handle("/", api)
mux.Handle("/health/", probes)
apiSvc, err := httpserver.NewService(s, settings, mux)

With TLS required, a probes server built on this package serves HTTPS and needs a certificate. Kubelet's httpGet probes with scheme: HTTPS don't verify the certificate and negotiate HTTP/2 through ALPN, so they work against an HTTP2Only Service too. Whether a job's or a consumer's pod serves its probes with this package, and so needs a certificate, is the wave-4 proposal's decision. So are exposing probes by default, which it can do with an Option that wraps Init, and whether probe requests get spans. Its stories make any change it needs here.

Proposal: failure injection (wave 5) strips scenarios from what a server receives, before the end user's code sees them. Its seam in this package is the chain the Service wraps around the end user's handler. otelhttp extracts the trace context and the baggage into the request's context, then the panic layer runs, then the handler. Stripping belongs after the extraction and before the handler, inside the Service, so every handler it serves, REST's included, is covered without the end user wrapping anything. The wave-5 proposal designs that layer and whatever configures it, and its stories add them here.

Design decisions

Unchanged, except that this bullet replaces the one with the same heading:

• No options on NewService. Every knob is a field of Settings, so it can be sourced like the rest. HTTP2Only is a field for that reason. A later proposal that needs a knob that isn't a setting, such as a filter on instrumentation for probes, adds what it needs in its own stories.

These bullets replace "Plain HTTP by default, on :8080", "HTTP/2 over TLS, HTTP/1.1 in plain text" and "The App takes no duration fields and requires Telemetry":

• TLS is required, on ":8443". Before the reset TLS was always on, with a self-signed certificate when none was configured, so a service that forgot its certificate started anyway and failed every request. The fallback was the mistake, not TLS. Now TLS has no default, and a service without a certificate fails Init with exit 78, naming its variable. Traffic is encrypted all the way to the pod, whatever sits in front of it. The cost is the one the post's question named: every service needs a certificate, even behind a mesh that already encrypts, and a proxy in front of the pod connects to it over TLS. ":8443" is the port before the reset.
• HTTP/2 always, with HTTP/1.1 as a fallback. The Service sets Protocols itself and puts h2 first in ALPN, so HTTP/2 doesn't depend on net/http's default or on the config's NextProtos. GODEBUG=http2server=0 still turns HTTP/2 off in net/http, whatever Protocols says. That setting is an operator overriding Go, and the Service doesn't second-guess it.
• HTTP2Only is a setting, off by default. net/http has served HTTP/2 alone since Go 1.24, through http.Server.Protocols. It is off by default because an HTTP/1.1 client is disconnected with no response and nothing in the logs. Two examples are an AWS ALB target group on its default protocol version, HTTP1, and a Go client with its own Transport that sets neither ForceAttemptHTTP2 nor Protocols. It is a Settings field, so an operator turns it on from the manifest once every client speaks HTTP/2.
• No h2c. TLS is required, so there is no plain text to serve unencrypted HTTP/2 on, and no setting for it.
• The Service orders ALPN itself. net/http adds h2 and http/1.1 after whatever protocols a config's NextProtos already lists, and the server's order wins. So a config that listed http/1.1 would prefer HTTP/1.1. The Service puts h2 first on a clone. It keeps any other protocol the config lists after them, such as autocert's acme-tls/1.
• CheckTLS checks what HTTP/2 needs. HTTP/2 is always on, so a config it can't use is as broken as one with no certificate. See Settings.
• No App: a Main and its defaults as one Option. humus.Main runs every runtime's Services. As humus.Main's revision 2 sets out for every runtime package, this package's Main does nothing but apply Defaults() before the end user's options. A test passes the same Option, through httpservertest.Start or humustest.WithApp.
• The package ships httpservertest. Its Start means a test can't forget the defaults. Its NewClient exists because HTTP/2 over a test Host takes a Transport setting that is easy to miss, and missing it fails only against an HTTP2Only Service.

Testing

These tests are sketches and have not been run. Every test runs inside a testing/synctest bubble, on a humustest Host, and asserts with testify/require. serves is a helper of the test file:

// serves returns an Init whose only Service serves h on DefaultSettings,
// over TLS with cert.
func serves(cert *humustest.Certificate, h http.Handler) func(context.Context, *humus.Setup) ([]humus.Service, error) {
	return func(_ context.Context, s *humus.Setup) ([]humus.Service, error) {
		settings := httpserver.DefaultSettings()
		settings.TLS = &tls.Config{Certificates: []tls.Certificate{cert.TLS()}}
		svc, err := httpserver.NewService(s, settings, h)
		if err != nil {
			return nil, err
		}
		return []humus.Service{svc}, nil
	}
}

The Service passes the contract:

func TestService_meets_the_Service_contract(t *testing.T) {
	humustest.TestService(t, func(t *testing.T, s *humus.Setup) humus.Service {
		settings := httpserver.DefaultSettings()
		cert := humustest.NewCA(t).Issue(t, "localhost")
		settings.TLS = &tls.Config{Certificates: []tls.Certificate{cert.TLS()}}
		svc, err := httpserver.NewService(s, settings, http.NotFoundHandler())
		require.NoError(t, err)
		return svc
	})
}

The Service keeps serving after Serve's context is done, so TestService's subtest for Shutdown after Serve has returned on its own reports SKIP, as TestService documents.

A whole App on the package's defaults, over HTTP/2, with its telemetry recorded:

func TestMain_serves_HTTP2_and_records_the_span(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		ca := humustest.NewCA(t)
		h := humustest.NewHost(t)
		tel := humustest.NewTelemetry()
		mux := http.NewServeMux()
		mux.HandleFunc("GET /hello/{name}", func(w http.ResponseWriter, r *http.Request) {
			_, _ = io.WriteString(w, "hello, "+r.PathValue("name"))
		})

		p, status := httpservertest.Start(t, serves(ca.Issue(t, "localhost"), mux),
			humustest.WithApp(humus.WithHost(h.Host()), humus.WithTelemetry(tel.Builder())),
		)
		<-h.Listening("localhost:8443")

		resp, err := httpservertest.NewClient(h, ca.Pool()).Get("https://localhost:8443/hello/humus")
		require.NoError(t, err)
		body, err := io.ReadAll(resp.Body)
		require.NoError(t, err)
		require.NoError(t, resp.Body.Close())
		require.Equal(t, 2, resp.ProtoMajor)
		require.Equal(t, "hello, humus", string(body))

		p.Signal(syscall.SIGTERM)
		require.Zero(t, (<-status).Code())

		spans := tel.Spans()
		require.Len(t, spans, 1)
		require.Equal(t, "GET /hello/{name}", spans[0].Name())
		require.Empty(t, p.Stderr())
	})
}

The end user's Init from Settings, with no certificate configured:

func TestInit_without_a_certificate_exits_78(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		_, status := httpservertest.Start(t, Init)

		st := <-status
		require.Equal(t, humus.ExitConfig, st.Code())
		require.ErrorContains(t, st.Cause(), "config TLS_PEM_FILE: not found")
	})
}

The tests also show the following. Unless a bullet says otherwise, the App runs with httpservertest.Start, so with a 5s drain and a 10s grace period:

• An in-flight request finishes during shutdown. A handler that sleeps 3s, with SIGTERM sent while it sleeps, returns its whole response. Serve's context is done at once, the drain is 5s, and the response arrives 3s after the handler started.
• A request that arrives during the drain is served. A request sent 1s after SIGTERM gets its response.
• The grace period closes what it couldn't drain. With HUMUS_SHUTDOWN_GRACE_PERIOD=2s, set with humustest.WithEnv, a handler blocked on r.Context() sees its context cancelled exactly 5s + 2s after SIGTERM, and the App exits 1 with a *humus.GracePeriodError. Without it, the context is cancelled 15s after SIGTERM.
• A handler that ignores r.Context() is caught. It outlives the App once Close has dropped its connection, and synctest reports the goroutine still blocked when the test ends. That is the lifecycle's leak detector at work (#630, Testing), and the package doc says so, because the end user's tests run into it too.
• A panic is aborted, recorded, and doesn't stop the Service. Over HTTP/2 the client's stream is reset. The span has an error status and an exception event with the stack, an ERROR record carries the span's trace_id and span_id, nothing is written to p.Stderr(), and a second request succeeds.
• ErrorLog reaches telemetry. A plain-HTTP request to the Service produces an ERROR record with the scope github.com/z5labs/humus/httpserver and the handshake error, and nothing on p.Stderr().
• Protocols. A client that offers h2 and http/1.1 gets HTTP/2. One that offers only http/1.1 gets HTTP/1.1 by default. With HTTP2Only, its request fails with the connection closed and no response, nothing is logged, and NewClient's request still succeeds. A client that sends unencrypted HTTP/2 with prior knowledge gets no response, and a TLS handshake error is recorded.
• ALPN order. A config whose NextProtos is ["http/1.1"] still serves HTTP/2 to a client that offers both. One that lists acme-tls/1 still offers it, after h2 and http/1.1. NewService leaves the end user's config unchanged.
• TLS settings. DefaultSettings() alone fails NewService, naming TLS, and passes once TLS has a certificate. CheckTLS rejects nil, a config with no certificate, CipherSuites below TLS 1.3 that lack both HTTP/2 suites, and a MaxVersion below TLS 1.2, without naming the field. It accepts a config with only GetCertificate set, and CipherSuites without those suites when MinVersion is TLS 1.3.
• Main and Defaults. An internal test checks that Main passes humus.Main Defaults() before the end user's options: with humus.WithShutdownGracePeriod(30*time.Second), the grace period is 30s and the drain is still 5s.
• NewClient. Its requests are HTTP/2. Its handshake fails against a certificate from another CA. It works next to an App started with humustest.Start.
• A taken address fails Init. Two Services on ":8443" in one App exit 1 with EADDRINUSE, and the first one's listener is closed by its OnStop function: a dial to :8443 is then refused.
• A listener is closed when Init fails after NewService. An Init that fails after building the Service exits 1, and a dial to its address is refused.
• Port 0. A Service on ":0" reports the port the Host picked through Addr(), and a request to it succeeds.
• The settings convention (#631). Each check accepts an in-range value with nil and rejects an out-of-range one without naming the field. NewService names each bad field, and reports two bad fields together. DefaultSettings() with a TLS config passes NewService. The package never reads a bedrock.Environment and doesn't import bedrock/config.
• No globals. With OTel's global providers set to recorders of their own, a request records nothing in them.

The package's example is a package main with an app_test.go, or an Example function, as the charter requires, and both packages are held to the 80% coverage gate. A story documents the package on the docs site, in a new page, "Serving HTTP", next to the lifecycle and configuration pages. It covers TLS, HTTP2Only, Main and Defaults, and httpservertest. The same story updates the lifecycle and testing pages. Where they say this proposal decides the names, the durations and whether httpservertest exists, they will state the decisions and link to the new page.

Migration from today

Unchanged, except for these rows, plus a new row on HTTP/2:

Before the reset	With this proposal
rest.Run(ctx, opts...), its server half	httpserver.Main(Init, opts...), with the end user's Init returning the Service from httpserver.NewService. Its REST half moves to Proposal: REST endpoint DSL and OpenAPI (wave 4).
rest.Port(...), HUMUS_REST_PORT=8443	Settings.Addr, default ":8443", sourced from whatever variable the end user names
rest.TLSConfig(...), HUMUS_REST_TLS_PKCS12_FILE and _PASSWORD	Settings.TLS, required, a *tls.Config the end user decodes from a file read through s.Host.FS. PKCS#12 decoding is the end user's, with go-pkcs12 if they like.
a self-signed certificate when no TLS is configured	exit 78 when the end user's certificate variable isn't set; a config with no certificate, or one HTTP/2 can't use, fails CheckTLS
whatever bedrock's HTTP runtime and net/http's defaults negotiated	HTTP/2 always on and first in ALPN, with HTTP/1.1 as a fallback that Settings.HTTP2Only turns off

Wording fixes to the post

• The note at the top now reads: wave 3, building on Proposal: humus.Main as well as the four wave-1 proposals it names. It blocks two wave-4 proposals, Proposal: REST endpoint DSL and OpenAPI, which is built on this package, and Proposal: exposing health probes, which serves probes over HTTP with it. Proposal: failure injection (wave 5) strips scenarios at this server.
• In Background, "What wave 1 provides" also lists humus.Main: humus.Main, the Drain, and humustest.Start.
• Out of scope's first bullet ends "(wave 4)", and its second bullet reads "Serving health probes, and whether an HTTP service exposes them by default: Proposal: exposing health probes (wave 4)."
• Out of scope's "Authentication and authorization (the charter's "REST security", later)" reads "Authentication and authorization: Proposal: authentication and authorization (wave 5)."
• Out of scope's "The outbound HTTP client (the charter's "Later" list)" reads "The outbound HTTP client: Proposal: outbound HTTP clients (wave 4)."
• Out of scope gains a bullet: "Plain HTTP and unencrypted HTTP/2 (h2c). TLS is required."
• References gain humus.Main's revision 2, the charter's revision 7, http.Protocols, and RFC 9113, section 9.2, on TLS for HTTP/2.

Open questions

None.

[Zaba505]

Revision 2: the address is required

The first revision gave Addr a default of ":8443". This revision drops it. Addr is required, so humus imposes no network default: the end user picks the port in every service, as they pick the certificate.

• Addr has no default. DefaultSettings().Addr is "", and CheckAddr rejects it, since it has no port. The end user's chain for Addr has no .Default, so a variable that isn't set fails as not found, with exit 78.
• CheckAddr is otherwise unchanged. It accepts what Host.Listen accepts: an empty host for every name, and port 0 for a free port. A service that wants a free port sets ":0" explicitly.
• Addr and TLS are the two fields with no default, so DefaultSettings() alone fails NewService, naming both.
• DefaultSettings() stays. The timeouts, MaxHeaderBytes and HTTP2Only still have defaults, and it supplies them.

Each section below replaces the part of the same name in the first revision. Everything else in it stands.

Summary

The bullet on TLS becomes:

• TLS and the address are required. Neither Settings.TLS nor Settings.Addr has a default, so the Service imposes no port and serves nothing in plain text.

Proposed API

Settings' doc and its Addr field become:

// Settings configures a Service. Every field is a plain value, and a field's
// zero value means zero, never "use the default": start from DefaultSettings
// and change the fields you source. Addr and TLS have no default, so they are
// always among them.
type Settings struct {
	// Addr is the TCP address the Service listens on through the Host, as
	// host:port. It is required: humus imposes no port. An empty host listens
	// on every address. Port 0 picks a free port, which Service.Addr reports.
	//
	// Default: none. Valid: host:port, with a port from 0 to 65535.
	// Check: CheckAddr.
	Addr string

	// TLS, HTTP2Only, ReadHeaderTimeout, ReadTimeout, WriteTimeout,
	// IdleTimeout and MaxHeaderBytes are unchanged.
}

Settings

The table's Addr row becomes:

Field	Default	Valid	Check	Before the reset
Addr	none: required	host:port, port 0 to 65535	CheckAddr	HUMUS_REST_PORT, default 8443

The bullet "TLS is required, and has no default" becomes:

• Addr and TLS are required, and have no default. DefaultSettings().Addr is "" and DefaultSettings().TLS is nil. CheckAddr rejects the first, since it has no port, and CheckTLS rejects the second. So DefaultSettings() alone doesn't pass NewService. The end user's chains for them have no .Default, so a variable that isn't set fails as not found, with exit 78, and config.Struct reports both at once.

In the end user's Init, the Addr line becomes:

			// No .Default: Addr is required, so an unset ADDR fails as not found.
			Addr: r.Get(env("ADDR").Validate(httpserver.CheckAddr)),

Design decisions

The bullet "TLS is required, on ":8443"" becomes "TLS is required", and loses its last sentence. This bullet is new:

• The address is required. A default port is a network decision made for every service that doesn't override it. A pod's container port, its Service's target port and its probes are set in the manifest, so the address the process listens on belongs with them, and the end user sources it from the same place. A service that forgets it fails Init with exit 78, naming the variable, rather than listening on a port nothing routes to. That reasoning is the same as for TLS: a default the end user didn't choose starts a service that may not work.

Testing

The sketches set the address explicitly. In serves and in the contract test, the settings gain a line after httpserver.DefaultSettings():

		settings.Addr = ":8443"

The whole-App sketch is otherwise unchanged: it still waits on h.Listening("localhost:8443").

The end user's Init, run with no environment, now reports both missing variables:

func TestInit_without_an_address_or_a_certificate_exits_78(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		_, status := httpservertest.Start(t, Init)

		st := <-status
		require.Equal(t, humus.ExitConfig, st.Code())
		require.ErrorContains(t, st.Cause(), "config ADDR: not found")
		require.ErrorContains(t, st.Cause(), "config TLS_PEM_FILE: not found")
	})
}

Two bullets change:

• TLS settings. DefaultSettings() alone fails NewService, naming Addr and TLS, and passes once both are set. The rest of the bullet is unchanged.
• The settings convention. DefaultSettings() with an address and a TLS config passes NewService. CheckAddr rejects "". The rest of the bullet is unchanged.

Migration from today

The rest.Port row becomes:

Before the reset	With this proposal
rest.Port(...), HUMUS_REST_PORT=8443	Settings.Addr, required, sourced from whatever variable the end user names

Open questions

None.

[Zaba505]

Closing: the HTTP server foundation has landed

The revisions' stories are done:

• story(httpserver): configure the HTTP server with Settings and a check per field #706 started httpserver with Settings and a check per field (story(httpserver): configure the HTTP server with Settings and a check per field #723);
• story(httpserver): serve an http.Handler over TLS as a humus Service #707 served an http.Handler over TLS as a humus Service (story(httpserver): serve an http.Handler over TLS as a humus Service #724);
• story(httpserver): trace and measure every request with the Setup's telemetry #708 traced and measured every request with the Setup's telemetry (story(httpserver): trace and measure every request with the Setup's telemetry #725);
• story(httpserver): record a handler's panic and abort its response #709 recorded a handler's panic and aborted its response (story(httpserver): record a handler's panic and abort its response #733);
• story(httpserver): run an HTTP service on its defaults with httpserver.Main #710 added httpserver.Main, which runs an HTTP service on its defaults (story(httpserver): run an HTTP service on its defaults with httpserver.Main #734);
• story(httpservertest): test an HTTP service on its defaults, with an HTTP/2 client #711 added httpservertest, which tests an HTTP service on its defaults with an HTTP/2 client (story(httpservertest): test an HTTP service on its defaults, with an HTTP/2 client #735);
• story(docs): document serving HTTP #712 documented serving HTTP (story(docs): document serving HTTP #736).

Revision 2 left no open questions. What it hands on belongs to the later proposals, as "What waves 4 and 5 build on this" lists: REST endpoint DSL and OpenAPI and exposing health probes in wave 4, and failure injection in wave 5.