Proposal: the humus App lifecycle

[Zaba505]

Note

The Summary, Proposed API, design decisions, testing, migration, open questions and prototype in this post are superseded by the revision below. It applies the review and answers all nine open questions. App.Name and Setup.Logger are removed, and Host.ReadFile becomes a filesystem on *os.File. Health leaves the lifecycle, the drain delay becomes an App field with no humus default, and running past the grace period gets its own cause type. Telemetry's integration with the lifecycle belongs to #632, and humustest's to #633. The rest of this post is unchanged and kept for context.

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model. It depends on the charter and blocks every app-type proposal. It takes over the scope of #624. It builds on the other wave-1 proposals: configuration (#631), telemetry (#632), humustest (#633) and health (#634).

Summary

Every humus app type does the same work around its own:

• read its settings;
• set up telemetry;
• listen for signals;
• build the user's dependencies;
• serve;
• shut down within a grace period;
• release resources;
• turn the outcome into an exit status.

This proposes one small lifecycle, exported from the root package, that every app type plugs into and that can run several services in one process:

func main() {
	bedrock.Main(humus.App{
		Name:      "shoppingcart",
		Host:      humus.OSHost(),
		Telemetry: telemetry.FromEnvironment, // Proposal: OpenTelemetry without globals
		Init:      app.Init,
	})
}

A tested prototype is at the end.

Background

• Each app type has its own lifecycle today, and each depends on removed bedrock APIs:
  • rest.Run wraps a Builder stack in DefaultRunner, RecoverPanics and NotifyOnSignal for os.Interrupt, os.Kill and SIGTERM (rest/rest.go#L201-L224). os.Kill can't be caught, so listing it does nothing.
  • job.App and queue.App are Run(context.Context) error (job/job.go, queue/queue.go).
  • internal/httpserver shuts down with Shutdown(context.Background()), which can hang (internal/httpserver/server.go#L40). internal/grpcserver calls GracefulStop with no bound at all (internal/grpcserver/server.go#L37).
  • grpc has had no lifecycle since story(issue-484): rest migrate to bedrock v0210 #485.
• User dependencies are built outside the App. The shoppingcart main reads DATABASE_URL with os.Getenv and opens a pgx pool before calling rest.Run (example/rest/shoppingcart/main.go#L30). Under the process model nothing reads the real environment, and a test of the whole service can't swap that pool out.
• What bedrock now provides and expects:
  • Main prints nothing. The ExitStatus cause is for tests, so the App writes anything a user should see to p.Stderr.
  • Panics are never recovered.
  • Signals arrive through p.NotifyContext, and bedrocktest drives them in tests.
  • example/httpserver is the reference for a grace period and for returning ExitWithCause(0, context.Cause(ctx)).
  • The network, the filesystem, the clock and randomness are left to the framework.

Proposed API

package humus

// Service is a long-running part of an App: an HTTP server, a gRPC server,
// a queue consumer or a job.
type Service interface {
	// Serve runs the Service until Shutdown stops it, until it fails, or, for
	// a Service that finishes on its own such as a job, until it is done. It
	// returns nil when Shutdown stopped it or it finished. ctx is done once
	// the App starts shutting down.
	Serve(ctx context.Context) error

	// Shutdown stops the Service gracefully. If ctx is done before the
	// Service has stopped, Shutdown stops it at once and returns an error
	// that wraps ctx's error.
	Shutdown(ctx context.Context) error
}

// Host is what humus hands an App beyond its process: the network and the
// filesystem. Every field is required.
type Host struct {
	Listen   func(network, address string) (net.Listener, error)                    // like net.Listen
	Dial     func(ctx context.Context, network, address string) (net.Conn, error)   // like (*net.Dialer).DialContext
	ReadFile func(name string) ([]byte, error)                                      // like os.ReadFile
}

// OSHost returns the Host of the current OS process.
func OSHost() Host

// App is a bedrock.App that runs Services with humus's lifecycle. Every
// field is required.
type App struct {
	Name      string            // names the App in logs and telemetry
	Host      Host              // OSHost() in main; humustest's in-memory Host in tests
	Telemetry telemetry.Builder // Proposal: OpenTelemetry without globals
	Init      func(ctx context.Context, s *Setup) ([]Service, error)
}

func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus

// Setup is what Init is given.
type Setup struct {
	Process   bedrock.Process
	Host      Host
	Telemetry telemetry.Telemetry
	Logger    *slog.Logger
	// unexported: OnStop functions and health state
}

// OnStop registers f to run once every Service has stopped, or once Init
// has failed. Functions run in reverse order of registration, like deferred
// calls, within the grace period.
func (s *Setup) OnStop(f func(context.Context) error)

// Liveness and Readiness add monitors to the App's probes, and Probes
// returns them; see Proposal: health monitors and probes.
func (s *Setup) Liveness(m health.Monitor)
func (s *Setup) Readiness(m health.Monitor)
func (s *Setup) Probes() health.Probes

const (
	ExitFailure = 1  // Init, a Service, its shutdown or an OnStop function failed
	ExitConfig  = 78 // a setting is missing or malformed: EX_CONFIG from sysexits.h
)

// Settings is the lifecycle's configuration; see Proposal: configuration for humus apps.
type Settings struct {
	ShutdownGracePeriod time.Duration
	ShutdownDrainDelay  time.Duration
}

func ReadSettings(ctx context.Context, env bedrock.Environment) (Settings, error)

Each app-type package provides Services rather than an App of its own. Their constructors take a *humus.Setup, for example rest.NewService(ctx, s, api), and are proposed in the wave-2 discussions. health is imported by the root package, so it takes explicit parts instead.

What Run does, in order

1. Read the lifecycle Settings from p.Environment. A malformed value exits 78.
2. Build telemetry with App.Telemetry. A malformed OTEL_* value exits 78, and any other failure exits 1.
3. Call ctx, stop := p.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM), before Init, so a signal during a slow Init (for example, retrying a database connection) cancels it.
4. Call Init(ctx, setup). If it fails, run the OnStop functions registered so far, shut telemetry down, and exit 78 when a *config.Error is in the error chain, or 1 otherwise.
5. Call Serve on every Service concurrently, and mark the App ready.
6. Wait for a signal, or for the first Service to return.
7. Call stop(). From now on a second interrupt or SIGTERM takes its default action and ends the process at once.
8. Mark the App not ready, and wait ShutdownDrainDelay.
9. Call Shutdown on every Service concurrently, within ShutdownGracePeriod.
10. Run the OnStop functions in reverse order, within the same grace period.
11. Shut telemetry down with its own timeout, so the spans of a slow shutdown are still exported.
12. Return 0 with the signal as the cause, or with no cause when a Service finished on its own. Otherwise return 1 with every error joined.

Every failure is written to p.Stderr, one line per error, and is also the cause of the status.

Settings

Variable	Default	Meaning
HUMUS_SHUTDOWN_GRACE_PERIOD	10s	bounds every Service's Shutdown and the OnStop functions
HUMUS_SHUTDOWN_DRAIN_DELAY	0s	time between readiness turning false and Shutdown starting

Exit codes

Code	When
0	a signal stopped the App and it shut down in time, or a Service such as a job finished
1	Init, a Service, its shutdown (including running past the grace period) or an OnStop function failed
2	not set by humus: Go's exit status for an unrecovered panic, and bedrock's idiom for a flag usage error
78	a setting is missing or malformed, including config values read by Init

shoppingcart under this proposal

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	url, err := config.Env(s.Process.Environment, "DATABASE_URL").Read(ctx)
	if err != nil {
		return nil, err // exit 78; stderr: init: config DATABASE_URL: not found
	}
	cfg, err := pgxpool.ParseConfig(url)
	if err != nil {
		return nil, err
	}
	cfg.ConnConfig.DialFunc = s.Host.Dial
	pool, err := pgxpool.NewWithConfig(ctx, cfg)
	if err != nil {
		return nil, err
	}
	s.OnStop(func(context.Context) error { pool.Close(); return nil })
	s.Readiness(health.MonitorFunc(func(ctx context.Context) (bool, error) {
		return pool.Ping(ctx) == nil, nil
	}))

	api, err := rest.NewService(ctx, s, app.API(database.New(pool))) // Proposal: REST endpoint DSL and OpenAPI
	if err != nil {
		return nil, err
	}
	return []humus.Service{api}, nil
}

humus can only document one leak in this example: pgxpool.ParseConfig reads PGHOST, PGPASSWORD and the other PG* variables from the real environment, and PGPASSFILE from disk. A dependency that reads the real process behind an App's back belongs on the "ambient leaks" list in humus's documentation, as it does in bedrock's.

Design decisions

• One exported lifecycle in the root package. A process that serves REST and gRPC, or a Kafka consumer next to a health endpoint, needs the Services to share one lifecycle. A per-package App would have each package re-implement it, and would make composing them impossible.
• Services, not Apps, per package. Serve/Shutdown is the shape of http.Server and of gRPC's Serve/GracefulStop. A Service returning from Serve ends the App, so a job is just a Service that returns.
• Init owns the user's dependencies. OnStop works like t.Cleanup: functions run in reverse order, and also run when Init fails partway through.
• Host is the second boundary. It holds what bedrock leaves to the framework and a test must be able to swap out: listening, dialling and reading files. Dial is included for the OTLP exporter, for user clients such as pgx's DialFunc, and for the later outbound clients. The clock comes from synctest and randomness from cryptotest, so neither needs a field.
• Every field is required. This follows bedrock's preconditions: no nil defaults, so main states every seam it uses.
• Signals are interrupt and SIGTERM. os.Kill is dropped because it can't be caught. Calling stop() as shutdown starts means a second signal ends the process at once, which bedrock documents for NotifyContext.
• The lifecycle never recovers a panic. This is bedrock's rule. Recovering a panic inside one request is a protocol decision: net/http already recovers handler panics, while grpc-go does not. The HTTP and gRPC proposals decide it.
• Exit codes are a small fixed table. 78 (EX_CONFIG) separates "fix the configuration" from "the service failed", for operators and for negative tests. errors.AsType[*config.Error] classifies config errors that Init returns, too.
• Telemetry shuts down last, with its own timeout. That way telemetry still records the shutdown itself.

Testing

Every humus App runs whole under synctest, on bedrocktest's Process and humustest's Host. This test is from the prototype below and passes under -race:

func TestApp_SIGTERMLetsAnInFlightRequestFinish(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost()
		entered := make(chan struct{})
		app := humus.App{
			Name: "hello",
			Host: host.Host(),
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				mux := http.NewServeMux()
				mux.HandleFunc("GET /slow", func(w http.ResponseWriter, r *http.Request) {
					close(entered)
					time.Sleep(3 * time.Second)
					_, _ = io.WriteString(w, "done")
				})
				svc, err := newHTTPService(s, ":8080", mux)
				if err != nil {
					return nil, err
				}
				return []humus.Service{svc}, nil
			},
		}

		p, status := start(t, app)
		synctest.Wait() // the App is listening

		body := get(host.HTTPClient(), "http://hello:8080/slow")
		<-entered
		p.Signal(syscall.SIGTERM)

		require.Equal(t, "done", <-body)
		st := <-status
		require.Equal(t, 0, st.Code(), p.Stderr())
		require.ErrorIs(t, st.Cause(), context.Canceled)
		require.EqualError(t, st.Cause(), "terminated signal received")
	})
}

The prototype also shows the following:

• A 2s grace period ends shutdown after exactly 2s of fake time, with exit 1 and context.DeadlineExceeded in the cause.
• A malformed HUMUS_SHUTDOWN_GRACE_PERIOD exits 78 with a cause that names it, and never calls Init.
• A missing DATABASE_URL read in Init exits 78 and runs OnStop in reverse order.
• A finished job ends the App with 0 and closes the other Services' listeners.
• A failed job exits 1 with its error on stderr.

One thing to know when writing such tests: synctest reports a deadlock when a goroutine the App started outlives the test body. One example is a handler that ignores its request context after the server force-closes the connection. That turns synctest into a leak detector for Services, at the cost of handlers having to honour r.Context().

Migration from today

• rest.Run(ctx, opts...) becomes humus.App plus a REST Service (Proposal: REST endpoint DSL and OpenAPI).
• job.App and job.Handler become a job Service (Proposal: jobs).
• queue.App and queue.Runtime become a queue Service (Proposal: queue processing and the Kafka runtime).
• humus.Logger(name) becomes s.Telemetry.Logger(name).
• The shoppingcart main becomes bedrock.Main(humus.App{...}), with the pool built in Init and closed in OnStop.

Open questions

1. Names. Setup, Host and Service, or something else? Runtime and Context are avoided on purpose, because bedrock retired both.
2. Exit codes. Should 78 be used for config errors, or should everything be 1 as in bedrock's example? Should running past the grace period get its own code?
3. Drain delay. Is a default of 0s right, or should it be a few seconds so Kubernetes can propagate endpoint removal?
4. Grace period budgets. One grace period for Shutdown and OnStop together, or separate budgets?
5. A long-running Service returning nil on its own, such as a consumer whose queue ended. Exit 0, or treat it as an error?
6. Flags. An Init that parses s.Process.Args with bedrock's idiom: how does a usage error map to exit 2?
7. Host.Dial. In scope now, or with the outbound clients?
8. humustest.Start(t, app, p). Does it belong in humustest, or in bedrocktest, since it only needs bedrock.App?
9. Linting the Host boundary. Should humus's forbidigo list ban net.Listen, net.Dial and os.ReadFile outside OSHost? That's stricter than bedrock, whose list covers only the process boundary. Moved here from the charter.

Out of scope

• Restarting or supervising a failed Service.
• Protocol details, such as listener addresses, TLS and per-request panics, which belong to the wave-2 proposals.
• Config files.

Prototype

The prototype below runs against bedrock at 131f577. It omits telemetry and health, which have their own proposals, and its logger is a JSON handler on p.Stdout. go test -race -count=3 passes, covering 97.5% of the lifecycle.

humus.go

// Package humus is a prototype of the lifecycle proposed in
// "Proposal: the humus App lifecycle". It is not the humus module.
package humus

import (
	"context"
	"errors"
	"fmt"
	"log/slog"
	"net"
	"os"
	"slices"
	"sync"
	"syscall"
	"time"

	"github.com/z5labs/bedrock"
	"github.com/z5labs/bedrock/config"
)

// Exit codes of an App. The zero ExitStatus is success.
const (
	// ExitFailure is the code of an App whose Init, Service, shutdown or
	// OnStop function failed.
	ExitFailure = 1

	// ExitConfig is the code of an App that failed to start because a
	// setting was missing or malformed. It is EX_CONFIG from sysexits.h.
	ExitConfig = 78
)

// Service is a long-running part of an App, such as an HTTP server, a gRPC
// server, a queue consumer or a job.
type Service interface {
	// Serve runs the Service until Shutdown stops it, until it fails, or,
	// for a Service that finishes on its own such as a job, until it is
	// done. It returns nil when Shutdown stopped it or it finished. ctx is
	// done once the App starts shutting down.
	Serve(ctx context.Context) error

	// Shutdown stops the Service gracefully. If ctx is done before the
	// Service has stopped, Shutdown stops it at once and returns an error
	// that wraps ctx's error.
	Shutdown(ctx context.Context) error
}

// Host is what humus hands an App beyond its process: the network and the
// filesystem. bedrock leaves both to the framework built on it, so humus
// passes them in explicitly, and a whole App runs in a test without
// opening a socket or a file. Every field is required.
type Host struct {
	// Listen opens a listener, like net.Listen.
	Listen func(network, address string) (net.Listener, error)

	// Dial opens a connection, like (*net.Dialer).DialContext.
	Dial func(ctx context.Context, network, address string) (net.Conn, error)

	// ReadFile reads a file, like os.ReadFile.
	ReadFile func(name string) ([]byte, error)
}

// OSHost returns the Host of the current OS process: net.Listen, a
// net.Dialer and os.ReadFile.
func OSHost() Host {
	var d net.Dialer
	return Host{
		Listen:   net.Listen,
		Dial:     d.DialContext,
		ReadFile: os.ReadFile,
	}
}

// App is a bedrock.App that runs Services with humus's lifecycle. Every
// field is required.
type App struct {
	// Name names the App in its logs and telemetry.
	Name string

	// Host is the network and filesystem the App's Services use. main sets
	// it to OSHost(), and a test sets it to humustest's in-memory Host.
	Host Host

	// Init builds the Services the App runs, and the resources they use.
	// ctx is cancelled by an interrupt or SIGTERM, so a slow Init stops
	// when the App is asked to.
	Init func(ctx context.Context, s *Setup) ([]Service, error)
}

// Setup is what an App's Init is given.
type Setup struct {
	// Process is the process the App runs in.
	Process bedrock.Process

	// Host is the App's Host.
	Host Host

	// Logger writes JSON to Process.Stdout. In humus proper it comes from
	// the App's telemetry.
	Logger *slog.Logger

	onStop []func(context.Context) error
}

// OnStop registers f to run once every Service has stopped, or once Init
// has failed. The functions run in the reverse order of registration, like
// deferred calls, within the shutdown grace period.
func (s *Setup) OnStop(f func(context.Context) error) {
	s.onStop = append(s.onStop, f)
}

func (s *Setup) stop(ctx context.Context) error {
	var errs []error
	for _, f := range slices.Backward(s.onStop) {
		errs = append(errs, f(ctx))
	}
	return errors.Join(errs...)
}

// Settings is the lifecycle's configuration.
type Settings struct {
	// ShutdownGracePeriod bounds the shutdown of every Service and the
	// OnStop functions that follow it. HUMUS_SHUTDOWN_GRACE_PERIOD, 10s.
	ShutdownGracePeriod time.Duration
}

// ReadSettings reads the Settings from env. A malformed variable is an
// error naming it, never a fallback to its default.
func ReadSettings(ctx context.Context, env bedrock.Environment) (Settings, error) {
	return config.Struct(func(r *config.Resolver) Settings {
		return Settings{
			ShutdownGracePeriod: r.Get(config.Env(env, "HUMUS_SHUTDOWN_GRACE_PERIOD").Parse(time.ParseDuration).Default(10 * time.Second)),
		}
	}).Read(ctx)
}

// errServiceReturned is the cause of a shutdown started by a Service
// returning from Serve on its own.
var errServiceReturned = errors.New("a service returned")

// Run implements bedrock.App.
//
// It reads its settings, registers for an interrupt and SIGTERM, calls
// Init, and serves every Service until a signal arrives or one of them
// returns. It then shuts every Service down within the grace period, runs
// the OnStop functions, and returns:
//
//   - ExitConfig, when a setting, or a config value read by Init, is
//     missing or malformed;
//   - ExitFailure, when Init, a Service, its shutdown or an OnStop function
//     fails, including a shutdown that outlives the grace period;
//   - 0 otherwise, with the signal that stopped the App as the cause.
//
// Every failure is written to p.Stderr, since bedrock.Main prints nothing,
// and is the cause of the status, for tests. A panic is never recovered.
func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus {
	settings, err := ReadSettings(ctx, p.Environment)
	if err != nil {
		return fail(p, ExitConfig, err)
	}

	log := slog.New(slog.NewJSONHandler(p.Stdout, nil)).With(slog.String("app", a.Name))

	ctx, stop := p.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)
	defer stop()

	s := &Setup{Process: p, Host: a.Host, Logger: log}
	services, err := a.Init(ctx, s)
	if err != nil {
		graceCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), settings.ShutdownGracePeriod)
		defer cancel()

		code := ExitFailure
		if _, ok := errors.AsType[*config.Error](err); ok {
			code = ExitConfig
		}
		return fail(p, code, errors.Join(fmt.Errorf("init: %w", err), s.stop(graceCtx)))
	}

	// serving is cancelled by a signal, or by the first Service to return.
	serving, stopServing := context.WithCancelCause(ctx)
	defer stopServing(nil)

	serveErrs := make([]error, len(services))
	var served sync.WaitGroup
	for i, svc := range services {
		served.Go(func() {
			serveErrs[i] = svc.Serve(serving)
			stopServing(errServiceReturned)
		})
	}
	log.InfoContext(ctx, "serving", slog.Int("services", len(services)))

	<-serving.Done()
	cause := context.Cause(serving)

	// From here on, a second interrupt or SIGTERM takes its default action
	// and ends the process at once.
	stop()
	log.InfoContext(ctx, "shutting down", slog.Any("cause", cause))

	graceCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), settings.ShutdownGracePeriod)
	defer cancel()

	shutdownErrs := make([]error, len(services))
	var shutdown sync.WaitGroup
	for i, svc := range services {
		shutdown.Go(func() {
			shutdownErrs[i] = svc.Shutdown(graceCtx)
		})
	}
	shutdown.Wait()
	served.Wait()

	err = errors.Join(errors.Join(serveErrs...), errors.Join(shutdownErrs...), s.stop(graceCtx))
	if err != nil {
		return fail(p, ExitFailure, err)
	}

	log.InfoContext(ctx, "stopped")
	if errors.Is(cause, errServiceReturned) {
		return bedrock.Exit(0)
	}
	return bedrock.ExitWithCause(0, cause)
}

// fail writes err to the standard error of p and returns a status with
// code and err as its cause.
func fail(p bedrock.Process, code int, err error) bedrock.ExitStatus {
	if _, werr := fmt.Fprintln(p.Stderr, err); werr != nil {
		return bedrock.ExitWithCause(code, errors.Join(err, werr))
	}
	return bedrock.ExitWithCause(code, err)
}

humus_test.go

package humus_test

import (
	"context"
	"errors"
	"fmt"
	"io"
	"log/slog"
	"net/http"
	"syscall"
	"testing"
	"testing/synctest"
	"time"

	"github.com/stretchr/testify/require"
	"github.com/z5labs/bedrock"
	"github.com/z5labs/bedrock/bedrocktest"
	"github.com/z5labs/bedrock/config"
	"github.com/z5labs/humus"
	"github.com/z5labs/humus/humustest"
)

// newHTTPService is the smallest humus.Service over net/http. The real one
// belongs to "Proposal: HTTP server foundation".
func newHTTPService(s *humus.Setup, addr string, h http.Handler) (humus.Service, error) {
	ln, err := s.Host.Listen("tcp", addr)
	if err != nil {
		return nil, err
	}
	srv := &http.Server{
		Handler:  h,
		ErrorLog: slog.NewLogLogger(s.Logger.Handler(), slog.LevelError),
	}
	return serviceFuncs{
		serve: func(context.Context) error {
			if err := srv.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
				return err
			}
			return nil
		},
		shutdown: func(ctx context.Context) error {
			if err := srv.Shutdown(ctx); err != nil {
				return errors.Join(fmt.Errorf("http shutdown: %w", err), srv.Close())
			}
			return nil
		},
	}, nil
}

type serviceFuncs struct {
	serve    func(context.Context) error
	shutdown func(context.Context) error
}

func (s serviceFuncs) Serve(ctx context.Context) error    { return s.serve(ctx) }
func (s serviceFuncs) Shutdown(ctx context.Context) error { return s.shutdown(ctx) }

// job is a Service that finishes on its own.
func job(f func(ctx context.Context) error) humus.Service {
	return serviceFuncs{serve: f, shutdown: func(context.Context) error { return nil }}
}

func start(t *testing.T, app humus.App, opts ...bedrocktest.Option) (*bedrocktest.Process, <-chan bedrock.ExitStatus) {
	p := bedrocktest.NewProcess(t, opts...)
	status := make(chan bedrock.ExitStatus, 1)
	go func() { status <- app.Run(t.Context(), p.Process()) }()
	return p, status
}

func get(c *http.Client, url string) <-chan string {
	res := make(chan string, 1)
	go func() {
		resp, err := c.Get(url)
		if err != nil {
			res <- "error: " + err.Error()
			return
		}
		defer resp.Body.Close()
		b, err := io.ReadAll(resp.Body)
		if err != nil {
			res <- "error: " + err.Error()
			return
		}
		res <- string(b)
	}()
	return res
}

func TestApp_SIGTERMLetsAnInFlightRequestFinish(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost()
		entered := make(chan struct{})
		app := humus.App{
			Name: "hello",
			Host: host.Host(),
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				mux := http.NewServeMux()
				mux.HandleFunc("GET /slow", func(w http.ResponseWriter, r *http.Request) {
					close(entered)
					time.Sleep(3 * time.Second)
					_, _ = io.WriteString(w, "done")
				})
				svc, err := newHTTPService(s, ":8080", mux)
				if err != nil {
					return nil, err
				}
				return []humus.Service{svc}, nil
			},
		}

		p, status := start(t, app)
		synctest.Wait() // the App is listening

		body := get(host.HTTPClient(), "http://hello:8080/slow")
		<-entered
		p.Signal(syscall.SIGTERM)

		require.Equal(t, "done", <-body)
		st := <-status
		require.Equal(t, 0, st.Code(), p.Stderr())
		require.ErrorIs(t, st.Cause(), context.Canceled)
		require.EqualError(t, st.Cause(), "terminated signal received")
		require.Empty(t, p.Stderr())
	})
}

func TestApp_ShutdownIsBoundedByTheGracePeriod(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost()
		entered := make(chan struct{})
		app := humus.App{
			Name: "hello",
			Host: host.Host(),
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				mux := http.NewServeMux()
				mux.HandleFunc("GET /stuck", func(w http.ResponseWriter, r *http.Request) {
					close(entered)
					// Stuck until the server force-closes the connection.
					<-r.Context().Done()
				})
				svc, err := newHTTPService(s, ":8080", mux)
				if err != nil {
					return nil, err
				}
				return []humus.Service{svc}, nil
			},
		}

		p, status := start(t, app, bedrocktest.WithEnv(map[string]string{"HUMUS_SHUTDOWN_GRACE_PERIOD": "2s"}))
		synctest.Wait()

		begin := time.Now()
		body := get(host.HTTPClient(), "http://hello:8080/stuck")
		<-entered
		p.Signal(syscall.SIGTERM)

		st := <-status
		require.Equal(t, humus.ExitFailure, st.Code())
		require.ErrorIs(t, st.Cause(), context.DeadlineExceeded)
		require.Contains(t, p.Stderr(), "http shutdown: context deadline exceeded")
		require.Equal(t, 2*time.Second, time.Since(begin))
		require.Contains(t, <-body, "error:")
	})
}

func TestApp_MalformedSettingFailsStartupAndNamesIt(t *testing.T) {
	t.Parallel()

	called := false
	app := humus.App{
		Name: "hello",
		Host: humustest.NewHost().Host(),
		Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
			called = true
			return nil, nil
		},
	}
	p := bedrocktest.NewProcess(t, bedrocktest.WithEnv(map[string]string{"HUMUS_SHUTDOWN_GRACE_PERIOD": "30"}))

	st := app.Run(t.Context(), p.Process())

	require.Equal(t, humus.ExitConfig, st.Code())
	cerr, ok := errors.AsType[*config.Error](st.Cause())
	require.True(t, ok)
	require.Equal(t, "HUMUS_SHUTDOWN_GRACE_PERIOD", cerr.Name)
	require.Contains(t, p.Stderr(), "config HUMUS_SHUTDOWN_GRACE_PERIOD:")
	require.False(t, called, "Init must not run when a setting is malformed")
}

func TestApp_InitConfigErrorExits78AndRunsOnStop(t *testing.T) {
	t.Parallel()

	var stopped []string
	app := humus.App{
		Name: "carts",
		Host: humustest.NewHost().Host(),
		Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
			s.OnStop(func(context.Context) error { stopped = append(stopped, "first"); return nil })
			s.OnStop(func(context.Context) error { stopped = append(stopped, "second"); return nil })
			_, err := config.Env(s.Process.Environment, "DATABASE_URL").Read(ctx)
			return nil, err
		},
	}
	p := bedrocktest.NewProcess(t)

	st := app.Run(t.Context(), p.Process())

	require.Equal(t, humus.ExitConfig, st.Code())
	require.ErrorIs(t, st.Cause(), config.ErrNotFound)
	require.Contains(t, p.Stderr(), "init: config DATABASE_URL: not found")
	require.Equal(t, []string{"second", "first"}, stopped)
}

func TestApp_AJobEndsTheAppAndStopsTheOtherServices(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost()
		app := humus.App{
			Name: "batch",
			Host: host.Host(),
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				probe, err := newHTTPService(s, ":8081", http.NotFoundHandler())
				if err != nil {
					return nil, err
				}
				work := job(func(ctx context.Context) error {
					time.Sleep(time.Second)
					return nil
				})
				return []humus.Service{probe, work}, nil
			},
		}

		p, status := start(t, app)

		st := <-status
		require.Equal(t, 0, st.Code(), p.Stderr())
		require.NoError(t, st.Cause())

		_, err := host.Dial(t.Context(), "tcp", "localhost:8081")
		require.ErrorIs(t, err, syscall.ECONNREFUSED, "the probe server is closed with the App")
	})
}

func TestApp_AFailedJobExitsWithItsError(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		errBoom := errors.New("boom")
		app := humus.App{
			Name: "batch",
			Host: humustest.NewHost().Host(),
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				return []humus.Service{job(func(context.Context) error { return errBoom })}, nil
			},
		}

		p, status := start(t, app)

		st := <-status
		require.Equal(t, humus.ExitFailure, st.Code())
		require.ErrorIs(t, st.Cause(), errBoom)
		require.Equal(t, "boom\n", p.Stderr())
	})
}

References

• story(humus): adopt the bedrock process-model App across rest, grpc, job and queue #624, closed in favour of this proposal
• bedrock's doc.go: the boundary, panics and "Beyond the boundary"
• bedrock's example/httpserver
• bedrocktest
• sysexits.h, for EX_CONFIG

[Zaba505]

Feedback on the lifecycle. Each point says what should change here and what it changes in the other wave-1 proposals.

1. Remove App.Name

The service name belongs to telemetry, and the lifecycle doesn't need to expose it.

• Name only feeds the default for OTEL_SERVICE_NAME, and the app attribute on the prototype's logger, which repeats the service.name resource attribute. humustest ignores host names, so tests don't depend on it either.
• With no OTEL_SERVICE_NAME set, use the OTel spec's default, unknown_service:<executable name>, with the executable name taken from p.Args[0]. bedrock's Process.Args starts with the program name, so this reads nothing from the real process, and under bedrocktest it is always unknown_service:bedrocktest.
• Proposal: OpenTelemetry without globals #632: Builder and ReadSettings lose their name parameter, and the default in the OTEL_SERVICE_NAME row changes.

2. Replace Host.ReadFile with a full filesystem

ReadFile only reads whole files into memory, and an App can't create a directory or a temp file through it, for example to store what a job downloaded before processing it. Listen and Dial are general primitives; the filesystem part of Host should be one too. Host is meant for user code as well (as Dial is for pgx), so anything it can't do ends up calling os directly.

Standardize on *os.File:

type Host struct {
	Listen func(network, address string) (net.Listener, error)
	Dial   func(ctx context.Context, network, address string) (net.Conn, error)
	FS     FS
}

// FS is the filesystem of a Host. Names are OS paths.
type FS interface {
	Open(name string) (*os.File, error)
	OpenFile(name string, flag int, perm os.FileMode) (*os.File, error)
	CreateTemp(dir, pattern string) (*os.File, error)
	MkdirTemp(dir, pattern string) (string, error)
	Mkdir(name string, perm os.FileMode) error
	MkdirAll(name string, perm os.FileMode) error
	Remove(name string) error
	RemoveAll(name string) error
	Rename(oldname, newname string) error
	Stat(name string) (os.FileInfo, error)
	ReadDir(name string) ([]os.DirEntry, error)

	// DirFS returns an fs.FS view of dir, for fs.WalkDir, fs.Glob,
	// template.ParseFS and the other std helpers.
	DirFS(dir string) fs.FS
}

The method set is a starting point. It follows os and os.Root, so its semantics are already familiar.

• OSHost implements it with plain os calls. It can't be os.OpenRoot("/"): os.Root refuses to follow absolute symlinks even when rooted at /, and fails with path escapes from parent.
• humustest implements it on a real directory from t.TempDir(), confined with os.Root so that .. and symlinks can't leave it. It accepts absolute paths under that directory, so a file's Name() can be passed back in, as in production.
  • The API becomes humustest.NewHost(t, opts...), mirroring bedrocktest.NewProcess(t, opts...).
  • WithFiles writes its files into the directory, and something like host.Path("etc/tls/cert.pem") returns the absolute path to put in a setting.
• Cleanup is the t.Cleanup that t.TempDir() registers. It runs when a test fails or panics, and it fails the test when the removal fails. Nothing in-process survives a SIGKILL or a -timeout crash, so CI should point TMPDIR at a directory scoped to the job. On GitHub Actions, RUNNER_TEMP is emptied at the start and end of every job.
• CreateTemp and MkdirTemp with dir == "" use the FS's temp directory: os.TempDir() for OSHost, and a directory inside the test's directory for humustest.

Why *os.File rather than an in-memory filesystem behind interfaces:

• Libraries that need a real file or a path keep working in tests: SQLite drivers, archive libraries, programs started with os/exec.
• humustest has no fake filesystem to build and keep in line with the real one.
• An *os.File is always an OS handle (type File struct { *file // os specific }), so it can't be faked. That makes this a one-way decision: moving to an in-memory filesystem later would change every signature.

What it costs:

• The testing rule changes. The charter's "Testable whole" principle and Proposal: humustest — whole-service testing on top of bedrocktest #633's rule 2 go from "no real file" to "no file outside the test's own temp directory".
• No fault injection. A test can't make a write fail with a full disk or an I/O error. Code that needs those tests puts an io.Writer seam in its own code.
• File times and fake time. Inside a synctest bubble time.Now() is fake, but file modification times are real. Code that compares ModTime() with time.Now() needs Chtimes in its tests.
• Open question 9 should be revisited against the whole os file API, not only os.ReadFile.

3. Take health out of the lifecycle

Setup.Liveness, Setup.Readiness and Setup.Probes, and the "mark ready" and "mark not ready" steps, tie the lifecycle to health. They suggest that a health endpoint is part of every App, and jobs get methods they never use. The lifecycle doesn't need them:

• Any Service returning ends the App, so an App is never partly up, and a health Service doesn't need the lifecycle to report the state of other Services.
• Listeners are opened in Init, as in the prototype, so connections wait in the backlog until Serve runs. A health Service can report ready once its own Serve starts.
• A health Service reports not ready as soon as its Serve context is done, and keeps answering probes until its own Shutdown (see point 4). grpc-go's health server already has Shutdown() for the gRPC side.
• HUMUS_SHUTDOWN_DRAIN_DELAY stays in the lifecycle as a plain pause before Shutdown. It isn't about probes: Kubernetes removes a terminating pod from its endpoints whether or not it has a readiness probe, and the delay gives that change time to propagate.
• Once the root package no longer imports health, health can take a *humus.Setup like any other runtime package.

The root package doesn't have to be the API most users touch. Today most users only use a runtime package (rest, job, queue), and that can stay true. Each runtime package provides a convenience App that is built on humus.App, not a lifecycle of its own, and that sets its own defaults: a REST App includes a health Service unless it's turned off, and a job App leaves it out. Using humus.App directly, for example to run REST and gRPC in one process, is the advanced path, and the docs should say it needs to be understood before it's used.

• This proposal: the decision "Services, not Apps, per package" becomes "Services for composition, plus a convenience App built on humus.App". Every wave-2 proposal specifies its convenience App and its defaults.
• Proposal: health monitors and probes #634: "The lifecycle owns readiness" is reversed, and the answer to its open question 1 becomes "explicit at the root, a default in the runtime packages".

4. Keep Shutdown, and define the two signals

Shutdown(ctx) carries the grace-period deadline that http.Server.Shutdown and gRPC's graceful stop need, so it stays. A Service does get two signals at different times, though, and the contract should say what each one means:

• Serve's context is done when shutdown starts, before the drain delay.
• A Service may stop on it, as a queue consumer or a job would, or keep serving until Shutdown, as HTTP, gRPC and health would.

In the prototype, the job helper ignores Shutdown and the HTTP Service ignores the context, and nothing says which is right. Point 3 depends on this being written down.

5. Remove Setup.Logger

Logs are a telemetry signal, like traces and metrics, so the logger belongs to Telemetry, not to Setup. s.Telemetry already provides one, so Setup.Logger is a second way to get the same thing.

• Init and the Services get a logger with s.Telemetry.Logger(name), the same way they get a tracer or a meter. In the prototype, newHTTPService builds its ErrorLog from that logger's Handler().
• The lifecycle's own records ("serving", "shutting down", "stopped") come from Telemetry.Logger too, named after the root package like any other instrumentation scope.
• Proposal: OpenTelemetry without globals #632: no change. s.Telemetry.Logger(name) is already its migration path for humus.Logger(name).

The result

type App struct {
	Host      Host
	Telemetry telemetry.Builder
	Init      func(ctx context.Context, s *Setup) ([]Service, error)
}

type Setup struct {
	Process   bedrock.Process
	Host      Host
	Telemetry telemetry.Telemetry
	// unexported: OnStop functions
}

Run loses its readiness steps. After a signal, or a Service returning, it calls stop(), waits ShutdownDrainDelay, calls Shutdown on every Service within ShutdownGracePeriod, runs the OnStop functions, and then shuts telemetry down.

[Zaba505]

Revision: the review and the answers to the open questions

This revision applies the review and the answers to the nine open questions. Each section below replaces the section of the same name in the post. Background, Out of scope and References are unchanged and still apply.

Two integrations are owned by other proposals, and this one doesn't specify them:

• Telemetry, owned by Proposal: OpenTelemetry without globals. It covers App.Telemetry and Setup.Telemetry, building telemetry before Init, logging the lifecycle's records and failures through it, and shutting it down last.
• humustest, owned by Proposal: humustest. The lifecycle's own tests use bedrocktest and fake Services, so they don't wait on humustest.

Decisions

#	Question	Decision
1	Names: Setup, Host and Service?	Keep them.
2	78 for config errors? A separate code for running past the grace period?	Keep 78 for config errors. Running past the grace period exits 1, like any other failed shutdown, but its cause is a *humus.GracePeriodError, so a test can assert on it.
3	Drain delay default?	None in the root. App.DrainDelay is a plain field, and the root reads no variable for it. Each runtime package's App sets its own default.
4	One grace period, or separate budgets?	One. The App shuts down as a whole. Kubernetes kills the container's process when its grace period runs out, however long any one Service needs.
5	A long-running Service returning nil on its own?	Exit 0. It's a successful shutdown.
6	Flag usage errors and exit 2?	humus never exits 2. A usage error is an Init failure, which exits 1 and is logged through telemetry, as every failure is (see #632).
7	Host.Dial now, or with the outbound clients?	Now.
8	humustest.Start or bedrocktest?	humustest, and humustest is self-contained. A test imports humustest, never bedrocktest. #633 owns it.
9	Ban net.Listen, net.Dial and os's file API outside OSHost?	No. Now that FS hands out *os.File, a ban on os's file API would get in the way of using those files.

The proposal has no open questions left.

Summary

Every humus app type does the same work around its own:

• read its settings;
• set up telemetry;
• listen for signals;
• build the user's dependencies;
• serve;
• shut down within a grace period;
• release resources;
• turn the outcome into an exit status.

This proposes one small lifecycle, exported from the root package, that every app type is built on and that can run several services in one process. Setting up telemetry is added to it by Proposal: OpenTelemetry without globals.

func main() {
	bedrock.Main(humus.App{
		Host:       humus.OSHost(),
		DrainDelay: 5 * time.Second, // humus has no default
		Init:       app.Init,
	})
}

Most users never write this. They use a runtime package's App (REST, gRPC, a job, a queue), which is built on humus.App and sets its own defaults.

Proposed API

package humus

// Service is a long-running part of an App: an HTTP server, a gRPC server,
// a queue consumer or a job.
type Service interface {
	// Serve runs the Service until Shutdown stops it, until it fails, or, for
	// a Service that finishes on its own such as a job, until it is done. It
	// returns nil when Shutdown stopped it or it finished.
	//
	// ctx is done as soon as the App starts shutting down, before the drain
	// delay. A Service may stop on it, as a job or a queue consumer does, or
	// keep serving until Shutdown, as an HTTP, gRPC or health Service does.
	Serve(ctx context.Context) error

	// Shutdown stops the Service gracefully. The App calls it on every
	// Service after the drain delay, including one whose Serve has already
	// returned. If ctx is done before the Service has stopped, Shutdown
	// stops it at once and returns an error that wraps ctx's error.
	Shutdown(ctx context.Context) error
}

// Host is what humus hands an App beyond its process: the network and the
// filesystem. Every field is required.
type Host struct {
	Listen func(network, address string) (net.Listener, error)                  // like net.Listen
	Dial   func(ctx context.Context, network, address string) (net.Conn, error) // like (*net.Dialer).DialContext
	FS     FS
}

// FS is the filesystem of a Host. Names are OS paths.
type FS interface {
	Open(name string) (*os.File, error)
	OpenFile(name string, flag int, perm os.FileMode) (*os.File, error)
	CreateTemp(dir, pattern string) (*os.File, error)
	MkdirTemp(dir, pattern string) (string, error)
	Mkdir(name string, perm os.FileMode) error
	MkdirAll(name string, perm os.FileMode) error
	Remove(name string) error
	RemoveAll(name string) error
	Rename(oldname, newname string) error
	Stat(name string) (os.FileInfo, error)
	ReadDir(name string) ([]os.DirEntry, error)

	// DirFS returns an fs.FS view of dir, for fs.WalkDir, fs.Glob,
	// template.ParseFS and the other std helpers.
	DirFS(dir string) fs.FS
}

// OSHost returns the Host of the current OS process.
func OSHost() Host

// App is a bedrock.App that runs Services with humus's lifecycle. Host and
// Init are required.
type App struct {
	Host Host // OSHost() in main; humustest's Host in tests

	// DrainDelay is the pause between shutdown starting and Shutdown being
	// called on the Services. Zero means no pause. humus reads no variable
	// for it and has no default; a runtime package's App sets its own.
	DrainDelay time.Duration

	Init func(ctx context.Context, s *Setup) ([]Service, error)
}

func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus

// Setup is what Init is given.
type Setup struct {
	Process bedrock.Process
	Host    Host
	// unexported: OnStop functions
}

// OnStop registers f to run once every Service has stopped, or once Init
// has failed. Functions run in reverse order of registration, like deferred
// calls, within the grace period.
func (s *Setup) OnStop(f func(context.Context) error)

const (
	ExitFailure = 1  // Init, a Service, its shutdown or an OnStop function failed
	ExitConfig  = 78 // a setting is missing or malformed: EX_CONFIG from sysexits.h
)

// GracePeriodError is the cause of an App whose Services or OnStop
// functions were still running when the grace period ran out. The App
// exits ExitFailure.
type GracePeriodError struct {
	GracePeriod time.Duration // the grace period that ran out
	Err         error         // every error the shutdown returned, joined
}

func (e *GracePeriodError) Error() string
func (e *GracePeriodError) Unwrap() error

// Settings is the lifecycle's configuration; see Proposal: configuration for humus apps.
type Settings struct {
	ShutdownGracePeriod time.Duration
}

func ReadSettings(ctx context.Context, env bedrock.Environment) (Settings, error)

Proposal: OpenTelemetry without globals adds App.Telemetry and Setup.Telemetry, and the steps of Run that build, use and shut down telemetry.

The method set of FS is the one the first stories implement. A later package that needs another method adds it in its own proposal.

Each runtime package provides Services, whose constructors take a *humus.Setup (for example rest.NewService(ctx, s, api)), and an App built on humus.App with its own defaults. Both are proposed in the wave-2 discussions. The root package doesn't import health, so health takes a *humus.Setup like any other runtime package.

What Run does, in order

1. Read the lifecycle Settings from p.Environment. A malformed value exits 78.
2. Call ctx, stop := p.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM), before Init, so a signal during a slow Init (for example, retrying a database connection) cancels it.
3. Call Init(ctx, setup). If it fails, run the OnStop functions registered so far within the grace period, and exit 78 when a *config.Error is in the error chain, or 1 otherwise.
4. Call Serve on every Service concurrently.
5. Wait for a signal, or for the first Service to return.
6. Call stop(), and cancel the context passed to Serve. From now on a second interrupt or SIGTERM takes its default action and ends the process at once.
7. Wait App.DrainDelay.
8. Call Shutdown on every Service concurrently, within ShutdownGracePeriod.
9. Run the OnStop functions in reverse order, within the same grace period.
10. Return 0 with the signal as the cause, or with no cause when a Service returned nil on its own, whether it was a job or a long-running Service such as a consumer whose queue ended. If the grace period ran out, return 1 with a *GracePeriodError. Otherwise return 1 with every error joined.

Every failure is the cause of the status, and is written to p.Stderr, one line per error. Once #632 integrates telemetry, every failure after telemetry is built is logged through it instead. p.Stderr is then used only before telemetry exists, or when shutting it down fails.

Settings

Variable	Default	Meaning
HUMUS_SHUTDOWN_GRACE_PERIOD	10s	bounds every Service's Shutdown and the OnStop functions

The root reads no variable for the drain delay. A runtime package's App may read HUMUS_SHUTDOWN_DRAIN_DELAY with its own default and pass it as App.DrainDelay, and each wave-2 proposal says whether it does.

Exit codes

Code	When	Cause
0	a signal stopped the App and it shut down in time, or a Service returned nil on its own	the signal, or none
1	Init, a Service, its shutdown or an OnStop function failed	every error, joined
1	the shutdown ran past the grace period	*GracePeriodError
2	never set by humus: Go's exit status for an unrecovered panic. A flag usage error is an Init failure and exits 1.
78	a setting is missing or malformed, including config values read by Init	a *config.Error in the chain

shoppingcart under this proposal

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	url, err := config.Env(s.Process.Environment, "DATABASE_URL").Read(ctx)
	if err != nil {
		return nil, err // exit 78: init: config DATABASE_URL: not found
	}
	cfg, err := pgxpool.ParseConfig(url)
	if err != nil {
		return nil, err
	}
	cfg.ConnConfig.DialFunc = s.Host.Dial
	pool, err := pgxpool.NewWithConfig(ctx, cfg)
	if err != nil {
		return nil, err
	}
	s.OnStop(func(context.Context) error { pool.Close(); return nil })

	api, err := rest.NewService(ctx, s, app.API(database.New(pool))) // Proposal: REST endpoint DSL and OpenAPI
	if err != nil {
		return nil, err
	}
	return []humus.Service{api}, nil
}

The readiness monitor on the pool moves to the health Service; see Proposal: health monitors and probes.

humus can only document one leak in this example: pgxpool.ParseConfig reads PGHOST, PGPASSWORD and the other PG* variables from the real environment, and PGPASSFILE from disk. A dependency that reads the real process behind an App's back belongs on the "ambient leaks" list in humus's documentation, as it does in bedrock's.

Design decisions

• One exported lifecycle in the root package. A process that serves REST and gRPC, or a Kafka consumer next to a health endpoint, needs the Services to share one lifecycle. A per-package lifecycle would have each package re-implement it, and would make composing them impossible.
• Services for composition, plus an App per runtime package. Each runtime package provides Services, and an App built on humus.App that sets its own defaults: a REST App includes a health Service unless it's turned off, and a job App leaves it out. Most users only touch a runtime package's App, as they do today. Using humus.App directly, for example to run REST and gRPC in one process, is the advanced path, and the docs say it needs to be understood first. A Service returning from Serve ends the App, so a job is just a Service that returns.
• Telemetry and humustest integrate themselves. Proposal: OpenTelemetry without globals #632 adds telemetry to the lifecycle, and Proposal: humustest — whole-service testing on top of bedrocktest #633 runs it in tests. Neither is in this proposal's stories, so the lifecycle lands first and each integration lands with its own package.
• Init owns the user's dependencies. OnStop works like t.Cleanup: functions run in reverse order, and also run when Init fails partway through.
• Host is the second boundary. It holds what bedrock leaves to the framework and a test must be able to swap out: listening, dialling and the filesystem. Dial is in scope now, for the OTLP exporter, for user clients such as pgx's DialFunc, and for the later outbound clients. The clock comes from synctest and randomness from cryptotest, so neither needs a field.
• The filesystem is *os.File, not an in-memory fake. ReadFile only read whole files into memory, and an App couldn't create a directory or a temp file through it.
  • Libraries that need a real file or a path keep working in tests: SQLite drivers, archive libraries, programs started with os/exec.
  • humustest has no fake filesystem to build and keep in line with the real one.
  • It's a one-way decision. An *os.File is always an OS handle, so moving to an in-memory filesystem later would change every signature.
  • OSHost uses plain os calls. It can't be os.OpenRoot("/"), because os.Root refuses to follow absolute symlinks even when rooted at /.
  • What it costs: the testing rule becomes "no file outside the test's own temp directory"; a test can't make a write fail with a full disk, so code that needs that test puts an io.Writer seam in its own code; and file modification times are real under synctest, so code that compares ModTime() with time.Now() needs Chtimes in its tests.
• No lint rule for the Host boundary. forbidigo doesn't ban net.Listen, net.Dial or os's file API outside OSHost. Now that FS hands out *os.File, user code works with os types directly, and a ban would get in its way. That matches bedrock, whose list covers only the process boundary.
• Health is not part of the lifecycle. Readiness steps would tie every App, jobs included, to a health endpoint, and the lifecycle doesn't need them:
  • Any Service returning ends the App, so an App is never partly up.
  • Listeners are opened in Init, so connections wait in the backlog until Serve runs. A health Service reports ready once its own Serve starts, reports not ready once its Serve context is done, and keeps answering probes until its own Shutdown.
• Two shutdown signals, at different times. Serve's context is done when shutdown starts, before the drain delay, and Shutdown is called after it, with the grace period as its deadline. A Service may stop on the first, as a job or a consumer does, or keep serving until the second, as HTTP, gRPC and health do.
• No drain-delay default in the root. Whether to pause depends on the app type. Kubernetes removes a terminating pod from its endpoints whether or not it has a readiness probe, and a REST or gRPC App needs time for that to propagate; a job doesn't. A runtime package's App picks its default. A direct humus.App sets the field in main, and can't change it from the environment.
• One grace period for the whole shutdown. Shutdown and the OnStop functions share one budget, because the App shuts down as a whole. A platform like Kubernetes kills the container's process when its own grace period runs out, not one Service at a time.
• Every field is required, except DrainDelay. This follows bedrock's preconditions: no nil defaults, so main states every seam it uses. DrainDelay's zero value means no pause.
• No App.Name, and no Setup.Logger. The name only fed the default for OTEL_SERVICE_NAME, and logs are a telemetry signal like traces and metrics. Both belong to Proposal: OpenTelemetry without globals #632.
• Failures go to telemetry once it exists. Logging every failure through telemetry is Proposal: OpenTelemetry without globals #632's integration. Before it, and in this proposal's stories, failures are written to p.Stderr. Tests assert on the cause, not on output, so they hold either way.
• Signals are interrupt and SIGTERM. os.Kill is dropped because it can't be caught. Calling stop() as shutdown starts means a second signal ends the process at once, which bedrock documents for NotifyContext.
• The lifecycle never recovers a panic. This is bedrock's rule. Recovering a panic inside one request is a protocol decision: net/http already recovers handler panics, while grpc-go does not. The HTTP and gRPC proposals decide it.
• Few exit codes, typed causes. 78 (EX_CONFIG) separates "fix the configuration" from "the service failed", for operators. Everything else that fails exits 1, and a test tells failures apart by their cause: errors.AsType[*config.Error] for a config error, errors.AsType[*humus.GracePeriodError] for a shutdown that ran out of time. humus never exits 2; a flag usage error is an Init failure.

Testing

The lifecycle runs whole under synctest on bedrocktest's Process, with fake Services, so its tests need neither the network nor humustest. Running whole Apps on humustest's Host is Proposal: humustest's integration. The prototype's grace-period test becomes:

func TestApp_ShutdownIsBoundedByTheGracePeriod(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		// stuck serves until shutdown starts, then never finishes shutting down.
		stuck := serviceFuncs{
			serve: func(ctx context.Context) error {
				<-ctx.Done()
				return nil
			},
			shutdown: func(ctx context.Context) error {
				<-ctx.Done()
				return fmt.Errorf("stuck shutdown: %w", ctx.Err())
			},
		}
		app := humus.App{
			Host: humus.OSHost(), // the fake Service never uses it
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				return []humus.Service{stuck}, nil
			},
		}

		p := bedrocktest.NewProcess(t, bedrocktest.WithEnv(map[string]string{"HUMUS_SHUTDOWN_GRACE_PERIOD": "2s"}))
		status := make(chan bedrock.ExitStatus, 1)
		go func() { status <- app.Run(t.Context(), p.Process()) }()

		<-p.Notified(syscall.SIGTERM)
		begin := time.Now()
		p.Signal(syscall.SIGTERM)

		st := <-status
		require.Equal(t, humus.ExitFailure, st.Code())
		gerr, ok := errors.AsType[*humus.GracePeriodError](st.Cause())
		require.True(t, ok)
		require.Equal(t, 2*time.Second, gerr.GracePeriod)
		require.ErrorIs(t, gerr, context.DeadlineExceeded)
		require.Equal(t, 2*time.Second, time.Since(begin))
		require.Contains(t, p.Stderr(), "stuck shutdown: context deadline exceeded")
	})
}

The lifecycle's tests show the following:

• SIGTERM cancels Serve's context and calls Shutdown, and the App exits 0 with the signal as the cause.
• A 2s grace period ends shutdown after exactly 2s of fake time, with exit 1 and a *GracePeriodError cause.
• A DrainDelay of 3s calls Shutdown exactly 3s after the signal, and Serve's context is already done by then.
• A malformed HUMUS_SHUTDOWN_GRACE_PERIOD exits 78 with a *config.Error that names it, and never calls Init.
• A missing DATABASE_URL read in Init exits 78 and runs OnStop in reverse order.
• A Service returning nil on its own, whether a job or a long-running one, ends the App with 0 and no cause, and Shutdown is called on every other Service.
• A failed job exits 1 with its error as the cause.

One thing to know when writing such tests: synctest reports a deadlock when a goroutine the App started outlives the test body. That turns synctest into a leak detector for Services. For an HTTP Service, the cost is that handlers have to honour r.Context(), because a handler that ignores it after the server force-closes the connection outlives the test.

Migration from today

• rest.Run(ctx, opts...) becomes the REST package's App, built on humus.App (Proposal: REST endpoint DSL and OpenAPI).
• job.App and job.Handler become a job Service and a job App (Proposal: jobs).
• queue.App and queue.Runtime become a queue Service and a queue App (Proposal: queue processing and the Kafka runtime).
• humus.Logger(name) becomes s.Telemetry.Logger(name) (Proposal: OpenTelemetry without globals #632).
• os.ReadFile and the rest of os's file API in an App become s.Host.FS.
• The shoppingcart main becomes bedrock.Main on the REST package's App, with the pool built in Init and closed in OnStop.

Open questions

None. The answers are in the Decisions table above.

Changes to the other proposals

These follow from the decisions above, and each is posted as a revision on its own proposal.

• Charter (Proposal: humus on the bedrock process model #629): the "Testable whole" principle changes from "no real file" to "no file outside the test's own temp directory". The engineering rules gain no ban on the Host boundary.
• Configuration (Proposal: configuration for humus apps #631): a setting that names a file is read through the Host's FS, so httpserver.ReadSettings takes an FS instead of a readFile function.
• Telemetry (Proposal: OpenTelemetry without globals #632):
  • It owns telemetry's integration with the lifecycle: App.Telemetry, Setup.Telemetry, building telemetry before Init, logging the lifecycle's records and failures through it, and shutting it down last.
  • Builder, FromEnvironment, NoopBuilder and ReadSettings lose their name parameter. OTEL_SERVICE_NAME defaults to unknown_service:<executable name>, taken from p.Args[0].
• humustest (Proposal: humustest — whole-service testing on top of bedrocktest #633):
  • It owns humustest's integration with the lifecycle.
  • NewHost(t, opts...) serves files from a directory made by t.TempDir() and confined with os.Root, and host.Path(name) returns the absolute path to put in a setting.
  • Start takes humustest's own options and returns humustest's own process handle, so a test never imports bedrocktest.
  • TestService also checks that calling Shutdown after Serve has returned is safe.
• Health (Proposal: health monitors and probes #634): "the lifecycle owns readiness" is reversed. Health is a Service that takes a *humus.Setup, and its open question 1 is answered "explicit at the root, a default in the runtime packages".
• Wave 2: every app-type proposal specifies its App: its defaults, whether it includes a health Service, and its drain delay.

Prototype

The prototype in the post predates this revision. Its shutdown sequence still holds, but it has App.Name, Host.ReadFile and Setup.Logger, and has no drain delay or *GracePeriodError. The lifecycle story implements this revision rather than copying the prototype.

[Zaba505]

Correction: decision 8 is reversed by #633

Decision 8 in the revision said that Start lives in humustest, and that humustest is self-contained, so a test never imports bedrocktest. Proposal: humustest's second revision reverses that:

• Start lives in bedrocktest (story(bedrocktest): Start runs an App on a new Process and returns its status bedrock#600), since it needs nothing but bedrock.App.
• humustest builds on bedrocktest; it does not wrap it. A whole-App test imports bedrocktest for the process and humustest for the Host.

The last item under "Changes to the other proposals → humustest (#633)" changes the same way. Start takes bedrocktest's options and returns *bedrocktest.Process and a status channel.

This proposal's stories are unaffected. #637 and #638 already test with bedrocktest and fake Services. The whole-App tests on humustest's Host are #645.