Proposal: humus.Main

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model. The charter's revision 5 makes this proposal wave 2 on its own, and moves the four runtime proposals to wave 3 so that they can build on it. It builds on Proposal: the humus App lifecycle, Proposal: OpenTelemetry without globals and Proposal: humustest.

Summary

Every runtime package was about to ship its own copy of the same App. This proposal replaces all four with one entry point in the root package:

• humus.Main(init, opts...) runs an App on the OS process. Only Init is required. The Host, the telemetry Builder, the grace period and the drain all have defaults, and functional options override them.
• One set of defaults fits every runtime. The drain becomes a function, humus.Drain, which ends early once no Service is still serving. So a job pays nothing for the drain that an API needs, and the runtime packages need no defaults of their own. They export Services, and the end user's Init returns them.
• humustest.Start(t, init, opts...) runs the App that Main would run, on a test Host, with the same options. humustest.NewApp builds that App for a test that drives bedrocktest itself.

func main() {
	humus.Main(app.Init, humus.WithTelemetry(buildTelemetry))
}

func TestExport(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		tel := humustest.NewTelemetry()
		_, status := humustest.Start(t, app.Init, humus.WithTelemetry(tel.Builder()))
		require.Zero(t, (<-status).Code())
	})
}

Background

• Every wave-2 runtime proposal wraps humus.App the same way. Each one has an App with Host, Telemetry and an Init, whose Run builds a humus.App with the runtime's defaults:

  Proposal	Its App	Drain delay	Grace period
  HTTP server foundation	httpserver.App	5s	10s
  gRPC services	humusgrpc.App	5s	15s
  jobs	job.App	0	20s
  queue processing and the Kafka runtime	kafka.App	0	20s

  Their Inits differ only in what they return. A process that runs two of them, such as a job next to a metrics server, can't use either App and falls back to humus.App.

• The drain delay is why the defaults differ. humus.App.Run sleeps the drain delay whenever an App with Services shuts down, including when a Service returned on its own (app.go). An API needs that sleep: Kubernetes removes a terminating pod from a Service's endpoints asynchronously, and an API that stops accepting connections at once refuses requests that are still being routed to it. A job or a queue consumer has nothing routed to it, and with a 5s drain delay every job would wait 5s after it finished. So each runtime picked its own drain delay, and an App type to carry it.

• main and every whole-App test repeat the same literal. Today main writes bedrock.Main(humus.App{Host: humus.OSHost(), ShutdownGracePeriod: …, DrainDelay: …, Telemetry: buildTelemetry, Init: Init}), and a test writes the same literal with a humustest Host, passed to bedrocktest.Start. Nothing but Init differs from one service to the next.

• Kubernetes' budget is 30s. A pod gets SIGTERM, then SIGKILL after terminationGracePeriodSeconds, which defaults to 30s. The drain, the grace period and telemetry's shutdown (telemetry.DefaultSettings().ShutdownTimeout is 5s) all have to fit inside it.

Proposed API

package humus // github.com/z5labs/humus

// Main runs the App that init and opts describe on the OS process, and exits
// the process with the App's status. It never returns.
//
// The App starts from Main's defaults, and opts change them in order:
//
//   - Host: OSHost().
//   - Telemetry: a Builder that calls telemetry.New with
//     telemetry.DefaultSettings(), the Host's Dial and the Process's stdout.
//   - ShutdownGracePeriod: 15s.
//   - Drain: DrainDelay(5 * time.Second).
//
// HUMUS_SHUTDOWN_GRACE_PERIOD and HUMUS_SHUTDOWN_DRAIN_DELAY override the
// grace period and the drain, as for any App. Main is bedrock.Main of that
// App. It panics if init is nil.
func Main(init func(ctx context.Context, s *Setup) ([]Service, error), opts ...Option)

// Option changes a field of the App that Main or humustest builds.
type Option func(*App)

// WithHost sets App.Host.
func WithHost(h Host) Option

// WithTelemetry sets App.Telemetry.
func WithTelemetry(b telemetry.Builder) Option

// WithShutdownGracePeriod sets App.ShutdownGracePeriod, the grace period
// when HUMUS_SHUTDOWN_GRACE_PERIOD is not set.
func WithShutdownGracePeriod(d time.Duration) Option

// WithDrain sets App.Drain, the drain when HUMUS_SHUTDOWN_DRAIN_DELAY is not
// set.
func WithDrain(d Drain) Option

// Drain waits, once the App has started shutting down, until it is safe to
// stop the Services: until load balancers have stopped sending the App new
// work. Run calls it after Serve's context is done and before it calls
// Shutdown on the Services, with the Setup that Init was given.
//
// Run calls it only when a Service is still serving once shutdown starts.
// ctx is done once every Service's Serve has returned, because nothing is
// left to drain, and a Drain returns when it is done or when ctx is done,
// whichever comes first. Run waits for it to return, and the grace period
// starts after it. An error it returns is joined into the App's cause and
// the App exits ExitFailure, and Shutdown is still called on every Service.
type Drain func(ctx context.Context, s *Setup) error

// DrainDelay returns a Drain that waits d, or until ctx is done. It panics
// if d is negative.
func DrainDelay(d time.Duration) Drain

type App struct {
	// Host, ShutdownGracePeriod, Telemetry and Init are unchanged.

	// Drain is the App's drain when the environment does not set
	// HUMUS_SHUTDOWN_DRAIN_DELAY. A nil Drain doesn't wait. It replaces
	// DrainDelay.
	Drain Drain
}

package humustest // github.com/z5labs/humus/humustest

// NewApp returns the App that humus.Main would run for init and opts, with a
// new Host of this package's in place of humus.OSHost(). opts are applied
// after that, so humus.WithHost replaces it. Call NewApp inside the
// testing/synctest bubble that runs the App, since the Host is built there.
// It panics if init is nil.
func NewApp(t testing.TB, init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humus.Option) humus.App

// Start runs NewApp(t, init, opts...) with bedrocktest.Start, on a new
// Process with no arguments and an empty environment.
func Start(t testing.TB, init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humus.Option) (*bedrocktest.Process, <-chan bedrock.ExitStatus)

humus.App stays as it is, apart from Drain. It is what an Option changes, what humustest.NewApp returns, and what humustest.TestService runs. A main that wants no defaults at all can still write bedrock.Main(humus.App{…}).

A service, end to end

A runtime package exports its Service, and the end user's Init returns it. A job, with job.NewService as Proposal: jobs proposes it:

func main() {
	humus.Main(Init, humus.WithTelemetry(buildTelemetry))
}

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	pool, err := openPool(ctx, s) // through s.Host.Dial
	if err != nil {
		return nil, err
	}
	s.OnStop(func(context.Context) error { pool.Close(); return nil })

	e := &exporter{pool: pool}
	return []humus.Service{job.NewService(s, e.Run)}, nil
}

An API's Init returns its server's Service in the same way, with whatever constructor the HTTP server foundation settles on. A process that runs a job next to a metrics server returns both, with nothing else to change.

Its test passes the test's own Host and telemetry recorder as options, and runs with the same defaults as production:

func TestInit_exports_every_row(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost(t)
		tel := humustest.NewTelemetry()
		_, status := humustest.Start(t, Init,
			humus.WithHost(host.Host()),
			humus.WithTelemetry(tel.Builder()),
		)

		st := <-status
		require.Zero(t, st.Code())
		require.Equal(t, "job", tel.Spans()[0].Name())
	})
}

A test that needs the environment, arguments or stdin passes the App to bedrocktest itself:

p, status := bedrocktest.Start(t, humustest.NewApp(t, Init, humus.WithTelemetry(tel.Builder())),
	bedrocktest.WithEnv(map[string]string{"EXPORT_BUCKET": "nightly"}),
)

The drain

The drain is the pause between shutdown starting and Shutdown being called on the Services. It becomes a function, so that what it waits for is the end user's choice, and so that it can stop waiting once nothing is left to drain.

• It ends early once nothing is serving. Run cancels the Drain's context once every Service's Serve has returned. A Service that stops on Serve's context, such as a job, a queue consumer or the health Service, returns at once or once its work in flight is done. A Service that keeps serving until Shutdown, such as an HTTP or gRPC server, doesn't return until after the drain. So:

  The App's Services	The drain lasts
  a job, finished on its own	nothing: Run doesn't call the Drain
  a job, interrupted by a signal	until the run returns, at most the Drain's delay
  a queue consumer	until its records in flight are done, at most the Drain's delay
  an HTTP or gRPC server, with or without the health Service	the Drain's whole delay
  a job and an HTTP server	the Drain's whole delay, unless both return first

  This generalises a rule the lifecycle already has: an Init that returns no Services ends the App without a drain, because there is nothing to drain.

• DrainDelay(d) is the default kind of drain. It waits d, or until nothing is serving. Main's default is DrainDelay(5 * time.Second).

• A Drain of the end user's own waits for something else. For example, one that polls a load balancer's API until the pod's target is draining, and goes through s.Host.Dial so that it runs on a test Host too:

  humus.Main(Init, humus.WithDrain(func(ctx context.Context, s *humus.Setup) error {
  	return waitUntilDeregistered(ctx, s.Host.Dial, targetID(s.Process))
  }))

• It gets the Setup, not what Init built. The Drain is chosen in main, before Init runs. Work at shutdown that needs Init's dependencies, such as deregistering from service discovery, belongs in a Service that stops on Serve's context. That Service sees shutdown start at the same moment, and the Drain then waits for whatever has to follow it, such as clients' cached lookups expiring.

• HUMUS_SHUTDOWN_DRAIN_DELAY replaces the Drain. When it is set, Run uses DrainDelay of its value in place of App.Drain, whatever that is. It is still a time.ParseDuration string that must not be negative, and a bad value still exits 78 before Init. An operator can always replace a Drain that misbehaves, from the same manifest that sets terminationGracePeriodSeconds.

The defaults

	Main	Why
Host	OSHost()	Main runs only in main. A test builds its App with humustest, which uses a test Host.
Telemetry	telemetry.New with DefaultSettings(), writing to the Process's stdout	Logs go to stdout as JSON lines, and spans get real IDs without being exported, as the telemetry proposal set out. A service that exports anything passes its own Builder.
ShutdownGracePeriod	15s	Every Service's Shutdown and every OnStop function share it.
Drain	DrainDelay(5 * time.Second)	Kubernetes' endpoint removal reaches every node within a few seconds, as the HTTP and gRPC proposals found.

The worst case is 5s of drain, 15s of grace period and 5s of telemetry shutdown: 25s, inside Kubernetes' default 30s, with a margin for the process to exit. For each runtime:

• HTTP and gRPC get 5s of drain, as they proposed. HTTP's grace period goes from 10s to 15s.
• A job or a queue consumer has up to 20s to stop after SIGTERM: the drain lasts until its Serve returns, at most 5s, and the grace period follows. That is the 20s that the jobs and Kafka proposals chose, and it costs nothing when the work stops at once.

Changes to Run

humus.App.Run changes in two of its steps. Everything else in it is unchanged.

1. Step 1 reads the drain from HUMUS_SHUTDOWN_DRAIN_DELAY. When the variable is set, the drain is DrainDelay of its value. When it is not, the drain is App.Drain, and a nil App.Drain doesn't wait.
2. Step 9 calls the Drain instead of sleeping. If any Serve is still running, Run calls the Drain with the Setup, and with a context that keeps the values of Run's context, is not cancelled with it, and is done once every Serve has returned. It collects the Serve results that arrive in the meantime, waits for the Drain to return, and then shuts the Services down within the grace period. A Drain error is logged at ERROR and joined into the cause as drain: and the error.

As today, neither a cancelled root context nor a second signal cuts the drain short: the second signal takes its default action and ends the process.

Design decisions

• One entry point, not one App per runtime. The runtime packages' Apps differed only in their defaults. With a drain that ends once nothing is serving, one set of defaults fits all four, so the runtime packages export Services and nothing else. An App that mixes runtimes needs nothing new.
• Option is func(*App). humustest.NewApp applies the same options that main passes to Main, so a test runs the App production runs. An option changes an exported field, so an end user can write their own as a one-line function literal.
• The defaults live in Main, not in App. humus.App's fields keep their meaning: a zero ShutdownGracePeriod fails startup, and a nil Drain doesn't wait, as the configuration proposal's "zero means zero" asks. Main is the one place with opinions. Both Main and humustest read the defaults from an internal package, so neither has to export them.
• The Host defaults to OSHost() only in Main. humus.App requires a Host so that a test can't run on the real host by mistake. Main never runs in a test, and humustest.NewApp starts from a test Host, so the default can't leak into one.
• The telemetry default is DefaultSettings(). A service that forgets its Builder logs to stdout and exports no traces. It doesn't fail. A team that exports anything already writes a Builder to name the service and point it at a collector.
• The drain is a function; the grace period stays a duration. What the drain waits for depends on what sends the App traffic, so it varies. The grace period is a bound on the shutdown, and nothing but its length varies.
• The Drain isn't bounded by the grace period. The grace period starts after it, as it did after the drain delay. A Drain that ignores its context holds the App up until a second signal, or Kubernetes' SIGKILL, ends the process. HUMUS_SHUTDOWN_DRAIN_DELAY replaces it without a code change.
• A Drain error fails the App, but not the shutdown. It is joined into the cause, as an OnStop error is, so the exit status shows it. Shutdown is still called, because the Services have to stop either way.
• humustest builds on bedrocktest, as humustest's second revision decided. Start returns bedrocktest's Process and status channel, so a test that sends signals or reads stdout doesn't import bedrocktest. A test that sets the environment, arguments or stdin passes NewApp's App to bedrocktest.Start with bedrocktest's options. See open question 1.
• humustest.NewApp keeps Main's telemetry default. A whole-App test that passes no Builder sees on p.Stdout() the JSON lines production would write. The default builds no exporter that dials out, and runs inside synctest. A test that asserts on spans or metrics passes humustest.NewTelemetry().Builder().
• No humus.NewApp. humustest.NewApp covers tests, and Main covers main. A program that wants an App value of its own writes the humus.App literal.

Testing

These tests are sketches and have not been run. They use humustest's Host and bedrocktest's process, inside synctest, so every duration below is fake time.

The drain ends once nothing is serving:

func TestApp_the_drain_ends_once_every_Serve_has_returned(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		start := time.Now()
		var shutdownAt time.Duration
		svc := &fakeService{
			serve: func(ctx context.Context) error {
				<-ctx.Done()
				time.Sleep(2 * time.Second) // finishing the work in flight
				return nil
			},
			shutdown: func(context.Context) error {
				shutdownAt = time.Since(start)
				return nil
			},
		}

		p, status := humustest.Start(t, func(context.Context, *humus.Setup) ([]humus.Service, error) {
			return []humus.Service{svc}, nil
		})
		synctest.Wait()
		p.Signal(syscall.SIGTERM)

		require.Zero(t, (<-status).Code())
		require.Equal(t, 2*time.Second, shutdownAt) // not the default 5s
	})
}

A Service that keeps serving until Shutdown gets the whole drain:

func TestApp_a_Service_that_serves_until_Shutdown_gets_the_whole_drain(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		start := time.Now()
		var shutdownAt time.Duration
		stopped := make(chan struct{})
		svc := &fakeService{
			serve: func(context.Context) error { <-stopped; return nil },
			shutdown: func(context.Context) error {
				shutdownAt = time.Since(start)
				close(stopped)
				return nil
			},
		}

		p, status := humustest.Start(t, func(context.Context, *humus.Setup) ([]humus.Service, error) {
			return []humus.Service{svc}, nil
		})
		synctest.Wait()
		p.Signal(syscall.SIGTERM)

		require.Zero(t, (<-status).Code())
		require.Equal(t, 5*time.Second, shutdownAt)
	})
}

The tests also show the following:

• A Service that returns on its own, as a finished job does, ends the App at the same fake instant: the Drain is never called.
• HUMUS_SHUTDOWN_DRAIN_DELAY=1s replaces a custom Drain, which is never called, and HUMUS_SHUTDOWN_DRAIN_DELAY=-1s exits 78 before Init.
• A Drain that returns an error exits 1, with drain: and the error in the cause, logged at ERROR, and Shutdown is still called on every Service.
• A Drain is given the Setup that Init was given.
• A nil App.Drain doesn't wait: Shutdown is called at the same fake instant as the signal.
• DrainDelay with a negative duration panics, and so do Main and humustest.NewApp with a nil init.
• humustest.NewApp with no options returns an App with a 15s grace period, a 5s DrainDelay, a humustest Host and Main's telemetry default, and its first log line appears on p.Stdout() as JSON.
• humustest.NewApp with humus.WithHost and humus.WithTelemetry uses the Host and Builder it was given.
• The App that Main runs has OSHost(), the 15s grace period and the 5s DrainDelay, checked by an internal test of the function Main builds it with, since Main itself exits the process.

Changes to the other proposals

This proposal changes the four runtime proposals, which the charter's revision 5 moves to wave 3. No correction is posted on their discussions. Each one revises itself when it is taken up again:

• HTTP server foundation, gRPC services, jobs and queue processing and the Kafka runtime lose their App types. Each package exports its Service, and the end user's Init returns it. The work their Apps did in Init, such as building the HTTP Service from the Settings and handler Init returned, moves into the end user's Init. Their default durations are replaced by Main's.
• Jobs (Proposal: jobs #682): open question 1, whether job.App requires a Builder, is answered by Main's telemetry default.
• Exposing health probes (now wave 4) can't add a field to a runtime App, since there are none. It decides how an App's probes are exposed in terms of Main, its options and the Services the end user's Init returns.

This proposal's own stories change the root package, humustest, their tests, and the docs pages for the lifecycle, configuration, telemetry and testing.

Migration from today

• bedrock.Main(humus.App{Host: humus.OSHost(), ShutdownGracePeriod: 30 * time.Second, DrainDelay: 5 * time.Second, Telemetry: buildTelemetry, Init: Init}) becomes humus.Main(Init, humus.WithTelemetry(buildTelemetry), humus.WithShutdownGracePeriod(30*time.Second)).
• App.DrainDelay: d becomes App.Drain: humus.DrainDelay(d), or humus.WithDrain(humus.DrainDelay(d)) with Main.
• bedrocktest.Start(t, humus.App{Host: host.Host(), Telemetry: tel.Builder(), ShutdownGracePeriod: …, Init: Init}) becomes humustest.Start(t, Init, humus.WithHost(host.Host()), humus.WithTelemetry(tel.Builder())), which also runs with production's grace period and drain.
• An App whose Services all stop on Serve's context now shuts down as soon as they have returned, instead of after the whole drain delay.
• HUMUS_SHUTDOWN_DRAIN_DELAY keeps its name and format. When it is set, it now replaces any Drain, not just a duration.

Open questions

1. Should humustest.Start take bedrocktest's options? As proposed, Start takes humus's options only, and a test that sets the environment, arguments or stdin writes bedrocktest.Start(t, humustest.NewApp(t, Init, opts...), bedrocktest.WithEnv(env)). Settings tests do that often. The alternative is an option type of humustest's own, covering both humus's options and the process's, which re-wraps bedrocktest's options and goes against humustest building on bedrocktest without wrapping it. Recommendation: as proposed.
2. Are 15s and 5s the right defaults? The other choice is a 20s grace period, which is what the jobs and Kafka proposals chose. With the 5s drain and 5s of telemetry, it uses all of Kubernetes' 30s and leaves no margin for the process to exit. Either way an operator overrides both with the lifecycle's variables. Recommendation: 15s and 5s.

Out of scope

• Flags. Main passes the process's arguments to Init in Setup.Process, as today. Parsing them is the end user's.
• Health probes. Exposing health probes decides how probes reach an orchestrator, now in terms of Main and the end user's Services.
• A drain that watches traffic, such as one that waits until no new connection has arrived for a second. It needs a count from the HTTP or gRPC Service, which those proposals would add. A Drain of the end user's own could then use it.

References

• Proposal: humus on the bedrock process model, the charter, and its wave plan
• Proposal: the humus App lifecycle: the drain delay, the grace period and their variables
• Proposal: configuration for humus apps: zero means zero
• Proposal: OpenTelemetry without globals: DefaultSettings(), and the choice each runtime App was to make about its Builder
• Proposal: humustest's second revision: humustest builds on bedrocktest
• The wave-2 runtime proposals: #680, #681, #682, #683
• bedrock v0.23.1's Main and bedrocktest.Start
• Kubernetes: pod termination

[Zaba505]

Revision: the open questions answered

This revision answers both open questions:

1. humustest.Start takes the process's options too. A test sets the environment, arguments and stdin through humustest, in the same call as humus's options, so it never imports bedrocktest. Both kinds of option go in one list, so humustest has an Option type of its own, which carries humus's options with WithApp. humustest.NewApp is no longer exported, since Start now covers every test that needed it.
2. Main defaults the grace period, and has no drain. As the post found, not every runtime needs a drain. A job, a queue consumer and the health Service stop on Serve's context, and only a Service that keeps serving until Shutdown, such as an HTTP or gRPC server, needs one. A runtime package whose Services need a drain builds a Main of its own on humus.Main, with its own defaults, in wave 3.

Each section below replaces the section of the same name in the post, except The drain, which changes in one bullet. Background, Changes to Run, Out of scope and References are unchanged and still apply.

Summary

Every runtime package was about to ship its own copy of the same App. This proposal replaces them with one entry point in the root package:

• humus.Main(init, opts...) runs an App on the OS process. Only Init is required. The Host, the telemetry Builder and the grace period have defaults, and functional options override them. There is no drain by default.
• The drain becomes a function, humus.Drain, which ends early once no Service is still serving. A runtime package whose Services need a drain builds a Main of its own on humus.Main, with a drain among its defaults. The runtime packages export Services, a Main where they need other defaults, and no App.
• humustest.Start(t, init, opts...) runs the App that Main would run, on a test Host. It takes humus's options and the process's options in one call.

func main() {
	humus.Main(app.Init, humus.WithTelemetry(buildTelemetry))
}

func TestExport(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		tel := humustest.NewTelemetry()
		_, status := humustest.Start(t, app.Init,
			humustest.WithApp(humus.WithTelemetry(tel.Builder())),
			humustest.WithEnv(map[string]string{"EXPORT_BUCKET": "nightly"}),
		)
		require.Zero(t, (<-status).Code())
	})
}

Proposed API

package humus // github.com/z5labs/humus

// Main runs the App that init and opts describe on the OS process, and exits
// the process with the App's status. It never returns.
//
// The App starts from Main's defaults, and opts change them in order:
//
//   - Host: OSHost().
//   - Telemetry: a Builder that calls telemetry.New with
//     telemetry.DefaultSettings(), the Host's Dial and the Process's stdout.
//   - ShutdownGracePeriod: 20s.
//   - Drain: none.
//
// HUMUS_SHUTDOWN_GRACE_PERIOD and HUMUS_SHUTDOWN_DRAIN_DELAY override the
// grace period and the drain, as for any App. Main is bedrock.Main of that
// App. It panics if init is nil.
func Main(init func(ctx context.Context, s *Setup) ([]Service, error), opts ...Option)

// Option changes fields of the App that Main or humustest builds.
type Option func(*App)

// WithHost sets App.Host.
func WithHost(h Host) Option

// WithTelemetry sets App.Telemetry.
func WithTelemetry(b telemetry.Builder) Option

// WithShutdownGracePeriod sets App.ShutdownGracePeriod, the grace period
// when HUMUS_SHUTDOWN_GRACE_PERIOD is not set.
func WithShutdownGracePeriod(d time.Duration) Option

// WithDrain sets App.Drain, the drain when HUMUS_SHUTDOWN_DRAIN_DELAY is not
// set.
func WithDrain(d Drain) Option

// Drain waits, once the App has started shutting down, until it is safe to
// stop the Services: until load balancers have stopped sending the App new
// work. Run calls it after Serve's context is done and before it calls
// Shutdown on the Services, with the Setup that Init was given.
//
// Run calls it only when a Service is still serving once shutdown starts.
// ctx is done once every Service's Serve has returned, because nothing is
// left to drain, and a Drain returns when it is done or when ctx is done,
// whichever comes first. Run waits for it to return, and the grace period
// starts after it. An error it returns is joined into the App's cause and
// the App exits ExitFailure, and Shutdown is still called on every Service.
type Drain func(ctx context.Context, s *Setup) error

// DrainDelay returns a Drain that waits d, or until ctx is done. It panics
// if d is negative.
func DrainDelay(d time.Duration) Drain

type App struct {
	// Host, ShutdownGracePeriod, Telemetry and Init are unchanged.

	// Drain is the App's drain when the environment does not set
	// HUMUS_SHUTDOWN_DRAIN_DELAY. A nil Drain doesn't wait. It replaces
	// DrainDelay.
	Drain Drain
}

package humustest // github.com/z5labs/humus/humustest

// Option changes the App that Start runs, or the Process it runs on.
type Option func(*startOptions)

// WithApp applies opts to the App, in order, after Start's defaults.
func WithApp(opts ...humus.Option) Option

// WithEnv, WithArgs and WithStdin set the Process's environment, its
// arguments after the program name, and its standard input. They work like
// bedrocktest's options of the same names.
func WithEnv(values map[string]string) Option
func WithArgs(args ...string) Option
func WithStdin(r io.Reader) Option

// Start runs the App that humus.Main would run for init with
// bedrocktest.Start, on a new Process. The App has a new Host of this
// package's in place of humus.OSHost(), and opts change the App and the
// Process: humus.WithHost, passed with WithApp, replaces that Host. Call
// Start inside the testing/synctest bubble that runs the App, since the Host
// is built there. It panics if init is nil.
func Start(t testing.TB, init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...Option) (*bedrocktest.Process, <-chan bedrock.ExitStatus)

humus.App stays as it is, apart from Drain. It is what an Option changes, what Start runs, and what humustest.TestService runs. A main that wants no defaults at all can still write bedrock.Main(humus.App{…}).

A service, end to end

A runtime package exports its Service, and the end user's Init returns it. A job, with job.NewService as Proposal: jobs proposes it, runs on humus.Main's defaults:

func main() {
	humus.Main(Init, humus.WithTelemetry(buildTelemetry))
}

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	bucket, err := config.Env(s.Process.Environment, "EXPORT_BUCKET").Read(ctx)
	if err != nil {
		return nil, err
	}
	pool, err := openPool(ctx, s) // through s.Host.Dial
	if err != nil {
		return nil, err
	}
	s.OnStop(func(context.Context) error { pool.Close(); return nil })

	e := &exporter{pool: pool, bucket: bucket}
	return []humus.Service{job.NewService(s, e.Run)}, nil
}

Its test passes the test's own Host and telemetry recorder, and the environment Init reads, in one call. It runs with the same defaults as production:

func TestInit_exports_every_row(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost(t)
		tel := humustest.NewTelemetry()
		_, status := humustest.Start(t, Init,
			humustest.WithApp(humus.WithHost(host.Host()), humus.WithTelemetry(tel.Builder())),
			humustest.WithEnv(map[string]string{"EXPORT_BUCKET": "nightly"}),
		)

		st := <-status
		require.Zero(t, st.Code())
		require.Equal(t, "job", tel.Spans()[0].Name())
	})
}

A runtime package whose Services need a drain builds a Main of its own, and exports its defaults as one humus.Option, so that a test runs with them too. As a sketch of the pattern, with the HTTP server foundation's proposed durations; that proposal decides the names and values:

package httpserver

// Defaults sets an HTTP service's defaults over humus.Main's: a 5s drain
// and a 10s grace period.
func Defaults() humus.Option {
	return func(a *humus.App) {
		a.Drain = humus.DrainDelay(5 * time.Second)
		a.ShutdownGracePeriod = 10 * time.Second
	}
}

// Main is humus.Main with Defaults applied before opts.
func Main(init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humus.Option) {
	humus.Main(init, append([]humus.Option{Defaults()}, opts...)...)
}

The end user's main calls httpserver.Main(Init, humus.WithTelemetry(buildTelemetry)), and its test calls humustest.Start(t, Init, humustest.WithApp(httpserver.Defaults())). A process that runs a job next to an HTTP server returns both Services from its Init, and uses httpserver.Main, because its HTTP server needs the drain.

The drain

Unchanged, except that Main has no drain. Its second bullet becomes:

• DrainDelay(d) is the usual kind of drain. It waits d, or until nothing is serving. Main has none, and a runtime package that needs one sets it in its own Main, as humus.WithDrain(humus.DrainDelay(5 * time.Second)) or as a field its defaults Option sets.

The defaults

	Main	Why
Host	OSHost()	Main runs only in main. A test runs its App with humustest.Start, which uses a test Host.
Telemetry	telemetry.New with DefaultSettings(), writing to the Process's stdout	Logs go to stdout as JSON lines, and spans get real IDs without being exported, as the telemetry proposal set out. A service that exports anything passes its own Builder.
ShutdownGracePeriod	20s	It fits inside Kubernetes' default terminationGracePeriodSeconds of 30s, with telemetry's 5s shutdown after it and a margin for the process to exit.
Drain	none	Only a Service that keeps serving until Shutdown needs a drain, and the package that provides it sets one.

The worst case is 20s of grace period and 5s of telemetry shutdown: 25s. For each runtime:

• A job or a queue consumer runs on humus.Main's defaults, which are the ones the jobs proposal and the Kafka proposal chose: no drain and a 20s grace period. Neither package needs a Main of its own.
• An HTTP or gRPC server needs a drain, so its package builds a Main of its own. A 5s drain takes 5s from the budget, so its grace period drops to fit: 5s, 10s and 5s for the HTTP server foundation's proposed defaults, and 5s, 15s and 5s for gRPC's.

Design decisions

• One entry point, not one App per runtime. The runtime packages' Apps differed only in their defaults. humus.Main takes the end user's Init, so the Services of any runtime run on it, and an App that mixes runtimes needs nothing new. A runtime package that needs other defaults wraps Main instead of defining an App.
• Main has no drain. A drain is for a Service that a load balancer routes to and that keeps serving until Shutdown. A job, a queue consumer and the health Service stop on Serve's context, so a drain there would only make the App wait. The packages whose Services need one set it.
• A runtime package's defaults are one humus.Option. An Option can set several fields, so the package's Main applies its defaults before the end user's options, and a test passes the same Option to humustest.Start. The end user's options come last, so they win over the package's.
• Option is func(*App). humustest.Start applies the same options that main passes to Main, so a test runs the App that production runs. An option changes exported fields, so an end user or a runtime package can write their own as a function literal.
• The defaults live in Main, not in App. humus.App's fields keep their meaning: a zero ShutdownGracePeriod fails startup, and a nil Drain doesn't wait, as the configuration proposal's "zero means zero" asks. Main is the one place with opinions. Both Main and humustest read the defaults from an internal package, so neither has to export them.
• The Host defaults to OSHost() only in Main. humus.App requires a Host so that a test can't run on the real host by mistake. Main never runs in a test, and humustest.Start starts from a test Host, so the default can't leak into one.
• The telemetry default is DefaultSettings(). A service that forgets its Builder logs to stdout and exports no traces. It doesn't fail. A team that exports anything already writes a Builder to name the service and point it at a collector.
• The drain is a function, and the grace period stays a duration. What the drain waits for depends on what sends the App traffic, so it varies. The grace period is a bound on the shutdown, and nothing but its length varies.
• The Drain isn't bounded by the grace period. The grace period starts after it, as it did after the drain delay. A Drain that ignores its context holds the App up until a second signal, or Kubernetes' SIGKILL, ends the process. HUMUS_SHUTDOWN_DRAIN_DELAY replaces it without a code change.
• A Drain error fails the App, but not the shutdown. It is joined into the cause, as an OnStop error is, so the exit status shows it. Shutdown is still called, because the Services have to stop either way.
• A test imports humustest, not bedrocktest. Start takes humus's options and the process's in one list, so humustest's Option carries both. WithEnv, WithArgs and WithStdin mirror bedrocktest's options. This reverses, for Start, humustest's second revision, under which humustest didn't wrap bedrocktest. Start still returns bedrocktest's Process and status channel unwrapped, and a test calls their methods without importing bedrocktest.
• WithApp carries humus's options, rather than humustest repeating them. A test passes the options main passes, and any runtime package's defaults, unchanged. humustest has no WithHost or WithTelemetry of its own.
• Start keeps Main's telemetry default. A whole-App test that passes no Builder sees on p.Stdout() the JSON lines production would write. The default builds no exporter that dials out, and runs inside synctest. A test that asserts on spans or metrics passes humustest.NewTelemetry().Builder().
• No exported NewApp, in either package. humustest.Start covers tests, and Main covers main. A program that wants an App value of its own writes the humus.App literal.

Testing

These tests are sketches and have not been run. They use humustest's Host and bedrocktest's process, inside synctest, so every duration below is fake time.

The drain ends once nothing is serving:

func TestApp_the_drain_ends_once_every_Serve_has_returned(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		start := time.Now()
		var shutdownAt time.Duration
		svc := &fakeService{
			serve: func(ctx context.Context) error {
				<-ctx.Done()
				time.Sleep(2 * time.Second) // finishing the work in flight
				return nil
			},
			shutdown: func(context.Context) error {
				shutdownAt = time.Since(start)
				return nil
			},
		}

		p, status := humustest.Start(t,
			func(context.Context, *humus.Setup) ([]humus.Service, error) {
				return []humus.Service{svc}, nil
			},
			humustest.WithApp(humus.WithDrain(humus.DrainDelay(5*time.Second))),
		)
		synctest.Wait()
		p.Signal(syscall.SIGTERM)

		require.Zero(t, (<-status).Code())
		require.Equal(t, 2*time.Second, shutdownAt) // not the drain's 5s
	})
}

A Service that keeps serving until Shutdown gets the whole drain:

func TestApp_a_Service_that_serves_until_Shutdown_gets_the_whole_drain(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		start := time.Now()
		var shutdownAt time.Duration
		stopped := make(chan struct{})
		svc := &fakeService{
			serve: func(context.Context) error { <-stopped; return nil },
			shutdown: func(context.Context) error {
				shutdownAt = time.Since(start)
				close(stopped)
				return nil
			},
		}

		p, status := humustest.Start(t,
			func(context.Context, *humus.Setup) ([]humus.Service, error) {
				return []humus.Service{svc}, nil
			},
			humustest.WithApp(humus.WithDrain(humus.DrainDelay(5*time.Second))),
		)
		synctest.Wait()
		p.Signal(syscall.SIGTERM)

		require.Zero(t, (<-status).Code())
		require.Equal(t, 5*time.Second, shutdownAt)
	})
}

The tests also show the following:

• With Start's defaults, a Service that keeps serving until Shutdown is shut down at the same fake instant as the signal: there is no drain.
• A Service that returns on its own, as a finished job does, ends the App at the same fake instant: the Drain is never called.
• HUMUS_SHUTDOWN_DRAIN_DELAY=1s, set with humustest.WithEnv, replaces a custom Drain, which is never called, and adds a drain to an App that has none. HUMUS_SHUTDOWN_DRAIN_DELAY=-1s exits 78 before Init.
• A Drain that returns an error exits 1, with drain: and the error in the cause, logged at ERROR, and Shutdown is still called on every Service.
• A Drain is given the Setup that Init was given.
• DrainDelay with a negative duration panics, and so do Main and humustest.Start with a nil init.
• humustest.Start with no options runs an App with a 20s grace period, no drain, a humustest Host and Main's telemetry default, and its first log line appears on p.Stdout() as JSON.
• humustest.WithApp(humus.WithHost(h), humus.WithTelemetry(b)) uses h and b. WithApp given twice applies both, in order, and a later option wins over an earlier one.
• humustest.WithEnv, WithArgs and WithStdin reach the App as Setup.Process's environment, Args[1:] and stdin.
• The App that Main runs has OSHost(), the 20s grace period and no drain. An internal test checks the function Main builds it with, since Main itself exits the process.

Changes to the other proposals

This proposal changes the four runtime proposals, which the charter's revision 5 moves to wave 3. No correction is posted on their discussions. Each one revises itself when it is taken up again:

• All four lose their App types. Each package exports its Service, and the end user's Init returns it. The work their Apps did in Init, such as building the HTTP Service from the Settings and handler Init returned, moves into the end user's Init.
• Jobs and queue processing and the Kafka runtime run on humus.Main's defaults, which match the ones they chose, and need no Main of their own. Jobs' open question 1, whether job.App requires a Builder, is answered by Main's telemetry default.
• HTTP server foundation and gRPC services each add a Main built on humus.Main and export their defaults as one humus.Option, with a drain and a grace period that fit Kubernetes' 30s together.
• Exposing health probes (now wave 4) can't add a field to a runtime App, since there are none. It decides how an App's probes are exposed in terms of humus.Main, the runtime packages' Mains, their options and the Services the end user's Init returns.

The charter's revision 5 is updated to match this revision. This proposal's own stories change the root package, humustest, their tests, and the docs pages for the lifecycle, configuration, telemetry and testing.

Migration from today

• bedrock.Main(humus.App{Host: humus.OSHost(), ShutdownGracePeriod: 30 * time.Second, DrainDelay: 5 * time.Second, Telemetry: buildTelemetry, Init: Init}) becomes humus.Main(Init, humus.WithTelemetry(buildTelemetry), humus.WithShutdownGracePeriod(30*time.Second), humus.WithDrain(humus.DrainDelay(5*time.Second))).
• App.DrainDelay: d becomes App.Drain: humus.DrainDelay(d). An App that set no DrainDelay sets no Drain.
• bedrocktest.Start(t, humus.App{Host: host.Host(), Telemetry: tel.Builder(), ShutdownGracePeriod: …, Init: Init}, bedrocktest.WithEnv(env)) becomes humustest.Start(t, Init, humustest.WithApp(humus.WithHost(host.Host()), humus.WithTelemetry(tel.Builder())), humustest.WithEnv(env)), which also runs with production's grace period.
• An App whose Services all stop on Serve's context now shuts down as soon as they have returned, even with a drain, instead of after the whole drain delay.
• HUMUS_SHUTDOWN_DRAIN_DELAY keeps its name and format. When it is set, it now replaces any Drain, not just a duration.

Open questions

None.

[Zaba505]

Revision 2: the Drain's contract, test packages, and what the integration found

A prototype of the first revision ran both of its sketch tests unchanged. No-App versions of the four wave-3 runtime packages, built on it with an end user's service and whole-App tests for each, found what the first revision left undecided. This revision decides it:

1. A Drain returns nil when its context is done. A done context means nothing is left to drain, so a Drain that returns ctx.Err() then fails the App. Run may call a Drain whose context is done at once, so a Drain checks it before it acts.
2. The Drain's context gives the shutdown's cause. humus.ShutdownCause(ctx) returns the signal's cause after a signal, or an error wrapping humus.ErrServiceReturned when a Service returned first. A Drain that waits on something outside the App, such as a load balancer deregistering the pod, returns at once when no signal came, since nothing outside is draining the App.
3. A runtime package may ship a test package. A package with a Main of its own may also ship a test package whose Start is humustest.Start with the package's defaults, along with helpers such as a client wired to the test Host. The HTTP server foundation and gRPC services proposals decide whether they do. A test that calls humustest.Start directly still passes the package's defaults with WithApp.
4. An Option may wrap App.Init. Main and humustest.Start set App.Init before they apply the options, so a package's Option can add a Service next to the end user's. A runtime package's Main does nothing but apply its defaults before the end user's options and call humus.Main.
5. A process picks its Main by what a load balancer routes to. A process uses an HTTP or gRPC package's Main when that server is one a load balancer routes to. A job or a queue consumer next to a probes or metrics server stays on humus.Main.

One correction to the first revision: answer 2 said "As the post found, not every runtime needs a drain." The post found that the early end already made a drain free for a Service that stops on Serve's context. Main has no drain because a runtime package's defaults are where its drain belongs, not to save the other runtimes a cost.

Each section below replaces the section of the same name, in the first revision where it has one and in the post otherwise. Background, Out of scope and References are unchanged, apart from the wording fixes at the end.

Summary

Every runtime package was about to ship its own copy of the same App. This proposal replaces them with one entry point in the root package:

• humus.Main(init, opts...) runs an App on the OS process. Only Init is required. The Host, the telemetry Builder and the grace period have defaults, and functional options override them. There is no drain by default.
• The drain becomes a function, humus.Drain. Its context is done once no Service is still serving, and it says why the shutdown started. A runtime package whose Services need a drain builds a Main of its own on humus.Main, with a drain among its defaults. The runtime packages export Services, a Main where they need other defaults, and no App.
• humustest.Start(t, init, opts...) runs the App that Main would run, on a test Host. It takes humus's options and the process's options in one call.

func main() {
	humus.Main(app.Init, humus.WithTelemetry(buildTelemetry))
}

func TestExport(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		tel := humustest.NewTelemetry()
		_, status := humustest.Start(t, app.Init,
			humustest.WithApp(humus.WithTelemetry(tel.Builder())),
			humustest.WithEnv(map[string]string{"EXPORT_BUCKET": "nightly"}),
		)
		require.Zero(t, (<-status).Code())
	})
}

Proposed API

Unchanged, apart from these declarations. In the root package, Main's and Drain's docs change, and ShutdownCause and ErrServiceReturned are new:

package humus // github.com/z5labs/humus

// Main runs the App that init and opts describe on the OS process, and exits
// the process with the App's status. It never returns.
//
// The App starts from Main's defaults, and opts change them in order:
//
//   - Host: OSHost().
//   - Telemetry: a Builder that calls telemetry.New with
//     telemetry.DefaultSettings(), the Host's Dial and the Process's stdout.
//   - ShutdownGracePeriod: 20s.
//   - Drain: none.
//
// Init is init, and is set before opts are applied, so an Option sees it
// and may wrap it, for example to add a Service.
//
// HUMUS_SHUTDOWN_GRACE_PERIOD and HUMUS_SHUTDOWN_DRAIN_DELAY override the
// grace period and the drain, as for any App. Main applies every option
// before it runs anything, and is then bedrock.Main of the App. It panics if
// init or one of opts is nil.
func Main(init func(ctx context.Context, s *Setup) ([]Service, error), opts ...Option)

// Drain waits, once the App has started shutting down, until it is safe to
// stop the Services: until load balancers have stopped sending the App new
// work. Run calls it after Serve's context is done and before it calls
// Shutdown on the Services, with the Setup that Init was given.
//
// Run calls it when, as shutdown starts, it hasn't yet received every
// Serve's result. ctx is done once every Service's Serve has returned,
// because nothing is left to drain, and it may be done as soon as Run calls
// the Drain, so a Drain checks it before it acts. A Drain returns when it is done or when
// ctx is done, whichever comes first, and returns nil when ctx is done: one
// that returns ctx.Err() then fails the App.
//
// ShutdownCause(ctx) says why the shutdown started. A Drain that waits on
// something outside the App, such as a load balancer deregistering the
// pod, returns nil when the cause wraps ErrServiceReturned: no signal came,
// so nothing outside is draining the App.
//
// Run waits for the Drain to return, and the grace period starts after it.
// An error it returns is joined into the App's cause and the App exits
// ExitFailure, and Shutdown is still called on every Service. ctx keeps the
// values of Run's context but is not cancelled with it.
type Drain func(ctx context.Context, s *Setup) error

// ErrServiceReturned is in a shutdown's cause when a Service's Serve
// returned before any signal came, and so started the shutdown.
var ErrServiceReturned = errors.New("humus: a Service returned before any signal")

// ShutdownCause returns why the App started shutting down, when ctx is the
// context Run gives a Drain or one derived from it, and nil for any other
// context. After a signal, it is the cause the signal gave the Process's
// NotifyContext. When a Service's Serve returned first, it wraps
// ErrServiceReturned and what Serve returned, if that was an error. When
// the context passed to Run was cancelled, it is that context's cause.
func ShutdownCause(ctx context.Context) error

// DrainDelay returns a Drain that waits d, or until ctx is done, and returns
// nil either way. It doesn't look at ShutdownCause. It panics if d is
// negative.
func DrainDelay(d time.Duration) Drain

In humustest, the docs of WithEnv and Start change:

package humustest // github.com/z5labs/humus/humustest

// WithEnv, WithArgs and WithStdin set the Process's environment, its
// arguments after the program name, and its standard input. They work like
// bedrocktest's options of the same names: if one is given more than once,
// the last one wins. A helper that sets a base environment and lets a test
// add to it merges the maps itself.
func WithEnv(values map[string]string) Option
func WithArgs(args ...string) Option
func WithStdin(r io.Reader) Option

// Start runs the App that humus.Main would run for init with
// bedrocktest.Start, on a new Process. The App has a new Host of this
// package's in place of humus.OSHost(), and opts change the App and the
// Process: humus.WithHost, passed with WithApp, replaces that Host. The
// Host and Init are set before opts are applied, so an Option given with
// WithApp sees both and may wrap them. Call Start inside the
// testing/synctest bubble that runs the App, since the Host is built there.
// It panics if init or one of opts is nil.
func Start(t testing.TB, init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...Option) (*bedrocktest.Process, <-chan bedrock.ExitStatus)

A service, end to end

Unchanged up to the HTTP sketch. The paragraph after the sketch becomes:

The end user's main calls httpserver.Main(Init, humus.WithTelemetry(buildTelemetry)), and its test calls humustest.Start(t, Init, humustest.WithApp(httpserver.Defaults())). The package may also ship a test package whose Start passes its defaults for the test, as its proposal decides. A process uses httpserver.Main when its HTTP server is one a load balancer routes to, as an API next to a job is. A job pod whose only HTTP server serves probes or metrics stays on humus.Main.

An end user whose main passes options that a test should keep, such as a grace period, puts them in one function that returns them. main passes its result to Main, and the test passes it with WithApp before its own Host and telemetry.

The drain

The drain is the pause between shutdown starting and Shutdown being called on the Services. It is a function, so that what it waits for is the end user's choice, and so that it can stop waiting once nothing is left to drain.

• It ends early once nothing is serving. Run cancels the Drain's context once every Service's Serve has returned. A Service that stops on Serve's context, such as a job, a queue consumer or the health Service, returns at once or once its work in flight is done. A Service that keeps serving until Shutdown, such as an HTTP or gRPC server, doesn't return until after the drain. So, for an App with a Drain set:

  The App's Services	The drain lasts
  a job, finished on its own	nothing: Run doesn't call the Drain
  a job, finished on its own, next to the health Service	nothing: Run calls the Drain, and its context is done at once
  a job, interrupted by a signal	until the run returns, at most the Drain's delay
  a queue consumer	until its records in flight are done, at most the Drain's delay
  an HTTP or gRPC server, with or without the health Service	the Drain's whole delay
  a job and an HTTP server	the Drain's whole delay, unless both return first

  On humus.Main's defaults there is no Drain, so none of these drain. This generalises a rule the lifecycle already has: an Init that returns no Services ends the App without a drain, because there is nothing to drain.

• DrainDelay(d) is the usual kind of drain. It waits d, or until nothing is serving. Main has none, and a runtime package that needs one sets it in its own Main, as humus.WithDrain(humus.DrainDelay(5 * time.Second)) or as a field its defaults Option sets. It doesn't look at the shutdown's cause: after a Service returned on its own, it still gives load balancers the delay to notice that the pod is no longer ready.

• A Drain of the end user's own waits for something else. For example, one that polls a load balancer's API until the pod's target is draining, and goes through s.Host.Dial so that it runs on a test Host too:

  humus.Main(Init, humus.WithDrain(func(ctx context.Context, s *humus.Setup) error {
  	if errors.Is(humus.ShutdownCause(ctx), humus.ErrServiceReturned) {
  		return nil // no signal came, so nothing is deregistering the pod
  	}
  	return waitUntilDeregistered(ctx, s.Host.Dial, targetID(s.Process))
  }))

  waitUntilDeregistered returns nil when ctx is done.

• The Drain's context says why the shutdown started. After a signal, Kubernetes is terminating the pod and removing it from load balancers. When a Service returned on its own, such as a job next to an API, nothing outside is draining the App, and the API keeps the Drain's context alive. A Drain that waits for deregistration would then wait for something nothing triggers, so it checks ShutdownCause first.

• It gets the Setup, not what Init built. The Drain is chosen in main, before Init runs. Work at shutdown that needs Init's dependencies, such as deregistering from service discovery, belongs in a Service that stops on Serve's context. That Service sees shutdown start at the same moment, and the Drain then waits for whatever has to follow it, such as clients' cached lookups expiring. A package's Option can also wrap Init and set a Drain that shares what it records.

• HUMUS_SHUTDOWN_DRAIN_DELAY replaces the Drain. When it is set, Run uses DrainDelay of its value in place of App.Drain, whatever that is. It is still a time.ParseDuration string that must not be negative, and a bad value still exits 78 before Init. An operator can always replace a Drain that misbehaves, from the same manifest that sets terminationGracePeriodSeconds.

The defaults

	Main	Why
Host	OSHost()	Main runs only in main. A test runs its App with humustest.Start, which uses a test Host.
Telemetry	telemetry.New with DefaultSettings(), writing to the Process's stdout	Logs go to stdout as JSON lines, and spans get real IDs without being exported, as the telemetry proposal set out. A service that exports anything passes its own Builder.
ShutdownGracePeriod	20s	It fits inside Kubernetes' default terminationGracePeriodSeconds of 30s, with telemetry's 5s shutdown after it and a margin for the process to exit.
Drain	none	Only a Service that a load balancer routes to, and that keeps serving until Shutdown, needs a drain, and the package that provides it sets one.

The worst case is 20s of grace period and 5s of telemetry shutdown: 25s. For each runtime:

• A job or a queue consumer runs on humus.Main's defaults, which are the ones the jobs proposal and the Kafka proposal chose: no drain and a 20s grace period. Neither package needs a Main of its own. Next to a probes or metrics server, it stays on them: nothing routes to that server, and a drain would make a finished job wait the whole delay.
• An HTTP or gRPC server needs a drain, so its package builds a Main of its own. A 5s drain takes 5s from the budget, so its grace period drops to fit: 5s, 10s and 5s for the HTTP server foundation's proposed defaults, and 5s, 15s and 5s for gRPC's.
• A job or a consumer next to a server that is routed to runs on that server's package's Main. Its work in flight at a signal gets the drain and that package's grace period: 15s on the HTTP server foundation's proposed defaults, not 20s. An end user who needs more passes humus.WithShutdownGracePeriod after the package's defaults, within the 25s budget.
• An HTTP and a gRPC server in one process use either package's Main. Each package's defaults set both the drain and the grace period, so passing the other package's defaults as well replaces both: the last one wins.

Changes to Run

Unchanged, except that step 9's context also gives ShutdownCause why the shutdown started:

2. Step 9 calls the Drain instead of sleeping. If a Serve result hasn't reached Run yet, Run calls the Drain with the Setup, and with a context that keeps the values of Run's context, is not cancelled with it, is done once every Serve has returned, and gives ShutdownCause the shutdown's cause. It collects the Serve results that arrive in the meantime, waits for the Drain to return, and then shuts the Services down within the grace period. A Drain error is logged at ERROR and joined into the cause as drain: and the error.

Design decisions

Unchanged, except that these bullets replace the ones with the same heading:

• Main has no drain. A drain is for a Service that a load balancer routes to and that keeps serving until Shutdown. A job, a queue consumer and the health Service stop on Serve's context, and a probes or metrics server isn't routed to. A runtime package's defaults are where its drain belongs, so the packages whose Services need one set it.
• A runtime package's defaults are one humus.Option. An Option can set several fields, so the package's Main applies its defaults before the end user's options, and a test passes the same Option to humustest.Start. The end user's options come last, so they win over the package's. A package may also ship a test package that passes its defaults for the test, so the test can't forget them.
• A test needs no bedrocktest options. Start takes humus's options and the process's in one list, so humustest's Option carries both. WithEnv, WithArgs and WithStdin mirror bedrocktest's options. This reverses, for Start, humustest's second revision, under which humustest didn't wrap bedrocktest. Start still returns bedrocktest's Process and status channel unwrapped. A test calls their methods without importing bedrocktest, and a helper that names their types imports bedrocktest and bedrock, as humustest builds on them.

These bullets are new:

• A Drain returns nil when its context is done. The context being done is how a Drain learns that nothing is left to drain, which is a normal end, not a failure. Run doesn't discard context.Canceled for it, so an error a Drain returns always fails the App, and a Drain's own failure can't be mistaken for that normal end.
• The shutdown's cause is on the Drain's context. Calling the Drain only after a signal would drop the drain of an API whose other Service failed, while load balancers are still routing to it. With the cause, each Drain decides: DrainDelay waits either way, and a Drain that waits on a load balancer waits only after a signal. The Drain keeps its signature.
• An Option may wrap Init. With no runtime App left to hold a Service, wrapping Init is how a package, or exposing health probes in wave 4, adds a Service by default, or registers on one that Init returns. Main and Start set Init first, so the wrapper sees the end user's.

Testing

The first revision's two sketches stay. They ran unchanged on a prototype, inside synctest, so every duration is fake time. The stories' tests also show the following:

• With Start's defaults, a Service that keeps serving until Shutdown is shut down at the same fake instant as the signal: there is no drain.
• A Service that returns on its own, as a finished job does, ends the App at the same fake instant: the Drain is never called. Next to a Service that stops on Serve's context, the Drain is called once, and its context is done at the same fake instant.
• A Drain that returns ctx.Err() once its context is done exits 1, with drain: context canceled in the cause.
• ShutdownCause on the Drain's context is the signal's cause after SIGTERM, and doesn't wrap ErrServiceReturned. It wraps ErrServiceReturned after a Service returned nil, wraps both ErrServiceReturned and the error after a Service returned an error, and is the cause of Run's context when that was cancelled. On any other context it is nil.
• A Drain that waits for a load balancer, next to an API, returns at once when a job returned on its own, and waits for the load balancer after SIGTERM.
• The grace period starts after the Drain: a Drain that takes 3s, with HUMUS_SHUTDOWN_GRACE_PERIOD=2s, gives Shutdown's context a deadline 5s after the signal.
• A Serve error during the drain exits 1, with serve: and the error in the cause.
• The Drain's context keeps a value set on Run's context, and isn't done when Run's context is cancelled.
• An Init that returns no Services ends the App without calling the Drain.
• HUMUS_SHUTDOWN_DRAIN_DELAY=1s, set with humustest.WithEnv, replaces a custom Drain, which is never called, and adds a drain to an App that has none. HUMUS_SHUTDOWN_DRAIN_DELAY=-1s exits 78 before Init.
• A Drain that returns an error exits 1, with drain: and the error in the cause, logged at ERROR, and Shutdown is still called on every Service.
• A Drain is given the Setup that Init was given.
• DrainDelay with a negative duration panics, and so do Main and humustest.Start with a nil init or a nil option.
• humustest.Start with no options runs an App with a 20s grace period, no drain, a humustest Host and Main's telemetry default, and its first log line appears on p.Stdout() as JSON.
• humustest.WithApp(humus.WithHost(h), humus.WithTelemetry(b)) uses h and b. WithApp given twice applies both, in order, and a later option wins over an earlier one. A second WithEnv replaces the first.
• An Option given to Main or to Start sees App.Init and may wrap it: one that appends a Service runs that Service next to the end user's. An Option given to Start also sees its test Host, and may wrap it.
• humustest.WithEnv, WithArgs and WithStdin reach the App as Setup.Process's environment, Args[1:] and stdin.
• The App that Main runs has OSHost(), the 20s grace period and no drain. An internal test checks the function Main builds it with, since Main itself exits the process.

Changes to the other proposals

This proposal changes the four runtime proposals, which the charter's revision 5 moves to wave 3. No correction is posted on their discussions. Each one revises itself when it is taken up again:

• All four lose their App types. Each package exports its Service, and the end user's Init returns it. The work their Apps did in Init, such as building the HTTP Service from the Settings and handler Init returned, moves into the end user's Init.
• Main's telemetry default replaces "Telemetry is required" in the HTTP server foundation, gRPC services and queue processing and the Kafka runtime, for the reason under Design decisions. It also answers jobs' open question 1, whether job.App requires a Builder.
• Jobs and queue processing and the Kafka runtime run on humus.Main's defaults, which match the ones they chose, and need no Main of their own.
• The HTTP server foundation and gRPC services each add a Main built on humus.Main and export their defaults as one humus.Option, with a drain and a grace period that fit Kubernetes' 30s together. Each decides whether it also ships a test package that starts an App with its defaults, with a client wired to the test Host. Such a package is named after its own, as httpservertest, since httptest is the name of net/http/httptest. Its client works without its Start, so that a test of a process that runs both servers can use one package's Start and both clients.
• Exposing health probes (now wave 4) can't add a field to a runtime App, since there are none. It decides how an App's probes are exposed in terms of humus.Main, the runtime packages' Mains, their options, which may wrap Init, and the Services the end user's Init returns. A probes server for a job or consumer pod isn't routed to, so it brings no drain: it stops on Serve's context, and the pod stays on humus.Main.

The charter's revision 6 matches this revision. This proposal's own stories change the root package, humustest and their tests, and health's tests, which build humus.App{DrainDelay: …}. They also change the docs: the lifecycle, configuration, telemetry, testing and health pages, the docs index, the README, and the package docs of the root package, health and humustest. The story that replaces DrainDelay changes health's tests and the health page in the same pull request.

Migration from today

• bedrock.Main(humus.App{Host: humus.OSHost(), ShutdownGracePeriod: 30 * time.Second, DrainDelay: 5 * time.Second, Telemetry: buildTelemetry, Init: Init}) becomes humus.Main(Init, humus.WithTelemetry(buildTelemetry), humus.WithShutdownGracePeriod(30*time.Second), humus.WithDrain(humus.DrainDelay(5*time.Second))).
• App.DrainDelay: d becomes App.Drain: humus.DrainDelay(d). An App that set no DrainDelay sets no Drain.
• bedrocktest.Start(t, humus.App{Host: host.Host(), Telemetry: tel.Builder(), ShutdownGracePeriod: …, Init: Init}, bedrocktest.WithEnv(env)) becomes humustest.Start(t, Init, humustest.WithApp(humus.WithHost(host.Host()), humus.WithTelemetry(tel.Builder())), humustest.WithEnv(env)). It runs with Main's defaults, so a test of the main above also passes that main's grace period and drain, as A service, end to end describes.
• An App whose Services all stop on Serve's context now shuts down as soon as they have returned, even with a drain, instead of after the whole drain delay.
• HUMUS_SHUTDOWN_DRAIN_DELAY keeps its name and format. When it is set, it now replaces any Drain, not just a duration.

Wording fixes to the post

• In Background and References, "the wave-2 runtime proposals" now means wave 3.
• Out of scope's Health probes bullet reads "in terms of humus.Main, the runtime packages' Mains, their options and the end user's Services".

Open questions

None.

[Zaba505]

Closing: humus.Main has landed

Revision 2's stories are done:

• story(humus): replace the drain delay with a Drain that ends once nothing is serving #688 replaced the drain delay with a Drain that ends once nothing is serving (story(humus): replace the drain delay with a Drain that ends once nothing is serving #693);
• story(humus): tell a Drain why the shutdown started #689 told a Drain why the shutdown started, with ShutdownCause and ErrServiceReturned (story(humus): tell a Drain why the shutdown started #694);
• story(humus): run an App on the OS process with humus.Main #690 added humus.Main and its options (story(humus): run an App on the OS process with humus.Main #695);
• story(humustest): Start runs the App humus.Main would run, on a test Host #691 added humustest.Start, which runs the App humus.Main would run, on a test Host (story(humustest): Start runs the App humus.Main would run, on a test Host #696);
• story(docs): document humus.Main, the Drain and humustest.Start #692 documented humus.Main, the Drain and humustest.Start (story(docs): document humus.Main, the Drain and humustest.Start #697).

Revision 2 left no open questions. What it hands on belongs to the wave-3 proposals, as its "Changes to the other proposals" lists. All four drop their App when they're taken up again. The HTTP server foundation and gRPC services each add a Main of their own, and decide whether to ship a test package.