Proposal: jobs

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 2. It builds on Proposal: the humus App lifecycle, Proposal: configuration for humus apps, Proposal: OpenTelemetry without globals and Proposal: humustest. It depends on no other wave-2 proposal. Proposal: exposing health probes (wave 3) decides whether a job's App ever exposes probes, and adds that to this package if it does.

Summary

The charter lists job as "keep or drop". This proposes keeping it, smaller than a framework and bigger than nothing:

• job.NewService turns a func(context.Context) error into a humus.Service that runs it once, stops it on either shutdown signal, and records one span around the run.
• job.App is a bedrock.App built on humus.App, with a job's defaults: no drain delay, a grace period that fits Kubernetes' default, and no health Service.
• The run's error is the exit status. A run that returns nil exits 0. A run that fails, or that a signal interrupts before it finishes, exits 1, so Kubernetes never records an interrupted job as a success.

func main() {
	bedrock.Main(job.App{
		Host:      humus.OSHost(),
		Telemetry: buildTelemetry,
		Init:      app.Init, // returns the run
	})
}

The package has no Settings, reads nothing and opens nothing. It is a leaf of the root package, as the lifecycle proposal sets out for every runtime package.

Background

• Today's job is a 36-line wrapper. Handler, HandlerFunc and an App whose Run(context.Context) error calls the handler (job/job.go#L10-L36). It has the v0.21.0 runtime shape that bedrock removed, and it sets up no telemetry, as the charter notes. Nothing else in it survives the process model.
• The lifecycle already runs jobs. The lifecycle's revision says "a job is just a Service that returns": a Serve that returns nil ends the App with exit 0 and no cause, and one that returns an error ends it with exit 1. So the question for this proposal is what a package adds on top of that.
• A correct job Service is fiddly to write by hand. humustest.TestService holds every Service to five rules: Serve returns nil after Shutdown; Shutdown with a done context returns at once; Shutdown is safe before Serve, after Serve has returned and twice; a Serve after Shutdown returns; and no goroutine outlives Serve. A job also has to tell "Shutdown stopped me" apart from "a signal interrupted me", because the two need different exit statuses (see below). That is about 60 lines that every job service would otherwise copy.
• Kubernetes reads a job's exit code. A Job's pod that exits 0 is Succeeded, and the Job completes. Any other code is a failure, counted against backoffLimit and retried under restartPolicy: OnFailure or by a new pod under Never. A pod that is evicted, preempted or deleted gets SIGTERM, and then SIGKILL after terminationGracePeriodSeconds (30s by default). podFailurePolicy can act on exit codes (onExitCodes) and on the DisruptionTarget pod condition. So the exit code of an interrupted run decides whether the work is done again.
• The lifecycle exits 0 after a signal. humus.App.Run returns 0, with the signal as the cause, when a signal stopped the App and every Service shut down cleanly. That is right for an API. For a job it is right only if the run had finished.

Proposed API

package job // github.com/z5labs/humus/job

// Service is a humus.Service that runs a job once. It is safe for concurrent
// use.
type Service struct{ /* unexported */ }

// NewService returns a Service that runs run once, when its Serve is called.
// run's context is done when the App starts shutting down or when Shutdown is
// called, whichever comes first, and carries the span of the run.
//
// NewService panics if run is nil.
func NewService(s *humus.Setup, run func(ctx context.Context) error) *Service

// Serve calls run on Serve's own goroutine, inside a span, and returns once
// run has returned:
//
//   - nil when run returned nil;
//   - nil when Shutdown was called before ctx was done and run returned an
//     error for which errors.Is(err, context.Canceled) holds, because
//     Shutdown stopped it;
//   - run's error, unchanged, otherwise. That includes a run that a signal
//     interrupted, which returns ctx's error.
//
// A Service is served once. A Serve called after Shutdown, or after an
// earlier Serve, returns nil at once without calling run.
func (s *Service) Serve(ctx context.Context) error

// Shutdown cancels run's context and waits for Serve to return. It returns
// nil once Serve has returned, or at once when Serve was never called or had
// already returned. If ctx is done first, it returns an error that wraps
// ctx's error, and Serve keeps waiting for run.
func (s *Service) Shutdown(ctx context.Context) error

// App is a bedrock.App that runs one job with humus's lifecycle. Host,
// Telemetry and Init are required.
//
// It is humus.App with a job's defaults: a drain delay of 0, a grace period
// of 20s, and no health Service. HUMUS_SHUTDOWN_GRACE_PERIOD and
// HUMUS_SHUTDOWN_DRAIN_DELAY override them, as for any humus.App.
type App struct {
	// Host is the network and filesystem the job runs on: humus.OSHost() in
	// main, and a humustest Host in a test.
	Host humus.Host

	// Telemetry builds the App's telemetry.SDK, as humus.App's does.
	Telemetry telemetry.Builder

	// Init builds the job's dependencies, registers their release with
	// s.OnStop, and returns the run. It is humus.App's Init, except that it
	// returns one run instead of Services.
	Init func(ctx context.Context, s *humus.Setup) (func(ctx context.Context) error, error)
}

// Run implements bedrock.App. It runs a humus.App whose Init calls a.Init and
// returns NewService of the run.
func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus

That is the whole package: one Service, one App, no named function type, no Settings.

The run

The run is a plain func(ctx context.Context) error. A struct with a Run method passes its method value, j.Run, so the package needs neither today's Handler interface nor HandlerFunc. Init builds the run's dependencies and closes over them:

func Init(ctx context.Context, s *humus.Setup) (func(context.Context) error, error) {
	bucket, err := config.Env(s.Process.Environment, "EXPORT_BUCKET").Read(ctx)
	if err != nil {
		return nil, err // exit 78
	}
	pool, err := openPool(ctx, s) // through s.Host.Dial
	if err != nil {
		return nil, err
	}
	s.OnStop(func(context.Context) error { pool.Close(); return nil })

	e := &exporter{pool: pool, bucket: bucket, fs: s.Host.FS}
	return e.Run, nil
}

Exit statuses

job.App adds no exit codes. The lifecycle's table applies, and the run decides which row:

The run	Exit	Cause
returns nil	0	none
returns an error	1	serve: and the run's error
is interrupted by a signal, and returns ctx's error	1	serve: context canceled, or the signal when the run returns context.Cause(ctx)
is interrupted by a signal, but returns nil	0	the signal
ignores its context past the grace period	1	a *humus.GracePeriodError
panics	2	none: Go's runtime prints the panic and exits
Init fails reading a setting	78	the *config.Error

The fourth row is the run's own decision: it finished, or it chose to treat stopping early as done. A run that stops early because its context is done returns an error, as almost all Go code already does with ctx.Err().

Which signal a job honours

The lifecycle gives a Service two signals: Serve's context, done as soon as shutdown starts, and Shutdown, called after the drain delay. A job stops on the first, as a queue consumer does.

• Serve's context is the run's parent. A signal cancels the run at once, and the run has the whole grace period to stop. With job.App's drain delay of 0, Shutdown is called straight after and waits for it.
• Shutdown also cancels the run. In an App, the run's context is always done before Shutdown is called, so this only matters when a Service is driven directly, as TestService does, or when a future caller calls Shutdown first. Serve returns nil in that case, because the Service contract says a Service that Shutdown stopped returns nil.
• Who cancelled first decides Serve's result. The run's context is made with context.WithCancelCause, and Shutdown cancels it with an unexported cause. context.Cause reports the first cause, so after a signal it is the signal, and Serve returns the run's error; after a bare Shutdown it is Shutdown's, and Serve returns nil when the error is a cancellation. Any other error from the run is returned either way.

Telemetry

NewService takes the tracer s.Telemetry.TracerProvider.Tracer("github.com/z5labs/humus/job"). Nothing reads OTel's globals.

• One span per run, named job, of kind internal. It starts when Serve calls the run and ends when the run returns. When the run returns an error, the span's status is Error with the error's text as its description. A run that Shutdown stopped, which Serve reports as nil, still has the error status: the span records what the run returned.
• The run's context carries the span. Every span the run starts is its child, and every record it logs with that context, such as otelslog's InfoContext(ctx, …), carries its trace_id. With telemetry.DefaultSettings(), traces are not exported but spans still get real IDs, so one job's console logs share a trace ID even without a collector.
• No log records. The lifecycle already logs a failed Serve at ERROR, and "serving", "shutting down" and "stopped" at INFO. A record from job would repeat them.
• No metrics. A job runs once per process. A run count or duration emitted from inside it is better read from the orchestrator, which sees runs that never started, such as a pod that failed to schedule.

The App's defaults

	job.App	Why
DrainDelay	0	Nothing routes traffic to a job, so there is nothing to drain.
ShutdownGracePeriod	20s	It fits inside Kubernetes' default terminationGracePeriodSeconds of 30s, with room for the telemetry shutdown that follows it (telemetry.DefaultSettings().ShutdownTimeout is 5s).
Telemetry	required	See open question 1.
health	none	A Job's pod has no Service routing to it, and wave 2 serves no probes.

The defaults are not fields of job.App. An operator changes them with HUMUS_SHUTDOWN_GRACE_PERIOD and HUMUS_SHUTDOWN_DRAIN_DELAY, which humus.App.Run reads. A team that wants a different default in code uses humus.App with job.NewService, which is five more lines.

Running more than the job

job.App runs one job and nothing else. A process that runs a job next to other Services, such as a metrics endpoint or a sidecar-like consumer, uses humus.App directly and returns job.NewService(s, run) among its Services. Any Service returning ends the App, so the job ending ends the process, as it should.

Proposal: exposing health probes (wave 3) decides whether a job's App ever exposes probes on an HTTP server of its own. If it does, it adds a field to job.App, which doesn't change Init's shape or break an existing job.App literal.

Design decisions

• Keep the package, small. Dropping it would leave every job service to write the same Service: the once-only Serve, the Shutdown that stops the run and waits, the rules TestService checks, the span and the exit-status rule for an interrupted run. The alternative, recorded here, is to drop job and document a twenty-line Service in the lifecycle docs. It is rejected because the interrupted-run rule is easy to get wrong, and getting it wrong marks unfinished work as done.
• An interrupted run exits 1. A run that a signal stops before it finishes returns its context's error, and the App exits 1. Kubernetes then counts the pod as failed and retries the work, rather than completing the Job. This needs nothing from the lifecycle: a Serve that returns an error already exits 1. An operator who doesn't want disruptions counted against backoffLimit uses podFailurePolicy with the DisruptionTarget condition and action: Ignore.
• No new exit code for an interruption. A shell reports 143 for a process killed by SIGTERM, and a job could mimic it. The lifecycle keeps its table small and tells failures apart by cause, so an interruption is exit 1 with the cancellation in its cause. See open question 3.
• Exit 78 is a podFailurePolicy rule. A config error fails every retry the same way. An operator can stop retrying it with onExitCodes: {operator: In, values: [78]} and action: FailJob. The docs page shows that rule.
• The run's error is returned unchanged. Serve doesn't wrap it, and the lifecycle prefixes serve: . A run that wants the signal in its cause returns context.Cause(ctx) instead of ctx.Err().
• The run runs on Serve's goroutine. NewService starts no goroutine, so no goroutine of the package outlives Serve. Goroutines the run starts are the end user's.
• A plain func, not an interface. func(context.Context) error is the shape of the run, and a method value adapts any type. An interface with one method would add a name and a Func adapter for nothing.
• NewService panics on a nil run, and returns no error. The package has no settings to check, so a nil run is the only misuse, and it is a programming error, as a nil function is for Setup.OnStop.
• No Settings. The package opens nothing and has no tunable. The grace period and drain delay belong to the lifecycle. As for health, there is no Settings, no DefaultSettings and no checks.
• No panic recovery. The lifecycle never recovers a panic, and a job is one unit of work, so there is no request to isolate. A panicking run crashes the process with Go's exit status 2 and the stack trace on stderr, which Kubernetes counts as a failure. Recovering, recording the panic on the span and panicking again was considered, and rejected: the process exits before the batch span processor exports the span, so it would cost code and show nothing.
• Retries are the end user's. Kubernetes retries a failed pod under backoffLimit. A run that wants to retry a step in-process uses whatever it likes inside the run, such as bedrock's retry module (v0.24.0). job doesn't wrap it, so retries stay visible in the end user's code.
• The package is in the core module. It imports only the root package, telemetry and the OTel trace API, all of which the core module already requires. No third-party dependency is added.

Testing

These tests are sketches and have not been run.

A whole job App runs under synctest, with bedrocktest's process, humustest's Host and humustest's telemetry recorder:

func TestApp_a_run_that_returns_nil_exits_0(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		tel := humustest.NewTelemetry()
		app := job.App{
			Host:      humustest.NewHost(t).Host(),
			Telemetry: tel.Builder(),
			Init: func(context.Context, *humus.Setup) (func(context.Context) error, error) {
				return func(context.Context) error {
					time.Sleep(time.Minute) // fake time: returns at once
					return nil
				}, nil
			},
		}

		_, status := bedrocktest.Start(t, app)

		st := <-status
		require.Zero(t, st.Code())
		require.NoError(t, st.Cause())
		spans := tel.Spans()
		require.Len(t, spans, 1)
		require.Equal(t, "job", spans[0].Name())
		require.Equal(t, time.Minute, spans[0].EndTime().Sub(spans[0].StartTime()))
	})
}

func TestApp_SIGTERM_interrupts_the_run_and_exits_1(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		tel := humustest.NewTelemetry()
		app := job.App{
			Host:      humustest.NewHost(t).Host(),
			Telemetry: tel.Builder(),
			Init: func(context.Context, *humus.Setup) (func(context.Context) error, error) {
				return func(ctx context.Context) error {
					<-ctx.Done()
					return context.Cause(ctx)
				}, nil
			},
		}

		p, status := bedrocktest.Start(t, app)
		synctest.Wait() // the run is blocked on its context
		p.Signal(syscall.SIGTERM)

		st := <-status
		require.Equal(t, humus.ExitFailure, st.Code())
		require.ErrorIs(t, st.Cause(), context.Canceled)
		require.ErrorContains(t, st.Cause(), "serve: terminated signal received")
		require.Equal(t, codes.Error, tel.Spans()[0].Status().Code)
	})
}

job.Service passes humustest.TestService. TestService needs a Service that runs until it is stopped, so the test's run blocks on its context:

func TestService_meets_the_Service_contract(t *testing.T) {
	humustest.TestService(t, func(t *testing.T, s *humus.Setup) humus.Service {
		return job.NewService(s, func(ctx context.Context) error {
			<-ctx.Done()
			return ctx.Err()
		})
	})
}

That exercises both stop paths: the "Serve returns nil after Shutdown" rule relies on a bare Shutdown being reported as nil, and the "after Serve has returned on its own" rule cancels Serve's context first.

The tests also show the following:

• A run that returns an error exits 1, with that error in the cause, and its span has status Error with the error's text.
• A run that a signal interrupts but that returns nil exits 0, with the signal as the cause.
• A run that ignores its context ends the App after the grace period, with exit 1 and a *humus.GracePeriodError.
• HUMUS_SHUTDOWN_GRACE_PERIOD=5s overrides the 20s default, and the App's drain delay is 0: Shutdown is called on the job at the same fake instant as the signal.
• Init failing with a *config.Error exits 78 and never runs the job.
• A span the run starts from its context is a child of the job span, and a record it logs with that context carries the job span's trace ID.
• Serve called twice runs the job once. Shutdown with a done context returns an error that wraps it, and Serve still returns once the run does.
• NewService with a nil run panics.

Migration from today

• job.NewApp(h), which bedrock ran as a Runtime, becomes bedrock.Main(job.App{Host: humus.OSHost(), Telemetry: buildTelemetry, Init: Init}), with Init returning the run.
• job.Handler and job.HandlerFunc go away. A type with Handle(ctx) error passes its method value, h.Handle, as the run.
• The handler's dependencies, built in main or in the Builder before, are built in Init and released with s.OnStop.
• A handler that ran with no telemetry now runs inside the job span, from s.Telemetry.
• A job interrupted by SIGTERM used to return whatever its handler returned to bedrock's old runner. It now exits 1 unless the run returns nil.
• The docs site gains a page on jobs: the run, the exit-status table, Kubernetes' backoffLimit and podFailurePolicy for exit 78 and for disruptions, and running a job beside other Services with humus.App. It is one of this proposal's stories.

Open questions

1. Does job.App require a telemetry.Builder? The telemetry proposal leaves each runtime App to choose between requiring one and defaulting to telemetry.New with DefaultSettings(). The default would name every job unknown_service, and New writes to the process's stdout, which the Builder is given but job.App would have to pass on itself. Recommendation: required, as for humus.App, so main names the service.
2. Is 20s the right grace period? The other choice is 25s, which uses all of Kubernetes' default except the 5s telemetry timeout. 20s leaves a margin for the process to exit after telemetry shuts down. Either way the operator overrides it with HUMUS_SHUTDOWN_GRACE_PERIOD, and should raise terminationGracePeriodSeconds with it.
3. Should an interrupted run get its own exit code? Exit 1 makes Kubernetes retry it, which is what matters, and the cause tells it apart in a test. A separate code, such as 143, would let podFailurePolicy treat interruptions differently from real failures by exit code. DisruptionTarget already covers the evictions and preemptions that cause most interruptions. Recommendation: no new code.
4. Should the run's span take a parent from the environment? The OpenTelemetry specification describes TRACEPARENT as an environment-variable carrier, so a job started by another traced process can join its trace. humus reads no variables, so it would be a field the end user fills, such as a parent context on job.App. Recommendation: not now. Add it when a service needs it.

Out of scope

• Scheduling. A Kubernetes CronJob, or any other scheduler, starts the process. job runs one run per process.
• Retries inside a run, and idempotency across Kubernetes' retries. Both are the end user's.
• Distributed locks and CronJob's concurrencyPolicy.
• Health probes. Proposal: exposing health probes, in wave 3, decides whether a job's App exposes them.
• Several runs in sequence or in parallel in one App. The end user's run calls whatever it likes, or composes several job.Service values with humus.App.

References

• Proposal: humus on the bedrock process model, the charter, and its wave plan
• Proposal: the humus App lifecycle: "a job is just a Service that returns", the two shutdown signals, and the exit-code table
• Proposal: OpenTelemetry without globals: the choice each runtime App makes about its Builder
• Proposal: humustest: TestService and the telemetry recorder
• Today's job/job.go
• Kubernetes: Jobs, including the pod failure policy and backoffLimit, and pod termination
• bedrock v0.24.0's retry module

[Zaba505]

Revision: the open questions answered, on humus.Main

The charter's revision 5 moved this proposal to wave 3, behind Proposal: humus.Main, which has landed. This revision applies humus.Main and answers the four open questions:

1. There is no job.App, so no Builder to require. humus.Main runs the Services of any runtime, and defaults the telemetry Builder to telemetry.New with DefaultSettings(). So job is NewService and nothing else, and the end user's Init returns job.NewService(s, run) to humus.Main.
2. 20s is the grace period, and it is humus.Main's. humus.Main has a 20s grace period and no drain, which are the defaults this proposal chose for a job. So job has no Main and no defaults of its own.
3. An interrupted run gets no exit code of its own. It exits 1, as recommended. podFailurePolicy's DisruptionTarget condition covers the evictions and preemptions that cause most interruptions.
4. The run's span joins the trace in TRACEPARENT, now. Serve extracts a trace context from the Process's environment with the SDK's propagator, as OpenTelemetry's environment-variable carriers describe, and starts the run's span as its child. When none is set, or it is malformed, the span is a new root, and there is no error.

Each section below replaces the section of the same name in the post. Background, Out of scope and References are unchanged, apart from the wording fixes at the end.

Summary

The charter lists job as "keep or drop". This proposes keeping it, smaller than a framework and bigger than nothing:

• job.NewService turns a func(context.Context) error into a humus.Service that runs it once, stops it on either shutdown signal, and records one span around the run. The span joins the trace of the process that started the job, when its environment carries one.
• A job runs on humus.Main's defaults. No drain, a grace period that fits Kubernetes' default, and no health Service, which are what a job needs. The package has no App and no Main.
• The run's error is the exit status. A run that returns nil exits 0. A run that fails, or that a signal interrupts before it finishes, exits 1, so Kubernetes never records an interrupted job as a success.

func main() {
	humus.Main(app.Init, humus.WithTelemetry(buildTelemetry))
}

app.Init returns []humus.Service{job.NewService(s, run)}. The package has no Settings, defines no variable names and opens nothing. It is a leaf of the root package, as the lifecycle proposal sets out for every runtime package.

Proposed API

Service, Serve and Shutdown are unchanged. job.App is gone, and NewService's doc gains a paragraph on the run's span:

package job // github.com/z5labs/humus/job

// NewService returns a Service that runs run once, when its Serve is called.
// run's context is done when the App starts shutting down or when Shutdown is
// called, whichever comes first, and carries the span of the run.
//
// The span's parent is the trace context that s.Telemetry.Propagator
// extracts from s.Process.Environment, with each of the propagator's keys
// read as OpenTelemetry's environment-variable carriers name it: traceparent
// as TRACEPARENT. When the environment carries none, or a malformed one, the
// span is the root of a new trace.
//
// NewService panics if run is nil.
func NewService(s *humus.Setup, run func(ctx context.Context) error) *Service

That is the whole package: one Service and its constructor, no App, no Main, no named function type, no Settings.

The run

The run is a plain func(ctx context.Context) error. A struct with a Run method passes its method value, j.Run, so the package needs neither today's Handler interface nor HandlerFunc. The end user's Init builds the run's dependencies, closes over them, and returns the run's Service:

func main() {
	humus.Main(Init, humus.WithTelemetry(buildTelemetry))
}

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	bucket, err := config.Env(s.Process.Environment, "EXPORT_BUCKET").Read(ctx)
	if err != nil {
		return nil, err // exit 78
	}
	pool, err := openPool(ctx, s) // through s.Host.Dial
	if err != nil {
		return nil, err
	}
	s.OnStop(func(context.Context) error { pool.Close(); return nil })

	e := &exporter{pool: pool, bucket: bucket, fs: s.Host.FS}
	return []humus.Service{job.NewService(s, e.Run)}, nil
}

Exit statuses

job adds no exit codes. The lifecycle's table applies, and the run decides which row:

The run	Exit	Cause
returns nil	0	none
returns an error	1	serve: and the run's error
is interrupted by a signal, and returns ctx's error	1	serve: context canceled, or the signal when the run returns context.Cause(ctx)
is interrupted by a signal, but returns nil	0	the signal
ignores its context past the grace period	1	a *humus.GracePeriodError
panics	2	none: Go's runtime prints the panic and exits
Init fails reading a setting	78	the *config.Error

The fourth row is the run's own decision: it finished, or it chose to treat stopping early as done. A run that stops early because its context is done returns an error, as almost all Go code already does with ctx.Err().

Which signal a job honours

The lifecycle gives a Service two signals: Serve's context, done as soon as shutdown starts, and Shutdown, called after the drain. A job stops on the first, as a queue consumer does.

• Serve's context is the run's parent. A signal cancels the run at once. On humus.Main's defaults there is no drain, so Shutdown is called straight after and waits for the run, which has the whole grace period to stop. Next to a server that a load balancer routes to, on that server's package's Main, Shutdown is called after the drain, and the run has the drain and the grace period to stop.
• Shutdown also cancels the run. In an App, the run's context is always done before Shutdown is called, so this only matters when a Service is driven directly, as TestService does, or when a future caller calls Shutdown first. Serve returns nil in that case, because the Service contract says a Service that Shutdown stopped returns nil.
• Who cancelled first decides Serve's result. The run's context is made with context.WithCancelCause, and Shutdown cancels it with an unexported cause. context.Cause reports the first cause, so after a signal it is the signal, and Serve returns the run's error; after a bare Shutdown it is Shutdown's, and Serve returns nil when the error is a cancellation. Any other error from the run is returned either way.

Telemetry

NewService takes the tracer s.Telemetry.TracerProvider.Tracer("github.com/z5labs/humus/job") and the propagator s.Telemetry.Propagator. Nothing reads OTel's globals.

• One span per run, named job, of kind internal. It starts when Serve calls the run and ends when the run returns. When the run returns an error, the span's status is Error with the error's text as its description. A run that Shutdown stopped, which Serve reports as nil, still has the error status: the span records what the run returned.
• The span joins the trace of the process that started the job. OpenTelemetry's environment-variable carriers, a release candidate, have a process pass its trace context to a child it starts in environment variables. Serve calls the propagator's Extract on its context, over a carrier on s.Process.Environment, and starts the run's span from the result. The carrier reads each key the propagator asks for under the name the specification gives it: ASCII letters uppercased, every character but an ASCII letter, digit or _ replaced with _, _ for an empty key, and a _ before a leading digit. So the W3C trace context and baggage propagator that telemetry.New and humustest build reads TRACEPARENT, TRACESTATE and BAGGAGE, and a Builder with a propagator of its own, such as B3, reads that propagator's keys the same way, x-b3-traceid as X_B3_TRACEID.
• An unset or malformed trace context starts a new trace. The propagator extracts nothing, as it does for an HTTP request with a bad traceparent header, and the run's span is a root. Serve returns no error for it and logs nothing.
• Only the run joins the trace. The extracted context is the parent of the run's span and of nothing else: the lifecycle's records and the spans Init starts stay outside it. The baggage is on the run's context, so the run reads it with baggage.FromContext(ctx).
• The span stays internal. A remote parent doesn't change its kind. The process that set TRACEPARENT started the job: it isn't a client whose request the job serves, or a producer whose message the job consumes.
• The run's context carries the span. Every span the run starts is its child, and every record it logs with that context, such as otelslog's InfoContext(ctx, …), carries its trace_id. With telemetry.DefaultSettings(), traces are not exported but spans still get real IDs, so one job's console logs share a trace ID even without a collector.
• No log records. The lifecycle already logs a failed Serve at ERROR, and "serving", "shutting down" and "stopped" at INFO. A record from job would repeat them.
• No metrics. A job runs once per process. A run count or duration emitted from inside it is better read from the orchestrator, which sees runs that never started, such as a pod that failed to schedule.

The carrier is unexported:

// envCarrier is a propagation.TextMapCarrier over a Process's environment.
// It reads each key under its name as an environment-variable carrier, and
// writes nothing.
type envCarrier struct{ env bedrock.Environment }

// Get returns the value of key's variable, or "" when it is not set.
func (c envCarrier) Get(key string) string

// Set does nothing: Extract never calls it, and the environment is read-only.
func (envCarrier) Set(key, value string) {}

// Keys returns nil, since a bedrock.Environment can't list its variables.
// The specification lets it return only names that are already normalized,
// and W3C trace context and baggage never call it.
func (envCarrier) Keys() []string { return nil }

The defaults

This section replaces The App's defaults. job has no defaults of its own. A job runs on humus.Main's, which are the ones this proposal chose:

	humus.Main	Why it fits a job
ShutdownGracePeriod	20s	It fits inside Kubernetes' default terminationGracePeriodSeconds of 30s, with room for the telemetry shutdown that follows it (telemetry.DefaultSettings().ShutdownTimeout is 5s).
Drain	none	Nothing routes traffic to a job, so there is nothing to drain.
Telemetry	telemetry.New with DefaultSettings()	A job that names itself or exports anything passes its own Builder with humus.WithTelemetry.
health	none	A Job's pod has no Service routing to it. Exposing health probes, in wave 4, decides whether a job pod exposes them.

An operator changes them with HUMUS_SHUTDOWN_GRACE_PERIOD and HUMUS_SHUTDOWN_DRAIN_DELAY, as for any App. A team that wants a different default in code passes humus.WithShutdownGracePeriod to humus.Main.

Running more than the job

A process that runs a job next to other Services returns them all from the end user's Init, with job.NewService(s, run) among them. Any Service returning ends the App, so the job ending ends the process, as it should.

Which Main it uses follows humus.Main's revision 2. Next to a probes or metrics server, a job stays on humus.Main: nothing routes to that server, and a drain would make a finished job wait the whole delay. Next to a server that a load balancer routes to, such as an API, it uses that server's package's Main, and after a signal its run gets that package's drain and grace period.

Proposal: exposing health probes (wave 4) decides whether a job pod exposes probes, in terms of humus.Main, its options and the Services the end user's Init returns. If that needs anything from this package, its stories add it.

Design decisions

Unchanged, except that these bullets replace the ones with the same heading:

• No new exit code for an interruption. A shell reports 143 for a process killed by SIGTERM, and a job could mimic it. The lifecycle keeps its table small and tells failures apart by cause, so an interruption is exit 1 with the cancellation in its cause. An operator who wants interruptions treated apart from failures uses podFailurePolicy with the DisruptionTarget condition.
• No Settings. The package opens nothing and has no tunable. The grace period and the drain belong to the lifecycle. As for health, there is no Settings, no DefaultSettings and no checks.
• The package is in the core module. It imports only bedrock, the root package, telemetry, and OTel's trace and propagation APIs, all of which the core module already requires. No third-party dependency is added.

These bullets are new:

• No App and no Main. humus.Main runs the Services of any runtime, and its defaults are a job's. An App would only repeat them, and a Main would apply nothing.
• The trace context comes through the SDK's propagator. Parsing TRACEPARENT with a W3C propagator of job's own would ignore the Builder's choice. With the SDK's, a service whose Builder propagates B3 reads B3's variables, and the baggage comes in with the trace context.
• The carrier is job's own. contrib's envcar reads the process's environment with os.Getenv and os.Environ, which humus's lint configuration bans: humus reads the process only through bedrock.Process. job's carrier reads s.Process.Environment, so a test sets TRACEPARENT with humustest.WithEnv.
• No opt-out. Only whatever starts the process sets these variables, and it sets them so that the job joins its trace. Kubernetes sets none of them, so a CronJob's run starts a new trace unless its pod spec sets one.
• job reads OpenTelemetry's variable names, not humus's. It is the second place in humus that reads the environment, after the lifecycle's two variables. humus still defines no variable names: the propagator's keys and the carriers specification name them. The configuration page says humus reads exactly two variables and that no humus package adds another. This proposal's docs story rewrites that section.

Testing

These tests are sketches and have not been run. A whole App runs under synctest with humustest.Start, which gives it a humustest Host and humus.Main's defaults. runs is a helper of the test file:

// runs returns an Init whose only Service is a job that runs run.
func runs(run func(context.Context) error) func(context.Context, *humus.Setup) ([]humus.Service, error) {
	return func(_ context.Context, s *humus.Setup) ([]humus.Service, error) {
		return []humus.Service{job.NewService(s, run)}, nil
	}
}

func TestService_a_run_that_returns_nil_exits_0(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		tel := humustest.NewTelemetry()
		_, status := humustest.Start(t, runs(func(context.Context) error {
			time.Sleep(time.Minute) // fake time: returns at once
			return nil
		}), humustest.WithApp(humus.WithTelemetry(tel.Builder())))

		st := <-status
		require.Zero(t, st.Code())
		require.NoError(t, st.Cause())
		spans := tel.Spans()
		require.Len(t, spans, 1)
		require.Equal(t, "job", spans[0].Name())
		require.False(t, spans[0].Parent().IsValid()) // no TRACEPARENT: a new trace
		require.Equal(t, time.Minute, spans[0].EndTime().Sub(spans[0].StartTime()))
	})
}

func TestService_SIGTERM_interrupts_the_run_and_exits_1(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		tel := humustest.NewTelemetry()
		p, status := humustest.Start(t, runs(func(ctx context.Context) error {
			<-ctx.Done()
			return context.Cause(ctx)
		}), humustest.WithApp(humus.WithTelemetry(tel.Builder())))
		synctest.Wait() // the run is blocked on its context
		p.Signal(syscall.SIGTERM)

		st := <-status
		require.Equal(t, humus.ExitFailure, st.Code())
		require.ErrorContains(t, st.Cause(), "serve: terminated signal received")
		require.Equal(t, codes.Error, tel.Spans()[0].Status().Code)
	})
}

func TestService_joins_the_trace_in_TRACEPARENT(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		tel := humustest.NewTelemetry()
		var schedule string
		_, status := humustest.Start(t, runs(func(ctx context.Context) error {
			schedule = baggage.FromContext(ctx).Member("schedule").Value()
			return nil
		}),
			humustest.WithApp(humus.WithTelemetry(tel.Builder())),
			humustest.WithEnv(map[string]string{
				"TRACEPARENT": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
				"BAGGAGE":     "schedule=nightly",
			}),
		)

		require.Zero(t, (<-status).Code())
		span := tel.Spans()[0]
		require.Equal(t, "4bf92f3577b34da6a3ce929d0e0e4736", span.SpanContext().TraceID().String())
		require.Equal(t, "00f067aa0ba902b7", span.Parent().SpanID().String())
		require.True(t, span.Parent().IsRemote())
		require.Equal(t, trace.SpanKindInternal, span.SpanKind())
		require.Equal(t, "nightly", schedule)
	})
}

job.Service passes humustest.TestService. TestService needs a Service that runs until it is stopped, so the test's run blocks on its context:

func TestService_meets_the_Service_contract(t *testing.T) {
	humustest.TestService(t, func(t *testing.T, s *humus.Setup) humus.Service {
		return job.NewService(s, func(ctx context.Context) error {
			<-ctx.Done()
			return ctx.Err()
		})
	})
}

That exercises both stop paths: the "Serve returns nil after Shutdown" rule relies on a bare Shutdown being reported as nil, and the "after Serve has returned on its own" rule cancels Serve's context first.

The tests also show the following:

• A run that returns an error exits 1, with that error in the cause, and its span has status Error with the error's text.
• A run that a signal interrupts but that returns nil exits 0, with the signal as the cause.
• A run that ignores its context ends the App after the grace period, with exit 1 and a *humus.GracePeriodError.
• HUMUS_SHUTDOWN_GRACE_PERIOD=5s, set with humustest.WithEnv, overrides the 20s default, and on humus.Main's defaults there is no drain: Shutdown is called on the job at the same fake instant as the signal.
• Init failing with a *config.Error exits 78 and never runs the job.
• A span the run starts from its context is a child of the job span, and a record it logs with that context carries the job span's trace ID.
• A malformed TRACEPARENT starts a new trace, as an unset one does: exit 0, with nothing about it in the logs or on stderr.
• With a valid TRACEPARENT, TRACESTATE reaches the span's trace state.
• Only the run's span joins the trace: a span that Init starts is in a trace of its own.
• The carrier reads traceparent as TRACEPARENT, x-b3-traceid as X_B3_TRACEID, an empty key as _, and a key with a leading digit under a leading _. It never reads a variable under the key's own spelling.
• Serve called twice runs the job once. Shutdown with a done context returns an error that wraps it, and Serve still returns once the run does.
• NewService with a nil run panics.

Migration from today

• job.NewApp(h), which bedrock ran as a Runtime, becomes humus.Main(Init, humus.WithTelemetry(buildTelemetry)), with Init returning []humus.Service{job.NewService(s, run)}.
• job.Handler and job.HandlerFunc go away. A type with Handle(ctx) error passes its method value, h.Handle, as the run.
• The handler's dependencies, built in main or in the Builder before, are built in Init and released with s.OnStop.
• A handler that ran with no telemetry now runs inside the job span, from s.Telemetry, which joins the trace in TRACEPARENT when the process's environment sets one.
• A job interrupted by SIGTERM used to return whatever its handler returned to bedrock's old runner. It now exits 1 unless the run returns nil.
• The docs site gains a page on jobs: the run, the exit-status table, Kubernetes' backoffLimit and podFailurePolicy for exit 78 and for disruptions, running a job beside other Services and which Main to use, and joining the trace of the process that started the job. The same story rewrites the configuration page's "The variables humus reads", which says humus reads exactly two variables. It is one of this proposal's stories.

Wording fixes to the post

• The note at the top now reads: wave 3, building on Proposal: humus.Main as well as the four proposals it names, and depending on no other wave-3 proposal. Proposal: exposing health probes is wave 4, and decides whether a job pod exposes probes.
• Out of scope's Health probes bullet reads "Proposal: exposing health probes, in wave 4, decides whether a job pod exposes them."
• Out of scope's last bullet ends "or composes several job.Service values in the end user's Init".

Open questions

None.

[Zaba505]

Closing: jobs have landed

The revision's stories are done:

• story(job): run a job once as a humus Service with job.NewService #702 added job.NewService, which runs a job once as a humus Service (story(job): run a job once as a humus Service with job.NewService #705);
• story(job): join the trace of the process that started the job #703 joined the trace of the process that started the job, through TRACEPARENT (story(job): join the trace of the process that started the job #713);
• story(docs): document jobs #704 documented jobs (story(docs): document jobs #714).

The revision left no open questions. What it hands on belongs to Proposal: exposing health probes, in wave 4, which decides whether a job pod exposes probes.