Proposal: OpenTelemetry without globals

[Zaba505]

Note

This post is superseded by two revisions below, read in order: telemetry owns its lifecycle integration, and the open questions answered, and settings the end user sources, which settles every open question. humus reads no variables for telemetry. The end user's Builder fills a plain telemetry.Settings and calls telemetry.New, and Services receive a telemetry.SDK. By default only logs are exported, to p.Stdout, as JSON lines in a format humus owns, with the trace context on every record logged in a span. The Background, Out of scope and References sections still apply. The rest of this post is kept for context.

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model. It depends on the charter and on Proposal: configuration for humus apps. It supplies the Telemetry field of Proposal: the humus App lifecycle, and it takes over the scope of #623.

Summary

A new package, telemetry, builds an App's OpenTelemetry providers with direct SDK calls, from settings read through p.Environment.

• Every Service receives the providers as one telemetry.Telemetry value, and the lifecycle shuts them down last.
• humus never reads or sets OTel's global providers, propagator or error handler, and never replaces slog.Default.
• Console exporters write to p.Stdout.
• Tests read spans, metrics and logs from in-memory recorders.

Background

• humus relies on OTel's global state everywhere:

  • rest builds its providers with bedrock's runtime/otel, whose BuildRuntime registers them globally (rest/rest.go#L256-L266). The error handler it registers discards every error: otel.ErrorHandlerFunc(func(err error) {}). Failed exports are invisible today.
  • grpc reads otel.GetMeterProvider, otel.GetTracerProvider and otel.GetTextMapPropagator (grpc/api.go#L45-L50).
  • queue/kafka gets its logger, tracer and meter from package-level functions over the globals (queue/kafka/otel.go#L17-L27).
  • humus.Logger and humus.LogHandler use otelslog's global logger provider (humus.go#L19-L26), and so does internal/grpchealth.

• The real process leaks in. resource.Default() reads OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES from the real environment (rest/rest.go#L235), and the stdout log exporter is wired to os.Stdout (rest/rest.go#L341).

• bedrock lists these as ambient leaks. Its package documentation says "OpenTelemetry's resource.Default reads the real environment, and its global providers are process-wide state". runtime/otel is gone, and MIGRATING.md maps each Build* function onto its SDK constructor.

• The SDK reads the real environment on its own (v1.46.0):

  • sdktrace.NewTracerProvider applies OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG and the OTEL_SPAN_* limits before its options (sdk/trace/provider.go:110).
  • The batch processors read OTEL_BSP_* and OTEL_BLRP_*.
  • The periodic reader reads OTEL_METRIC_EXPORT_*, and the meter provider reads its exemplar filter and cardinality limit.
  • The OTLP exporters read OTEL_EXPORTER_OTLP_* before their options, including certificate paths, which they then read from disk.
  • Experimental features are switched on by OTEL_GO_X_*.

  Explicit options override most of these, but not all.

• What story(issue-484): rest migrate to bedrock v0210 #485 deleted. Before it, humus had log-level filtering (internal/otel/filter.go) and its own stdout log exporter (feat(issue-181): implement custom stdout log exporter #182).

Proposed API

package telemetry // github.com/z5labs/humus/telemetry

// Telemetry is the set of OpenTelemetry providers an App hands its
// Services. It replaces OTel's globals: every humus package takes one.
type Telemetry struct {
	TracerProvider trace.TracerProvider
	MeterProvider  metric.MeterProvider
	LoggerProvider log.LoggerProvider
	Propagator     propagation.TextMapPropagator
}

// Logger returns a *slog.Logger named name whose records go to
// t.LoggerProvider, filtered by the App's log level.
func (t Telemetry) Logger(name string) *slog.Logger

// Noop returns a Telemetry that discards everything.
func Noop() Telemetry

// SDK is a Telemetry built from the OTel SDK, with the function that flushes
// and stops it.
type SDK struct {
	Telemetry
	Shutdown func(ctx context.Context) error
}

// Dialer opens connections, like (*net.Dialer).DialContext. It is the Dial
// field of humus.Host.
type Dialer func(ctx context.Context, network, address string) (net.Conn, error)

// Builder builds an App's telemetry. It is the Telemetry field of humus.App.
type Builder func(ctx context.Context, p bedrock.Process, dial Dialer, name string) (SDK, error)

// FromEnvironment is the Builder of a production App: ReadSettings from
// p.Environment, then New with p.Stdout as the console.
func FromEnvironment(ctx context.Context, p bedrock.Process, dial Dialer, name string) (SDK, error)

// NoopBuilder is a Builder whose Telemetry discards everything, for tests
// that don't assert on telemetry.
func NoopBuilder(ctx context.Context, p bedrock.Process, dial Dialer, name string) (SDK, error)

type Exporter string

const (
	ExporterNone    Exporter = "none"
	ExporterConsole Exporter = "console"
	ExporterOTLP    Exporter = "otlp"
)

type Settings struct {
	ServiceName        string               // OTEL_SERVICE_NAME
	ResourceAttributes []attribute.KeyValue // OTEL_RESOURCE_ATTRIBUTES
	Traces             Exporter             // OTEL_TRACES_EXPORTER
	Metrics            Exporter             // OTEL_METRICS_EXPORTER
	Logs               Exporter             // OTEL_LOGS_EXPORTER
	OTLP               OTLPSettings         // OTEL_EXPORTER_OTLP_ENDPOINT, _PROTOCOL, _HEADERS, _TIMEOUT, _INSECURE
	Sampler            sdktrace.Sampler     // OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG
	LogLevel           slog.Level           // HUMUS_LOG_LEVEL
	ShutdownTimeout    time.Duration        // HUMUS_TELEMETRY_SHUTDOWN_TIMEOUT
}

func ReadSettings(ctx context.Context, env bedrock.Environment, name string) (Settings, error)

// New builds SDK providers from s. Console exporters write to stdout, and
// OTLP exporters connect through dial.
func New(ctx context.Context, s Settings, dial Dialer, stdout io.Writer) (SDK, error)

A Service gets its instrumentation from s.Telemetry, never from the globals:

h := otelhttp.NewHandler(mux, "rest",
	otelhttp.WithTracerProvider(s.Telemetry.TracerProvider),
	otelhttp.WithMeterProvider(s.Telemetry.MeterProvider),
	otelhttp.WithPropagators(s.Telemetry.Propagator),
)

Settings

Variable	Default	Meaning
OTEL_SERVICE_NAME	the App's Name	service.name
OTEL_RESOURCE_ATTRIBUTES	(none)	extra resource attributes, key=value,…
OTEL_TRACES_EXPORTER	none	none, console or otlp
OTEL_METRICS_EXPORTER	none	none, console or otlp
OTEL_LOGS_EXPORTER	console	none, console or otlp
OTEL_EXPORTER_OTLP_ENDPOINT	localhost:4317	collector address
OTEL_EXPORTER_OTLP_PROTOCOL	grpc	grpc or http/protobuf
OTEL_EXPORTER_OTLP_HEADERS	(none)	key=value,…
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG	parentbased_always_on	the sampler, as in the OTel spec
HUMUS_LOG_LEVEL	info	debug, info, warn or error
HUMUS_TELEMETRY_SHUTDOWN_TIMEOUT	5s	how long the final flush may take

The exporter defaults keep today's behaviour: traces and metrics are discarded, and logs go to standard output. The OTel spec's default is otlp; see the open questions.

Design decisions

• Injected, never global. Every humus package takes a Telemetry, and third-party instrumentation gets its providers passed explicitly: otelhttp's options, grpc's stats/opentelemetry options, kotel, and otelslog.WithLoggerProvider. Lint forbids otel.Set*, otel.Get*, otel.Tracer, otel.Meter and global.SetLoggerProvider in humus. A user whose libraries read only the globals can set them in Init. That makes process-wide state their choice, and humus offers no helper for it.
• Standard variable names, read through p.Environment. humus passes every relevant SDK option explicitly, so the SDK's own environment reads are overridden: the sampler, span limits, batch settings, OTLP headers, TLS, compression, timeout and resource. The residual leaks, such as OTEL_GO_X_* and anything the SDK starts reading in a later release, are documented. A conformance test proves that explicit options win for the variables that matter. It runs on its own, not in parallel, against the real environment.
• The resource is built from Settings, with resource.NewWithAttributes and semconv. There is no resource.Default and no host or process detectors, because detectors read the filesystem and the real process.
• Console output goes to p.Stdout, so bedrocktest captures it.
• The log level is applied before the bridge, so a filtered record costs nothing downstream.
• Telemetry shuts down last, with its own timeout. The order is Services, then OnStop, then telemetry, so spans that record a slow shutdown are still exported.
• The package is named telemetry, not otel, so it doesn't clash with go.opentelemetry.io/otel at every import site.
• The propagator is W3C trace context plus baggage, as today. Stripping untrusted baggage, which Feature request: request-scoped failure injection (Netflix FIT) on OpenTelemetry baggage bedrock#578 calls for, belongs to a later proposal.

Testing

humustest.NewTelemetry() (Proposal: humustest) is a Builder over the real SDK with in-memory exporters: tracetest's span recorder, a sdkmetric.ManualReader and an in-memory log exporter. A test sees exactly what production would export:

func TestCreateCart_RecordsASpan(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost()
		tel := humustest.NewTelemetry()
		app := humus.App{Name: "carts", Host: host.Host(), Telemetry: tel.Builder(), Init: app.Init}

		p, status := humustest.Start(t, app)
		synctest.Wait()
		resp, err := host.HTTPClient().Post("http://carts:8080/carts", "application/json", nil)
		require.NoError(t, err)
		require.NoError(t, resp.Body.Close())
		p.Signal(syscall.SIGTERM)
		require.Zero(t, (<-status).Code())

		require.Equal(t, "POST /carts", tel.Spans()[0].Name())
	})
}

An end-to-end test can instead set OTEL_LOGS_EXPORTER=console with bedrocktest.WithEnv and assert on p.Stdout().

Migration from today

• rest.OTLPExporter(target) becomes OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER set to otlp, plus OTEL_EXPORTER_OTLP_ENDPOINT.
• humus.Logger(name) and humus.LogHandler(name) become s.Telemetry.Logger(name) and its Handler().
• grpc and queue/kafka take a Telemetry instead of reading the globals (in their wave-2 proposals).
• bedrock's runtime/otel Build* functions become the SDK calls inside telemetry.New, as mapped in bedrock's MIGRATING.md.

Open questions

1. Exporter defaults. Keep today's (none, none, console), or follow the spec's default of otlp, which tries to reach a collector on localhost:4317?
2. Console log format. OTel's stdoutlog JSON is verbose. Should humus bring back its own compact exporter (feat(issue-181): implement custom stdout log exporter #182), or use slog's JSON handler and bridge to the provider only for OTLP?
3. The global error handler. The SDK reports export errors through otel.Handle. Today humus registers a handler that throws them away; the default handler writes to the real stderr through log. Neither can be injected per provider. Should humus accept the leak, set the handler once (process-wide state), or log those errors through the App's logger by some other route?
4. Resource detectors (host, process, container, Kubernetes): opt-in, or not at all?
5. Telemetry.Logger. Should Telemetry expose a Logger method, or only the LoggerProvider plus a bridge helper?
6. OTEL_SDK_DISABLED. Honour it?
7. The package name: telemetry, otel or observability?

Out of scope

• Instrumentation of outbound clients, and FIT (Feature request: request-scoped failure injection (Netflix FIT) on OpenTelemetry baggage bedrock#578).
• A Prometheus exporter, profiling, metric views and exemplars.

References

• story(otel): inline OpenTelemetry setup in humus #623, closed in favour of this proposal
• bedrock's doc.go, "Ambient leaks"
• bedrock's MIGRATING.md, bedrock/runtime/otel
• OpenTelemetry SDK environment variables

[Zaba505]

Revision: telemetry owns its lifecycle integration

This revision applies the decisions in the lifecycle proposal's revision:

• This proposal owns telemetry's integration with the lifecycle, which the lifecycle leaves out: App.Telemetry, Setup.Telemetry, building telemetry before Init, logging through it, and shutting it down last. The new Lifecycle integration section specifies it.
• humus.App has no Name, so the Builder and ReadSettings lose their name parameter, and OTEL_SERVICE_NAME falls back to the OTel spec's default.
• Setup.Logger is gone, so s.Telemetry.Logger(name) is the only way to get a logger.

Each section below replaces the section of the same name in the post, and Lifecycle integration is new. Summary, Background, Migration from today, Out of scope and References are unchanged and still apply.

Proposed API

package telemetry // github.com/z5labs/humus/telemetry

// Telemetry is the set of OpenTelemetry providers an App hands its
// Services. It replaces OTel's globals: every humus package takes one.
type Telemetry struct {
	TracerProvider trace.TracerProvider
	MeterProvider  metric.MeterProvider
	LoggerProvider log.LoggerProvider
	Propagator     propagation.TextMapPropagator
}

// Logger returns a *slog.Logger named name whose records go to
// t.LoggerProvider, filtered by the App's log level.
func (t Telemetry) Logger(name string) *slog.Logger

// Noop returns a Telemetry that discards everything.
func Noop() Telemetry

// SDK is a Telemetry built from the OTel SDK, with the function that flushes
// and stops it.
type SDK struct {
	Telemetry
	Shutdown func(ctx context.Context) error
}

// Dialer opens connections, like (*net.Dialer).DialContext. It is the Dial
// field of humus.Host.
type Dialer func(ctx context.Context, network, address string) (net.Conn, error)

// Builder builds an App's telemetry. It is the Telemetry field of humus.App.
type Builder func(ctx context.Context, p bedrock.Process, dial Dialer) (SDK, error)

// FromEnvironment is the Builder of a production App: ReadSettings from
// p.Environment, then New with p.Stdout as the console. When
// OTEL_SERVICE_NAME is unset, it names the service
// unknown_service:<executable name>, from the base name of p.Args[0].
func FromEnvironment(ctx context.Context, p bedrock.Process, dial Dialer) (SDK, error)

// NoopBuilder is a Builder whose Telemetry discards everything, for tests
// that don't assert on telemetry.
func NoopBuilder(ctx context.Context, p bedrock.Process, dial Dialer) (SDK, error)

type Exporter string

const (
	ExporterNone    Exporter = "none"
	ExporterConsole Exporter = "console"
	ExporterOTLP    Exporter = "otlp"
)

type Settings struct {
	ServiceName        string               // OTEL_SERVICE_NAME; empty when unset
	ResourceAttributes []attribute.KeyValue // OTEL_RESOURCE_ATTRIBUTES
	Traces             Exporter             // OTEL_TRACES_EXPORTER
	Metrics            Exporter             // OTEL_METRICS_EXPORTER
	Logs               Exporter             // OTEL_LOGS_EXPORTER
	OTLP               OTLPSettings         // OTEL_EXPORTER_OTLP_ENDPOINT, _PROTOCOL, _HEADERS, _TIMEOUT, _INSECURE
	Sampler            sdktrace.Sampler     // OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG
	LogLevel           slog.Level           // HUMUS_LOG_LEVEL
	ShutdownTimeout    time.Duration        // HUMUS_TELEMETRY_SHUTDOWN_TIMEOUT
}

func ReadSettings(ctx context.Context, env bedrock.Environment) (Settings, error)

// New builds SDK providers from s. Console exporters write to stdout, and
// OTLP exporters connect through dial. An empty s.ServiceName becomes
// unknown_service, as in the OTel spec when the executable name is unknown.
func New(ctx context.Context, s Settings, dial Dialer, stdout io.Writer) (SDK, error)

A Service gets its instrumentation from s.Telemetry, never from the globals:

h := otelhttp.NewHandler(mux, "rest",
	otelhttp.WithTracerProvider(s.Telemetry.TracerProvider),
	otelhttp.WithMeterProvider(s.Telemetry.MeterProvider),
	otelhttp.WithPropagators(s.Telemetry.Propagator),
)

Settings

Variable	Default	Meaning
OTEL_SERVICE_NAME	unknown_service:<executable name>	service.name
OTEL_RESOURCE_ATTRIBUTES	(none)	extra resource attributes, key=value,…
OTEL_TRACES_EXPORTER	none	none, console or otlp
OTEL_METRICS_EXPORTER	none	none, console or otlp
OTEL_LOGS_EXPORTER	console	none, console or otlp
OTEL_EXPORTER_OTLP_ENDPOINT	localhost:4317	collector address
OTEL_EXPORTER_OTLP_PROTOCOL	grpc	grpc or http/protobuf
OTEL_EXPORTER_OTLP_HEADERS	(none)	key=value,…
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG	parentbased_always_on	the sampler, as in the OTel spec
HUMUS_LOG_LEVEL	info	debug, info, warn or error
HUMUS_TELEMETRY_SHUTDOWN_TIMEOUT	5s	how long the final flush may take

The exporter defaults keep today's behaviour: traces and metrics are discarded, and logs go to standard output. The OTel spec's default is otlp; see the open questions.

Lifecycle integration

The lifecycle lands without telemetry. This proposal's stories add it:

package humus

type App struct {
	// Host, DrainDelay and Init, as in the lifecycle proposal.

	// Telemetry builds the App's telemetry. It is required:
	// telemetry.FromEnvironment in main, and telemetry.NoopBuilder or
	// humustest's recorder in tests.
	Telemetry telemetry.Builder
}

type Setup struct {
	// Process and Host, as in the lifecycle proposal.

	// Telemetry is the App's telemetry. Init and the Services get their
	// tracers, meters and loggers from it.
	Telemetry telemetry.Telemetry
}

Run changes in three places:

1. Telemetry is built right after the lifecycle's settings are read, with App.Telemetry(ctx, p, Host.Dial), before NotifyContext and Init. A malformed setting exits 78, and any other failure exits 1. Both are written to p.Stderr, since there is no logger yet.
2. Failures are logged instead of written to stderr. Every later failure is logged through Telemetry.Logger("github.com/z5labs/humus") instead of being written to p.Stderr. The lifecycle also logs its own records there: "serving", "shutting down" and "stopped".
3. Telemetry shuts down last. It shuts down after the OnStop functions, or after a failed Init and its OnStop functions, within HUMUS_TELEMETRY_SHUTDOWN_TIMEOUT rather than the grace period. A failure to shut it down is written to p.Stderr and joined into the cause, and turns an exit 0 into 1.

The integration's tests show the following:

• A malformed OTEL_TRACES_EXPORTER exits 78 with a *config.Error that names it, and never calls Init.
• A failed Init is logged, and appears in humustest.Telemetry's Logs() rather than p.Stderr().
• A span recorded by an OnStop function is exported, because telemetry shuts down after them.
• A telemetry shutdown that fails turns exit 0 into 1, with its error on stderr.

Design decisions

• Injected, never global. Every humus package takes a Telemetry, and third-party instrumentation gets its providers passed explicitly: otelhttp's options, grpc's stats/opentelemetry options, kotel, and otelslog.WithLoggerProvider. Lint forbids otel.Set*, otel.Get*, otel.Tracer, otel.Meter and global.SetLoggerProvider in humus. A user whose libraries read only the globals can set them in Init. That makes process-wide state their choice, and humus offers no helper for it.
• Standard variable names, read through p.Environment. humus passes every relevant SDK option explicitly, so the SDK's own environment reads are overridden: the sampler, span limits, batch settings, OTLP headers, TLS, compression, timeout and resource. The residual leaks, such as OTEL_GO_X_* and anything the SDK starts reading in a later release, are documented. A conformance test proves that explicit options win for the variables that matter. It runs on its own, not in parallel, against the real environment.
• The service name comes from OTEL_SERVICE_NAME, or the spec's default. The App has no name of its own. With the variable unset, FromEnvironment uses unknown_service:<executable name>, taking the executable name from the base name of p.Args[0]. bedrock's Process.Args starts with the program name, so this reads nothing from the real process. Under humustest it's unknown_service:bedrocktest. ReadSettings can't fill in the default, because the environment doesn't know the program's name.
• The resource is built from Settings, with resource.NewWithAttributes and semconv. There is no resource.Default and no host or process detectors, because detectors read the filesystem and the real process.
• Console output goes to p.Stdout, so humustest captures it.
• The log level is applied before the bridge, so a filtered record costs nothing downstream.
• The lifecycle logs through telemetry. Its own records ("serving", "shutting down", "stopped") and every failure after telemetry is built go to Telemetry.Logger("github.com/z5labs/humus"), not to p.Stderr. With the default OTEL_LOGS_EXPORTER=console, those failures appear on p.Stdout. With none, a failure after startup shows only in the exit code.
• Telemetry shuts down last, with its own timeout. The order is Services, then OnStop, then telemetry, so the spans and logs that record a slow shutdown, including its failures, are still exported.
• The package is named telemetry, not otel, so it doesn't clash with go.opentelemetry.io/otel at every import site.
• The propagator is W3C trace context plus baggage, as today. Stripping untrusted baggage, which Feature request: request-scoped failure injection (Netflix FIT) on OpenTelemetry baggage bedrock#578 calls for, belongs to a later proposal.

Testing

humustest.NewTelemetry() (Proposal: humustest) is a Builder over the real SDK with in-memory exporters: tracetest's span recorder, a sdkmetric.ManualReader and an in-memory log exporter. A test sees exactly what production would export:

func TestCreateCart_RecordsASpan(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost(t)
		tel := humustest.NewTelemetry()
		app := humus.App{Host: host.Host(), Telemetry: tel.Builder(), Init: app.Init}

		p := humustest.Start(t, app)
		synctest.Wait()
		resp, err := host.HTTPClient().Post("http://carts:8080/carts", "application/json", nil)
		require.NoError(t, err)
		require.NoError(t, resp.Body.Close())
		p.Signal(syscall.SIGTERM)
		require.Zero(t, p.Wait().Code())

		require.Equal(t, "POST /carts", tel.Spans()[0].Name())
	})
}

An end-to-end test can instead set OTEL_LOGS_EXPORTER=console with humustest.WithEnv and assert on p.Stdout().

Open questions

1. Exporter defaults. Keep today's (none, none, console), or follow the spec's default of otlp, which tries to reach a collector on localhost:4317? Now that the lifecycle logs failures through telemetry, the logs default also decides where an operator first sees a failed App.
2. Console log format. OTel's stdoutlog JSON is verbose. Should humus bring back its own compact exporter (feat(issue-181): implement custom stdout log exporter #182), or use slog's JSON handler and bridge to the provider only for OTLP?
3. The global error handler. The SDK reports export errors through otel.Handle. Today humus registers a handler that throws them away; the default handler writes to the real stderr through log. Neither can be injected per provider. Should humus accept the leak, set the handler once (process-wide state), or log those errors through the App's logger by some other route?
4. Resource detectors (host, process, container, Kubernetes): opt-in, or not at all?
5. Telemetry.Logger. Should Telemetry expose a Logger method, or only the LoggerProvider plus a bridge helper? The lifecycle's revision now uses s.Telemetry.Logger(name) throughout, so dropping the method would change that proposal too.
6. OTEL_SDK_DISABLED. Honour it?
7. The package name: telemetry, otel or observability?

[Zaba505]

Revision 2: the open questions answered, and settings the end user sources

This revision answers the seven open questions of the lifecycle integration revision. It also applies where Proposal: configuration for humus apps landed in its final revision: humus packages take plain Go values, and the end user sources them. So FromEnvironment and ReadSettings are gone. The end user's own Builder fills telemetry.Settings and calls telemetry.New.

Each section below replaces the section of the same name in that revision, and Summary and Migration from today replace the post's. Background, Out of scope and References from the post still apply.

Decisions on the open questions

#	Question	Decision
1	Exporter defaults	Keep them. Only logs are exported by default, to p.Stdout. Traces aren't exported, but the TracerProvider is still the SDK's, so spans get real IDs and the logs carry them. Metrics are a no-op. These are the values DefaultSettings() returns, not defaults of variables.
2	Console log format	humus's own JSON format, written by an unexported exporter. OTel's stdoutlog does log the trace context, but its format is not standardized. humus controls the JSON it writes by default. See below.
3	The global error handler	Set it. Once the SDK is built, Run sets OTel's process-wide error handler, which writes each export error to p.Stderr.
4	Resource detectors	The end user's choice, in Settings. Settings.ResourceOptions takes resource.Options, such as resource.WithHost() or a contrib cloud detector. By default no detector runs.
5	Telemetry.Logger	Dropped, with no bridge helper. SDK exposes the LoggerProvider only. The end user either uses the OTel Logs API directly or bridges it to their own logging library.
6	OTEL_SDK_DISABLED	Supported for compatibility, as Settings.Disabled. Because of #631, humus reads no variables, so the end user's Builder maps OTEL_SDK_DISABLED onto the field. The Go SDK doesn't read the variable itself (checked at v1.47.0), so nothing leaks around it.
7	Package name	telemetry, and the type is telemetry.SDK. It replaces telemetry.Telemetry. The old SDK wrapper merges into it.
new	OTEL_* after #631	humus reads none of them. There is no FromEnvironment, and no helper like it. The reference example maps the spec's variables onto Settings.

Question 2: humus's console log format

OTel's stdoutlog does log the trace context. A probe against v0.23.0 shows TraceID, SpanID and TraceFlags on every record, and they have been there since its first release, v0.2.0-alpha (April 2024). But its format is not standardized. The package doc says: "The exporter is intended to be used for testing and debugging; it is not meant for production use. Additionally, it does not provide an interchange format for OpenTelemetry that is supported with any stability or compatibility guarantees." It's verbose as well: every line repeats the whole resource.

Console logs are humus's default, in production too, so humus owns their format, as #182 did. New uses an unexported exporter when Logs is ExporterConsole, and humus doesn't depend on stdoutlog. Each record is one line of JSON:

{"timestamp":"2026-10-16T06:56:32.447829601Z","severity_text":"INFO","severity_number":9,"body":"cart created","trace_id":"e2f668b6c06f2de82e61e2517c2d7a07","span_id":"ff25eb972a5557d5","trace_flags":"01","scope":"carts","attributes":{"cart.id":"c-123"}}

The keys are written in this order:

Key	Value	Present
timestamp	the record's timestamp, or its observed timestamp when the timestamp is unset, as RFC 3339 with nanoseconds, in UTC	always
severity_text	the record's severity text, or, when that's empty, the short name of its severity number, such as INFO2	when the severity is set
severity_number	the OTel severity number, 1 to 24, for filtering by threshold	when the severity is set
event_name	the record's event name	when set
body	a string as is, a map as an object, a slice (typed slices included) as an array, and bytes as base64	when set
trace_id, span_id, trace_flags	lowercase hex	when the record has a valid span context
scope	the instrumentation scope's name	always
attributes	an object, with nested maps and slices as objects and arrays	when there are any

• The keys are the OTel log data model's field names, in snake_case. For the trace context, that's what the stable spec on trace context in non-OTLP log formats recommends: top-level trace_id, span_id and trace_flags, in lowercase hex. So a log pipeline can link a line to its trace by the names the spec gives.
• A record without a span context has no trace fields, rather than zeros. The spec makes all three optional, and a zero ID would point to a trace that doesn't exist.
• There is no resource. OTLP carries the resource. On stdout, whatever collects the logs, such as a Kubernetes log agent, adds its own metadata. Repeating the resource on every line is what makes stdoutlog verbose.
• Every value is valid JSON. A NaN or infinite float is written as the string "NaN", "+Inf" or "-Inf".
• The format is part of humus's API. The telemetry package doc specifies it, a golden test pins it, and changing it is a breaking change.

Traces and metrics on the console keep OTel's stdouttrace and stdoutmetric. They're off by default, and on the console they're for debugging, so humus makes no promise about their format.

Summary

A new package, telemetry, builds an App's OpenTelemetry providers with direct SDK calls. It builds them from a plain Settings value that the end user sources.

• Every Service receives the providers as one telemetry.SDK value, and the lifecycle shuts them down last.
• humus reads no variables for telemetry. The end user's Builder fills Settings from the process however it likes, then calls telemetry.New.
• By default only logs are exported, to p.Stdout, as JSON lines in a format humus owns. Traces aren't exported, but spans still get real IDs, so every record logged in a span carries its trace context, even on a laptop with no collector. Metrics are a no-op.
• humus never reads or sets OTel's global providers or propagator, and never replaces slog.Default. It sets one piece of global state, OTel's error handler, so that export errors reach p.Stderr.
• Tests read spans, metrics and logs from in-memory recorders.

Proposed API

package telemetry // github.com/z5labs/humus/telemetry

// SDK is the set of OpenTelemetry providers an App hands its Services. It
// replaces OTel's globals: every humus package takes one.
type SDK struct {
	TracerProvider trace.TracerProvider
	MeterProvider  metric.MeterProvider
	LoggerProvider log.LoggerProvider
	Propagator     propagation.TextMapPropagator

	// Shutdown flushes and stops the providers. humus.App.Run calls it after
	// the OnStop functions. Init and the Services never call it.
	Shutdown func(ctx context.Context) error
}

// Noop returns an SDK whose providers discard everything and whose Shutdown
// does nothing. Its propagator is still W3C trace context plus baggage, so a
// request's trace passes through a service whose telemetry is off.
func Noop() SDK

// Dialer opens connections, like (*net.Dialer).DialContext. It is the Dial
// field of humus.Host.
type Dialer func(ctx context.Context, network, address string) (net.Conn, error)

// Builder builds an App's SDK. It is the Telemetry field of humus.App. The
// end user writes it: it sources Settings from p and calls New.
type Builder func(ctx context.Context, p bedrock.Process, dial Dialer) (SDK, error)

// NoopBuilder is a Builder that returns Noop(), for tests that don't assert
// on telemetry.
func NoopBuilder(ctx context.Context, p bedrock.Process, dial Dialer) (SDK, error)

type Exporter string

const (
	ExporterNone    Exporter = "none"
	ExporterConsole Exporter = "console"
	ExporterOTLP    Exporter = "otlp"
)

type Protocol string

const (
	ProtocolGRPC         Protocol = "grpc"
	ProtocolHTTPProtobuf Protocol = "http/protobuf"
)

// Settings configures New. Every field is a plain value, and a field's zero
// value means zero, never "use the default".
type Settings struct {
	Disabled        bool
	ServiceName     string
	ResourceOptions []resource.Option
	Traces          Exporter
	Metrics         Exporter
	Logs            Exporter
	OTLP            OTLPSettings
	Sampler         sdktrace.Sampler
	LogLevel        log.Severity
	ShutdownTimeout time.Duration
}

type OTLPSettings struct {
	Endpoint string
	Protocol Protocol
	Headers  map[string]string
	TLS      *tls.Config
	Timeout  time.Duration
}

// DefaultSettings returns New's defaults.
func DefaultSettings() Settings

// New builds SDK providers from s. It runs every check, and reports every bad
// field in one error, naming each one. With s.Disabled, it ignores every
// other field and returns Noop(). Console exporters write to stdout, which
// need not be safe for concurrent use. OTLP exporters connect through dial.
func New(ctx context.Context, s Settings, dial Dialer, stdout io.Writer) (SDK, error)

// The checks, one for each field with a valid range, as #631 sets out. Each
// reports what is wrong, without naming the field.
func CheckServiceName(name string) error
func CheckExporter(e Exporter) error // Traces, Metrics and Logs share it
func CheckOTLPEndpoint(addr string) error
func CheckOTLPProtocol(p Protocol) error
func CheckOTLPTimeout(d time.Duration) error
func CheckSampler(s sdktrace.Sampler) error
func CheckLogLevel(l log.Severity) error
func CheckShutdownTimeout(d time.Duration) error

The end user writes the Builder, in the same way they source any other package's settings (#631). This one keeps the OTel spec's variable names, but the names are the end user's choice:

func buildTelemetry(ctx context.Context, p bedrock.Process, dial telemetry.Dialer) (telemetry.SDK, error) {
	d := telemetry.DefaultSettings()
	env := func(name string) config.Value[string] { return config.Env(p.Environment, name) }
	exporter := func(name string, def telemetry.Exporter) config.Value[telemetry.Exporter] {
		return env(name).
			Map(func(s string) telemetry.Exporter { return telemetry.Exporter(s) }).
			Default(def).
			Validate(telemetry.CheckExporter)
	}

	s, err := config.Struct(func(r *config.Resolver) telemetry.Settings {
		s := d
		s.Disabled = r.Get(env("OTEL_SDK_DISABLED").Parse(strconv.ParseBool).Default(d.Disabled))
		s.ServiceName = r.Get(env("OTEL_SERVICE_NAME").Default("carts").Validate(telemetry.CheckServiceName))
		s.Traces = r.Get(exporter("OTEL_TRACES_EXPORTER", d.Traces))
		s.Metrics = r.Get(exporter("OTEL_METRICS_EXPORTER", d.Metrics))
		s.Logs = r.Get(exporter("OTEL_LOGS_EXPORTER", d.Logs))
		s.ResourceOptions = []resource.Option{resource.WithHost(), resource.WithContainer()} // opting in to detectors
		return s
	}).Read(ctx)
	if err != nil {
		return telemetry.SDK{}, err // exit 78
	}
	return telemetry.New(ctx, s, dial, p.Stdout)
}

func main() {
	bedrock.Main(humus.App{
		Host:                humus.OSHost(),
		ShutdownGracePeriod: 10 * time.Second,
		Telemetry:           buildTelemetry,
		Init:                app.Init,
	})
}

A Service gets its instrumentation from s.Telemetry, never from the globals. It brings its own log bridge:

h := otelhttp.NewHandler(mux, "rest",
	otelhttp.WithTracerProvider(s.Telemetry.TracerProvider),
	otelhttp.WithMeterProvider(s.Telemetry.MeterProvider),
	otelhttp.WithPropagators(s.Telemetry.Propagator),
)
logger := otelslog.NewLogger("carts", otelslog.WithLoggerProvider(s.Telemetry.LoggerProvider))
// In a handler: logger.InfoContext(r.Context(), "cart created"), so the record carries the request's span.

Settings

The field godoc replaces the variable table. The last column lists the OTel spec variable that has the same meaning, for an end user who wants to stay compatible. humus reads none of them.

Field	Default	Valid	Check	Spec variable
Disabled	false	any		OTEL_SDK_DISABLED
ServiceName	"unknown_service"	not empty	CheckServiceName	OTEL_SERVICE_NAME
ResourceOptions	none, so no detector runs	any		OTEL_RESOURCE_ATTRIBUTES, as resource.WithAttributes
Traces	ExporterNone, under which spans still get IDs but aren't exported	none, console, otlp	CheckExporter	OTEL_TRACES_EXPORTER
Metrics	ExporterNone	the same	CheckExporter	OTEL_METRICS_EXPORTER
Logs	ExporterConsole	the same	CheckExporter	OTEL_LOGS_EXPORTER
OTLP.Endpoint	"localhost:4317"	host:port	CheckOTLPEndpoint	OTEL_EXPORTER_OTLP_ENDPOINT, which is a URL in the spec
OTLP.Protocol	ProtocolGRPC	grpc, http/protobuf	CheckOTLPProtocol	OTEL_EXPORTER_OTLP_PROTOCOL
OTLP.Headers	none	any		OTEL_EXPORTER_OTLP_HEADERS
OTLP.TLS	nil, which means plaintext	any		OTEL_EXPORTER_OTLP_CERTIFICATE and the related variables
OTLP.Timeout	10s	positive	CheckOTLPTimeout	OTEL_EXPORTER_OTLP_TIMEOUT
Sampler	ParentBased(AlwaysSample())	not nil	CheckSampler	OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG
LogLevel	log.SeverityInfo	SeverityTrace1 to SeverityFatal4	CheckLogLevel	(none)
ShutdownTimeout	5s	positive	CheckShutdownTimeout	(none)

HUMUS_LOG_LEVEL and HUMUS_TELEMETRY_SHUTDOWN_TIMEOUT are gone. LogLevel is an OTel severity now, not a slog.Level, because humus has no slog in its API.

Lifecycle integration

package humus

type App struct {
	// Host, ShutdownGracePeriod, DrainDelay and Init, as in the lifecycle
	// and configuration proposals.

	// Telemetry builds the App's SDK. It is required: the end user's own
	// Builder in main, and telemetry.NoopBuilder or humustest's recorder in
	// tests.
	Telemetry telemetry.Builder
}

type Setup struct {
	// Process and Host, as in the lifecycle proposal.

	// Telemetry is the App's SDK. Init and the Services get their tracers,
	// meters and loggers from it.
	Telemetry telemetry.SDK
}

Run changes in four places:

1. Telemetry is built after NotifyContext, before Init. Run calls App.Telemetry(ctx, p, Host.Dial) with the context that a signal cancels. A Builder that runs detectors can be slow, because a cloud detector calls its metadata server. So a signal cancels it, as it cancels a slow Init. The previous revision built telemetry before NotifyContext. If the error chain holds a *config.Error, the App exits 78. Any other failure exits 1, and so does an SDK with a nil field. These failures are written to p.Stderr, because there is no SDK yet.
2. OTel's error handler is set. It writes each error to p.Stderr as one line that starts with telemetry: . This is the only call to otel.SetErrorHandler in humus. An end user who wants a different handler sets it in Init, which runs after this.
3. Failures are logged instead of written to stderr. Every later failure is emitted at ERROR, with the OTel Logs API, through SDK.LoggerProvider.Logger("github.com/z5labs/humus"). So are the lifecycle's own records, "serving", "shutting down" and "stopped", at INFO. humus has no slog bridge of its own.
4. Telemetry shuts down last. SDK.Shutdown runs after the OnStop functions, or after a failed Init and its OnStop functions. New bounds it with Settings.ShutdownTimeout, separately from the grace period. A failure to shut it down is written to p.Stderr and joined into the cause, and it turns an exit 0 into 1.

The integration's tests show the following:

• A malformed OTEL_TRACES_EXPORTER, read by the end user's Builder, exits 78 with a *config.Error that names it, and Init is never called.
• A signal during a slow Builder cancels it, and Init is never called.
• A failed Init is logged, and appears in humustest.Telemetry's Logs() rather than in p.Stderr().
• A span recorded by an OnStop function is exported, because telemetry shuts down after them.
• On humustest's Host, a span exported over OTLP to an address that nothing listens on writes telemetry: and the connection error to p.Stderr.
• A telemetry shutdown that fails turns exit 0 into 1, with its error on stderr.

Design decisions

• Injected, never global, with one exception. Every humus package takes a telemetry.SDK, and third-party instrumentation gets its providers passed explicitly: otelhttp's options, grpc's stats/opentelemetry options, kotel, and otelslog.WithLoggerProvider. Lint forbids otel.Set*, otel.Get*, otel.Tracer, otel.Meter and global.SetLoggerProvider in humus. The one exception is the call to otel.SetErrorHandler in Run, which is marked with a nolint that names this proposal. A user whose libraries read only the global providers can set them in Init.
• The end user sources Settings (Proposal: configuration for humus apps #631). humus has no FromEnvironment, reads no OTEL_* variable and has no parser for the spec's formats. The reference example maps the spec's variables onto Settings. The cost: a platform that injects OTEL_* variables, such as the OpenTelemetry Operator, has no effect on an App unless its Builder reads them.
• Every SDK option is passed explicitly, so the SDK's own reads of the real environment are overridden: the sampler, the span and log record limits, the batch settings, the OTLP headers, TLS, compression, the timeout and the resource. sdk/log is stable as of v1.47.0, and it reads OTEL_BLRP_* and OTEL_LOGRECORD_* like the rest. A conformance test proves that explicit options win for the variables that matter. It runs against the real environment, on its own and not in parallel. The leaks that remain are documented: OTEL_GO_X_*, OTel's internal logger (which writes its own errors to the real stderr), and any detector the end user opts into.
• Console logs use humus's own format, as covered under question 2.
• The trace context comes from the log call's context. A record has a span context only when the log call is given the span's context. With otelslog, that means logger.InfoContext(ctx, …). logger.Info(…) uses context.Background(), so the record has no trace fields. That's true of any exporter.
• With traces at none, the TracerProvider is still the SDK's. New builds it with the Sampler and the resource, but with no span processor. So every span gets a real trace ID and span ID, and the logs carry them, but nothing is exported. That's the usual local setup: you want to group the logs on your stdout by request, without running a collector. A no-op TracerProvider would leave every ID at zero, unless a request brought a trace in.
  • Sampling is unchanged. The provider keeps the configured Sampler, so the sampled flag passed on to downstream services is the same as with an exporter. A sampled span is still recorded in memory until it ends, and is then dropped.
  • The cost is a gap in a shared trace. In production, a service at none in the middle of a traced call chain passes on span IDs that no backend ever receives. The trace ID is still the caller's, so the trace holds together, but the downstream spans point to a parent that's missing. A service that takes part in a traced system sets Traces to otlp.
• No logging bridge in humus. SDK exposes the LoggerProvider, and the end user picks the OTel Logs API or a bridge (otelslog, otelzap, otellogrus and so on). humus's own records use the Logs API directly. humus never replaces slog.Default.
• The log level is applied in the LoggerProvider. New wraps the log processor, so that its Enabled reports false below LogLevel. A bridge that asks Enabled before it builds a record skips filtered records at no cost, and otelslog's handler does ask.
• Export errors go to p.Stderr. Telemetry can't report its own failure. A failing OTLP log exporter would swallow the errors about itself, and every error would queue another log record that is bound to fail. So export errors follow the same rule as build and shutdown failures. The handler is process-wide: in a process with several Apps, the last App to start gets every App's export errors. In parallel tests, one App can see another's errors, so a test that asserts on export errors doesn't run in parallel.
• Disabled means Noop(). No exporter is built and nothing is dialled. The propagator stays, because the spec disables signals, not propagation. Unlike traces at none, a disabled SDK makes no IDs of its own: it passes on the span context that a request brought in.
• The resource is ServiceName plus the end user's ResourceOptions. By default no detector runs, because detectors read the real host and bypass the Host: the host, process and container detectors read /etc, /proc and cgroups, and cloud detectors dial their metadata servers. ServiceName wins over anything a detector finds. DefaultSettings can't see the Process, so it names the service unknown_service, which is the spec's name when the executable is unknown. The end user's Builder names the service, and unknown_service: plus the base name of p.Args[0] takes one line there.
• SDK is one plain struct, Shutdown included. An end user who needs something Settings doesn't cover, such as a Prometheus reader or metric views, builds the SDK value in their Builder, and humustest's recorder does the same. Shutdown is a field so that such an SDK can carry its own.
• Telemetry shuts down last, with its own timeout. The order is Services, then OnStop, then telemetry, so the spans and logs that record a slow shutdown, its failures included, are still exported.
• The package is named telemetry, not otel, so it doesn't clash with go.opentelemetry.io/otel at every import site. The type is SDK, so that call sites don't read telemetry.Telemetry.
• The propagator is W3C trace context plus baggage, as today. Stripping untrusted baggage, which Feature request: request-scoped failure injection (Netflix FIT) on OpenTelemetry baggage bedrock#578 calls for, belongs to a later proposal.

Testing

Proposal: humustest's last comment hands the telemetry recorder to this proposal, so this proposal adds it to humustest:

package humustest

// Telemetry records everything an App emits, using the real SDK with
// in-memory exporters: tracetest's span recorder, an sdkmetric.ManualReader
// and an in-memory log exporter. It records every severity.
type Telemetry struct{ /* unexported */ }

func NewTelemetry() *Telemetry
func (t *Telemetry) Builder() telemetry.Builder
func (t *Telemetry) Spans() []sdktrace.ReadOnlySpan
func (t *Telemetry) Metrics(ctx context.Context) (metricdata.ResourceMetrics, error)
func (t *Telemetry) Logs() []sdklog.Record

A whole-App test sees exactly what production would export:

func TestCreateCart_RecordsASpan(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost(t)
		tel := humustest.NewTelemetry()
		app := humus.App{
			Host:                host.Host(),
			ShutdownGracePeriod: 10 * time.Second,
			Telemetry:           tel.Builder(),
			Init:                app.Init,
		}

		p, status := bedrocktest.Start(t, app)
		synctest.Wait()
		resp, err := host.HTTPClient().Post("http://carts:8080/carts", "application/json", nil)
		require.NoError(t, err)
		require.NoError(t, resp.Body.Close())
		p.Signal(syscall.SIGTERM)
		require.Zero(t, (<-status).Code())

		require.Equal(t, "POST /carts", tel.Spans()[0].Name())
	})
}

A golden test pins the console format from question 2, byte for byte. It passes against a prototype of the exporter on sdk/log v1.47.0, where the stable Logs API takes bodies and attributes as attribute.Value:

func TestConsoleExporter_WritesOneJSONLinePerRecord(t *testing.T) {
	t.Parallel()

	var out bytes.Buffer
	lp := sdklog.NewLoggerProvider(sdklog.WithProcessor(sdklog.NewSimpleProcessor(newConsoleExporter(&out))))
	sc := trace.NewSpanContext(trace.SpanContextConfig{
		TraceID:    trace.TraceID{0xe2, 0xf6, 0x68, 0xb6, 0xc0, 0x6f, 0x2d, 0xe8, 0x2e, 0x61, 0xe2, 0x51, 0x7c, 0x2d, 0x7a, 0x07},
		SpanID:     trace.SpanID{0xff, 0x25, 0xeb, 0x97, 0x2a, 0x55, 0x57, 0xd5},
		TraceFlags: trace.FlagsSampled,
	})

	var r log.Record
	r.SetTimestamp(time.Date(2026, 10, 16, 6, 56, 32, 447829601, time.UTC))
	r.SetSeverity(log.SeverityInfo)
	r.SetSeverityText("INFO")
	r.SetBody(attribute.StringValue("cart created"))
	r.AddAttributes(attribute.String("cart.id", "c-123"))
	lp.Logger("carts").Emit(trace.ContextWithSpanContext(t.Context(), sc), r)
	require.NoError(t, lp.Shutdown(t.Context()))

	require.Equal(t, `{"timestamp":"2026-10-16T06:56:32.447829601Z","severity_text":"INFO","severity_number":9,`+
		`"body":"cart created","trace_id":"e2f668b6c06f2de82e61e2517c2d7a07","span_id":"ff25eb972a5557d5",`+
		`"trace_flags":"01","scope":"carts","attributes":{"cart.id":"c-123"}}`+"\n", out.String())
}

The console exporter's other tests show the following:

• A record without a span context has no trace_id, span_id or trace_flags.
• A record with an empty severity text gets the short name of its severity number.
• Maps, slices and bytes, in the body and in attributes, are written as objects, arrays and base64.
• A NaN attribute is written as "NaN", and the line is still valid JSON.
• Through New with DefaultSettings(), so traces at none and logs at the console, a record logged with otelslog's InfoContext, inside a span of the SDK's own TracerProvider, carries that span's trace_id and span_id. The only output is that log line.

The package's own tests follow #631's list:

• Each check accepts an in-range value with nil, and rejects an out-of-range value without naming the field.
• New given a field out of range names that field, and two bad fields are reported together.
• DefaultSettings() passes New.
• With Traces at none, spans have valid IDs and the configured sampler's decision, nothing is exported, and nothing is dialled.
• Disabled returns Noop() and dials nothing.
• LogLevel makes Enabled false below it.
• The package never reads a bedrock.Environment and doesn't import bedrock/config.

An end-to-end test can leave Logs at the console default and assert on p.Stdout().

Migration from today

• rest.OTLPExporter(target) becomes Traces, Metrics and Logs set to ExporterOTLP, with OTLP.Endpoint set to target, in the end user's Builder.
• humus.Logger(name) and humus.LogHandler(name) become the end user's own bridge, such as otelslog.NewLogger(name, otelslog.WithLoggerProvider(s.Telemetry.LoggerProvider)) and otelslog.NewHandler(…).
• feat(issue-181): implement custom stdout log exporter #182's console exporter, which story(issue-484): rest migrate to bedrock v0210 #485 deleted, comes back as the format under question 2. It uses the spec's top-level trace_id and span_id, not feat(issue-181): implement custom stdout log exporter #182's otel group with trace.id and span.id, and it isn't built on slog.
• OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES, which resource.Default read from the real environment, are read only if the end user's Builder maps them.
• grpc and queue/kafka take a telemetry.SDK instead of reading the globals, in their wave-2 proposals.
• bedrock's runtime/otel Build* functions become the SDK calls inside telemetry.New, as mapped in bedrock's MIGRATING.md.

Consequences for the other proposals

• Proposal: the humus App lifecycle #630 (lifecycle). Its migration line "humus.Logger(name) becomes s.Telemetry.Logger(name)" is replaced by the bridge line above. Telemetry is now built after NotifyContext, not before it.
• Proposal: humus on the bedrock process model #629 (charter). Principle 3 and the forbidigo list gain one exception: Run's call to otel.SetErrorHandler.
• Proposal: humustest — whole-service testing on top of bedrocktest #633 (humustest). humustest.Telemetry is specified under Testing above.
• Wave 2. Runtime packages take a telemetry.SDK from Setup.Telemetry. Each runtime package's App says whether the end user must give it a Builder, or whether it defaults to telemetry.New with DefaultSettings().

Open questions

None.