Proposal: humustest — whole-service testing on top of bedrocktest

[Zaba505]

Note

The Summary, Proposed API, design decisions, testing, migration, open questions and prototype in this post are superseded by the revision below. It applies the decisions of the lifecycle proposal, and records that this proposal owns humustest's integration with the lifecycle. The Host serves real files from the test's own temp directory, and humustest wraps bedrocktest so a test imports humustest alone. Open questions 1 and 6 are answered. The rest of this post is unchanged and kept for context.

A second revision supersedes the same sections of the first. It answers the remaining open questions and reverses the answer to question 1. Start moves to bedrocktest (z5labs/bedrock#600), so humustest no longer wraps bedrocktest, and a test imports both. The network uses the host name in each address. Dial refuses when nothing is listening, and tests wait for a listener with synctest.Wait or Host.Listening. TLS keys, certificates and passwords are generated in each test. The Kafka question moves to the Kafka proposal.

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model. It depends on Proposal: the humus App lifecycle, which defines Host and Service. It also sets the testing requirements that every package proposal has to meet.

Summary

humus is being rebuilt on the process model so that a whole service can run inside a test.

• bedrocktest supplies the process.
• humustest supplies everything else:
  • an in-memory Host, for the network and files;
  • an in-memory telemetry recorder;
  • a helper that starts an App;
  • a contract test that every Service must pass.

With both, every humus App runs in a testing/synctest bubble without a socket, a file, the real environment or real time.

Background

• bedrocktest builds a fake Process with NewProcess(t, WithArgs, WithEnv, WithStdin). It drives signals with Notified and Signal, and records output in Stdout and Stderr. It is strict:

  • Signal with no live registration fails the test;
  • a NotifyContext that is never stopped fails the test.

  It starts no goroutines, so it works inside synctest (bedrocktest).

• bedrock's reference service test hand-rolls an in-memory listener over net.Pipe (example/httpserver/app_test.go). Without humustest, every humus user would write the same one.

• humus tests today:

  • the Kafka runtime is tested against testcontainers behind -tags testcontainers (queue/kafka/kafka_testcontainer.go), which needs docker;
  • story(kafka): add unit tests using kfake #431 proposes kfake instead;
  • the README's coverage badge reads 55.9%.

• What the lifecycle prototype found:

  • synctest, net.Pipe and bedrocktest make shutdown tests deterministic: a 2s grace period ends after exactly 2s of fake time;
  • synctest reports a deadlock when a goroutine the App started outlives the test body, so it doubles as a leak detector for Services.

Proposed API

package humustest

// Host is an in-memory humus.Host: a network of net.Pipe connections and a
// fixed set of files. It starts no goroutines, so it works inside a
// testing/synctest bubble; build it inside the bubble that uses it.
type Host struct{ /* unexported */ }

type Option func(*Host)

// WithFiles sets the files ReadFile returns, keyed by the exact name
// ReadFile is called with, so absolute OS paths work.
func WithFiles(files map[string][]byte) Option

func NewHost(opts ...Option) *Host

// Host returns the humus.Host to run an App on, like bedrocktest's
// Process.Process.
func (h *Host) Host() humus.Host

// Listen listens on the port of address. The host part is ignored, and
// port 0 picks a free port. A port already in use fails with EADDRINUSE.
func (h *Host) Listen(network, address string) (net.Listener, error)

// Dial connects to the listener on the port of address. It fails with
// ECONNREFUSED when nothing listens there.
func (h *Host) Dial(ctx context.Context, network, address string) (net.Conn, error)

// ReadFile returns a file set with WithFiles, or an error satisfying
// errors.Is(err, fs.ErrNotExist).
func (h *Host) ReadFile(name string) ([]byte, error)

// HTTPClient returns an http.Client that dials the Host, with keep-alives
// off, so no idle connection outlives a request.
func (h *Host) HTTPClient() *http.Client

// Telemetry records everything an App emits, using the real SDK with
// in-memory exporters.
type Telemetry struct{ /* unexported */ }

func NewTelemetry() *Telemetry
func (t *Telemetry) Builder() telemetry.Builder
func (t *Telemetry) Spans() []sdktrace.ReadOnlySpan
func (t *Telemetry) Metrics(ctx context.Context) (metricdata.ResourceMetrics, error)
func (t *Telemetry) Logs() []sdklog.Record

// Start runs app on a bedrocktest.Process built from opts, in a new
// goroutine, and returns the process and a channel that receives the
// App's status.
func Start(t testing.TB, app bedrock.App, opts ...bedrocktest.Option) (*bedrocktest.Process, <-chan bedrock.ExitStatus)

// TestService runs the humus.Service contract against the Services that
// newService builds. A package that provides a Service calls it from its
// own tests.
func TestService(t *testing.T, newService func(t *testing.T, s *humus.Setup) humus.Service)

TestService checks, inside synctest, that a Service obeys this contract:

• Serve returns nil after Shutdown.
• Shutdown with a context that is already done stops the Service at once and returns an error that wraps the context's error.
• Shutdown called before Serve, and Shutdown called twice, are both safe.
• No goroutine the Service started outlives Serve.

gRPC helpers, such as a grpc.DialOption that dials the Host, live with Proposal: gRPC services, so grpc-go stays out of the module graph of humus users who don't use gRPC.

What every package proposal must show

1. Its App or Service runs whole under synctest with bedrocktest and humustest.
2. No test calls t.Setenv, opens a real socket or touches the real filesystem, except integration tests behind a build tag.
3. Negative tests assert why something failed (the cause, with errors.Is or errors.AsType, and stderr), not only the exit code.
4. Every Service passes humustest.TestService.

Design decisions

• humustest mirrors bedrocktest. NewHost(opts...) and .Host() work like NewProcess(t, opts...) and .Process().
• Only ports matter. A Service listens on ":8080", and a client dials "name:8080". Name resolution is not modelled.
• Connection refused, not wait-for-listener. Dial fails with ECONNREFUSED when nothing listens, as a real dial does. A test calls synctest.Wait() after starting an App, which returns once the App is blocked in Accept.
• Files are a map keyed by the exact name, not an fs.FS. Settings carry absolute OS paths, and fs.FS rejects names that start with /.
• The real SDK, not the noop API, records telemetry. A test sees the spans, metrics and logs that production would export.
• humustest checks itself against the real thing. A conformance test runs the same scenarios against the in-memory network and against a real loopback network, as bedrocktest's conformance_unix_test.go does for signals.

Testing

The lifecycle prototype's tests already use the Host below. For example:

func TestApp_AJobEndsTheAppAndStopsTheOtherServices(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost()
		app := humus.App{
			Name: "batch",
			Host: host.Host(),
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				probe, err := newHTTPService(s, ":8081", http.NotFoundHandler())
				if err != nil {
					return nil, err
				}
				work := job(func(ctx context.Context) error {
					time.Sleep(time.Second)
					return nil
				})
				return []humus.Service{probe, work}, nil
			},
		}

		p, status := start(t, app)

		st := <-status
		require.Equal(t, 0, st.Code(), p.Stderr())
		require.NoError(t, st.Cause())

		_, err := host.Dial(t.Context(), "tcp", "localhost:8081")
		require.ErrorIs(t, err, syscall.ECONNREFUSED, "the probe server is closed with the App")
	})
}

Migration from today

• Hand-written signal fakes become bedrocktest.
• Real listeners in tests become humustest.Host.
• kafka_testcontainer.go stays for integration tests behind the build tag. Unit tests move to whichever fake Proposal: queue processing and the Kafka runtime chooses (story(kafka): add unit tests using kfake #431).

Open questions

1. Start's home. It only needs bedrock.App, so should it live in bedrocktest instead?
2. Dial before a listener exists. Should it wait for one rather than refuse?
3. Host names. Should the network model them, so two Services can listen on the same port under different names?
4. TLS in tests. A self-signed certificate needs randomness (testing/cryptotest). Should humustest provide a fixed test CA instead?
5. Kafka. kfake listens on real loopback ports, so it can't run inside synctest. kgo.Dialer can dial through the Host, but can kfake listen on it? If not, what is the unit-test story for the Kafka runtime?
6. Coverage. Should humustest itself be held to 100% statement coverage?

Out of scope

• Fakes for specific third-party services, such as databases.
• Load and performance testing.

Prototype

This is the in-memory Host the lifecycle prototype runs on (see Proposal: the humus App lifecycle for its tests). It omits Telemetry, Start and TestService.

humustest/humustest.go

// Package humustest is a prototype of the in-memory Host proposed in
// "Proposal: humustest". It is not the humus module.
package humustest

import (
	"context"
	"errors"
	"fmt"
	"io/fs"
	"maps"
	"net"
	"net/http"
	"strconv"
	"sync"
	"syscall"

	"github.com/z5labs/humus"
)

// Option configures a Host built by NewHost.
type Option func(*Host)

// WithFiles sets the files the Host's ReadFile returns, keyed by the exact
// name ReadFile is called with. The map is copied.
func WithFiles(files map[string][]byte) Option {
	files = maps.Clone(files)
	return func(h *Host) {
		h.files = files
	}
}

// Host is an in-memory humus.Host for tests: a network of net.Pipe
// connections and a fixed set of files. It touches nothing of the real
// host, so a test that uses it can run in parallel, and it starts no
// goroutines, so it can run inside a testing/synctest bubble. Build it
// inside the bubble that uses it.
type Host struct {
	files map[string][]byte

	mu        sync.Mutex
	listeners map[int]*listener
	nextPort  int
}

// NewHost returns a Host with no listeners and the files of opts.
func NewHost(opts ...Option) *Host {
	h := &Host{
		listeners: make(map[int]*listener),
		nextPort:  49152,
	}
	for _, opt := range opts {
		opt(h)
	}
	return h
}

// Host returns the humus.Host to run an App on. Every field is set.
func (h *Host) Host() humus.Host {
	return humus.Host{
		Listen:   h.Listen,
		Dial:     h.Dial,
		ReadFile: h.ReadFile,
	}
}

// Listen listens on the port of address; the host part is ignored, so
// ":8080" and "localhost:8080" are the same address. Port 0 picks a free
// port, which the listener's Addr reports.
func (h *Host) Listen(network, address string) (net.Listener, error) {
	port, err := portOf(address)
	if err != nil {
		return nil, &net.OpError{Op: "listen", Net: network, Err: err}
	}

	h.mu.Lock()
	defer h.mu.Unlock()
	if port == 0 {
		for h.listeners[h.nextPort] != nil {
			h.nextPort++
		}
		port = h.nextPort
	}
	if h.listeners[port] != nil {
		return nil, &net.OpError{Op: "listen", Net: network, Err: syscall.EADDRINUSE}
	}
	l := &listener{
		host:   h,
		port:   port,
		conns:  make(chan net.Conn),
		closed: make(chan struct{}),
	}
	h.listeners[port] = l
	return l, nil
}

// Dial connects to the listener on the port of address. With no listener
// on that port it fails with ECONNREFUSED, as a real dial would.
func (h *Host) Dial(ctx context.Context, network, address string) (net.Conn, error) {
	port, err := portOf(address)
	if err != nil {
		return nil, &net.OpError{Op: "dial", Net: network, Err: err}
	}

	h.mu.Lock()
	l := h.listeners[port]
	h.mu.Unlock()
	if l == nil {
		return nil, &net.OpError{Op: "dial", Net: network, Err: syscall.ECONNREFUSED}
	}

	server, client := net.Pipe()
	select {
	case l.conns <- server:
		return client, nil
	case <-l.closed:
		return nil, errors.Join(&net.OpError{Op: "dial", Net: network, Err: syscall.ECONNREFUSED}, server.Close(), client.Close())
	case <-ctx.Done():
		return nil, errors.Join(ctx.Err(), server.Close(), client.Close())
	}
}

// ReadFile returns the file set with WithFiles, or an error satisfying
// errors.Is(err, fs.ErrNotExist).
func (h *Host) ReadFile(name string) ([]byte, error) {
	b, ok := h.files[name]
	if !ok {
		return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
	}
	return append([]byte(nil), b...), nil
}

// HTTPClient returns an http.Client that dials the Host. Keep-alives are
// off, so no idle connection outlives a request.
func (h *Host) HTTPClient() *http.Client {
	return &http.Client{
		Transport: &http.Transport{
			DialContext:       h.Dial,
			DisableKeepAlives: true,
		},
	}
}

func portOf(address string) (int, error) {
	_, p, err := net.SplitHostPort(address)
	if err != nil {
		return 0, err
	}
	port, err := strconv.ParseUint(p, 10, 16)
	if err != nil {
		return 0, fmt.Errorf("invalid port %q", p)
	}
	return int(port), nil
}

type listener struct {
	host      *Host
	port      int
	conns     chan net.Conn
	closed    chan struct{}
	closeOnce sync.Once
}

func (l *listener) Accept() (net.Conn, error) {
	select {
	case c := <-l.conns:
		return c, nil
	case <-l.closed:
		return nil, net.ErrClosed
	}
}

func (l *listener) Close() error {
	l.closeOnce.Do(func() {
		close(l.closed)
		l.host.mu.Lock()
		delete(l.host.listeners, l.port)
		l.host.mu.Unlock()
	})
	return nil
}

func (l *listener) Addr() net.Addr {
	return &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: l.port}
}

References

• bedrocktest
• bedrock's example/httpserver/app_test.go
• testing/synctest
• story(kafka): add unit tests using kfake #431 (kfake)

[Zaba505]

Revision: real files, and a self-contained humustest

This revision applies the decisions in the lifecycle proposal's revision:

• This proposal owns humustest's integration with the lifecycle. The lifecycle's own tests use bedrocktest and fake Services, so the lifecycle lands without humustest. This proposal's stories add the tests that run whole Apps on humustest's Host.
• Host.ReadFile is replaced by Host.FS, a filesystem on *os.File, so humustest's Host serves real files from the test's own temp directory instead of a map.
• humustest is self-contained. A test imports humustest, never bedrocktest, which answers open question 1.
• The lifecycle calls Shutdown on every Service, including one whose Serve has already returned, and logs failures through telemetry rather than stderr.

Each section below replaces the section of the same name in the post. Background, Out of scope and References are unchanged and still apply.

Summary

humus is being rebuilt on the process model so that a whole service can run inside a test. humustest supplies everything a test needs:

• a test process, wrapping bedrocktest's, that starts an App;
• a Host: an in-memory network, and a filesystem confined to the test's own temp directory;
• an in-memory telemetry recorder;
• a contract test that every Service must pass.

With it, every humus App runs in a testing/synctest bubble without a socket, the real environment, real time or a file outside the test's own temp directory.

Proposed API

package humustest

// Host is a humus.Host for tests: a network of net.Pipe connections, and a
// filesystem in a directory from t.TempDir(). The network starts no
// goroutines, so the Host works inside a testing/synctest bubble; build it
// inside the bubble that uses it.
type Host struct{ /* unexported */ }

type HostOption func(*Host)

// WithFiles writes files into the Host's directory, keyed by their path
// relative to it, such as "etc/tls/server.p12". Parent directories are
// created.
func WithFiles(files map[string][]byte) HostOption

// NewHost returns a Host whose directory comes from t.TempDir(), so it is
// removed when the test ends.
func NewHost(t testing.TB, opts ...HostOption) *Host

// Host returns the humus.Host to run an App on, like bedrocktest's
// Process.Process.
func (h *Host) Host() humus.Host

// Listen listens on the port of address. The host part is ignored, and
// port 0 picks a free port. A port already in use fails with EADDRINUSE.
func (h *Host) Listen(network, address string) (net.Listener, error)

// Dial connects to the listener on the port of address. It fails with
// ECONNREFUSED when nothing listens there.
func (h *Host) Dial(ctx context.Context, network, address string) (net.Conn, error)

// FS returns the Host's filesystem. It is confined to the Host's directory
// with os.Root, so ".." and symlinks can't leave it. It accepts absolute
// paths under that directory, so a file's Name() can be passed back in, as
// in production. CreateTemp and MkdirTemp with dir == "" use a directory
// inside it.
func (h *Host) FS() humus.FS

// Path returns the absolute OS path of name, which is relative to the
// Host's directory, to put in a setting.
func (h *Host) Path(name string) string

// HTTPClient returns an http.Client that dials the Host, with keep-alives
// off, so no idle connection outlives a request.
func (h *Host) HTTPClient() *http.Client

// Telemetry records everything an App emits, using the real SDK with
// in-memory exporters.
type Telemetry struct{ /* unexported */ }

func NewTelemetry() *Telemetry
func (t *Telemetry) Builder() telemetry.Builder
func (t *Telemetry) Spans() []sdktrace.ReadOnlySpan
func (t *Telemetry) Metrics(ctx context.Context) (metricdata.ResourceMetrics, error)
func (t *Telemetry) Logs() []sdklog.Record

// Process is the process an App runs on in a test. It wraps bedrocktest's
// Process, so a test imports humustest alone.
type Process struct{ /* unexported */ }

type ProcessOption func(*processOptions)

// WithArgs, WithEnv and WithStdin work like bedrocktest's options of the
// same names.
func WithArgs(args ...string) ProcessOption
func WithEnv(env map[string]string) ProcessOption
func WithStdin(r io.Reader) ProcessOption

// Start runs app on a new Process, in a new goroutine, and returns the
// Process. To use it with testing/synctest, call it inside the bubble.
func Start(t testing.TB, app bedrock.App, opts ...ProcessOption) *Process

// Notified, Signal, Stdout and Stderr work like bedrocktest's.
func (p *Process) Notified(sig os.Signal) <-chan struct{}
func (p *Process) Signal(sig os.Signal)
func (p *Process) Stdout() string
func (p *Process) Stderr() string

// Wait blocks until the App returns, and returns its status.
func (p *Process) Wait() bedrock.ExitStatus

// TestService runs the humus.Service contract against the Services that
// newService builds. A package that provides a Service calls it from its
// own tests.
func TestService(t *testing.T, newService func(t *testing.T, s *humus.Setup) humus.Service)

TestService checks, inside synctest, that a Service obeys this contract:

• Serve returns nil after Shutdown.
• Shutdown with a context that is already done stops the Service at once and returns an error that wraps the context's error.
• Shutdown is safe when called before Serve, after Serve has returned on its own, and twice.
• No goroutine the Service started outlives Serve.

gRPC helpers, such as a grpc.DialOption that dials the Host, live with Proposal: gRPC services, so grpc-go stays out of the module graph of humus users who don't use gRPC.

What every package proposal must show

1. Its App or Service runs whole under synctest with humustest.
2. No test calls t.Setenv, opens a real socket or touches a file outside its own temp directory, except integration tests behind a build tag.
3. Negative tests assert why something failed (the cause, with errors.Is or errors.AsType), not only the exit code. A test that checks a logged failure reads it from humustest.Telemetry.
4. Every Service passes humustest.TestService.

Design decisions

• humustest is self-contained. A test imports humustest, never bedrocktest. Process wraps bedrocktest's Process, and its options and methods mirror bedrocktest's. NewHost(t, opts...) and .Host() work like NewProcess(t, opts...) and .Process(). Start is here rather than in bedrocktest, because a humus test needs the Host too.
• Only ports matter. A Service listens on ":8080", and a client dials "name:8080". Name resolution is not modelled.
• Connection refused, not wait-for-listener. Dial fails with ECONNREFUSED when nothing listens, as a real dial does. A test calls synctest.Wait() after starting an App, which returns once the App is blocked in Accept.
• Files are real files in the test's own temp directory. Host.FS returns *os.File, which is always an OS handle and can't be faked:
  • humustest has no fake filesystem to build and keep in line with the real one;
  • libraries that need a real file or a path keep working in tests: SQLite drivers, archive libraries, programs started with os/exec;
  • os.Root confines the filesystem to the directory, and Path gives a setting the absolute path it would carry in production.
• Cleanup is t.TempDir's. It runs when a test fails or panics, and it fails the test when the removal fails. Nothing in-process survives a SIGKILL or a -timeout crash, so CI points TMPDIR at a directory scoped to the job. On GitHub Actions, RUNNER_TEMP is emptied at the start and end of every job.
• What real files cost. A test can't make a write fail with a full disk or an I/O error, so code that needs that test puts an io.Writer seam in its own code. File modification times are real inside a synctest bubble, where time.Now() is fake, so code that compares ModTime() with time.Now() needs Chtimes in its tests.
• The real SDK, not the noop API, records telemetry. A test sees the spans, metrics and logs that production would export. Since the lifecycle logs failures through telemetry, that includes the App's failures.
• humustest checks itself against the real thing. A conformance test runs the same scenarios against the in-memory network and against a real loopback network, as bedrocktest's conformance_unix_test.go does for signals.

Testing

Whole-App tests run on this API. These include the lifecycle prototype's network tests, which the lifecycle's own tests leave out. For example:

func TestApp_AJobEndsTheAppAndStopsTheOtherServices(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost(t)
		app := humus.App{
			Host:      host.Host(),
			Telemetry: telemetry.NoopBuilder,
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				probe, err := newHTTPService(s, ":8081", http.NotFoundHandler())
				if err != nil {
					return nil, err
				}
				work := job(func(ctx context.Context) error {
					time.Sleep(time.Second)
					return nil
				})
				return []humus.Service{probe, work}, nil
			},
		}

		st := humustest.Start(t, app).Wait()
		require.Equal(t, 0, st.Code())
		require.NoError(t, st.Cause())

		_, err := host.Dial(t.Context(), "tcp", "localhost:8081")
		require.ErrorIs(t, err, syscall.ECONNREFUSED, "the probe server is closed with the App")
	})
}

Proposal: configuration for humus apps shows WithFiles and Path in its settings tests.

Migration from today

• Hand-written signal fakes become humustest's Process.
• Real listeners in tests become humustest.Host.
• Test files on disk are written with WithFiles into the test's own directory.
• kafka_testcontainer.go stays for integration tests behind the build tag. Unit tests move to whichever fake Proposal: queue processing and the Kafka runtime chooses (story(kafka): add unit tests using kfake #431).

Open questions

Question 1 (Start's home) is answered above: humustest, which is self-contained. Question 6 (coverage) is settled by the charter's 80% gate for every library package. The rest are still open:

2. Dial before a listener exists. Should it wait for one rather than refuse?
3. Host names. Should the network model them, so two Services can listen on the same port under different names?
4. TLS in tests. A self-signed certificate needs randomness (testing/cryptotest). Should humustest provide a fixed test CA instead?
5. Kafka. kfake listens on real loopback ports, so it can't run inside synctest. kgo.Dialer can dial through the Host, but can kfake listen on it? If not, what is the unit-test story for the Kafka runtime?

Prototype

The prototype in the post predates this revision. Its network (Listen, Dial, HTTPClient and the listener) still holds. Its files (WithFiles over a map, and ReadFile) are replaced by the temp-directory FS above, and NewHost now takes t. The humustest stories implement this revision.

[Zaba505]

Revision 2: the open questions answered

This revision answers every open question that the first revision left open, and reverses its answer to question 1:

1. Start lives in bedrocktest. It needs nothing but bedrock.App, so story(bedrocktest): Start runs an App on a new Process and returns its status bedrock#600 adds bedrocktest.Start. humustest no longer wraps bedrocktest's Process. A whole-App test imports bedrocktest for the process and humustest for the Host, the telemetry recorder and the Service contract.
2. Dial refuses when nothing listens. Listen and Dial are independent. A Dial to an address with no listener fails with ECONNREFUSED, as a real dial does, and never waits for one. This was already the proposed behaviour, and it is now settled. The test makes sure a listener exists before it dials. Inside a synctest bubble it calls synctest.Wait(). Anywhere else, or to wait for one particular listener, it receives from the new Host.Listening, which works like bedrocktest's Process.Notified.
3. Host names come from the address. Listen already receives the host in its address argument. The first revision ignored it and kept only the port. The Host now keeps both, so two Services can listen on the same port under different names.
4. TLS material is generated per test. humustest generates a CA inside the test. Every key, serial number and password is random. humustest holds no fixed key, certificate or password, and neither does any test that uses it.
5. Kafka belongs to the Kafka proposal. Whether kfake can listen on the Host, and what the Kafka runtime's unit tests run against otherwise, is for Proposal: queue processing and the Kafka runtime (story(kafka): add unit tests using kfake #431) to answer in a later wave.

Each section below replaces the section of the same name in the first revision. Background, Out of scope and References still apply.

Summary

humus is being rebuilt on the process model so that a whole service can run inside a test. bedrocktest supplies the process, and starts an App on it with Start. humustest supplies everything else:

• a Host: an in-memory network, and a filesystem confined to the test's own temp directory;
• a certificate authority generated for each test;
• an in-memory telemetry recorder;
• a contract test that every Service must pass.

With both, every humus App runs in a testing/synctest bubble. It needs no socket, no real environment, no real time and no file outside the test's own temp directory.

Proposed API

package humustest

// Host is a humus.Host for tests: a network of net.Pipe connections, and a
// filesystem in a directory from t.TempDir(). The network starts no
// goroutines, so the Host works inside a testing/synctest bubble; build it
// inside the bubble that uses it.
type Host struct{ /* unexported */ }

type HostOption func(*Host)

// WithFiles writes files into the Host's directory, keyed by their path
// relative to it, such as "etc/tls/server.p12". Parent directories are
// created.
func WithFiles(files map[string][]byte) HostOption

// NewHost returns a Host whose directory comes from t.TempDir(), so it is
// removed when the test ends.
func NewHost(t testing.TB, opts ...HostOption) *Host

// Host returns the humus.Host to run an App on, like bedrocktest's
// Process.Process.
func (h *Host) Host() humus.Host

// Listen listens on the host and port of address. An empty host, 0.0.0.0
// or :: listens on every name. Port 0 picks a free port. Listen fails with
// EADDRINUSE when a listener already holds the same name and port, or
// when one of the two listens on every name for that port.
func (h *Host) Listen(network, address string) (net.Listener, error)

// Dial connects to the listener for the host and port of address: one
// bound to that exact name, or else one that listens on every name for
// that port. It fails with ECONNREFUSED when there is none. It never
// waits for a listener.
func (h *Host) Dial(ctx context.Context, network, address string) (net.Conn, error)

// Listening returns a channel that is closed once a listener exists that a
// Dial to address would reach. A test receives from it before dialling, so
// that it doesn't dial before the App listens. It works like bedrocktest's
// Process.Notified: once closed, the channel stays closed, and after that
// listener closes, Listening returns a new channel, closed when the next
// one is made. The channel is created on the first call, and no goroutine
// is started, so Listening works inside a testing/synctest bubble.
func (h *Host) Listening(address string) <-chan struct{}

// FS returns the Host's filesystem. It is confined to the Host's directory
// with os.Root, so ".." and symlinks can't leave it. It accepts absolute
// paths under that directory, so a file's Name() can be passed back in, as
// in production. CreateTemp and MkdirTemp with dir == "" use a directory
// inside it.
func (h *Host) FS() humus.FS

// Path returns the absolute OS path of name, which is relative to the
// Host's directory, to put in a setting.
func (h *Host) Path(name string) string

// HTTPClient returns an http.Client that dials the Host, with keep-alives
// off, so no idle connection outlives a request.
func (h *Host) HTTPClient() *http.Client

// CA is a certificate authority generated for one test. Its key, serial
// number and validity come from crypto/rand and the clock when NewCA is
// called. Nothing about it is fixed.
type CA struct{ /* unexported */ }

// NewCA generates a CA with a new random ECDSA P-256 key and a random
// 128-bit serial number. It is valid from an hour before time.Now() to a
// day after. Inside a synctest bubble that is the bubble's clock, so build
// the CA inside the bubble that uses it. NewCA fails the test through t
// if generation fails.
func NewCA(t testing.TB) *CA

// Pool returns a CertPool holding only the CA's certificate, for a
// client's tls.Config.RootCAs.
func (ca *CA) Pool() *x509.CertPool

// Issue returns a certificate for names, which are DNS names or IP
// literals, signed by the CA, with a new random key and serial number.
func (ca *CA) Issue(t testing.TB, names ...string) *Certificate

// Certificate is a leaf certificate issued by a CA.
type Certificate struct{ /* unexported */ }

// TLS returns the certificate and its chain, for a server's
// tls.Config.Certificates.
func (c *Certificate) TLS() tls.Certificate

// PKCS12 encodes the certificate, its key and the CA's certificate as a
// PKCS#12 bundle, encrypted with a new password from crypto/rand.Text.
// It returns the bundle and the password. Each call makes a new password.
func (c *Certificate) PKCS12(t testing.TB) (bundle []byte, password string)

// Telemetry records everything an App emits, using the real SDK with
// in-memory exporters.
type Telemetry struct{ /* unexported */ }

func NewTelemetry() *Telemetry
func (t *Telemetry) Builder() telemetry.Builder
func (t *Telemetry) Spans() []sdktrace.ReadOnlySpan
func (t *Telemetry) Metrics(ctx context.Context) (metricdata.ResourceMetrics, error)
func (t *Telemetry) Logs() []sdklog.Record

// TestService runs the humus.Service contract against the Services that
// newService builds. A package that provides a Service calls it from its
// own tests.
func TestService(t *testing.T, newService func(t *testing.T, s *humus.Setup) humus.Service)

Process, its options, Start and Wait are gone from humustest. bedrocktest provides them (z5labs/bedrock#600):

func Start(tb testing.TB, app bedrock.App, opts ...bedrocktest.Option) (*bedrocktest.Process, <-chan bedrock.ExitStatus)

The TestService contract is unchanged from the first revision. gRPC helpers still live with Proposal: gRPC services.

What every package proposal must show

1. Its App or Service runs whole under synctest with bedrocktest and humustest.
2. No test calls t.Setenv, opens a real socket or touches a file outside its own temp directory, except integration tests behind a build tag.
3. Negative tests assert why something failed (the cause, with errors.Is or errors.AsType), not only the exit code. A test that checks a logged failure reads it from humustest.Telemetry.
4. Every Service passes humustest.TestService.
5. No test, fixture or testdata file contains a secret: a private key, a certificate, a password or a token. A test generates the secrets it needs, such as with humustest.NewCA.

Design decisions

• humustest builds on bedrocktest; it does not wrap it. Start needs only bedrock.App, so it is in bedrocktest, where every bedrock App's tests can use it. humustest adds only what a humus App needs beyond the process. A test imports both. NewHost(t, opts...) and .Host() still mirror NewProcess(t, opts...) and .Process().
• Host names come from the address, and are compared as written. Listen and Dial use the host part of address, which they always received. The rules follow a real network: a name and port can be held once, and a listener on every name (empty host, 0.0.0.0 or ::) conflicts with any other listener on its port. Names are not resolved. localhost and 127.0.0.1 are different names, so a client dials the name a Service listens on, or the Service listens on every name.
• Dial refuses rather than waits. Listen and Dial are independent, as they are on a real network. A dial with no listener is a test that dialled too early, and ECONNREFUSED says so at once. Waiting would hide the mistake, and inside synctest it would turn into a deadlock report that doesn't name the cause. A test waits for the listener itself, before it dials:
  • Inside a synctest bubble, synctest.Wait() is the default. It returns once every goroutine in the bubble is blocked, which includes the App blocked in Accept.
  • <-host.Listening(address) waits for one particular listener. It also works outside a bubble, such as in the conformance test or a test that needs real time. It follows bedrocktest's Notified and Signal, where the test receives from Notified before it sends a signal.
• Secrets are generated, never fixed. A fixed test CA would put a private key in the repository, and copies of it would end up in users' test suites. NewCA generates the key, serial number and validity window on each call. PKCS12 encrypts each bundle with a new random password. Nothing is shared between tests, and testing/cryptotest isn't needed. Certificates are valid around time.Now() when they are made, so a CA built inside a synctest bubble is valid on the bubble's clock.
• Files are real files in the test's own temp directory, cleanup is t.TempDir's, and what real files cost are unchanged from the first revision.
• The real SDK, not the noop API, records telemetry, unchanged.
• humustest checks itself against the real thing. The conformance test runs the same scenarios against the in-memory network and a real loopback network. It now includes the name rules, using 127.0.0.1 and 127.0.0.2 as the two names. It also includes a TLS handshake, with a NewCA certificate, over both networks.

Testing

Whole-App tests start the App with bedrocktest.Start, and run it on a humustest Host:

func TestApp_AJobEndsTheAppAndStopsTheOtherServices(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		host := humustest.NewHost(t)
		app := humus.App{
			Host:      host.Host(),
			Telemetry: telemetry.NoopBuilder,
			Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
				probe, err := newHTTPService(s, ":8081", http.NotFoundHandler())
				if err != nil {
					return nil, err
				}
				work := job(func(ctx context.Context) error {
					time.Sleep(time.Second)
					return nil
				})
				return []humus.Service{probe, work}, nil
			},
		}

		_, status := bedrocktest.Start(t, app)
		st := <-status
		require.Equal(t, 0, st.Code())
		require.NoError(t, st.Cause())

		_, err := host.Dial(t.Context(), "tcp", "localhost:8081")
		require.ErrorIs(t, err, syscall.ECONNREFUSED, "the probe server is closed with the App")
	})
}

A TLS test generates everything it needs. Here newApp serves HTTPS on ":8443", and the setting names are only for illustration, since Proposal: REST services names the real ones:

func TestApp_ServesTLSFromAPKCS12File(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		ca := humustest.NewCA(t)
		bundle, password := ca.Issue(t, "api.internal").PKCS12(t)

		host := humustest.NewHost(t, humustest.WithFiles(map[string][]byte{
			"etc/tls/server.p12": bundle,
		}))
		fp, status := bedrocktest.Start(t, newApp(host), bedrocktest.WithEnv(map[string]string{
			"HUMUS_HTTP_TLS_PKCS12_FILE":     host.Path("etc/tls/server.p12"),
			"HUMUS_HTTP_TLS_PKCS12_PASSWORD": password,
		}))
		synctest.Wait()

		client := host.HTTPClient()
		client.Transport.(*http.Transport).TLSClientConfig = &tls.Config{RootCAs: ca.Pool()}
		resp, err := client.Get("https://api.internal:8443/health")
		require.NoError(t, err)
		require.NoError(t, resp.Body.Close())
		require.Equal(t, http.StatusOK, resp.StatusCode)

		fp.Signal(syscall.SIGTERM)
		require.Equal(t, 0, (<-status).Code())
	})
}

Migration from today

• Hand-written signal fakes, and goroutines that run an App, become bedrocktest.NewProcess and bedrocktest.Start.
• Real listeners in tests become humustest.Host.
• Test files on disk are written with WithFiles into the test's own directory.
• Certificates, keys and passwords in tests become humustest.NewCA. rest's self-signed fallback in buildSelfSignedTLSConfig is a production behaviour, and Proposal: REST services decides its future.
• kafka_testcontainer.go stays for integration tests behind the build tag. Its unit-test story belongs to Proposal: queue processing and the Kafka runtime.

Open questions

None. Questions 1 to 4 are answered above. Question 5 moves to Proposal: queue processing and the Kafka runtime. Question 6 is still settled by the charter's 80% gate.

Prototype

The first revision's note still holds, with one change: the prototype's Listen and Dial ignore the host part of the address, and the humustest stories implement the name rules above instead.

[Zaba505]

Stories, and telemetry's ownership

Telemetry belongs to #632. Each proposal fully owns its domain, including how that domain integrates with other packages. So humustest.Telemetry, along with NewTelemetry, Builder, Spans, Metrics and Logs, comes out of this proposal's API. Proposal: OpenTelemetry without globals adds telemetry support to humustest when it implements telemetry, and specifies that API itself. The rest of the second revision stands. Until #632 lands, whole-App tests read the lifecycle's failures from stderr, as #637 writes them.

Stories

Story	Blocked by
#641 in-memory network on a test Host: Listen, Dial, Listening, HTTPClient, the name rules and the loopback conformance test	#636
#642 the test Host's filesystem, confined to the test's temp directory: WithFiles, FS and Path	#641
#643 a certificate authority generated for each test: NewCA, Issue, TLS, and PKCS12 with a random password	#641
#644 TestService checks the humus.Service contract	#637, #642, z5labs/bedrock#600
#645 whole-App tests over humustest's network	#638, #644, z5labs/bedrock#600
#646 documenting how to test a whole humus App	#639, #641–#645

The stories settle three details the revisions left open:

• TestService runs each check as a humus.App started with bedrocktest.Start. Setup has no exported way to run the OnStop functions a Service's constructor registers, so the lifecycle runs them, as it does in production.
• PKCS12 adds software.sslmate.com/src/go-pkcs12. The standard library has no PKCS#12 encoder, and golang.org/x/crypto/pkcs12 only decodes. humus used go-pkcs12 to read PKCS#12 before the reset.
• The loopback half of the conformance test is behind a conformance build tag, so unit tests open no real socket. CI runs it.

#644 and #645 need bedrocktest.Start in a bedrock release that humus depends on. Closing z5labs/bedrock#600 alone isn't enough.