Proposal: gRPC services

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 2. It builds on Proposal: the humus App lifecycle, Proposal: configuration for humus apps, Proposal: OpenTelemetry without globals and Proposal: humustest. It depends on no other wave-2 proposal. It blocks Proposal: exposing health probes (wave 3), which serves gRPC health v1 on the server proposed here.

Summary

humus can't run a gRPC service today. grpc.NewApi builds a *grpc.Server and nothing serves it. This proposes a grpc package, in a nested module, with:

• a Service that serves a *grpc.Server on a listener from the App's Host. It keeps serving through the drain delay. Its Shutdown stops gracefully within the grace period and then stops hard.
• an App built on humus.App, for a process that serves gRPC and nothing else. Its defaults are a 5s drain delay and a 15s grace period.
• Settings with two fields, Addr and TLS. Everything else about the server is a grpc.ServerOption the end user passes.
• telemetry from s.Telemetry, through otelgrpc, with no OTel globals.
• per-RPC panic recovery, which fails only the RPC that panicked, with codes.Internal.
• grpctest.NewClient, a client that dials the test Host, so grpc-go stays out of the core module's graph.

func main() {
	bedrock.Main(humusgrpc.App{
		Host:      humus.OSHost(),
		Telemetry: buildTelemetry,
		Init:      app.Init, // returns a *humusgrpc.Service with the end user's services registered
	})
}

The examples import the package as humusgrpc, because almost every file that uses it also imports google.golang.org/grpc. Open question 1 asks whether to rename it.

grpc-go runs inside a testing/synctest bubble on humustest's in-memory network: its server, its client, otelgrpc and TLS all do. The tests in this proposal come from a prototype that passes go test -race -count=3, with 90% coverage.

Background

• Nothing serves a gRPC Api. grpc.NewApi builds a *grpc.Server and registers a health server on it (grpc/api.go#L35-L62). grpc/grpc.go, which ran it, was deleted in story(issue-484): rest migrate to bedrock v0210 #485. internal/grpcserver still has a runner, but nothing calls it.
• internal/grpcserver shuts down with no bound. It calls GracefulStop when its context is done, with no deadline and no fallback to Stop (internal/grpcserver/server.go#L29-L46). An RPC that never returns, such as a server stream, holds the process up until it is killed.
• Telemetry comes from the globals. NewApi passes otel.GetMeterProvider(), otel.GetTracerProvider() and otel.GetTextMapPropagator() to grpc-go's stats/opentelemetry (grpc/api.go#L42-L53).
• Health is registered implicitly. Api.RegisterService looks for a health.Monitor on each implementation and wires it into internal/grpchealth (grpc/api.go#L64-L71). internal/grpchealth logs through the global humus.Logger, and its Watch polls every 10 seconds with time.After (internal/grpchealth/server.go#L78-L105). Proposal: health monitors and probes moved all of this to Proposal: exposing health probes, in wave 3.
• What grpc-go does that humus's principles don't allow:
  • It exits the process. RegisterService after Serve, a duplicate service name, or an implementation of the wrong type calls grpclog.Fatalf, which calls os.Exit(1) (server.go#L770-L790, grpclog/grpclog.go#L96-L100). The charter's first principle says nothing in humus exits the process.
  • It doesn't recover a handler's panic, so one panicking RPC ends the process and every other RPC in flight. net/http recovers a handler's panic and fails only that request.
  • Stop doesn't wait for handlers by default. A handler can keep running after Stop returns, unless the server is built with grpc.WaitForHandlers(true). That option is still marked experimental.
  • It reads the real environment. grpclog reads GRPC_GO_LOG_SEVERITY_LEVEL, GRPC_GO_LOG_VERBOSITY_LEVEL and GRPC_GO_LOG_FORMATTER at init, and writes to the real os.Stderr (grpclog/loggerv2.go#L65-L81). internal/envconfig reads its GRPC_* variables the same way.
  • Its default resolver reaches the real network. grpc.NewClient("carts:9090") uses the dns resolver, which looks the name up with net.DefaultResolver whatever dialer the client has. In a synctest bubble, this crashes the test binary with fatal error: send on synctest channel from outside bubble.

Proposed API

// Package grpc serves gRPC with humus's lifecycle.
package grpc // import "github.com/z5labs/humus/grpc", a nested module

// Settings configures a Service. Every field is a plain value, and a
// field's zero value means zero, never "use the default".
type Settings struct {
	// Addr is the address the Service listens on, as host:port, through the
	// App's Host. An empty host listens on every name, and port 0 picks a
	// free port.
	//
	// Default: ":9090". Valid: host:port, with a port from 0 to 65535.
	// Check: CheckAddr.
	Addr string

	// TLS secures every connection. Nil serves gRPC over plaintext HTTP/2,
	// for a service behind a proxy or mesh that terminates TLS. A client
	// certificate is required by setting ClientAuth, as with any tls.Config.
	//
	// Default: nil. Valid: any.
	TLS *tls.Config
}

// DefaultSettings returns the defaults of a Service.
func DefaultSettings() Settings

// CheckAddr reports why addr is not a valid Addr, or nil if it is.
func CheckAddr(addr string) error

// Service is a humus.Service that serves gRPC. It is a
// grpc.ServiceRegistrar, so generated code registers services on it:
//
//	cartpb.RegisterCartServiceServer(svc, carts)
type Service struct{ /* unexported */ }

// NewService checks settings, listens on settings.Addr through s.Host, and
// builds the server. opts come after humus's own options: an option that
// grpc-go keeps one value of, such as grpc.Creds, replaces humus's, and an
// option that chains, such as grpc.ChainUnaryInterceptor, is added inside
// humus's.
//
// Every bad field is reported in one error, naming each one. The listener
// is closed by Shutdown, or by an OnStop function NewService registers
// with s, if the App never serves.
func NewService(s *humus.Setup, settings Settings, opts ...grpc.ServerOption) (*Service, error)

// RegisterService implements grpc.ServiceRegistrar. It panics, instead of
// exiting the process as grpc-go does, if it is called after Serve, if
// desc's service is already registered, or if impl doesn't implement
// desc.HandlerType.
func (s *Service) RegisterService(desc *grpc.ServiceDesc, impl any)

// GetServiceInfo returns the registered services, like
// (*grpc.Server).GetServiceInfo, so a Service can be given to
// reflection.Register.
func (s *Service) GetServiceInfo() map[string]grpc.ServiceInfo

// Serve serves gRPC until Shutdown. It keeps serving after ctx is done,
// through the App's drain delay.
func (s *Service) Serve(ctx context.Context) error

// Shutdown stops the server gracefully: it stops accepting connections,
// sends every client a GOAWAY, and waits for the RPCs in flight. If ctx is
// done first, it closes every connection, cancels every RPC, waits for
// every handler to return, and returns an error that wraps ctx's error.
func (s *Service) Shutdown(ctx context.Context) error

// App is a bedrock.App that serves gRPC with humus's lifecycle: a
// humus.App with a drain delay of 5s and a grace period of 15s. Host,
// Telemetry and Init are required.
type App struct {
	// Host is the network and filesystem the App runs on.
	Host humus.Host

	// Telemetry builds the App's telemetry.SDK. The end user's Builder
	// names the service.
	Telemetry telemetry.Builder

	// Init builds the end user's dependencies and returns the Service, with
	// their gRPC services registered on it.
	Init func(ctx context.Context, s *humus.Setup) (*Service, error)
}

// Run implements bedrock.App.
func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus

// Package grpctest connects a gRPC client to a humustest Host.
package grpctest // import "github.com/z5labs/humus/grpc/grpctest"

// NewClient returns a client of the Service listening on address in h. It
// dials through h, never resolves address, and is plaintext unless opts
// give it transport credentials. The client is closed when the test ends.
func NewClient(t testing.TB, h *humustest.Host, address string, opts ...grpc.DialOption) *grpc.ClientConn

The end user's side

The end user sources Settings in their Init, as Proposal: configuration for humus apps sets out, registers their services, and returns the Service:

func Init(ctx context.Context, s *humus.Setup) (*humusgrpc.Service, error) {
	d := humusgrpc.DefaultSettings()
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }

	settings, err := config.Struct(func(r *config.Resolver) humusgrpc.Settings {
		return humusgrpc.Settings{
			Addr: r.Get(env("GRPC_ADDR").Default(d.Addr).Validate(humusgrpc.CheckAddr)),
			// readFile and serverTLS are the end user's own helpers.
			TLS: r.Get(env("GRPC_TLS_P12_FILE").Parse(readFile(s.Host.FS)).Parse(serverTLS).Default(d.TLS)),
		}
	}).Read(ctx)
	if err != nil {
		return nil, err // exit 78, naming each bad variable
	}

	pool, err := newPool(ctx, s) // closed by an OnStop function
	if err != nil {
		return nil, err
	}

	svc, err := humusgrpc.NewService(s, settings, grpc.ChainUnaryInterceptor(authenticate))
	if err != nil {
		return nil, err
	}
	cartpb.RegisterCartServiceServer(svc, carts.New(pool))
	return svc, nil
}

The Service

NewService does three things, in order:

1. It runs CheckAddr, and returns every bad field in one error, naming each one. A Settings built in code is checked as well as one sourced with config.
2. It listens on settings.Addr with s.Host.Listen. The listener is opened in Init, so connections that arrive before Serve wait in the backlog, as the lifecycle expects. If Init fails after NewService, the App never calls Shutdown, so NewService registers an OnStop function that closes the listener. Closing an already closed listener is not an error.
3. It builds the *grpc.Server with humus's options first, and then opts:

Option	Why
grpc.StatsHandler(otelgrpc.NewServerHandler(...))	traces and metrics, from s.Telemetry; see Telemetry below
grpc.ChainUnaryInterceptor and grpc.ChainStreamInterceptor with panic recovery	the outermost interceptors, so a panic in the end user's interceptors is recovered too; see Panics below
grpc.Creds(credentials.NewTLS(settings.TLS))	only when settings.TLS is set
grpc.WaitForHandlers(true)	Stop waits for every handler to return, so no handler outlives Shutdown

opts is where everything else goes: the end user's interceptors, keepalive parameters, message size limits, a second stats handler. grpc-go keeps one value of some options and chains others, and NewService documents what that means here:

• an end user's grpc.Creds replaces Settings.TLS;
• an end user's interceptors run inside humus's, after recovery;
• an end user's grpc.WaitForHandlers(false) lets a handler outlive Shutdown. That breaks the humus.Service contract, and humustest.TestService reports it as a leak.

RegisterService must be called before Serve, as with *grpc.Server. grpc-go exits the process on three misuses. humus checks for the same three first and panics instead, with the service's name in the message. OnStop panics on a nil function in the same way: a programming error fails loudly, in the end user's Init, without leaving through os.Exit.

Serve honours only Shutdown, as an HTTP server does. Its context is done when shutdown starts, but the Service keeps accepting connections and RPCs through the drain delay. Kubernetes stops routing new connections to a terminating pod only once the endpoint removal has reached every node, and a connection that arrives in that window is still served. Serve returns nil once Shutdown has stopped the server.

Shutdown calls GracefulStop. It closes the listener and sends every connection a GOAWAY, so clients open new connections elsewhere, and it waits for the RPCs in flight to finish. gRPC connections are long-lived, and the GOAWAY is what moves a client off a terminating pod. If ctx is done first, Shutdown calls Stop. That closes every connection, cancels every RPC's context and, with WaitForHandlers, waits for every handler to return. Then Shutdown returns an error that wraps ctx.Err(), and the App exits 1 with a *humus.GracePeriodError. Shutdown is safe before Serve, after Serve has returned and twice.

The App

App.Run runs a humus.App:

func (a App) Run(ctx context.Context, p bedrock.Process) bedrock.ExitStatus {
	return humus.App{
		Host:                a.Host,
		Telemetry:           a.Telemetry,
		ShutdownGracePeriod: 15 * time.Second,
		DrainDelay:          5 * time.Second,
		Init: func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
			svc, err := a.Init(ctx, s)
			if err != nil {
				return nil, err
			}
			return []humus.Service{svc}, nil
		},
	}.Run(ctx, p)
}

• The defaults: a 5s drain delay and a 15s grace period. The drain delay gives Kubernetes' endpoint removal time to reach every node. The grace period gives in-flight RPCs time to finish. Telemetry's default ShutdownTimeout adds 5s, so the worst case is 25s, inside Kubernetes' default terminationGracePeriodSeconds of 30s. An operator changes either with HUMUS_SHUTDOWN_DRAIN_DELAY or HUMUS_SHUTDOWN_GRACE_PERIOD, which humus.App.Run already reads with these as defaults.
• Telemetry is required. It doesn't default to telemetry.New with DefaultSettings(). Those settings name the service unknown_service, which no production service wants, and the end user's Builder is where the service gets its name. This also keeps humus.App's rule that every field is required, so main states every seam it uses.
• Init returns the *Service, not a slice. The App keeps the Service it was given, so a later proposal can register on it before Serve, without changing the end user's Init. A process that runs gRPC next to something else, such as a Kafka consumer or an HTTP server, uses humus.App and NewService directly. The lifecycle proposal calls that the advanced path.
• No health in wave 2. The App doesn't register gRPC health v1 and doesn't run a health Service. See "The seam for health" below.

Settings

Field	Default	Valid	Check
Addr	":9090"	host:port, with a port from 0 to 65535	CheckAddr
TLS	nil, which serves plaintext HTTP/2	any

Settings holds what an operator changes from one deployment to the next: where to listen, and the certificate. How the server behaves, such as its interceptors, keepalive parameters and message size limits, is code, and is passed as a grpc.ServerOption. That keeps one way to set each of them. The end user who wants a variable for one sources it in their Init and passes the option.

Proposal: configuration for humus apps left optional settings to each package. Here, the answer is that TLS is optional, and nil means plaintext. Addr is not optional.

CheckAddr accepts what Host.Listen accepts: an empty host for every name, and port 0 for a free port. It rejects an address with no port, and a port that is not a number from 0 to 65535.

Telemetry

The Service gets everything from s.Telemetry, and never touches OTel's globals:

otelgrpc.NewServerHandler(
	otelgrpc.WithTracerProvider(s.Telemetry.TracerProvider),
	otelgrpc.WithMeterProvider(s.Telemetry.MeterProvider),
	otelgrpc.WithPropagators(s.Telemetry.Propagator),
)

What it emits, as the prototype records it with humustest.Telemetry:

• A server span per RPC, named after the method without its leading slash, such as grpc.testing.TestService/EmptyCall. Its parent comes from the request's metadata, through the SDK's propagator. It has the attributes rpc.system.name, rpc.method, server.port and rpc.response.status_code. Its status is Error for the codes OpenTelemetry's RPC conventions count as server errors, Internal among them.
• The metric rpc.server.call.duration, a histogram, from otelgrpc's instrumentation scope.
• A log record at ERROR for each recovered panic, from the logger github.com/z5labs/humus/grpc. It is emitted with the RPC's context, so it carries the RPC's trace_id and span_id. It has the attributes rpc.method, exception.message and exception.stacktrace.

The Service logs nothing else. The lifecycle already logs "serving", "shutting down" and "stopped".

Panics

The Service recovers a panic in a handler or an interceptor. The RPC fails with codes.Internal and the message internal error, so the panic's value never reaches the client. The panic is logged at ERROR with its stack trace, and the RPC's span records the error. Other RPCs, and the server, carry on.

The interceptors are the outermost ones, inside otelgrpc's stats handler. So the span sees the Internal status, and a panic in one of the end user's interceptors is recovered too.

The seam for health

Proposal: exposing health probes, in wave 3, decides how health.Probes are served over gRPC health v1, and whether this App does that by default. It needs two things from this proposal, and both are here:

• Registration. *Service is a grpc.ServiceRegistrar, so grpc_health_v1.RegisterHealthServer(svc, h) works on it before Serve. grpc.App holds the *Service the end user's Init returns, so the App can register a health server on it and return a health Service next to it.
• Ordering. The two shutdown signals already give health what it needs. The health Service's Serve context is done when shutdown starts, and that is when it turns not ready. This Service keeps serving through the drain delay, so a gRPC health check during the drain delay is answered, with not ready. grpc-go's own health server has Shutdown(), which sets every status to NOT_SERVING, if wave 3 uses it.

This proposal doesn't decide the answers to a health check, the service names, or Watch.

Design decisions

• One Service per *grpc.Server, built by humus. NewService builds the server, so it can install the stats handler, the recovery interceptors and WaitForHandlers before the end user's options. Taking a *grpc.Server from the end user would leave humus unable to know any of them were set.
• *Service is the registrar. Generated code calls RegisterXServer(registrar, impl), so the end user registers on the Service as they would on a *grpc.Server. A func(grpc.ServiceRegistrar) argument would be a second way to say the same thing.
• Settings are Addr and TLS; everything else is a grpc.ServerOption. grpc-go has dozens of server options. Copying them into Settings would add a field, a default and a check for each, and put a second name on each one. The two fields are what changes per deployment.
• Recover per RPC. The charter leaves recovery inside a request to each protocol, and the lifecycle never recovers. Without recovery, one panicking RPC ends the process, fails every other RPC in flight, and skips the graceful shutdown the lifecycle promises. net/http recovers a handler's panic and fails only that request, so a humus service fails a single request the same way over either protocol. It still fails loudly: an ERROR record with the stack, an error span, and Internal to the caller. The cost is that of net/http: state the handler left half-changed, such as a held lock, stays that way.
• Misuse panics instead of exiting. grpc-go calls os.Exit on three registration mistakes, which bypasses the lifecycle, the OnStop functions and telemetry. RegisterService checks for the same three first and panics. All three are programming errors, which fail in the first test or the first start.
• Stop waits for handlers. Without grpc.WaitForHandlers(true), a handler can outlive Shutdown, the App's grace period, and the process's telemetry. The option is experimental in grpc-go. If grpc-go removes it, the Service waits for its handlers itself.
• Serve through the drain delay, stop at Shutdown. A gRPC Service is in the lifecycle's second group, with HTTP: it keeps serving until Shutdown. GracefulStop is what tells clients with long-lived connections to go elsewhere.
• otelgrpc, not grpc-go's stats/opentelemetry. Both take injected providers. otelgrpc follows OpenTelemetry's semantic conventions for RPC, which the rest of an OTel setup and the HTTP server's instrumentation use. Its spans have attributes, and none of its API is marked experimental. grpc-go's tracing is in experimental/opentelemetry, its spans are named like Recv.grpc.testing.TestService.EmptyCall and carry no attributes, and its metrics follow gRPC's own gRFC A66 instead. An end user who wants A66 metrics passes opentelemetry.ServerOption in opts as a second stats handler.
• New dependencies. The grpc module adds google.golang.org/grpc, which the core module already has through the OTLP exporters, and go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc. The prototype uses otelgrpc v0.72.0 with OTel v1.47.0, the version the core module requires.
• A nested module. The charter names grpc as a nested module, so a service that doesn't use gRPC doesn't get grpc-go or otelgrpc in its module graph through it. grpc/go.mod requires the core module, with a replace to ../ so that CI builds it against the core module in the same commit. A release tags the core module first. The module is tagged grpc/vX.Y.Z.
• The test client lives here, as Proposal: humustest — whole-service testing on top of bedrocktest #633 decided. grpctest.NewClient is one function, but it encodes two traps. The dns resolver reaches the real network, and crashes a synctest bubble. And Host.Dial doesn't have the signature grpc.WithContextDialer takes. A test that dials a Service needs both right, every time. It is plaintext by default, and later transport credentials in opts replace that, so a TLS test passes credentials.NewTLS with the CA's pool.
• grpc-go's global logger is a documented leak. grpclog reads its variables from the real environment and writes to the real os.Stderr. Replacing it means grpclog.SetLoggerV2, which is global state, and humus sets none. This proposal's stories add grpc-go's GRPC_* variables and its stderr to the lifecycle page's list of ambient leaks. An end user who wants grpc-go's logs in their telemetry sets the logger in main.
• No reflection by default. Reflection exposes the service's schema to anyone who can reach the port. It is one line for the end user who wants it, reflection.Register(svc), which GetServiceInfo makes work.

Testing

The tests below come from a prototype in a scratch module that uses this module through a replace. It passes go test -race -count=3 with 90% coverage. They use grpc-go's interop/grpc_testing service. Its generated code ships with grpc-go, so the repository holds no .proto file and CI needs no protoc.

An RPC in flight when SIGTERM arrives finishes, and its span is recorded:

func TestApp_Run_an_RPC_in_flight_finishes_during_shutdown(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		tel := humustest.NewTelemetry()
		ts := &testServer{entered: make(chan struct{}), release: make(chan struct{})}
		app := humusgrpc.App{
			Host:      h.Host(),
			Telemetry: tel.Builder(),
			Init: func(ctx context.Context, s *humus.Setup) (*humusgrpc.Service, error) {
				svc, err := humusgrpc.NewService(s, humusgrpc.DefaultSettings())
				if err != nil {
					return nil, err
				}
				testpb.RegisterTestServiceServer(svc, ts)
				return svc, nil
			},
		}
		p, status := bedrocktest.Start(t, app)
		<-h.Listening(":9090")

		client := testpb.NewTestServiceClient(grpctest.NewClient(t, h, "carts:9090"))
		got := make(chan string, 1)
		go func() {
			resp, err := client.UnaryCall(t.Context(), &testpb.SimpleRequest{FillUsername: true})
			if err != nil {
				got <- err.Error()
				return
			}
			got <- resp.GetUsername()
		}()
		<-ts.entered // the handler is running

		p.Signal(syscall.SIGTERM)
		time.Sleep(7 * time.Second) // past the 5s drain delay: Shutdown is waiting for the RPC
		close(ts.release)

		require.Equal(t, "done", <-got)
		st := <-status
		require.Zero(t, st.Code(), st.Cause())
		require.Equal(t, "grpc.testing.TestService/UnaryCall", tel.Spans()[0].Name())
	})
}

An RPC that never returns is cut off at the grace period:

func TestApp_Run_shutdown_stops_at_the_grace_period(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		ts := &testServer{entered: make(chan struct{}), release: make(chan struct{})} // never released
		// newApp builds the App of the previous test, with ts registered.
		p, status := bedrocktest.Start(t, newApp(h, telemetry.NoopBuilder, ts),
			bedrocktest.WithEnv(map[string]string{"HUMUS_SHUTDOWN_GRACE_PERIOD": "2s"}))
		<-h.Listening(":9090")

		client := testpb.NewTestServiceClient(grpctest.NewClient(t, h, "carts:9090"))
		go func() { _, _ = client.UnaryCall(t.Context(), &testpb.SimpleRequest{FillUsername: true}) }()
		<-ts.entered

		begin := time.Now()
		p.Signal(syscall.SIGTERM)
		st := <-status

		require.Equal(t, humus.ExitFailure, st.Code())
		_, ok := errors.AsType[*humus.GracePeriodError](st.Cause())
		require.True(t, ok, st.Cause())
		require.Equal(t, 7*time.Second, time.Since(begin)) // 5s drain delay + 2s grace period
	})
}

The Service passes the contract every Service passes:

func TestService_contract(t *testing.T) {
	humustest.TestService(t, func(t *testing.T, s *humus.Setup) humus.Service {
		svc, err := humusgrpc.NewService(s, humusgrpc.DefaultSettings())
		require.NoError(t, err)
		testpb.RegisterTestServiceServer(svc, &testServer{})
		return svc
	})
}

Its subtest "Shutdown is safe after Serve has returned on its own" reports SKIP, as it does for an HTTP Service: Serve keeps serving after its context is done.

The tests also show the following. Each passes in the prototype.

• A panic in a handler fails that RPC with codes.Internal, and the next RPC succeeds. humustest.Telemetry's Logs() holds one ERROR record with the RPC's trace ID, and the RPC's span has status Error.
• TLS: with Settings.TLS holding a certificate that humustest.NewCA issued, a client built with grpctest.NewClient and credentials.NewTLS(&tls.Config{RootCAs: ca.Pool()}) succeeds. No key, certificate or password is written into the repository.
• Registration: RegisterService after Serve, twice for one service, or with an implementation of the wrong type panics, and the test process doesn't exit.
• Settings: an App whose Init passes Addr: "nope" exits 1 with init: Addr: must be host:port. Through the end user's config chain, the same value exits 78, naming the variable.
• The listener: an Init that fails after NewService leaves no listener open. A second App on the same Host listens on the same address.

The package's own tests follow #631's list: CheckAddr accepts and rejects with a table, NewService names each bad field, and DefaultSettings() passes NewService. An Example in the grpc package runs an App against grpc_testing, and the docs site gets a "gRPC services" page.

grpc-go has one rule for synctest that the docs page states. A client's target must not use the dns resolver, or the test binary crashes, as shown under Background. grpctest.NewClient uses passthrough, so a test that uses it is safe.

Migration from today

Today	Under this proposal
grpc.NewApi()	humusgrpc.NewService(s, settings, opts...), in the end user's Init
(*Api).RegisterService, called by generated code	(*Service).RegisterService. Generated RegisterXServer(svc, impl) calls don't change.
running an Api: not possible since #485	humusgrpc.App in bedrock.Main, or the Service in a humus.App
internal/grpcserver's unbounded GracefulStop	Shutdown, bounded by the grace period, then Stop
stats/opentelemetry fed from otel.Get*	otelgrpc fed from s.Telemetry
an implementation that is a health.Monitor is health-checked implicitly	nothing implicit. Proposal: exposing health probes (wave 3) decides how probes are served over gRPC health v1.
internal/grpchealth	Proposal: exposing health probes (wave 3)
grpc/internal/echopb in tests	grpc-go's interop/grpc_testing
a handler's panic ends the process	the RPC fails with Internal, and the panic is logged

Open questions

1. The package name. grpc collides with google.golang.org/grpc, which every file that registers a service, passes an option or returns a status also imports. So every end user aliases one of them. The options:

   • keep github.com/z5labs/humus/grpc as the charter names it, and alias it as humusgrpc in every example;
   • keep the nested module at grpc/, and put the server in github.com/z5labs/humus/grpc/grpcserver, next to grpc/grpctest. It matches the HTTP server foundation's httpserver, and leaves room for a gRPC client package in the same module later.

   I recommend the second. Only the import path changes; the API above is the same.

2. Streams that never end. GracefulStop waits for every RPC, streams included. A server stream meant to run until the client leaves, such as a subscription, only ends when Stop cancels it at the end of the grace period. Its handler's context isn't done until then. So a service with such a stream open exits 1 with a *GracePeriodError on every deployment. The options:

   • (a) keep it as proposed, and document that such a service sets a short grace period;
   • (b) when Shutdown starts, cancel the context of every streaming RPC, and let unary RPCs finish within the grace period. A client sees the stream end with Canceled, and reconnects to another pod, as it must after any disconnect. A finite stream, such as a large download, is cut short too.
   • (c) give handlers a signal that shutdown has started. That is new API, and each stream handler has to select on it.

   I lean to (b). The grace period exists for work that finishes, and a stream that never finishes can only end with an error either way. (b) ends it early, without failing the App.

3. The default address. :9090 is common for gRPC on Kubernetes, and doesn't clash with an HTTP server on 8080. grpc.io's examples use :50051. Is :9090 right?

4. The App's durations. App has no DrainDelay or ShutdownGracePeriod fields. An operator changes them with the lifecycle's variables, and code that needs other defaults uses humus.App with NewService. Should App have the two fields instead? Then a zero value would have to mean the package's default, which the settings convention rules out for Settings.

Out of scope

• gRPC health v1, and whether this App serves it by default: Proposal: exposing health probes, in wave 3. The seam it uses is described above.
• Outbound gRPC clients with FIT injection points: the charter's Later list. grpctest.NewClient is for tests only.
• Authentication and authorization. The end user's interceptors go in opts. A humus API for it comes later, after REST security.
• gRPC and HTTP on one port, through cmux or (*grpc.Server).ServeHTTP. ServeHTTP is experimental and lacks some of grpc-go's transport features. A process that serves both uses two Services on two ports, in one humus.App.
• gRPC-Gateway, which depends on the HTTP and REST proposals.
• xDS, ORCA and server-side load reporting.

References

• Pre-reset code: grpc/api.go, internal/grpcserver/server.go, internal/grpchealth/server.go
• Proposal: the humus App lifecycle, revision: Services and an App per runtime package, the two shutdown signals, panics left to each protocol
• Proposal: configuration for humus apps, final revision
• Proposal: OpenTelemetry without globals, revision 2
• Proposal: humustest: gRPC test helpers live with this proposal
• Proposal: health monitors and probes, revision 2, and its first revision's gRPC health v1 answers, which wave 3 takes over
• grpc-go v1.83.2: (*Server).GracefulStop, WaitForHandlers, grpclog, stats/opentelemetry, interop/grpc_testing
• otelgrpc, and OpenTelemetry's semantic conventions for RPC
• gRPC OpenTelemetry metrics (gRFC A66)

[Zaba505]

Revision: the open questions answered, on humus.Main

The charter's revision 5 moved this proposal to wave 3, behind Proposal: humus.Main, which has landed. This revision answers the four open questions, applies humus.Main, and takes in the charter's revision 7.

The open questions:

1. The server is grpcserver, at github.com/z5labs/humus/grpc/grpcserver, in the nested module at grpc/, as recommended. No example has to alias it, and it matches httpserver. The module's root holds no package.
2. Shutdown ends every stream, and the client gets Unavailable. This is option (b). When Shutdown starts, the Service cancels the context of every streaming RPC, and unary RPCs finish within the grace period. When a stream's handler then returns nil or a context error, the stream ends with codes.Unavailable and the message server is shutting down. The client reconnects, and lands on another pod. See Streams at shutdown.
3. TLS and the address are required. Neither field of Settings has a default, so DefaultSettings() goes. Nothing is served in plain text, and humus imposes no port. Kubernetes' gRPC probes connect in plain text, so Proposal: exposing health probes (wave 4) serves them on a plaintext port of its own, with only gRPC health v1 on it. See What waves 4 and 5 build on this.
4. The App's durations are answered by humus.Main. App is gone. grpcserver.Defaults() sets the drain and the grace period as one humus.Option, and grpcserver.Main applies it before the end user's options. So grpcserver.Main(Init, humus.WithShutdownGracePeriod(25*time.Second)) changes the grace period in code, and no zero value has to stand for the package's default.

What humus.Main and the charter change:

• humusgrpc.App is gone. grpcserver.Main is humus.Main with the package's defaults, a 5s drain and a 15s grace period. The end user's Init builds the Service with NewService and returns it.
• Telemetry is no longer required. humus.Main's telemetry default replaces it, as humus.Main's revision 2 sets out.
• The test package is grpcservertest, beside grpcserver, as httpservertest is beside httpserver in the HTTP server foundation's revision. It replaces grpctest. Its NewClient trusts a CA's pool, since every Service serves TLS.
• The waves. This proposal is in wave 3. It blocks exposing health probes and outbound gRPC clients in wave 4, and failure injection and authentication and authorization in wave 5.

Each section below replaces the section of the same name in the post. Background and References are unchanged.

Summary

humus can't run a gRPC service today. grpc.NewApi builds a *grpc.Server and nothing serves it. This proposes a nested module, github.com/z5labs/humus/grpc, with a package grpcserver:

• a Service that serves a *grpc.Server over TLS, on a listener from the Setup's Host. It keeps serving through the drain. When Shutdown starts, it ends every stream with Unavailable, stops gracefully within the grace period, and then stops hard.
• Settings with two fields, Addr and TLS, both required. Everything else about the server is a grpc.ServerOption the end user passes.
• Main, which is humus.Main with a gRPC service's defaults: a 5s drain and a 15s grace period, exported as one humus.Option, Defaults().
• telemetry from s.Telemetry, through otelgrpc, with no OTel globals.
• per-RPC panic recovery, which fails only the RPC that panicked, with codes.Internal.
• grpcservertest, which starts an App with those defaults on a test Host and gives a test a client that dials that Host over TLS. It is in the grpc module, so humustest doesn't import grpc-go.

func main() {
	grpcserver.Main(app.Init, humus.WithTelemetry(buildTelemetry))
}

app.Init sources the Settings, builds the Service with grpcserver.NewService, registers the end user's services on it, and returns it.

grpc-go runs inside a testing/synctest bubble on humustest's in-memory network: its server, its client, otelgrpc and TLS all do. The post's prototype passed go test -race -count=3 with 90% coverage. The tests this revision adds or changes are sketches and have not been run.

Proposed API

// Package grpcserver serves gRPC with humus's lifecycle.
package grpcserver // import "github.com/z5labs/humus/grpc/grpcserver", in the nested module github.com/z5labs/humus/grpc

// Settings configures a Service. Every field is a plain value, and a
// field's zero value means zero, never "use the default". Neither field
// has a default, so the end user sources both.
type Settings struct {
	// Addr is the address the Service listens on through the Setup's Host,
	// as host:port. It is required: humus imposes no port. An empty host
	// listens on every name, and port 0 picks a free port, which
	// Service.Addr reports.
	//
	// Default: none. Valid: host:port, with a port from 0 to 65535.
	// Check: CheckAddr.
	Addr string

	// TLS is the config the Service serves gRPC with. It is required: the
	// Service serves nothing in plain text. A client certificate is
	// required by setting ClientAuth, as with any tls.Config.
	//
	// Default: none. Valid: a config with Certificates, GetCertificate or
	// GetConfigForClient set, and a MaxVersion of TLS 1.2 or later, or
	// none. Check: CheckTLS.
	TLS *tls.Config
}

// CheckAddr reports why addr is not a valid Addr, or nil if it is.
func CheckAddr(addr string) error

// CheckTLS reports why c can't serve gRPC, or nil if it can. c must not be
// nil, must have Certificates, GetCertificate or GetConfigForClient set,
// and must allow TLS 1.2 or later, which HTTP/2 requires.
func CheckTLS(c *tls.Config) error

// Service is a humus.Service that serves gRPC over TLS. It is a
// grpc.ServiceRegistrar, so generated code registers services on it:
//
//	cartpb.RegisterCartServiceServer(svc, carts)
type Service struct{ /* unexported */ }

// NewService runs CheckAddr and CheckTLS on settings, reporting every bad
// field in one error that names each one, listens on settings.Addr through
// s.Host, and builds the server.
//
// opts come after humus's own options, and humus's transport credentials
// come after opts. An option that grpc-go keeps one value of, such as
// grpc.MaxRecvMsgSize, replaces humus's. An option that chains, such as
// grpc.ChainUnaryInterceptor, is added inside humus's. grpc.Creds in opts
// has no effect, so every connection is served with settings.TLS.
//
// The listener is closed by Shutdown, or by an OnStop function NewService
// registers with s, if the App never serves.
func NewService(s *humus.Setup, settings Settings, opts ...grpc.ServerOption) (*Service, error)

// Addr returns the address the Service listens on, with the port the Host
// picked when settings.Addr has port 0.
func (s *Service) Addr() net.Addr

// RegisterService, GetServiceInfo and Serve are unchanged.

// Shutdown ends every streaming RPC: it cancels each stream handler's
// context, and a stream whose handler then returns nil or a context error
// ends with codes.Unavailable. It stops the server gracefully: it stops
// accepting connections, sends every client a GOAWAY, and waits for the
// RPCs in flight. If ctx is done first, it closes every connection,
// cancels every RPC, waits for every handler to return, and returns an
// error that wraps ctx's error.
func (s *Service) Shutdown(ctx context.Context) error

// Defaults sets a gRPC service's defaults over humus.Main's: a drain of
// humus.DrainDelay(5 * time.Second) and a 15s grace period. The Host and
// the telemetry Builder stay humus.Main's.
func Defaults() humus.Option

// Main is humus.Main with Defaults applied before opts, so the end user's
// options win over the package's. It never returns.
func Main(init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humus.Option)

// Package grpcservertest tests gRPC services on a humustest Host.
package grpcservertest // import "github.com/z5labs/humus/grpc/grpcserver/grpcservertest"

// Start is humustest.Start with grpcserver.Defaults applied before opts, so
// a test runs its App with the drain and the grace period that
// grpcserver.Main gives it in production. Options given with
// humustest.WithApp come after the defaults, and win over them.
func Start(t testing.TB, init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humustest.Option) (*bedrocktest.Process, <-chan bedrock.ExitStatus)

// NewClient returns a client of the Service listening on address in h. It
// dials through h, never resolves address, and verifies the Service's
// certificate for address's host against roots. Transport credentials in
// opts replace those. The client is closed when the test ends. It works
// with or without Start.
func NewClient(t testing.TB, h *humustest.Host, address string, roots *x509.CertPool, opts ...grpc.DialOption) *grpc.ClientConn

App and DefaultSettings() are gone. CheckTLS, Service.Addr, Defaults, Main and grpcservertest are new.

The end user's side

The end user sources Settings in their Init, as Proposal: configuration for humus apps sets out, builds the Service, registers their services, and returns it:

func main() {
	grpcserver.Main(Init, humus.WithTelemetry(buildTelemetry))
}

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }

	settings, err := config.Struct(func(r *config.Resolver) grpcserver.Settings {
		return grpcserver.Settings{
			// No .Default: both fields are required, so an unset variable fails as not found.
			Addr: r.Get(env("GRPC_ADDR").Validate(grpcserver.CheckAddr)),
			// readFile and serverTLS are the end user's own helpers.
			TLS: r.Get(env("GRPC_TLS_P12_FILE").Parse(readFile(s.Host.FS)).Parse(serverTLS).Validate(grpcserver.CheckTLS)),
		}
	}).Read(ctx)
	if err != nil {
		return nil, err // exit 78, naming every bad variable
	}

	pool, err := newPool(ctx, s) // closed by an OnStop function
	if err != nil {
		return nil, err
	}

	svc, err := grpcserver.NewService(s, settings, grpc.ChainUnaryInterceptor(authenticate))
	if err != nil {
		return nil, err
	}
	cartpb.RegisterCartServiceServer(svc, carts.New(pool))
	return []humus.Service{svc}, nil
}

With no environment, config.Struct reports both variables at once, and the App exits 78.

The Service

Unchanged, except that the paragraphs below replace the ones with the same heading, and Streams at shutdown is new.

NewService does three things, in order:

1. It runs CheckAddr and CheckTLS, and returns every bad field in one error, naming each one. A Settings built in code is checked as well as one sourced with config.
2. It listens on settings.Addr with s.Host.Listen. The listener is opened in Init, so connections that arrive before Serve wait in the backlog, as the lifecycle expects. If Init fails after NewService, the App never calls Shutdown, so NewService registers an OnStop function that closes the listener. Closing an already closed listener is not an error.
3. It builds the *grpc.Server with humus's options, then opts, then humus's transport credentials:

Option	Why
grpc.StatsHandler(otelgrpc.NewServerHandler(...))	traces and metrics, from s.Telemetry; see Telemetry
grpc.ChainUnaryInterceptor and grpc.ChainStreamInterceptor, with panic recovery, and for streams the context that Shutdown cancels	the outermost interceptors, so a panic in the end user's interceptors is recovered too, and every stream handler's context is one Shutdown can cancel; see Panics and Streams at shutdown
grpc.WaitForHandlers(true)	Stop waits for every handler to return, so no handler outlives Shutdown
the end user's opts
grpc.Creds(credentials.NewTLS(settings.TLS))	last, so an end user's grpc.Creds can't replace it. credentials.NewTLS clones the config, adds h2 to its ALPN protocols, and makes TLS 1.2 the least version when the config sets none.

opts is where everything else goes: the end user's interceptors, keepalive parameters, message size limits, a second stats handler. grpc-go keeps one value of some options and chains others, and NewService documents what that means here:

• an end user's interceptors run inside humus's, after recovery;
• an end user's grpc.Creds has no effect. grpc-go keeps the last one, and humus's comes last;
• an end user's grpc.WaitForHandlers(false) lets a handler outlive Shutdown. That breaks the humus.Service contract, and humustest.TestService reports it as a leak.

Shutdown ends every stream, as Streams at shutdown describes, and calls GracefulStop. GracefulStop closes the listener and sends every connection a GOAWAY, so clients open new connections elsewhere, and it waits for the RPCs in flight to finish. gRPC connections are long-lived, and the GOAWAY is what moves a client off a terminating pod. If ctx is done first, Shutdown calls Stop. That closes every connection, cancels every RPC's context and, with WaitForHandlers, waits for every handler to return. Then Shutdown returns an error that wraps ctx.Err(), and the App exits 1 with a *humus.GracePeriodError. Shutdown is safe before Serve, after Serve has returned and twice.

Streams at shutdown. The stream interceptor gives every streaming RPC's handler a context of its own, derived from the RPC's, which Shutdown cancels. That covers server, client and bidirectional streams. When Shutdown starts:

• it cancels the context of every stream in flight. A stream that starts after that gets a context that is already cancelled;
• when a handler then returns nil, an error that is context.Canceled, or a status with codes.Canceled, the interceptor returns status.Error(codes.Unavailable, "server is shutting down") in its place. Without that, grpc-go would end the stream with Canceled (server.go#L1460-L1467). And a handler that returned nil, such as a download cut short, would end its stream with OK;
• a handler that returns a status of its own, such as Aborted with details, keeps it;
• unary RPCs are not cancelled. They finish within the grace period, as before.

GracefulStop keeps each connection open until its streams have ended, so the status reaches the client, in the stream's trailers:

resp, err := stream.Recv()
if status.Code(err) == codes.Unavailable {
	// The server is going away. Reconnect: the new stream lands on another pod.
}

A handler sees the shutdown through its context. A handler that is blocked in Recv, waiting for its client's next message, sees it only when the client sends, or when Stop closes the connection at the end of the grace period. A handler that has to notice shutdown while it waits receives in a goroutine and selects on its context, as it would for any call that takes no context.

The streams are cancelled when Shutdown starts, after the drain, and not when Serve's context is done. During the drain, a load balancer may still route a client that reconnects back to this pod.

Main and its defaults

This section replaces The App. humus.Main runs the Services of any runtime, so the package exports only what App added on top: its defaults, as one humus.Option, and a Main that applies them. They are, in full:

func Defaults() humus.Option {
	return func(a *humus.App) {
		a.Drain = humus.DrainDelay(5 * time.Second)
		a.ShutdownGracePeriod = 15 * time.Second
	}
}

func Main(init func(ctx context.Context, s *humus.Setup) ([]humus.Service, error), opts ...humus.Option) {
	humus.Main(init, append([]humus.Option{Defaults()}, opts...)...)
}

• The defaults are unchanged. The 5s drain gives Kubernetes' endpoint removal time to reach every node. The 15s grace period gives unary RPCs, and stream handlers ending on their contexts, time to finish. Streams that never end no longer use it up. With telemetry's 5s shutdown, they add up to 25s, the budget humus.Main keeps within Kubernetes' default terminationGracePeriodSeconds of 30s.
• The gRPC Service drains for the whole delay. It keeps serving until Shutdown, so its Serve hasn't returned during the drain, and the Drain's context isn't done early.
• The Host and the telemetry are humus.Main's. Telemetry was required on App, so that a service couldn't start as unknown_service by accident. humus.Main decided otherwise for every runtime: the default Builder calls telemetry.New with DefaultSettings() and writes to stdout, and a service that names itself or exports anything passes humus.WithTelemetry.
• The end user changes a default with an option. In grpcserver.Main(Init, humus.WithShutdownGracePeriod(25*time.Second)), the end user's option comes after Defaults(), so it wins. That is what open question 4 asked for: a duration set in code, without a zero value that stands for the package's default. An operator still sets HUMUS_SHUTDOWN_GRACE_PERIOD and HUMUS_SHUTDOWN_DRAIN_DELAY in the manifest that sets terminationGracePeriodSeconds.
• The end user's Init returns every Service. App's Init returned the *Service so that the App could hold it, and a process with anything else on it had to use humus.App. Now the end user's Init returns the Service along with any others the process runs. A process that runs a gRPC server next to a Kafka consumer returns both, and uses grpcserver.Main, because its gRPC server needs the drain.
• No health. Main runs no health Service and serves no probes. See What waves 4 and 5 build on this.

Settings

Field	Default	Valid	Check
Addr	none: required	host:port, with a port from 0 to 65535	CheckAddr
TLS	none: required	a config that has a certificate and allows TLS 1.2 or later	CheckTLS

Settings holds what an operator changes from one deployment to the next: where to listen, and the certificate. How the server behaves, such as its interceptors, keepalive parameters and message size limits, is code, and is passed as a grpc.ServerOption. That keeps one way to set each of them. The end user who wants a variable for one sources it in their Init and passes the option.

• Both fields are required, and there is no DefaultSettings(). The zero Settings has an Addr of "", which CheckAddr rejects because it has no port, and a nil TLS, which CheckTLS rejects. With no field to default, DefaultSettings() would return the zero Settings, so the package has none. The end user's chains have no .Default, so a variable that isn't set fails as not found, with exit 78.
• CheckTLS fails startup for what would otherwise fail every connection. With no certificate, every handshake fails. So does a MaxVersion below TLS 1.2: HTTP/2 requires TLS 1.2, and grpc-go's clients ask for at least that by default. The check fails Init instead, naming the variable that sourced the config.
• CheckAddr is unchanged. It accepts what Host.Listen accepts: an empty host for every name, and port 0 for a free port. It rejects an address with no port, "" included, and a port that is not a number from 0 to 65535.
• Optional settings (#631, question 2). No field is optional now. TLS was the only one, and it is required.

What waves 4 and 5 build on this

This section replaces The seam for health.

Proposal: exposing health probes (wave 4) decides how probes are served over gRPC health v1, and whether a gRPC service exposes them by default. Kubernetes' gRPC probes connect in plain text, and never present a client certificate. A probe over TLS, with mode: TLS, needs the GRPCContainerProbeTLS feature gate, which is alpha in Kubernetes 1.37 and off by default (Use TLS with gRPC probes). So on most clusters, a Service that requires TLS can't answer them. Wave 4 serves them on a second gRPC server, on a port of its own, in plain text, with only grpc.health.v1.Health registered:

readinessProbe:
  grpc:
    port: 8081 # the health server's port; the end user's services are served over TLS on their own port

The plaintext port answers health checks and nothing else, so the end user's services are never reachable in plain text. The two ports are two Services in one App, which humus already runs.

The seam is in this module:

• Serving and shutdown live in an internal package of the grpc module. That package listens through the Host, recovers panics, cancels streams, sets WaitForHandlers, and calls GracefulStop and then Stop. grpcserver.NewService is that code with TLS required and the end user's options. Wave 4 builds its plaintext health server on the same code, so it serves through the drain and shuts down as this Service does, and grpcserver doesn't offer plain text. Wave 4's gRPC health code is in the grpc module for that reason.
• Registration. *Service is still a grpc.ServiceRegistrar, so wave 4 can also register health v1 on the end user's Service before Serve, for gRPC clients' own health checks, or for probes with mode: TLS.
• Ordering. The two shutdown signals give health what it needs. The health Service's Serve context is done when shutdown starts, and that is when it turns not ready. Both gRPC servers keep serving through the drain, so a health check during the drain is answered, with not ready. A Watch is a stream, so Shutdown ends it with Unavailable.
• One server, where probes can use TLS. On a cluster with GRPCContainerProbeTLS on, a probe with mode: TLS reaches the end user's Service, and the second port isn't needed, unless the Service sets ClientAuth: the kubelet presents no client certificate. Wave 4's docs say when one server is enough.

This proposal doesn't decide the answers to a health check, the service names, Watch, or the health server's settings.

Proposal: failure injection (wave 5) strips scenarios from what a server receives, before the end user's code sees them. Its seam here is humus's interceptors. otelgrpc's stats handler extracts the trace context and the baggage into the RPC's context before any interceptor runs. Then humus's recovery runs, then the end user's interceptors, then the handler. Stripping belongs after recovery and before the end user's interceptors, inside the Service, so every RPC is covered without the end user adding an interceptor. The wave-5 proposal designs that interceptor and whatever configures it, and its stories add them here.

Design decisions

Unchanged, except for the following. These bullets replace "Serve through the drain delay, stop at Shutdown", "The test client lives here, as #633 decided" and, from The App, "Init returns the *Service, not a slice" and "Telemetry is required":

• Serve through the drain, and end streams and stop at Shutdown. A gRPC Service is in the lifecycle's second group, with HTTP: it keeps serving until Shutdown. GracefulStop is what tells clients with long-lived connections to go elsewhere, and ending the streams is what lets it finish.
• No App: a Main, and its defaults as one Option. humus.Main runs every runtime's Services. As humus.Main's revision 2 sets out for every runtime package, this package's Main does nothing but apply Defaults() before the end user's options. A test passes the same Option, through grpcservertest.Start or humustest.WithApp.
• The test package is grpcservertest. Its Start means a test can't forget the defaults. Its NewClient exists because a test that dials a Service has two traps to get right every time. The dns resolver reaches the real network, and crashes a synctest bubble. And Host.Dial doesn't have the signature grpc.WithContextDialer takes. NewClient makes no CA, as httpservertest makes none: how the end user's Init sources its certificate is the end user's decision. The test makes a humustest.CA, gives the App a certificate the way production does, and passes the CA's Pool() to NewClient. A test of wave 4's plaintext health server passes insecure.NewCredentials() in opts. NewClient works without Start, so a test of a process that runs an HTTP and a gRPC server can use one package's Start and both packages' clients.

These bullets are new:

• The package is grpcserver. grpc collides with google.golang.org/grpc, which every file that registers a service also imports. grpcserver matches httpserver, and leaves the module room for Proposal: outbound gRPC clients (wave 4).
• TLS is required. Traffic is encrypted all the way to the pod, whatever sits in front of it, as with the HTTP server foundation. The cost is that every service needs a certificate, even behind a mesh that already encrypts. Kubernetes' gRPC probes also need a plaintext port of their own until probes over TLS are on in the cluster. That port is wave 4's, and serves health alone.
• The address is required. A default port is a network decision made for every service that doesn't override it. A pod's container port, its Service's target port and its probes are set in the manifest, so the address the process listens on belongs with them, and the end user sources it from the same place. A service that forgets it fails Init with exit 78, naming the variable, rather than listening on a port nothing routes to.
• No DefaultSettings(). Under the configuration proposal, DefaultSettings() supplies a package's defaults. This package has none, so it would only return the zero Settings.
• humus's transport credentials come last. Settings.TLS is the one way to set how the Service secures its connections. A grpc.ServerOption is opaque, so NewService can't reject an end user's grpc.Creds. Instead, it places its own after opts, and grpc-go keeps the last one. So grpc.Creds(insecure.NewCredentials()) in opts has no effect.
• Shutdown ends streams, with Unavailable. Without that, a stream that runs until its client leaves holds GracefulStop up until the grace period, and every deployment exits 1 with a *GracePeriodError. Unavailable is gRPC's code for a condition the client corrects by retrying, and it is what grpc-go's client reports when a connection goes away. Canceled would tell the client that the call was cancelled, typically by the caller. otelgrpc marks a server span Error for Unavailable, as OpenTelemetry's RPC conventions say, so a rollout records an error span for each stream that was open. That is accurate: the server ended those streams.
• A stream ends through its handler's context. The Service doesn't make Recv return when the handler's context is cancelled. That would take a goroutine for each receive, which could still be blocked in grpc-go's RecvMsg after the handler has moved on, still writing into the handler's message, and grpc-go doesn't allow two RecvMsg calls on one stream at once. A handler that has to notice shutdown while it waits for its client does what it does for any call that takes no context.
• Addr() is API. It is the only way to learn the port when Addr has port 0, as with the HTTP server foundation.

Testing

These tests are sketches and have not been run. Every test runs inside a testing/synctest bubble, on a humustest Host, and asserts with testify/require. They use grpc-go's interop/grpc_testing service, imported as testpb. serves is a helper of the test file:

// serves returns an Init whose only Service listens on ":9090" over TLS
// with cert, with ts registered.
func serves(cert *humustest.Certificate, ts testpb.TestServiceServer) func(context.Context, *humus.Setup) ([]humus.Service, error) {
	return func(_ context.Context, s *humus.Setup) ([]humus.Service, error) {
		svc, err := grpcserver.NewService(s, grpcserver.Settings{
			Addr: ":9090",
			TLS:  &tls.Config{Certificates: []tls.Certificate{cert.TLS()}},
		})
		if err != nil {
			return nil, err
		}
		testpb.RegisterTestServiceServer(svc, ts)
		return []humus.Service{svc}, nil
	}
}

The Service passes the contract every Service passes:

func TestService_meets_the_Service_contract(t *testing.T) {
	humustest.TestService(t, func(t *testing.T, s *humus.Setup) humus.Service {
		cert := humustest.NewCA(t).Issue(t, "carts")
		svc, err := grpcserver.NewService(s, grpcserver.Settings{
			Addr: ":9090",
			TLS:  &tls.Config{Certificates: []tls.Certificate{cert.TLS()}},
		})
		require.NoError(t, err)
		testpb.RegisterTestServiceServer(svc, &testServer{})
		return svc
	})
}

Its subtest "Shutdown is safe after Serve has returned on its own" reports SKIP, as it does for an HTTP Service: Serve keeps serving after its context is done.

An RPC in flight when SIGTERM arrives finishes, and its span is recorded:

func TestMain_an_RPC_in_flight_finishes_during_shutdown(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		ca := humustest.NewCA(t)
		h := humustest.NewHost(t)
		tel := humustest.NewTelemetry()
		ts := &testServer{entered: make(chan struct{}), release: make(chan struct{})}

		p, exit := grpcservertest.Start(t, serves(ca.Issue(t, "carts"), ts),
			humustest.WithApp(humus.WithHost(h.Host()), humus.WithTelemetry(tel.Builder())),
		)
		<-h.Listening(":9090")

		client := testpb.NewTestServiceClient(grpcservertest.NewClient(t, h, "carts:9090", ca.Pool()))
		got := make(chan string, 1)
		go func() {
			resp, err := client.UnaryCall(t.Context(), &testpb.SimpleRequest{FillUsername: true})
			if err != nil {
				got <- err.Error()
				return
			}
			got <- resp.GetUsername()
		}()
		<-ts.entered // the handler is running

		p.Signal(syscall.SIGTERM)
		time.Sleep(7 * time.Second) // past the 5s drain: Shutdown is waiting for the RPC
		close(ts.release)

		require.Equal(t, "done", <-got)
		st := <-exit
		require.Zero(t, st.Code(), st.Cause())
		require.Equal(t, "grpc.testing.TestService/UnaryCall", tel.Spans()[0].Name())
	})
}

A stream that would never end is ended at Shutdown, with Unavailable, and the App exits 0:

// subscriber's StreamingOutputCall sends one response, then waits for its
// context, as a subscription does.
type subscriber struct{ testpb.UnimplementedTestServiceServer }

func (subscriber) StreamingOutputCall(_ *testpb.StreamingOutputCallRequest, stream testpb.TestService_StreamingOutputCallServer) error {
	if err := stream.Send(&testpb.StreamingOutputCallResponse{}); err != nil {
		return err
	}
	<-stream.Context().Done()
	return stream.Context().Err()
}

func TestMain_shutdown_ends_a_stream_with_Unavailable(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		ca := humustest.NewCA(t)
		h := humustest.NewHost(t)
		p, exit := grpcservertest.Start(t, serves(ca.Issue(t, "carts"), subscriber{}),
			humustest.WithApp(humus.WithHost(h.Host())),
		)
		<-h.Listening(":9090")

		client := testpb.NewTestServiceClient(grpcservertest.NewClient(t, h, "carts:9090", ca.Pool()))
		stream, err := client.StreamingOutputCall(t.Context(), &testpb.StreamingOutputCallRequest{})
		require.NoError(t, err)
		_, err = stream.Recv()
		require.NoError(t, err)

		begin := time.Now()
		p.Signal(syscall.SIGTERM)
		_, err = stream.Recv()
		require.Equal(t, codes.Unavailable, status.Code(err), err)
		require.Equal(t, 5*time.Second, time.Since(begin)) // the drain, then Shutdown ends the stream

		require.Zero(t, (<-exit).Code())
	})
}

The end user's Init, with no environment, reports both variables:

func TestInit_without_an_address_or_a_certificate_exits_78(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		_, exit := grpcservertest.Start(t, Init)

		st := <-exit
		require.Equal(t, humus.ExitConfig, st.Code())
		require.ErrorContains(t, st.Cause(), "config GRPC_ADDR: not found")
		require.ErrorContains(t, st.Cause(), "config GRPC_TLS_P12_FILE: not found")
	})
}

The post's test of the grace period is unchanged, except that it runs with grpcservertest.Start, serves and humustest.WithEnv(map[string]string{"HUMUS_SHUTDOWN_GRACE_PERIOD": "2s"}): a unary RPC that never returns is still cut off after the 5s drain and the 2s grace period, with a *humus.GracePeriodError.

The tests also show the following:

• Streams at shutdown. A handler that returns nil once its context is done ends its stream with Unavailable, not OK. A handler that returns status.Error(codes.Aborted, ...) keeps Aborted. A stream that starts after Shutdown has begun ends with Unavailable at once. A unary RPC in flight is not cancelled. A bidirectional handler blocked in Recv holds the shutdown until the grace period, and the App exits 1 with a *humus.GracePeriodError.
• TLS can't be turned off. A Service built with grpc.Creds(insecure.NewCredentials()) in opts still serves TLS. A client built with NewClient and the CA's pool succeeds, and a plaintext client fails.
• Settings. NewService with the zero Settings names Addr and TLS, in one error. CheckTLS rejects nil, a config with no certificate, and a MaxVersion of TLS 1.1, and accepts a config with Certificates, GetCertificate or GetConfigForClient. An App whose Init passes Addr: "nope" exits 1 with init: Addr: must be host:port. Through the end user's config chain, the same value exits 78, naming the variable.
• Port 0. A Service on ":0" reports the port the Host picked through Addr(), and an RPC to it succeeds.
• A panic, registration and the listener are unchanged from the post, each run with a certificate from a humustest.CA.

The package's own tests follow #631's list, except that there is no DefaultSettings(): CheckAddr and CheckTLS accept and reject with a table, and NewService names each bad field. An Example in grpcserver runs an App against grpc_testing, and the docs site gets a "gRPC services" page.

Migration from today

These rows replace the ones with the same first column:

Today	Under this proposal
grpc.NewApi()	grpcserver.NewService(s, settings, opts...), in the end user's Init
running an Api: not possible since #485	grpcserver.Main, with the Service among the Services the end user's Init returns
internal/grpcserver's unbounded GracefulStop	Shutdown: every stream ends with Unavailable, then GracefulStop, bounded by the grace period, then Stop
an implementation that is a health.Monitor is health-checked implicitly	nothing implicit. Proposal: exposing health probes (wave 4) serves gRPC health v1, on a plaintext port of its own.
internal/grpchealth	Proposal: exposing health probes (wave 4)

Out of scope

These bullets replace the ones with the same heading, and the last one is new:

• gRPC health v1, and whether a gRPC service serves it by default: Proposal: exposing health probes, in wave 4, on a plaintext port of its own. The seam it uses is described above.
• Outbound gRPC clients: Proposal: outbound gRPC clients (wave 4), in the grpc module. Their failure injection: Proposal: failure injection (wave 5). grpcservertest.NewClient is for tests only.
• Authentication and authorization. The end user's interceptors go in opts. humus's API for it is Proposal: authentication and authorization (wave 5).
• TLS and plain text on one port, by reading the first byte of each connection. Every RPC would then be reachable in plain text, unless a check on each RPC stopped it. Probes get a port of their own instead.

Open questions

None.

[Zaba505]

Closing: gRPC services have landed

The revision's stories are done:

• story(grpcserver): start the grpc module with Settings and a check per field #715 started the grpc module with grpcserver.Settings and a check per field (story(grpcserver): start the grpc module with Settings and a check per field #737);
• story(grpcservertest): dial a gRPC Service on a humustest Host over TLS #716 added grpcservertest, which dials a gRPC Service on a humustest Host over TLS (story(grpcservertest): dial a gRPC Service on a humustest Host over TLS #738);
• story(grpcserver): serve gRPC over TLS as a humus Service #717 served gRPC over TLS as a humus Service (story(grpcserver): serve gRPC over TLS as a humus Service #739);
• story(grpcserver): trace and measure every RPC with the Setup's telemetry #718 traced and measured every RPC with the Setup's telemetry (story(grpcserver): trace and measure every RPC with the Setup's telemetry #740);
• story(grpcserver): fail only the RPC that panicked, with Internal #719 failed only the RPC that panicked, with Internal (story(grpcserver): fail only the RPC that panicked, with Internal #741);
• story(grpcserver): run a gRPC service on its defaults with grpcserver.Main #720 added grpcserver.Main, which runs a gRPC service on its defaults (story(grpcserver): run a gRPC service on its defaults with grpcserver.Main #742);
• story(grpcserver): end every stream at Shutdown with Unavailable #721 ended every stream at Shutdown with Unavailable (story(grpcserver): end every stream at Shutdown with Unavailable #743);
• story(docs): document gRPC services #722 documented gRPC services (story(docs): document gRPC services #744).

The revision left no open questions. What it hands on belongs to the later proposals, as "What waves 4 and 5 build on this" lists: exposing health probes, with gRPC health v1, and outbound gRPC clients in wave 4, and failure injection in wave 5.