Proposal: outbound gRPC clients

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 4. It builds on wave 1 (the lifecycle, configuration, OpenTelemetry without globals and humustest) and on Proposal: gRPC services: its grpc module, and the passthrough resolver and Host dialler that grpcservertest.NewClient uses. It is one of wave 4's three client proposals, with outbound HTTP clients and producing to Kafka, and depends on neither. It blocks Proposal: failure injection (wave 5), which adds an injection point at the seam this proposal names.

Summary

A new package, grpcclient, in the grpc module, calls gRPC services from a humus service. The end user's Init builds a client from Settings, and generated code takes it:

carts, err := grpcclient.NewClient(s, settings) // s is the end user's *humus.Setup
if err != nil {
	return nil, err
}
api := &api{carts: cartpb.NewCartServiceClient(carts)}

• grpcclient.Client is a grpc.ClientConnInterface of humus's own, around a *grpc.ClientConn. It is not a humus.Service. It closes its connection with s.OnStop, after every Service has stopped.
• It dials only through s.Host. The target is passthrough:/// and Settings.Addr, and the dialler is s.Host.Dial, as in grpcservertest.NewClient and telemetry's OTLP connection. The Host resolves the name when it dials. grpc-go's dns resolver, which goes around the Host and crashes a synctest bubble, is never used.
• Every unary RPC is retried with bedrock's retry, and every attempt goes through bedrock's circuit. Unavailable is retried, and so is ResourceExhausted when the server sends grpc-retry-pushback-ms. The pushback is the least delay. grpc-go's own configurable retries are turned off, and its transparent retries, of RPCs the server never received, are kept.
• Streams go through the breaker too. Opening a stream is retried, because nothing has been sent yet. Once a stream is open it is never retried, and the breaker counts its final status.
• Telemetry comes from s.Telemetry alone: otelgrpc's client stats handler gives each attempt a client span and the RPC client metrics. A call the breaker rejects gets a failed span of its own, and each change of the breaker's state is one log record.
• Settings holds the address, the TLS config (nil means plain text), a timeout for each unary call, and the retry and breaker settings, typed with bedrock's retry.BudgetConfig and circuit.Config.

Background

• humus had no gRPC client before the reset. The grpc package had NewApi and RegisterService for servers only. The only *grpc.ClientConn in the tree was the OTLP exporters' connection, grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials())) (rest/rest.go#L358-L364). That uses the dns resolver and the real network, and its exporters used OTel's globals.
• grpc-go's defaults reach around the Host. A target with no scheme gets the dns resolver (internal/resolver/dns/dns_resolver.go). Whatever dialler the client has, it:
  • looks names up with net.DefaultResolver, which reads /etc/resolv.conf and /etc/hosts from the real filesystem and queries the real network, or, given a DNS server in the target's authority, dials that server with a plain net.Dialer;
  • looks up a _grpc_config. TXT record for a service config, unless the real environment sets GRPC_ENABLE_TXT_SERVICE_CONFIG=false;
  • is tuned with package-level state, dns.SetResolvingTimeout and dns.SetMinResolutionInterval, which humus never sets;
  • inside a testing/synctest bubble, crashes the test binary with "fatal error: send on synctest channel from outside bubble", as grpcservertest's doc records.
• The passthrough resolver avoids all of it. It hands the address to the dialler as it is. grpcservertest.NewClient does this for tests, and telemetry's OTLP connection does it in production. Proposal: gRPC services left outbound clients to this proposal: "grpcservertest.NewClient is for tests only."
• Every humus gRPC server requires TLS (#681's revision), so a humus client of a humus server always uses it.
• bedrock v0.24.0 ships retry and circuit as models with no I/O. Each leaves the protocol to humus. retry's doc says it does not "replay request bodies or parse Retry-After or gRPC pushback, which belong to the framework that wraps a transport", and ships no classifiers. circuit ships no transports, policies, classifiers or observers (circuit, retry).
• The charter's revision 7 puts retry and circuit in the three client proposals. It holds them to three points: each dials through Setup.Host and reports through Setup.Telemetry, each composes retry and circuit as bedrock's docs do, and each names the seam failure injection uses. For gRPC it names "interceptors that read grpc-retry-pushback-ms". It also says this proposal "decides how a client resolves names through the Host".

Proposed API

package grpcclient // github.com/z5labs/humus/grpc/grpcclient

// Settings configures a Client. Every field is a plain value, and a field's
// zero value means zero, never "use the default": start from DefaultSettings
// and change the fields you source. Addr has no default, so it is always
// among them.
type Settings struct {
	// Addr is the address the Client dials through the Setup's Host, as
	// host:port. The Host resolves the host when it dials. Its host is also
	// the name the server's certificate is verified for, and the RPCs'
	// :authority. It is required.
	//
	// Default: none. Valid: host:port, with a host and a port from 1 to
	// 65535. Check: CheckAddr.
	Addr string

	// TLS is the config the Client dials with. nil means plain text. A
	// client certificate in Certificates makes it mTLS. With RootCAs nil,
	// the server is verified against the system's roots.
	//
	// Default: nil, plain text. Valid: nil, or a config with a MaxVersion
	// of TLS 1.2 or later, or none. Check: CheckTLS.
	TLS *tls.Config

	// Timeout bounds each unary RPC, its retries included, unless the
	// caller's context has an earlier deadline. It doesn't apply to streams.
	//
	// Default: 10s. Valid: positive. Check: CheckTimeout.
	Timeout time.Duration

	// MaxAttempts bounds the attempts of each unary RPC, and of opening each
	// stream. 1 turns retries off.
	//
	// Default: 3. Valid: 1 or more. Check: CheckMaxAttempts.
	MaxAttempts int

	// BackoffBase and BackoffMax are the schedule between attempts:
	// retry.Exponential(BackoffBase, BackoffMax), exponential backoff with
	// full jitter.
	//
	// Default: 100ms and 5s. Valid: positive, and BackoffMax at least
	// BackoffBase, which NewClient checks. Check: CheckBackoffBase,
	// CheckBackoffMax.
	BackoffBase time.Duration
	BackoffMax  time.Duration

	// RetryBudget limits the retries of every RPC the Client makes, as
	// gRPC's retry throttling does. First attempts are never limited.
	//
	// Default: {MaxTokens: 10, TokenRatio: 0.1}. Valid: as retry.NewBudget
	// accepts. Check: CheckRetryBudgetMaxTokens, CheckRetryBudgetTokenRatio.
	RetryBudget retry.BudgetConfig

	// Breaker configures the Client's circuit breaker.
	//
	// Default: {Window: 20, Probes: 1, OpenFor: 10s, ProbeTimeout: 10s}.
	// Valid: as circuit.New accepts. Check: CheckBreakerWindow,
	// CheckBreakerProbes, CheckBreakerOpenFor, CheckBreakerProbeTimeout.
	Breaker circuit.Config

	// BreakerPolicy decides when the breaker opens and closes. It is set in
	// code, not sourced.
	//
	// Default: open after 5 failures in a row, or when half of the last 20
	// or more results failed; while half-open, reopen on a failed probe and
	// close after 3 successful ones. Valid: both functions set, which
	// NewClient checks. Check: none.
	BreakerPolicy circuit.Policy
}

// DefaultSettings returns the Client's defaults. Addr has none, so its Addr
// is "", which CheckAddr rejects: the end user always sources it.
func DefaultSettings() Settings

// The checks, one for each field with a valid range, as #631 sets out. Each
// reports what is wrong, without naming the field.
func CheckAddr(addr string) error
func CheckTLS(c *tls.Config) error
func CheckTimeout(d time.Duration) error
func CheckMaxAttempts(n int) error
func CheckBackoffBase(d time.Duration) error
func CheckBackoffMax(d time.Duration) error
func CheckRetryBudgetMaxTokens(n int) error
func CheckRetryBudgetTokenRatio(r float64) error
func CheckBreakerWindow(n int) error
func CheckBreakerProbes(n int) error
func CheckBreakerOpenFor(d time.Duration) error
func CheckBreakerProbeTimeout(d time.Duration) error

// Client is a grpc.ClientConnInterface, so generated code takes it:
//
//	cartpb.NewCartServiceClient(client)
//
// It is safe for concurrent use. One Client is one dependency: one
// connection, one circuit breaker and one retry budget.
type Client struct{ /* unexported */ }

// NewClient runs every check on settings, reporting every bad field in one
// error that names each one, rejects a BreakerPolicy with a nil function,
// and builds the Client. It opens no connection: the first RPC dials.
//
// The Client dials settings.Addr through s.Host, with grpc-go's passthrough
// resolver, so no name is looked up but by the Host. Its spans and metrics
// come from otelgrpc's client stats handler, given s.Telemetry's
// TracerProvider, MeterProvider and Propagator, and its own spans and log
// records from s.Telemetry, under the scope
// github.com/z5labs/humus/grpc/grpcclient.
//
// opts come after humus's own dial options, and the transport credentials
// built from settings.TLS come after opts, so a
// grpc.WithTransportCredentials in opts has no effect. An option that
// chains, such as grpc.WithChainUnaryInterceptor, runs inside the Client's
// retries and breaker, once per attempt.
//
// NewClient registers the connection's close with s.OnStop.
func NewClient(s *humus.Setup, settings Settings, opts ...grpc.DialOption) (*Client, error)

// Invoke makes a unary RPC, retrying it as the package doc describes, with
// each attempt through the circuit breaker. It is called by generated code.
func (c *Client) Invoke(ctx context.Context, method string, args, reply any, opts ...grpc.CallOption) error

// NewStream opens a stream through the circuit breaker, retrying the open
// as the package doc describes. The stream is never retried once open. It is
// called by generated code.
func (c *Client) NewStream(ctx context.Context, desc *grpc.StreamDesc, method string, opts ...grpc.CallOption) (grpc.ClientStream, error)

The package is in the nested grpc module, next to grpcserver. Its two new dependencies are bedrock's retry and circuit modules at v0.24.0. Both import only the standard library and OpenTelemetry. otelgrpc and grpc-go are already in the module.

The end user's side

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	d := grpcclient.DefaultSettings()
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }

	settings, err := config.Struct(func(r *config.Resolver) grpcclient.Settings {
		c := d
		// No .Default: Addr is required, so an unset variable fails as not found.
		c.Addr = r.Get(env("CARTS_ADDR").Validate(grpcclient.CheckAddr))
		c.TLS = r.Get(env("CARTS_CA_FILE").Parse(readFile(s.Host.FS)).Parse(clientTLS).Validate(grpcclient.CheckTLS))
		c.Timeout = r.Get(env("CARTS_TIMEOUT").Parse(time.ParseDuration).Default(d.Timeout).Validate(grpcclient.CheckTimeout))
		c.MaxAttempts = r.Get(env("CARTS_MAX_ATTEMPTS").Parse(strconv.Atoi).Default(d.MaxAttempts).Validate(grpcclient.CheckMaxAttempts))
		c.Breaker.OpenFor = r.Get(env("CARTS_BREAKER_OPEN_FOR").Parse(time.ParseDuration).Default(d.Breaker.OpenFor).Validate(grpcclient.CheckBreakerOpenFor))
		return c
	}).Read(ctx)
	if err != nil {
		return nil, err // exit 78, naming every bad variable
	}

	carts, err := grpcclient.NewClient(s, settings)
	if err != nil {
		return nil, err
	}

	api := &api{carts: cartpb.NewCartServiceClient(carts)}
	svc, err := httpserver.NewService(s, serverSettings, api.routes())
	if err != nil {
		return nil, err
	}
	return []humus.Service{svc}, nil
}

readFile and clientTLS are the end user's helpers, as in httpserver's and grpcserver's docs: clientTLS builds a *tls.Config whose RootCAs holds the CA it read. The end user's Init sources the fields it wants an operator to change. The rest keep DefaultSettings().

Dialling and names

NewClient calls grpc.NewClient("passthrough:///"+settings.Addr, ...) with:

• grpc.WithContextDialer, calling s.Host.Dial(ctx, "tcp", addr). With a dialler of its own, grpc-go also uses no HTTP CONNECT proxy, so HTTPS_PROXY is never read from the real environment. grpc.WithNoProxy() is passed as well, as telemetry's connection does;
• no default service config, and grpc.WithDisableRetry(), under Retries below.

The Host resolves the name. In main, humus.OSHost()'s Dial is a net.Dialer's, which looks the host up as net.Dial does, each time it dials. In a test, humustest's Host compares names as written. A name that exists only on the test Host works, inside a bubble or out.

One connection, picked first. The passthrough resolver hands grpc-go one address, so the Client has one subchannel and grpc-go's default pick_first policy. Every RPC goes over one HTTP/2 connection to whichever backend that dial reached. When the connection breaks, grpc-go dials again through the Host, with its own connection backoff, which starts at 1s and grows to 120s. That backoff is grpc-go's, and grpc.WithConnectParams in opts changes it.

Spreading the load. Behind a Kubernetes ClusterIP Service, each dial reaches one pod, chosen by kube-proxy, and stays there. Two things move a client to another pod:

• grpcserver's Shutdown sends every client a GOAWAY, so a rollout moves them;
• a server whose end user sets keepalive.ServerParameters{MaxConnectionAge: ...} in grpcserver.NewService's options sends a GOAWAY when a connection reaches that age. The client then dials again, and reaches whichever pod kube-proxy picks. gRFC A9 added MaxConnectionAge for this kind of rebalancing.

The docs page for this package says so, with an example. Spreading RPCs across every backend at once needs client-side load balancing, which needs a name resolved to several addresses. That is open question 1.

:authority and the certificate. grpc-go takes the authority from the target's endpoint, so RPCs carry Addr, and TLS verifies the server's certificate for Addr's host, as grpcservertest.NewClient does.

TLS

Settings.TLS nil means plain text, as kafka.Settings.TLS and telemetry's OTLPSettings.TLS mean. A client's peer decides whether TLS is needed. Every humus server requires it, but a humus service also calls services humus doesn't run: one behind a mesh's sidecar, a vendor's agent, a local emulator. Open question 2 asks whether to require it instead, as grpcserver does.

CheckTLS accepts nil, and rejects a config whose MaxVersion is below TLS 1.2, which HTTP/2 requires. A config needs no certificate: a client certificate is only for mTLS. credentials.NewTLS clones the config, so the end user's is never changed. grpc-go adds h2 to ALPN and enforces it.

A config with RootCAs nil verifies the server against the system's roots, which crypto/x509 reads from the real filesystem, and from SSL_CERT_FILE and SSL_CERT_DIR in the real environment. That is an ambient leak of every TLS client, kafka's included, and the lifecycle page's "Ambient leaks" lists it. A test never relies on system roots: it sets RootCAs to humustest.CA.Pool().

Retries

Invoke is bedrock's loop, with the call through the breaker in a function of its own, as retry's doc composes them:

func (c *Client) Invoke(ctx context.Context, method string, args, reply any, opts ...grpc.CallOption) error {
	ctx, cancel := context.WithTimeout(ctx, c.timeout)
	defer cancel()

	var err error
	for a := range c.retrier.Attempts(ctx) {
		var trailer metadata.MD
		err = c.attempt(ctx, method, func(ctx context.Context) error {
			return c.cc.Invoke(ctx, method, args, reply, append(opts, grpc.Trailer(&trailer))...)
		})
		a.Done(withPushback(err, trailer)) // the classifier reads grpc-retry-pushback-ms
	}
	return err
}

func (c *Client) attempt(ctx context.Context, method string, call func(context.Context) error) error {
	t, err := c.breaker.Allow(ctx)
	if err != nil {
		return c.rejected(ctx, method, err) // a failed span, and status Unavailable
	}
	defer t.Release()

	err = call(ctx) // failure injection's seam, in wave 5
	t.Done(err)
	return err
}

What is retried. The retry classifier sees each attempt's error:

Attempt's result	Outcome	Delay
OK	Success
rejected by the breaker (circuit.ErrRejected)	Stop
grpc-retry-pushback-ms negative or not a number	Stop: the server asked not to retry
Unavailable	Retry	the pushback, if any
ResourceExhausted with a pushback of 0 or more	Retry	the pushback
anything else, ResourceExhausted without a pushback included	Stop

• Unavailable is gRPC's code for "try again". It is what grpc-go returns when it can't reach the server, and what grpcserver ends streams with at shutdown. It is the code gRPC's own retry examples list.
• ResourceExhausted is retried only when the server says when. Without a pushback it is as likely to be a quota as an overload, and a quota isn't fixed by trying again a few hundred milliseconds later.
• The pushback is gRFC A6's. grpc-retry-pushback-ms is read from the attempt's trailers, and is the least delay passed back to retry, so the wait is the larger of it and the schedule's. A negative or malformed value means "don't retry", as A6 says.
• The caller giving up ends the call. With the call's context done, retry decides Expired for a Retry and takes no token. Timeout is applied to the context before the loop, so it bounds the whole call, its waits included, and a caller's earlier deadline wins.
• Every RPC is retried on these codes, whatever its method. A server that returns Unavailable after acting on a request breaks gRPC's meaning of the code. A call that must never be retried is made through a Client with MaxAttempts 1.

grpc-go's own retries. grpc-go retries an RPC when a service config gives its method a retryPolicy, with its own throttling and pushback. The Client passes grpc.WithDisableRetry(), so no service config, from the end user's grpc.WithDefaultServiceConfig or any other source, turns them on beside humus's. Each RPC is retried by one of them, never both, as the charter requires. humus's are the ones kept, because only they let each attempt go through the breaker, as bedrock composes retry and circuit. grpc-go's retries run inside a single Invoke, where a breaker would count one result for several attempts, and couldn't stop a retry with ErrRejected.

WithDisableRetry leaves grpc-go's transparent retries on. grpc-go retries an RPC by itself when no data reached the wire, or when the server reports it never processed it, such as a stream refused by a server that sent a GOAWAY. Those are part of one attempt, and safe whatever the method, so the Client keeps them.

The circuit breaker

One per Client. A Client is one dependency, one Addr, so it has one circuit.Breaker and one retry.Budget, built in NewClient, shared by every method and every goroutine. An end user who wants separate breakers for two gRPC services on one address builds two Clients.

What counts. The breaker's classifier sees each attempt's error, with the call's context:

Attempt's result	Outcome
OK	Success
the call's context was cancelled, or the code is Canceled	Ignored: the caller gave up
Unavailable, DeadlineExceeded, ResourceExhausted, Internal, Unknown, DataLoss	Failure
any other code, such as NotFound or InvalidArgument	Success: the server is up and answering

DeadlineExceeded counts whether the deadline was the Client's Timeout, the caller's own, or one the caller inherited from an inbound request. A dependency that hangs ends every call this way, so a breaker that ignored deadlines would never open on one. That is the classifier circuit's doc gives. An inbound deadline that is too short to call the dependency in counts against it too.

The policy. DefaultSettings().BreakerPolicy is the policy above, the one in circuit's doc. It is a field set in code rather than settings, because a policy's shape, which counts matter, is a design decision, as queue.Delivery is. An operator tunes the numbers that are settings: the window, the probes and the two periods. An end user who wants a policy of their own sets the field, and one who wants no breaker sets a Trip that never trips.

The default's ratio rule needs 20 results, which is the default Window. With a Window below 20, only its rule of 5 failures in a row trips it. The field's doc says so, and an end user who lowers the window sets a policy to match.

A rejected call. An attempt the breaker rejects fails with an error that matches circuit.ErrRejected with errors.Is, and whose gRPC status is Unavailable, with the rejection's message, such as "circuit: call rejected: open for another 4.2s". So status.Code(err) is Unavailable, as for an unreachable server, and a caller that needs to tell them apart uses errors.Is. The error type is unexported. The rejection is classified Stop, so it isn't retried at once, which would defeat the breaker.

Streams

NewStream runs the same loop with MaxAttempts and the backoff, but not Timeout, since a stream may rightly live as long as its caller wants. Each attempt opens the stream through the breaker:

• Opening is retried. NewStream fails before anything reaches the server: no connection, the breaker rejecting it, the channel in transient failure. So an Unavailable there is retried as for a unary RPC.
• An open stream is never retried. The end user's code sends and receives on it directly, and a retry would replay what it already saw. That is retry's rule that a call is committed once its result is handed to its caller.
• The breaker counts the stream's end. The breaker's ticket is handed to the stream, which settles it once: when RecvMsg returns an error, with io.EOF counted as a success, or, for a client-streaming RPC, when RecvMsg returns its one response. A stream its caller abandons without reading to the end, or cancelling its context, never settles its ticket. That costs nothing while the breaker is closed, and holds a probe slot for at most ProbeTimeout while it is half-open, as circuit's doc describes.

Telemetry

The Client takes everything from s.Telemetry and never reads or sets OTel's globals. The first of humus's dial options is otelgrpc's client stats handler:

grpc.WithStatsHandler(otelgrpc.NewClientHandler(
	otelgrpc.WithTracerProvider(s.Telemetry.TracerProvider),
	otelgrpc.WithMeterProvider(s.Telemetry.MeterProvider),
	otelgrpc.WithPropagators(s.Telemetry.Propagator),
))

• Spans. grpc-go calls a stats handler for each attempt, and each retry is a new Invoke. So each attempt gets one client span, named after the method, such as cart.v1.CartService/AddItem, with the attributes of OpenTelemetry's RPC conventions, among them rpc.system.name, rpc.method and rpc.response.status_code. The attempts of one call are siblings, children of the caller's span. The propagator injects the trace context and the baggage into each attempt's metadata, so the server joins the caller's trace.
• Rejected calls. A call the breaker rejects never reaches grpc-go, so the Client starts its own client span for it, under its own scope, named and attributed as otelgrpc would, with rpc.response.status_code set to UNAVAILABLE, and status Error with the rejection's message.
• Metrics. otelgrpc records rpc.client.call.duration for each attempt. The Client records no metrics of its own for retries or the breaker. A retry shows as an extra attempt, and a breaker's state as its log records.
• Logs. The Client logs through s.Telemetry.LoggerProvider.Logger("github.com/z5labs/humus/grpc/grpcclient"), with the OTel Logs API. Each change of the breaker's state is one record, at WARN when it opens and INFO otherwise. Its body is "circuit breaker open", "circuit breaker half-open" or "circuit breaker closed", and it is emitted in the context of the call that made the change, so it carries that call's trace. Its attributes are server.address and server.port, from Addr, circuit.previous_state, and the counts of the state it left: circuit.successes, circuit.failures, circuit.consecutive_successes, circuit.consecutive_failures and circuit.timed_out. circuit delivers transitions one at a time, in order, and outside its lock, so the observer needs no lock of its own.

grpc-go's own logger, and the GRPC_ variables of its internal/envconfig, are read from the real environment, as grpcserver's doc already records. So is OTEL_SEMCONV_STABILITY_OPT_IN, which otelgrpc reads. The client adds nothing new to that list.

Closing

NewClient registers cc.Close with s.OnStop. OnStop functions run after every Service has stopped, so no Service is still calling, and within the grace period. An RPC made after the close fails with Canceled and "grpc: the client connection is closing". Client has no Close of its own, so the end user's code can't close a connection that other Services share.

NewClient dials nothing, as grpc.NewClient doesn't. A dependency that is down when the service starts doesn't fail Init. Its RPCs fail, are retried, and open the breaker, which the logs show. A service that must not start without its dependency calls it in Init.

Settings

Field	Default	Valid	Check
Addr	none	host:port, a host, a port from 1 to 65535	CheckAddr
TLS	nil, plain text	nil, or MaxVersion TLS 1.2 or later, or none	CheckTLS
Timeout	10s	positive	CheckTimeout
MaxAttempts	3	1 or more	CheckMaxAttempts
BackoffBase	100ms	positive	CheckBackoffBase
BackoffMax	5s	positive, and at least BackoffBase	CheckBackoffMax, and NewClient
RetryBudget.MaxTokens	10	3 to retry.TokenLimit	CheckRetryBudgetMaxTokens
RetryBudget.TokenRatio	0.1	0.001 to retry.TokenLimit	CheckRetryBudgetTokenRatio
Breaker.Window	20	1 to circuit.MaxCount	CheckBreakerWindow
Breaker.Probes	1	1 to circuit.MaxCount	CheckBreakerProbes
Breaker.OpenFor	10s	positive	CheckBreakerOpenFor
Breaker.ProbeTimeout	10s	positive	CheckBreakerProbeTimeout
BreakerPolicy	humus's	both functions set	NewClient

• Addr has no default, like grpcserver.Settings.Addr: humus imposes no port. CheckAddr rejects an empty host too, which grpcserver's accepts: a client dials a host, it doesn't listen on every name. It rejects a scheme such as dns:///, since the Client never resolves names itself.
• Timeout must be positive. A zero would mean no deadline, and an RPC with none can hang forever, which no breaker ever counts. A caller's earlier deadline wins, but a later one doesn't extend Timeout: an end user whose calls need longer sets the field.
• The retry and breaker fields are bedrock's types, retry.BudgetConfig and circuit.Config, so their meaning is bedrock's documentation. Each leaf field with a range has a check of its own, named after its path, so an operator's bad value fails startup with exit 78, naming the variable, like any other field.
• The defaults are the charter's, shared with the HTTP and Kafka clients: 3 attempts, retry.Exponential(100ms, 5s), gRFC A6's example budget, and circuit's example policy.

The seam for failure injection

Proposal: failure injection (wave 5) adds an injection point to every client's calls. Its seam here is the single attempt, inside the breaker: in attempt, after Allow admits it and before c.cc.Invoke, and, for streams, before c.cc.NewStream. An injected abort is then an attempt's error like any other. The breaker counts it, the retry classifier decides on it, and its span records it, so a test of an injected fault sees what a real one would cause. An injected delay sits inside Timeout and the caller's deadline.

The attempt's context is also where wave 5 strips scenarios from calls to third parties. otelgrpc injects the propagator's baggage into the attempt's metadata when the attempt starts, so a context stripped before c.cc.Invoke carries no scenario across. The wave-5 proposal designs both, and whatever configures them, and its stories add them here.

Design decisions

• The package is grpcclient, in the grpc module. The charter puts outbound gRPC clients there, so an end user who doesn't use gRPC keeps grpc-go out of their module graph. It mirrors grpcserver: the client half, not the protocol. A package named grpc would clash with grpc-go at every call site.

• A Client of its own, not a *grpc.ClientConn. The retries and the breaker sit in Invoke and NewStream, outside everything in opts, whatever interceptor options the end user passes. A *grpc.ClientConn with humus's interceptors would let a grpc.WithUnaryInterceptor in opts run outside them, because grpc-go puts the single interceptor ahead of the chained ones. And a *grpc.ClientConn has a Close that one Service could call under the others. Open question 3 asks whether to return the *grpc.ClientConn anyway.

• Passthrough, not a resolver. It is the one way grpc-go reaches only the Host. It is what grpcservertest and telemetry already do, so production and tests dial the same way. Open question 1 covers what it leaves out.

• TLS nil means plain text. See TLS above, and open question 2.

• humus's retries, not grpc-go's. Only they put each attempt through the breaker. grpc-go's transparent retries stay, because they never repeat what a server saw.

• Unavailable is retried for every method, and nothing else is without a pushback. It keeps a retried call to what gRPC defines as safe to try again, without a per-method table to maintain.

• Streams are retried only while they open. See Streams above.

• The policy is code, the numbers are settings. See The circuit breaker above.

• No metrics of humus's own. otelgrpc's per-attempt duration already counts every attempt and its status, and a breaker's transitions are rare enough to be log records. A gauge of the breaker's state is out of scope.

• NewClient doesn't dial. A dependency's outage is the breaker's to handle, not a crash loop's. kafka.NewService opens no connection either.

• No test package. A test runs the end user's whole App, its client included, against a server on the same humustest Host, as Testing shows. grpcservertest.NewClient stays the way a test calls a Service directly.

• Changes elsewhere, made by this proposal's stories:

  • the docs site gets a page, "Calling gRPC services", next to the gRPC services page. It covers settings, TLS, retries, the breaker, streams, telemetry, spreading load with MaxConnectionAge, and testing;
  • the gRPC services page gets a short section on keepalive.ServerParameters{MaxConnectionAge}, pointing to the client page;
  • the lifecycle page's "Ambient leaks" gets the system roots and SSL_CERT_FILE and SSL_CERT_DIR;
  • the testing page gets a client next to a server on one Host;
  • grpcservertest's doc gets a sentence pointing production code to grpcclient.

  .claude/backlog.json already verifies the grpc module.

Testing

A prototype of the Client, under 300 lines, ran these tests under -race -count=5 in a scratch module. It used grpc-go v1.84.0, otelgrpc v0.72.0, bedrock's retry and circuit v0.24.0 and the humus checkout at 4289e89a, and every test passed. The prototype is not kept. Every test runs inside a testing/synctest bubble, on humustest's Host, with a CA generated for the test, and asserts with testify/require. Fake time makes the backoff, the pushback and OpenFor take no real time.

The end user's whole App: an Init that serves a backend with grpcserver, builds a client of it, and runs a job that calls it.

func run(t *testing.T, b *backend, work func(ctx context.Context, c testpb.TestServiceClient) error) (*humustest.Telemetry, error) {
	ca := humustest.NewCA(t)
	cert := ca.Issue(t, "backend")
	h := humustest.NewHost(t)
	tel := humustest.NewTelemetry()

	var workErr error
	init := func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
		srv, err := grpcserver.NewService(s, grpcserver.Settings{
			Addr: "backend:9090",
			TLS:  &tls.Config{Certificates: []tls.Certificate{cert.TLS()}},
		})
		if err != nil {
			return nil, err
		}
		testpb.RegisterTestServiceServer(srv, b)

		settings := grpcclient.DefaultSettings()
		settings.Addr = "backend:9090"
		settings.TLS = &tls.Config{RootCAs: ca.Pool()}
		client, err := grpcclient.NewClient(s, settings)
		if err != nil {
			return nil, err
		}

		j := job.NewService(s, func(ctx context.Context) error {
			workErr = work(ctx, testpb.NewTestServiceClient(client))
			return nil
		})
		return []humus.Service{srv, j}, nil
	}

	_, status := humustest.Start(t, init, humustest.WithApp(humus.WithHost(h.Host()), humus.WithTelemetry(tel.Builder())))
	st := <-status
	require.Zero(t, st.Code(), "%v", st.Cause())
	return tel, workErr
}

func TestClient_retries_Unavailable(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		b := &backend{answer: func(_ context.Context, n int32) error {
			if n < 3 {
				return status.Error(codes.Unavailable, "not yet")
			}
			return nil
		}}
		tel, err := run(t, b, func(ctx context.Context, c testpb.TestServiceClient) error {
			_, err := c.EmptyCall(ctx, &testpb.Empty{})
			return err
		})
		require.NoError(t, err)
		require.EqualValues(t, 3, b.calls.Load())
		require.Len(t, clientSpans(tel), 3) // one span per attempt
	})
}

backend is a grpc_testing.TestService whose EmptyCall answers with answer, and counts its calls. When the job returns, the App shuts down, and the Client's OnStop function closes its connection inside the bubble. A connection left open would be reported as a leak.

The prototype's tests also showed the following:

• The pushback is the least delay. A first answer of ResourceExhausted with grpc-retry-pushback-ms: 2000 is retried, and the call takes at least 2s of fake time.
• A negative pushback stops the retries. Unavailable with grpc-retry-pushback-ms: -1 is tried once.
• The breaker opens, rejects and closes. With MaxAttempts 1 and a backend that answers Unavailable, five calls fail and the sixth is rejected without reaching the backend. Its error matches circuit.ErrRejected, and its status code is Unavailable. After 10s of fake time and three successful calls, the scope's log records are WARN "circuit breaker open", INFO "circuit breaker half-open" and INFO "circuit breaker closed". The trace has nine client spans: five failures, the rejection's own span and three successes.
• A stream settles at its end. A server-streaming RPC reads its three messages and io.EOF.
• An address nothing listens on fails with Unavailable after its retries, and the App still exits 0.

The stories add:

• The settings convention (#631). Each check accepts an in-range value and rejects an out-of-range one without naming the field. NewClient names each bad field and reports them together. DefaultSettings() with an Addr passes. A nil policy function, and a BackoffMax below BackoffBase, are rejected. The package never reads a bedrock.Environment.
• Timeout. A backend blocked on its context fails the call with DeadlineExceeded after 10s, which the breaker counts. A caller's earlier deadline wins.
• Plain text and mTLS. A nil TLS reaches a plaintext server. A config with a client certificate reaches a grpcserver whose TLS sets ClientAuth.
• opts. A grpc.WithTransportCredentials(insecure.NewCredentials()) in opts has no effect. A chained interceptor in opts runs once per attempt. A grpc.WithDefaultServiceConfig with a retryPolicy doesn't add retries.
• Opening a stream is retried, an open stream isn't. A stream whose server fails after sending a message ends with that error, and the backend sees one call. Its status is counted by the breaker.
• No globals. With OTel's global providers set to recorders of their own, a call records nothing in them.

The package's example is an Example function in the grpc module, and the package is held to the 80% coverage gate.

Migration from today

Before the reset	With this proposal
no outbound gRPC client; the end user called grpc.NewClient themselves	grpcclient.NewClient(s, settings, opts...)
grpc.NewClient(addr, ...), with the dns resolver and the real network	passthrough:/// and Settings.Addr, dialled through s.Host
grpc.WithTransportCredentials(insecure.NewCredentials())	Settings.TLS nil
a retryPolicy in a service config	MaxAttempts, BackoffBase, BackoffMax and RetryBudget; grpc-go's configurable retries are off
otelgrpc on OTel's globals, if at all	otelgrpc's client stats handler with s.Telemetry
defer conn.Close() in main	s.OnStop, registered by NewClient
grpcservertest.NewClient	unchanged, for tests that call a Service directly

Open questions

1. Spreading RPCs across backends. The Client dials one address and keeps one connection, as proposed. That spreads load only per connection, so with few client pods and many server pods, most servers sit idle until MaxConnectionAge or a rollout moves the clients. The options are:

   • (a) keep passthrough and pick_first, and document MaxConnectionAge on the server, as proposed;
   • (b) add a LookupHost func(ctx context.Context, host string) ([]string, error) to humus.Host. OSHost would use net.DefaultResolver.LookupHost, and humustest's Host would answer from a table a test gives it, with a name not in the table resolving to itself. grpcclient would get a resolver of its own on it, with a Settings field for round_robin, so a headless Service's pods each get a connection. This changes the root package's Host, which every Host literal must then fill. This proposal's stories would make the change in humus, humustest and the lifecycle docs. It also adds a resolver to maintain, with its re-resolution timing;
   • (c) a net.Resolver with PreferGo that dials its DNS server through the Host. This isn't workable: it still reads /etc/resolv.conf from the real filesystem, queries over UDP, which the Host doesn't carry, and has no DNS server to reach on a test Host.

   Recommendation: (a). It needs no change to humus.Host, and production and tests dial the same way. MaxConnectionAge is needed with (b) as well, because grpc-go re-resolves only when a connection is lost. With (a), client-side load balancing is out of scope.

2. TLS. nil means plain text, as proposed, matching kafka.Settings.TLS and telemetry's OTLPSettings.TLS? Or required, as grpcserver requires it, so that an unset variable fails startup rather than dialling in plain text? Requiring it leaves no way to call a plaintext service with this package. Recommendation: nil means plain text. Every humus server requires TLS, so a humus-to-humus call with no TLS fails its first RPC, and the breaker's WARN record shows it.

3. The type NewClient returns. A *grpcclient.Client of its own, as proposed, or the *grpc.ClientConn, with the retries and the breaker in interceptors humus chains first? The *grpc.ClientConn gives the end user GetState and Connect, for a readiness monitor on the connection. It also gives them Close, and lets a grpc.WithUnaryInterceptor in opts run outside humus's. Recommendation: the Client. A readiness monitor calls the dependency, such as its grpc.health.v1 service, through it.

Out of scope

• Client-side load balancing, other resolvers, and xDS, if open question 1 keeps (a).
• Failure injection: Proposal: failure injection (wave 5), at the seam named above.
• Authentication and authorization: Proposal: authentication and authorization (wave 5). Until then, the end user passes per-RPC credentials, such as grpc.WithPerRPCCredentials, in opts.
• The health of a dependency. No client serves or feeds health, and an open breaker doesn't make the pod unready, as the charter's revision 7 sets out. A readiness monitor on a dependency is the end user's, as health monitors and probes left it.
• Retrying a stream once it is open, hedging, and a per-method retry table.
• Metrics of humus's own for retries and the breaker, such as a gauge of the breaker's state.
• grpc-go's HTTP CONNECT proxy support, ALTS, and Unix domain sockets. The Host dials TCP.
• Walkthroughs of a service that calls gRPC: Proposal: walkthroughs and a humus MIGRATING.md (wave 6).

References

• Proposal: humus on the bedrock process model, and its revision 7: the client proposals, their three points, and this proposal's question of names
• Proposal: gRPC services, and its revision: the grpc module, TLS required, grpcservertest.NewClient for tests only
• The lifecycle's revision, the configuration revision, telemetry's revision 2: Host, OnStop, Settings and checks, telemetry.SDK and outbound instrumentation left out of scope
• bedrock's retry and circuit v0.24.0, and their discussions, A retry package owned by bedrock bedrock#606 and A circuit breaker should be provided by bedrock bedrock#595; fit and Feature request: request-scoped failure injection (Netflix FIT) on OpenTelemetry baggage bedrock#578 for wave 5
• grpcservertest and its notes on the dns resolver; telemetry's OTLP connection
• grpc-go v1.84.0: the dns resolver, WithDisableRetry, name resolution, and keepalive.ServerParameters
• gRFC A6: client retries: retry throttling and grpc-retry-pushback-ms; gRFC A9: server-side connection management: MaxConnectionAge
• gRPC status codes
• otelgrpc v0.72.0, and OpenTelemetry's semantic conventions for RPC
• Before the reset: rest/rest.go#L358-L364, the OTLP connection on the dns resolver

[Zaba505]

Revision 1: the open questions answered

The open questions:

1. One address, one connection, picked first, as recommended: (a). The question was whether the limit is only for an IP address, or for a DNS name too. It is the same for both, as Dialling and names now sets out: passthrough hands grpc-go one address, whatever its form, and the Host resolves a name inside the dial. It is also what an end user who calls grpc.NewClient with nothing but an address already has. grpc-go's dns resolver and its default pick_first keep one connection too. Client-side load balancing is out of scope.
2. Plain text is allowed, as recommended, for outbound HTTP clients' reason: humus makes a service as secure as it can, but it can't control what the service connects to.
3. A Client of its own, as recommended. A type of humus's own can carry methods that fit it into the rest of humus, which a *grpc.ClientConn can't. The first is Connected, a readiness monitor on the Client's connection. It replaces the GetState and Connect a *grpc.ClientConn would have given, and leaves out its Close.

Outbound HTTP clients' revision 1 also reaches this proposal. Its "The other clients" gives grpcclient.Settings a Classify field, of the same name as httpclient's, shaped for what a gRPC attempt ends with, its pushback included. It also leaves instruments of humus's own to Proposal: humus's own metrics, which the charter's revision 9 puts in wave 5. Both are below.

The charter needs no revision. Connected is a monitor the end user adds, as revision 7 leaves a dependency's health, and an open breaker never makes it unhealthy.

Each section below replaces or adds to the part of the same name in the post. Background, The end user's side, Closing, The seam for failure injection, Migration from today and References are unchanged.

Summary

The third bullet, and the bullet on Settings, become, and a bullet is added after them:

• Every unary RPC is retried with bedrock's retry, and every attempt goes through bedrock's circuit. What a dependency's codes mean is a function, Settings.Classify. Its default retries Unavailable, and ResourceExhausted when the server sends grpc-retry-pushback-ms, and the pushback is the least delay. The end user sets their own for an API that means something else. grpc-go's own configurable retries are turned off, and its transparent retries, of RPCs the server never received, are kept.
• Settings holds the address, the TLS config (nil means plain text), a timeout for each unary call, the retry and breaker settings, typed with bedrock's retry.BudgetConfig and circuit.Config, and Classify.
• Connected returns a health.Monitor for readiness. It is healthy once the Client's connection has first been ready, so a release that can't reach its dependency never rolls out.

Proposed API

Settings gains a field, the package a type, and Client a method:

type Settings struct {
	// Addr, TLS, Timeout, MaxAttempts, BackoffBase, BackoffMax,
	// RetryBudget, Breaker and BreakerPolicy, unchanged.

	// Classify decides how each attempt's result counts: whether another
	// attempt follows, and whether the result counts against the
	// dependency. It is set in code, not sourced, and one function may be
	// set on every Client whose dependencies agree on what their codes
	// mean.
	//
	// Classify gets the RPC's full method name, such as
	// "/cart.v1.CartService/AddItem", the error the attempt ended with, nil
	// for OK, and the attempt's trailer, which holds grpc-retry-pushback-ms.
	// status.FromError gives the error's code and its details, such as a
	// google.rpc.RetryInfo. It isn't asked about what the Client decides
	// itself: a rejection by the breaker, and an error once the call's
	// context has ended.
	//
	// For a stream, it is asked about an open that failed, with no trailer,
	// and about the end of each stream that opened, with the stream's
	// trailer. Only the Breaker of a stream's end counts: an open stream is
	// never retried.
	//
	// It is called on the goroutine that made the call, or, for a stream's
	// end, the goroutine whose RecvMsg ended it, once for each attempt, so
	// it must be safe for concurrent use. A Classification with a zero
	// Outcome, or with retry.Retry and a negative RetryAfter, panics, as
	// bedrock's classifiers do.
	//
	// Default: the classification under "How an attempt's result counts".
	// Valid: set, which NewClient checks.
	Classify func(method string, err error, trailer metadata.MD) Classification
}

// Classification is how one attempt's result counts, as Settings.Classify
// decides it.
type Classification struct {
	// Retry is whether another attempt follows. retry.Success and
	// retry.Stop end the call. retry.Retry asks for another attempt, which
	// MaxAttempts, the budget and the call's deadline must all allow.
	Retry retry.Outcome

	// RetryAfter is, with retry.Retry, the least time to wait before the
	// next attempt, such as the server's pushback, or 0. A RetryAfter
	// longer than BackoffMax isn't waited for: the call ends, and returns
	// the attempt's error.
	RetryAfter time.Duration

	// Breaker is how the result counts in the dependency's circuit
	// breaker: circuit.Success, circuit.Failure, or circuit.Ignored for a
	// result that says nothing about the dependency.
	Breaker circuit.Outcome
}

// Connected returns a monitor that is healthy once the Client has
// connected to Addr: unhealthy until its connection is first ready, its
// TLS handshake done, and healthy from then until the connection is
// closed. A later outage doesn't make it unhealthy. It never returns an
// error. A service is made ready once its dependency is reachable with
//
//	health.Readiness(client.Connected())
//
// Until the connection is first ready, each check starts it, if no RPC
// has, and waits until it is ready or the check's context is done. It
// makes no RPC. It is for readiness only: restarting the pod doesn't bring
// a dependency back.
func (c *Client) Connected() health.Monitor

NewClient's doc says it rejects a nil Classify, next to a BreakerPolicy with a nil function. DefaultSettings' doc is unchanged: Classify is among its defaults. grpcclient imports health, which imports nothing but the standard library, as grpchealth already does.

Dialling and names

These are added after "The Host resolves the name":

A name spreads load as an address does. Passthrough hands grpc-go Addr as one address, whatever its form (passthrough.go#L55), so grpc-go opens one connection to it. The Host resolves a name inside the dial. OSHost().Dial is a net.Dialer's, which looks the name up, tries its addresses in the order the lookup returns them, and keeps the first connection that succeeds (net/dial.go's dialSerial). So:

• A ClusterIP Service's name resolves to the Service's one virtual IP, and kube-proxy picks a pod for each connection, as for the IP itself.
• A headless Service's name resolves to every ready pod, and the Client connects to the first that accepts. Clients in different pods may reach different backends, but each Client keeps one.
• When the connection ends, grpc-go dials the same name again, and the Host looks it up again, so a Client finds pods that weren't there when it first dialled. A pod's IP, given as Addr, never moves.

It is what grpc.NewClient with an address already does. An end user who passes nothing but an address gets grpc-go's dns resolver and its default pick_first, which also keeps one connection, even to a name with several addresses. The Client changes who looks the name up, not how many connections there are.

Client-side load balancing wouldn't help a ClusterIP Service. Its name has one address, whichever resolver looks it up. Spreading RPCs across a headless Service's pods would need grpc-go told every address, which only a lookup on the Host could give it, and that is out of scope.

The last paragraph of "Spreading the load" becomes:

The docs page for this package says so, with an example of MaxConnectionAge, for a ClusterIP Service and a headless one.

TLS

The first paragraph's last sentence, "Open question 2 asks whether to require it instead, as grpcserver does", is replaced by:

humus can't control what a service connects to, so plain text is allowed, as httpclient allows it. Addr is host:port and names no scheme, so a nil TLS is the one way to ask for plain text. A call in plain text to a server that requires TLS fails its first RPC, and Connected stays unhealthy, so a release that left out its TLS config doesn't roll out.

Retries

The code becomes:

func (c *Client) Invoke(ctx context.Context, method string, args, reply any, opts ...grpc.CallOption) error {
	ctx, cancel := context.WithTimeout(ctx, c.timeout)
	defer cancel()

	var err error
	for a := range c.retrier.Attempts(ctx) {
		var trailer metadata.MD
		r := c.attempt(ctx, method, &trailer, func(ctx context.Context) error {
			return c.cc.Invoke(ctx, method, args, reply, append(opts, grpc.Trailer(&trailer))...)
		})
		a.Done(r) // retry's classifier reads r's Classification
		err = r.err
	}
	return err
}

// attempt makes one attempt through the breaker, and classifies its
// result once, for both retry and circuit.
func (c *Client) attempt(ctx context.Context, method string, trailer *metadata.MD, call func(context.Context) error) *result {
	t, err := c.breaker.Allow(ctx)
	if err != nil {
		return c.rejected(ctx, method, err) // a failed span, status Unavailable, and Stop
	}
	defer t.Release()

	err = call(ctx) // failure injection's seam, in wave 5
	r := &result{err: err, class: c.classify(ctx, method, err, *trailer)} // the Client's own rules, then Settings.Classify
	t.Done(r) // circuit's classifier reads r's Classification
	return r
}

result is unexported. It carries an attempt's error and its Classification to both of bedrock's classifiers, as retry's doc says a result that is not an error reaches its classifier: "as an error value that the caller builds". So Classify runs once per attempt, and retry and circuit agree on it.

"What is retried" and its table are replaced by How an attempt's result counts, below. Of its bullets:

• "The pushback is gRFC A6's" moves to that section, under The pushback.
• "Every RPC is retried on these codes, whatever its method" becomes: The default retries every method on these codes. A server that returns Unavailable after acting on a request breaks gRPC's meaning of the code. grpc-go also returns Unavailable when the connection breaks after the request was sent, and then the server may have acted on it. Classify gets the method, so an end user's Classify returns retry.Stop for a method that must never be sent twice, where the default would retry.

The other bullets, and grpc-go's own retries, are unchanged.

How an attempt's result counts

This section is new. It replaces "What is retried" under Retries, and "What counts" under The circuit breaker.

Every attempt's result is classified twice: for retry, whether another attempt follows, and for circuit, whether the result counts against the dependency. Some of that is about the call, and the Client decides it the same way for every dependency. The rest is about what the dependency's codes mean, which differs from one API to another, and Settings.Classify decides it.

What the Client decides. Classify is never asked about these:

An attempt's result	Retry	Breaker
rejected by the breaker	Stop	not admitted
an error, once the call's context was cancelled: the caller gave up	Stop	Ignored
an error, once the call's deadline passed: Timeout, or a deadline the caller set or inherited	Stop	Failure
a stream that opened	Success	settled at the stream's end

• A rejection is never retried at once, which would defeat the breaker, as the charter requires of every client.
• Every deadline counts against the dependency, as in bedrock's own example classifier. A dependency that hangs ends every call this way, and a breaker that ignored deadlines would never open. The deadline may be one the inbound request brought.

And whatever Classify returns:

• a RetryAfter longer than BackoffMax ends the call, with the attempt's error (The pushback);
• the end of an open stream is never retried: only its Breaker counts (Streams).

What DefaultSettings().Classify decides. Every other result, by its code, which status.Code gives. An error with no gRPC status, such as one an interceptor in opts returns, is Unknown:

An attempt's code	Retry	Breaker
OK	Success	Success
Unavailable	Retry, after the pushback if there is one	Failure
ResourceExhausted with a pushback of 0 or more	Retry, after the pushback	Failure
ResourceExhausted without a pushback	Stop	Failure
DeadlineExceeded, Internal, Unknown, DataLoss	Stop	Failure
Canceled	Stop	Ignored
any other code, such as NotFound, InvalidArgument or Aborted	Stop	Success

A grpc-retry-pushback-ms that is negative or not a number makes any of these Stop: the server asked not to be retried.

• Unavailable is gRPC's code for "try again". It is what grpc-go returns when it can't reach the server, and what grpcserver ends streams with at shutdown.
• ResourceExhausted is retried only when the server says when. Without a pushback it is as likely to be a quota as an overload, and a quota isn't fixed by trying again a few hundred milliseconds later. Either way it counts against the dependency, which is shedding load.
• DeadlineExceeded while the call's own deadline is live is the server's: it gave up on something of its own, such as its dependency. It counts, as the call's own deadline does, and isn't retried, because the server's deadline is likely to end the next attempt the same way.
• Canceled while the call's context is live comes from the server cancelling the RPC, or the connection closing at OnStop. It says nothing about the dependency.
• Any other code is the dependency working. It answered, and the request was wrong or conflicted with its state, so the breaker counts the dependency healthy, and the default doesn't try again.
• The default reads only the code and the trailer. It never reads a status's details.

Changing it. The end user's Classify usually handles what their API means differently, and hands the rest to the default:

d := grpcclient.DefaultSettings()
classify := d.Classify

settings.Classify = func(method string, err error, trailer metadata.MD) grpcclient.Classification {
	if status.Code(err) == codes.Aborted {
		// The ledger aborts a write that races another write to the same
		// account. That write is over by the next attempt.
		return grpcclient.Classification{Retry: retry.Retry, Breaker: circuit.Success}
	}
	return classify(method, err, trailer)
}

An API that sends google.rpc.RetryInfo in its statuses' details, rather than a pushback, has a Classify that reads it and returns it as RetryAfter. The function is the end user's own, so the same one can be set on every Client whose dependencies share a convention. humus exports no classifier but the default in DefaultSettings, and no helpers to build one.

The pushback. grpc-retry-pushback-ms is gRFC A6's. The default reads it from the attempt's trailer and returns it as RetryAfter, so the wait is the larger of it and the schedule's. A negative or malformed value means "don't retry", as A6 says. A RetryAfter longer than BackoffMax, from any Classify, isn't waited for: the call ends, and returns the attempt's error, as httpclient does with a long Retry-After. A server that asks for longer than that is not one a call in flight should wait on, and a wait past the call's deadline would end it as Expired anyway. The prototype's 2s pushback is within the default BackoffMax of 5s.

The circuit breaker

"What counts" and its table are replaced by How an attempt's result counts, above. Its paragraph on DeadlineExceeded moves there, as the bullet "Every deadline counts against the dependency". The rest of the section is unchanged.

Streams

The bullets "Opening is retried" and "The breaker counts the stream's end" become:

• Opening is retried. NewStream fails before anything reaches the server: no connection, the breaker rejecting it, the channel in transient failure. A failed open is classified as a unary attempt is, and Classify gets it with no trailer, since nothing came back. A stream that opens is a Success for retry, without asking Classify.
• The breaker counts the stream's end. The breaker's ticket is handed to the stream, which settles it once, with the Breaker of Classify's answer: when RecvMsg returns an error, which Classify gets with Trailer(), and io.EOF given as nil, or, for a client-streaming RPC, when RecvMsg returns its one response, given as nil. A stream whose context its caller cancelled is Ignored, without asking Classify. A stream its caller abandons without reading to the end, or cancelling its context, never settles its ticket. That costs nothing while the breaker is closed, and holds a probe slot for at most ProbeTimeout while it is half-open, as circuit's doc describes.

Telemetry

The bullet on Metrics becomes:

• Metrics. otelgrpc records rpc.client.call.duration for each attempt. The Client records no metrics of its own for retries or the breaker in wave 4. A retry shows as an extra attempt, and a breaker's state as its log records. Whether humus records instruments of its own, such as a gauge of each breaker's state and of each budget's tokens, is decided for every client at once by Proposal: humus's own metrics (wave 5).

Connected makes no RPC, so it starts no span, and it logs nothing. A connection that fails shows in the RPCs that wait on it, and in grpc-go's own logger.

Readiness: Connected

This section is new, after Closing.

What it gates. A rollout. NewClient dials nothing, so without a monitor a release with a wrong Addr, a CA that doesn't trust the server, a missing TLS config, or a dependency it can't reach turns ready, replaces the pods that worked, and fails every RPC. With Connected in its readiness, it never turns ready, and the rollout stops at its progress deadline with the old pods still serving.

It latches. After the first connection, an outage of the dependency hits every pod at once. Turning them all unready would take the service out of its load balancer, so its callers would get no answer instead of this service's own error, and the parts of it that don't need the dependency would stop too. It would also block a rollout that might be the fix. The breaker is what handles an outage: it fails calls fast, and its log records show it. This is (*kafka.Service).Joined's rule, for the same reason, and the charter's rule that an open breaker doesn't make the pod unready. It also keeps the monitor steady through grpc-go's own reconnects: after a GOAWAY, from MaxConnectionAge or a rollout of the dependency, and after 30 minutes with no RPCs, when grpc-go closes an idle connection.

The first check dials. A check while the connection is idle calls the *grpc.ClientConn's Connect, so a service with Connected in its readiness dials at its first probe rather than its first RPC. The check then waits for the connection's state to change, until it is ready, the connection is closed, or the check's context is done. So a reachable dependency passes the first probe, and an unreachable one fails it when the probe's own timeout ends the check. grpchealth's Watch poller already counts a check that outlasts its 1s interval as NOT_SERVING, and waits on that check rather than starting another. grpc-go marks Connect experimental, so the Client's tests pin what it does, and a grpc-go upgrade that changes it fails them.

One latch per Client. Every monitor Connected returns reads the same latch, kept on the Client, so an HTTP probe and a gRPC one agree.

Closed. After s.OnStop closes the connection, the monitor is unhealthy. By then every Service has stopped, and the health Service has already turned readiness false.

It makes no RPC. The dependency sees no call, and the trace no span. A monitor that calls the dependency's grpc.health.v1 service is the end user's, a few lines over the Client:

health.MonitorFunc(func(ctx context.Context) (bool, error) {
	resp, err := healthpb.NewHealthClient(carts).Check(ctx, &healthpb.HealthCheckRequest{Service: "cart.v1.CartService"})
	return err == nil && resp.GetStatus() == healthpb.HealthCheckResponse_SERVING, nil
})

Such a monitor goes through the Client's retries and breaker, so it turns unhealthy whenever the breaker is open, which Connected never does.

Panics

This section is new.

The Client recovers nothing. Nothing of the end user's runs inside a call but Classify, BreakerPolicy, and the interceptors and other options in opts. A panic in any of them passes to the goroutine that made the call, or, for a stream's end, to the one whose RecvMsg ended it. circuit frees the attempt's slot and leaves the breaker consistent, and retry holds nothing between attempts. A panic in the Client itself is humus's bug.

Settings

This row is added to the table:

Field	Default	Valid	Check
Classify	the table under How an attempt's result counts	set	NewClient; set in code, not sourced

These bullets are added:

• TLS is nil for plain text, as kafka.Settings.TLS, telemetry.OTLPSettings.TLS and httpclient.Settings.TLS are. humus can't control what a service connects to: a mesh's sidecar, a vendor's agent, a local emulator. Every humus server requires TLS, so a call to one always sets it.
• Classify is set in code, like BreakerPolicy. What a dependency's codes mean is the end user's knowledge of their API, not an operator's choice. NewClient rejects a nil Classify, which would otherwise panic on the first call.

Design decisions

These bullets become:

• A Client of its own, not a *grpc.ClientConn. The retries and the breaker sit in Invoke and NewStream, outside everything in opts, whatever interceptor options the end user passes. A *grpc.ClientConn with humus's interceptors would let a grpc.WithUnaryInterceptor in opts run outside them, because grpc-go puts the single interceptor ahead of the chained ones. A *grpc.ClientConn also has a Close that one Service could call under the others. And a type of humus's own carries methods that fit it into the rest of humus, of which Connected is the first.
• Passthrough, not a resolver. It is the one way grpc-go reaches only the Host. It is what grpcservertest and telemetry already do, so production and tests dial the same way. A name spreads load as an address does, and as grpc.NewClient with an address does. Rejected: a LookupHost on humus.Host, with a resolver of the Client's own and round_robin. It changes every Host, adds a resolver to maintain, and helps only a headless Service, since a ClusterIP Service's name has one address. MaxConnectionAge would be needed with it as well, because grpc-go re-resolves only when a connection is lost.
• Plain text is allowed. humus can't control what a service connects to. Connected catches a missing TLS config to a server that requires one before the release rolls out.
• The classification is a function, Classify, from the start. A gRPC API picks its own codes, such as Aborted for a conflict a retry resolves, or google.rpc.RetryInfo where another sends a pushback, so nearly every Client ends up classifying its own dependency. The default keeps a retried call to what gRPC defines as safe to try again. The Client keeps what is about the call, not the API: the breaker's rejections, the caller giving up, deadlines and an open stream, so a Classify can't defeat the breaker or replay a stream. Classify gets the method, so a method that must never be sent twice needs no per-method table in Settings. Rejected: a fixed table, with a field added when users ask. Every Client of an API that differs would wait for it.
• No instruments of humus's own in wave 4. otelgrpc's per-attempt duration already counts every attempt and its status, and a breaker's transitions are rare enough to be log records. Whether humus adds instruments of its own is decided for every package at once, by Proposal: humus's own metrics (wave 5).

The bullet "Unavailable is retried for every method, and nothing else is without a pushback" is replaced by the bullet on Classify above.

These are added:

• Connected latches. It gates a rollout on a working configuration, and never takes every pod out of service during an outage that a restart or a rollback can't fix. Rejected: a monitor that turns unhealthy whenever the connection isn't ready. It makes every pod unready during a dependency's outage, flaps on grpc-go's reconnects, and blocks a rollout that might be the fix. Rejected too: a monitor that calls the dependency's grpc.health.v1, which the end user writes in a few lines over the Client, and which an open breaker would turn unhealthy.
• Connected is API because the end user can't build it. The Client hides its *grpc.ClientConn, so its connection's state is the Client's to report, as a consumer's join is kafka.Service's. It is the only client with a monitor: httpclient.NewClient returns an *http.Client, which pools many connections and has no one state to report.

The "Changes elsewhere" bullet's list becomes:

• the docs site gets a page, "Calling gRPC services", next to the gRPC services page. It covers settings, TLS and plain text, names and addresses behind a ClusterIP or a headless Service, retries, the classification table and what the Client decides itself, writing a Classify, the pushback, the breaker, streams, telemetry, Connected and a monitor of the dependency's grpc.health.v1, spreading load with MaxConnectionAge, and testing;
• the gRPC services page gets a short section on keepalive.ServerParameters{MaxConnectionAge}, pointing to the client page;
• the health page's readiness section gets Connected as a dependency's readiness monitor, and why it latches;
• the lifecycle page's "Ambient leaks" gets the system roots and SSL_CERT_FILE and SSL_CERT_DIR;
• the testing page gets a client next to a server on one Host;
• grpcservertest's doc gets a sentence pointing production code to grpcclient.

Testing

Connected was prototyped in a scratch package of the grpc module, at humus 4b786e46 with grpc-go v1.84.0. Its tests passed under -race -count=3, each inside a testing/synctest bubble on a humustest Host, with a CA generated for the test, against a grpcserver.Service:

• Nothing listening on Addr. The connection is idle until the first check, which starts it, and returns false and no error when its 2s context ends, after 2s of fake time. The Client's stats handler sees no RPC.
• A server listening. The first check returns true at once, with no RPC made.
• It latches. Once the server shuts down, its GOAWAY ends the connection, and the check still returns true.
• An untrusted server. With RootCAs that don't trust the server's certificate, the connection stays in transient failure, and the check returns false.
• Plain text to a TLS server. With a nil TLS, against the same server, the connection stays in transient failure, and the check returns false.
• Closed. After the connection is closed, the check returns false.

Classify was not prototyped. These bullets are added to what the stories add:

• An end user's Classify. One that retries Aborted and hands the rest to the default makes a call answered Aborted and then OK succeed, after two attempts. One that returns retry.Stop for one method makes that method's Unavailable fail after one attempt, while another method's is retried.
• What the Client decides stays the Client's. A Classify that returns retry.Retry and circuit.Failure for everything is never asked about a rejection, so the open breaker's calls aren't retried. Calls whose caller cancels them leave the breaker closed. A server-streaming RPC that fails after a message is tried once, even when Classify returns retry.Retry for its status.
• A RetryAfter beyond BackoffMax, returned by an end user's Classify, ends the call at once, after one attempt.
• A stream's end. Classify gets a stream's final status with its trailer, and its Breaker is what the breaker counts.
• NewClient rejects a nil Classify, naming the field, next to its other errors.
• A panic in Classify reaches the caller, and leaves the breaker consistent.
• Connected in an App. An Init that returns a health Service with health.Readiness(client.Connected()) is unready while nothing listens on Addr, and ready once the backend serves.

Open questions

All three are answered above. None are left.

Out of scope

These bullets become:

• Client-side load balancing, other resolvers, and xDS, including a lookup on humus.Host.
• The health of a dependency, beyond Connected. No client serves or feeds health on its own, and an open breaker doesn't make the pod unready, as the charter's revision 7 sets out. A monitor that calls the dependency, such as its grpc.health.v1 service, is the end user's.
• Retrying a stream once it is open, and hedging. A per-method retry table of humus's own is replaced by Classify, which gets the method.
• Instruments of humus's own, such as a gauge of the breaker's state: Proposal: humus's own metrics (wave 5).

This is added:

• Classifiers for well-known APIs, and helpers to build one. A Classify is the end user's function.

[Zaba505]

Closing: outbound gRPC clients have landed

The revision's stories are done:

• story(grpcclient): call one gRPC dependency through the Host with grpcclient.NewClient #801 called one gRPC dependency through the Host with grpcclient.NewClient, with Settings checked by CheckAddr and CheckTLS (story(grpcclient): call one gRPC dependency through the Host with grpcclient.NewClient #820);
• story(grpcclient): trace and measure every attempt with the Setup's telemetry #802 traced and measured every attempt with the Setup's telemetry, with otelgrpc's client stats handler as the first dial option (story(grpcclient): trace and measure every attempt with the Setup's telemetry #825);
• story(grpcclient): gate readiness on the Client's first connection with Connected #803 gated readiness on the Client's first connection with Connected, a monitor that latches once the connection has first been ready (story(grpcclient): gate readiness on the Client's first connection with Connected #829);
• story(grpcclient): retry Unavailable and pushed-back calls, as the end user's Classify decides #804 retried Unavailable and pushed-back calls as the end user's Classify decides, with grpc-go's configurable retries turned off and its transparent retries kept (story(grpcclient): retry Unavailable and pushed-back calls, as the end user's Classify decides #831);
• story(grpcclient): a circuit breaker fails calls at once while the dependency keeps failing #805 put a circuit breaker in front of calls while the dependency keeps failing, with Breaker and BreakerPolicy; a rejected call's status is Unavailable and its error matches circuit.ErrRejected (story(grpcclient): a circuit breaker fails calls at once while the dependency keeps failing #834);
• story(docs): document calling gRPC services #806 documented calling gRPC services, on a new page that follows gRPC services (story(docs): document calling gRPC services #845).

The revision left no open questions. One address gets one connection, picked first; plain text is allowed; and the end user gets a Client of humus's own, not a *grpc.ClientConn.

The rest is out of scope or belongs to other proposals, as "Out of scope" lists:

• Failure injection's seam is Proposal: failure injection (wave 5).
• Authentication and authorization is Proposal: authentication and authorization (wave 5). Until then, the end user passes per-RPC credentials in opts.
• Instruments of humus's own for retries and the breaker are Proposal: humus's own metrics (wave 5), as the charter's revision 9 places them.
• Walkthroughs of a service that calls gRPC are Proposal: walkthroughs and a humus MIGRATING.md (wave 6).
• Client-side load balancing, other resolvers and xDS; retrying an open stream, hedging and a per-method retry table; grpc-go's HTTP CONNECT proxy, ALTS and Unix domain sockets stay out of scope.