Proposal: outbound HTTP clients

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 4. It builds on Proposal: the humus App lifecycle, Proposal: configuration for humus apps, Proposal: OpenTelemetry without globals and Proposal: humustest, and on bedrock v0.24.0's retry and circuit. It depends on no other wave-4 proposal. It is discussed side by side with Proposal: outbound gRPC clients and Proposal: producing to Kafka, and keeps to the three points the charter's revision 7 sets for clients. It blocks Proposal: failure injection (wave 5), which adds an injection point to its calls.

Summary

A new package, httpclient, gives the end user's Init an *http.Client for one HTTP dependency, such as another service's API:

• httpclient.NewClient(s, settings) returns a plain *http.Client. Service code, generated clients and SDKs take it as they take any other. It dials only through s.Host.Dial, so a test runs the end user's whole App, its outbound calls included, on humustest's network.
• A client is for one dependency: the scheme, host and port of Settings.BaseURL. It sends nothing anywhere else. Settings.TLS is nil for plain text, as for a Kafka consumer and the OTLP exporters, and an https BaseURL requires it.
• Every call goes through bedrock's retry and circuit. A request that is safe to repeat is retried, up to 3 attempts, with exponential backoff, a retry budget, and the server's Retry-After. Every attempt goes through one circuit breaker for the dependency.
• Every attempt is a client span, and records OpenTelemetry's HTTP client metrics, through otelhttp and s.Telemetry. Each change of the breaker's state is a log record.
• A client is not a humus.Service. It opens no connection in NewClient, and closes its idle connections in a function it registers with s.OnStop.

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	paymentsSettings, err := readPaymentsSettings(ctx, s) // httpclient.Settings, sourced with config.Struct
	if err != nil {
		return nil, err
	}
	payments, err := httpclient.NewClient(s, paymentsSettings)
	if err != nil {
		return nil, err
	}

	api := newAPI(payments) // the end user's handler, which calls payments
	svc, err := httpserver.NewService(s, serverSettings, api)
	if err != nil {
		return nil, err
	}
	return []humus.Service{svc}, nil
}

Background

• Outbound calls were left to this proposal. OpenTelemetry without globals put "instrumentation of outbound clients, and FIT" out of scope, and the HTTP server foundation put "the outbound HTTP client" out of scope. The charter's revision 7 moved both here.

• Before the reset, humus had no outbound client. The v0.15.0 orders walkthrough built one in its Init (app/app.go#L28-L30):

  httpClient := &http.Client{
  	Transport: otelhttp.NewTransport(http.DefaultTransport),
  }

  The spans used OTel's global providers and propagator. http.DefaultTransport dials the real network and reads HTTP_PROXY from the real environment, which the lifecycle page already lists as an ambient leak. The one client served three dependencies, with no retries, no breaker and no timeout.

• bedrock ships the model and leaves the wiring to humus. bedrock v0.24.0 releases retry and circuit as nested modules. Neither does any I/O. Their discussions name what humus builds: "the http.RoundTripper wrapper, gRPC interceptors and OTel metrics" for the breaker (z5labs/bedrock#595), and "an http.RoundTripper that replays bodies with GetBody and reads Retry-After" for retries (z5labs/bedrock#606). A retry package owned by bedrock bedrock#606 includes a humus sketch of the composition, which this proposal starts from.

• net/http already retries one case, and it stays. http.Transport sends a request again on a new connection when a reused keep-alive connection fails before the server has answered, if the request is idempotent and its body can be rewound (Transport's doc). That happens inside one attempt, and can't be turned off. It recovers a stale pooled connection. It doesn't retry a failed call.

• What wave 1 provides. Host.Dial, which humustest makes in-memory. Setup.Telemetry, which carries the providers. Setup.OnStop, which releases what Init built. humustest.NewCA, so a test generates the certificates it serves and trusts. And from wave 3, httpserver, which serves a test's fake dependency on the same Host.

Proposed API

package httpclient // github.com/z5labs/humus/httpclient

// Settings configures a client of one dependency. Every field is a plain
// value, and a field's zero value means zero, never "use the default": start
// from DefaultSettings and change the fields you source. BaseURL has no
// default, so it is always among them.
type Settings struct {
	// BaseURL is the dependency. Its scheme, host and port are the origin
	// the client sends requests to, and the only one. Its path, if any, is
	// for service code to join request paths onto; the client doesn't use it.
	//
	// Default: none. Valid: an absolute http or https URL with a host, and
	// no user info, query or fragment. Check: CheckBaseURL.
	BaseURL *url.URL

	// TLS is the config the client dials the dependency with: the roots it
	// trusts in RootCAs, and for mTLS, a client certificate in
	// Certificates. nil means plain text. An https BaseURL requires a
	// config, and an http one requires nil. A config with no RootCAs trusts
	// the system's roots.
	//
	// Default: nil, which means plain text. Valid: nil, or a config that
	// verifies the server's certificate and allows TLS 1.2 or later.
	// Check: CheckTLS.
	TLS *tls.Config

	// Timeout bounds a whole call: every attempt, the waits between them,
	// and reading the response's body. A deadline the caller's context has
	// already, if it is earlier, wins. It is the http.Client's Timeout.
	//
	// Default: 10s. Valid: positive. Check: CheckTimeout.
	Timeout time.Duration

	// AttemptTimeout bounds one attempt, from dialling to the response's
	// headers. An attempt that runs past it fails with an error that wraps
	// context.DeadlineExceeded, and is retried as any failed attempt is.
	//
	// Default: 5s. Valid: positive. Check: CheckAttemptTimeout.
	AttemptTimeout time.Duration

	// MaxAttempts bounds the attempts of one call, the first included. 1
	// turns retries off.
	//
	// Default: 3. Valid: 1 or more. Check: CheckMaxAttempts.
	MaxAttempts int

	// BackoffBase and BackoffMax are retry.Exponential's arguments: the wait
	// after attempt n is drawn from [0, min(BackoffMax, BackoffBase·2ⁿ⁻¹)).
	// A Retry-After longer than BackoffMax isn't waited for.
	//
	// Default: 100ms and 5s. Valid: positive, and BackoffMax at least
	// BackoffBase. Check: CheckBackoffBase, CheckBackoffMax.
	BackoffBase time.Duration
	BackoffMax  time.Duration

	// RetryBudget limits the retries of every call to the dependency
	// together, as retry.Budget describes.
	//
	// Default: {MaxTokens: 10, TokenRatio: 0.1}. Valid: MaxTokens from 3 to
	// retry.TokenLimit; TokenRatio from 0.001 to retry.TokenLimit. Check:
	// CheckRetryBudgetMaxTokens, CheckRetryBudgetTokenRatio.
	RetryBudget retry.BudgetConfig

	// Breaker configures the dependency's circuit breaker.
	//
	// Default: {Window: 20, Probes: 1, OpenFor: 10s, ProbeTimeout: 10s}.
	// Valid: Window and Probes from 1 to circuit.MaxCount; OpenFor and
	// ProbeTimeout positive. Check: CheckBreakerWindow, CheckBreakerProbes,
	// CheckBreakerOpenFor, CheckBreakerProbeTimeout.
	Breaker circuit.Config

	// BreakerPolicy decides when the breaker opens and closes. It is set in
	// code, not sourced.
	//
	// Default: open after 5 failures in a row, or when half of the last 20
	// or more results failed; while half-open, reopen on a failed probe and
	// close after 3 successful ones. Valid: Trip and Probe both set, which
	// NewClient checks.
	BreakerPolicy circuit.Policy
}

// DefaultSettings returns the client's defaults. BaseURL has none, so its
// BaseURL is nil, which CheckBaseURL rejects: the end user always sources it.
func DefaultSettings() Settings

// The checks, one for each field that is sourced, as #631 sets out. Each
// reports what is wrong without naming the field.
func CheckBaseURL(u *url.URL) error
func CheckTLS(c *tls.Config) error
func CheckTimeout(d time.Duration) error
func CheckAttemptTimeout(d time.Duration) error
func CheckMaxAttempts(n int) error
func CheckBackoffBase(d time.Duration) error
func CheckBackoffMax(d time.Duration) error
func CheckRetryBudgetMaxTokens(n int) error
func CheckRetryBudgetTokenRatio(r float64) error
func CheckBreakerWindow(n int) error
func CheckBreakerProbes(n int) error
func CheckBreakerOpenFor(d time.Duration) error
func CheckBreakerProbeTimeout(d time.Duration) error

// NewClient returns an *http.Client for the dependency at settings.BaseURL.
//
// NewClient runs every check on settings, and rejects a BackoffMax below
// BackoffBase, a TLS that doesn't match BaseURL's scheme, and a
// BreakerPolicy with a nil function, reporting every bad field in one error
// that names each one. It builds the dependency's retry.Budget,
// retry.Retrier and circuit.Breaker, dials nothing, and registers a function
// with s.OnStop that closes the client's idle connections.
//
// The client dials through s.Host and reports through s.Telemetry. A
// request to any origin other than settings.BaseURL's fails without being
// sent.
func NewClient(s *humus.Setup, settings Settings) (*http.Client, error)

The package is in the core module. Its new dependencies are bedrock's retry and circuit modules, which depend on nothing but the standard library and testify. otelhttp is already in the core module, for httpserver.

The client

One client per dependency. A dependency is an origin: the scheme, host and port of BaseURL, with the scheme's default port when the URL has none. The end user's Init builds one client for each dependency, with its own Settings, and shares it between every Service that calls it. The breaker and the retry budget belong to the client, so a client is what bedrock calls "once per dependency, at startup". A request to another origin fails with httpclient: request to https://ledger:8443 on a client for https://payments:8443, and nothing is sent. A redirect to the same origin is followed, as http.Client does. A redirect to another origin isn't: the client returns the 3xx response to service code.

What it returns. A plain *http.Client. Service code depends on net/http's type, not on httpclient, and a unit test of that code passes any client. Its Transport is the stack described below, its Timeout is Settings.Timeout, and its CheckRedirect stops at a redirect to another origin.

The transport underneath is an http.Transport that humus builds:

• DialContext is s.Host.Dial. So the client reaches the dependency the way every humus App reaches the network, and a test's in-memory Host serves it.
• Proxy is nil. The client never reads HTTP_PROXY, HTTPS_PROXY or NO_PROXY.
• TLSClientConfig is a clone of Settings.TLS, for an https BaseURL. Protocols offers HTTP/2 and HTTP/1.1, so every server that offers h2 gets HTTP/2, a humus server set to HTTP2Only among them. An http BaseURL is HTTP/1.1, because humus serves no h2c.
• The pool keeps up to 100 idle connections, all to the one origin, for 90s, as http.DefaultTransport does for every host together. net/http's default of 2 idle connections per host would make a busy client open and close connections on every burst. These aren't settings: they are added when users ask for them.

Shutdown. The client isn't a Service, and has nothing to drain. A call in flight ends with its own context: a handler's call ends when its request's context does. The function NewClient registers with s.OnStop calls CloseIdleConnections, and the OnStop functions run after every Service has stopped, so nothing is still calling the dependency by then.

The layers

Each call goes through four layers, all unexported, outermost first:

1. Retry. It checks the origin, then ranges over Retrier.Attempts with the call's context. Before each attempt after the first, it rewinds the body with GetBody. After each, it builds the attempt's result: the transport's error, or an error for a status code the classifiers count. It passes that to Attempt.Done.
2. otelhttp.Transport. One client span per attempt. It injects the propagator's trace context and baggage into the attempt's headers, and records the attempt's metrics.
3. The breaker. Breaker.Allow, a deferred Release, the attempt, then Ticket.Done, which the response body settles. A rejected attempt returns the *circuit.RejectedError and sends nothing.
4. The attempt timeout. A timer of AttemptTimeout that cancels the attempt's context, with a cause that wraps context.DeadlineExceeded, unless the response's headers arrive first.

Then the http.Transport.

The order is bedrock's: "a circuit breaker goes inside the retry loop, so that each attempt is a call the breaker admits or rejects". otelhttp sits between them so that every attempt, rejected ones included, has a span. That's where HTTP differs from gRPC: an http.RoundTripper can wrap the breaker, so a rejected attempt's span is otelhttp's own, named and attributed as every other attempt's is, and humus starts no span of its own. The breaker sees the call's context, not the attempt's, so the classifier can tell the caller giving up from an attempt timing out.

Which requests are retried

A call is retried only when sending it twice is safe and possible:

• Safe. Its method is idempotent under RFC 9110: GET, HEAD, OPTIONS, TRACE, PUT or DELETE. Or it carries an Idempotency-Key or X-Idempotency-Key header, which net/http treats the same way.
• Possible. It has no body, or its GetBody is set. http.NewRequest sets GetBody for a *bytes.Buffer, a *bytes.Reader and a *strings.Reader.

Any other request makes one attempt. It still goes through the breaker, and Done still counts its result in the budget: the loop breaks after Done, as bedrock's doc shows.

net/http's own list leaves out PUT and DELETE. RFC 9110 defines both as idempotent, and a service that sends them relies on that already.

How an attempt's result counts

Every result goes to two classifiers: retry's, which decides whether another attempt follows, and the breaker's, which decides whether the result counts against the dependency.

An attempt's result	Retry	Breaker
a status below 500, other than 408 and 429	Success	Success
408 Request Timeout	Retry	Success
429 Too Many Requests	Retry, after Retry-After	Success
500 Internal Server Error	Stop	Failure
502 Bad Gateway, 504 Gateway Timeout	Retry	Failure
503 Service Unavailable	Retry, after Retry-After	Failure
any other 5xx, such as 501 Not Implemented	Stop	Success
a transport error: refused, reset, a failed TLS handshake, AttemptTimeout	Retry	Failure
a server certificate the client doesn't trust	Stop	Failure
rejected by the breaker	Stop	not admitted
the call's context cancelled: the caller gave up	Stop	Ignored
the call's deadline passed: Timeout, or a deadline the caller set	Stop	Failure
reading the body fails	the call is service code's by then	Failure, or Ignored when the caller gave up

• A 4xx is the dependency working. It answered, and the request was wrong, so another attempt won't fix it, and the breaker counts the dependency healthy.
• 408 and 429 are retried, and don't trip the breaker. A 408 says the server gave up waiting for the request, which RFC 9110 says a client may repeat. A 429 is the server's own flow control: it is answering, and says when to come back. The budget bounds those retries, and Retry-After paces them.
• 500 trips the breaker but isn't retried. It is usually the server's bug, which a second attempt meets again, while a dependency returning many 500s should still be shed. 502, 503 and 504 say the dependency or what is in front of it is unavailable, which is what both retries and breakers are for.
• Every deadline counts against the dependency, as in bedrock's own example classifier: a dependency that hangs ends every call this way, and a breaker that ignored deadlines would never open. The deadline may be one the inbound request brought.
• The status is classified before the body is read. A call can only be retried until its response is handed to service code. Once service code reads the body, a failure is the caller's to handle, as retry's doc says.

Retry-After

When the classifier retries a response that has a Retry-After header, its delay is the least time to wait, and the wait is the longer of it and the backoff. Both forms of RFC 9110 are read: a number of seconds, and an HTTP-date, which is measured from now and clamped at 0. A malformed header is ignored.

A Retry-After longer than BackoffMax isn't waited for: the classifier returns Stop, and the response goes to service code as it came. A server that asks for a minute is not one a request in flight should wait on, and that rule needs no setting of its own. A wait that would end after the call's deadline isn't started either: retry ends the call as Expired, and it returns the last response.

The response body

Two things happen to a response's body.

When another attempt follows, the client reads the body of the response it replaces into memory, up to 64 KiB, and closes it, before the wait. Closing it ends the attempt's span, settles its breaker ticket, and frees its connection during the backoff. If the wait is cut short, the buffered response is what the call returns. A body larger than 64 KiB isn't retried: the loop breaks, and the call returns the response with its body intact. Error bodies are small, and z5labs/bedrock#606 named this choice: "unless humus buffers a small body and closes the connection."

The breaker's ticket is settled by the body, as circuit's doc shows for an http.RoundTripper. The body settles it when service code reads it to the end, or closes it, with the status's result. A read that fails settles it with the error. So a dependency that sends its headers and then fails mid-body counts as failing. A body that is never closed leaves its ticket unsettled. That costs nothing while the breaker is closed, and holds a half-open probe slot for ProbeTimeout at most. net/http asks for every body to be closed anyway.

The sketch in z5labs/bedrock#606 settles the ticket once the headers arrive. That is simpler, but a failure after the headers is never counted.

Timeouts

• Timeout bounds the call. It is the http.Client's Timeout, which net/http puts on the request's context as a deadline, unless the caller's context has an earlier one, which then wins. So retry sees the deadline, and doesn't start a wait that would end after it. It covers reading the body, so a client that streams large bodies raises it.
• AttemptTimeout bounds one attempt until its headers, as retry's doc puts a deadline on each attempt. An attempt to a dependency that hangs fails after 5s and is retried, so with the defaults, a hung first attempt still leaves time for a second before Timeout. It stops counting once the headers arrive, so a slow body isn't cut off. Its cause wraps context.DeadlineExceeded, which the classifiers read as an attempt timing out, not the caller giving up. net/http returns the context's cause.

Settings

Field	Default	Valid	Check
BaseURL	none	absolute http or https URL with a host; no user info, query or fragment	CheckBaseURL
TLS	nil: plain text	nil, or a config that verifies the server and allows TLS 1.2 or later; set for https and nil for http	CheckTLS; the scheme in NewClient
Timeout	10s	positive	CheckTimeout
AttemptTimeout	5s	positive	CheckAttemptTimeout
MaxAttempts	3	1 or more	CheckMaxAttempts
BackoffBase	100ms	positive	CheckBackoffBase
BackoffMax	5s	positive, and at least BackoffBase	CheckBackoffMax; the order in NewClient
RetryBudget.MaxTokens	10	3 to retry.TokenLimit	CheckRetryBudgetMaxTokens
RetryBudget.TokenRatio	0.1	0.001 to retry.TokenLimit	CheckRetryBudgetTokenRatio
Breaker.Window	20	1 to circuit.MaxCount	CheckBreakerWindow
Breaker.Probes	1	1 to circuit.MaxCount	CheckBreakerProbes
Breaker.OpenFor	10s	positive	CheckBreakerOpenFor
Breaker.ProbeTimeout	10s	positive	CheckBreakerProbeTimeout
BreakerPolicy	bedrock's documented example	Trip and Probe set	NewClient; set in code, not sourced

• CheckBaseURL rejects user info. A password in the URL would sit in a setting, and in every error that names the client's origin. Credentials go in a header that service code sets.
• TLS is nil for plain text, as kafka.Settings.TLS and telemetry.OTLPSettings.TLS are. A URL names its scheme too, so NewClient requires the two to agree: an https BaseURL with a nil TLS, or an http one with a config, fails startup rather than quietly doing one of the two. Open question 2 asks whether plain text is allowed at all.
• CheckTLS rejects InsecureSkipVerify. A test trusts its humustest.NewCA through RootCAs, so nothing in humus needs to skip verification, and a setting that does fails startup.
• A config with no RootCAs trusts the system's roots, which crypto/x509 loads from the real host: from SSL_CERT_FILE and SSL_CERT_DIR in the real environment, and from the OS's certificate files. That's an ambient leak, and the lifecycle page lists it. A service that calls only its own services sets RootCAs to its CA and reads nothing ambient.
• Timeout is positive. A client with no bound on a call holds a goroutine and a connection for as long as a dependency hangs. It covers reading the body, because for HTTP the body is part of the call: a client that streams large bodies raises it.
• The breaker and the budget are bedrock's types, with a check for each field, so each field can be sourced from a variable of its own and fail as that variable. The checks' ranges are bedrock's.
• BreakerPolicy is set in code, like a Kafka consumer's Delivery. It is the end user's decision about how their dependency fails, rather than an operator's, and any Policy can be written. Four threshold settings would fix one shape of policy, and the operator rarely changes them. DefaultSettings gives bedrock's example policy, and NewClient rejects a nil Trip or Probe, which circuit.New would panic on.

In the end user's Init, sourced as #631 and the configuration page describe, with whatever variable names the service uses:

d := httpclient.DefaultSettings()
env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }

settings, err := config.Struct(func(r *config.Resolver) httpclient.Settings {
	c := d
	// No .Default: BaseURL is required.
	c.BaseURL = r.Get(env("PAYMENTS_URL").Parse(url.Parse).Validate(httpclient.CheckBaseURL))
	c.TLS = r.Get(env("PAYMENTS_CA_FILE").Parse(readFile(s.Host.FS)).Parse(trusting).Validate(httpclient.CheckTLS))
	c.Timeout = r.Get(env("PAYMENTS_TIMEOUT").Parse(time.ParseDuration).Default(d.Timeout).Validate(httpclient.CheckTimeout))
	c.MaxAttempts = r.Get(env("PAYMENTS_MAX_ATTEMPTS").Parse(strconv.Atoi).Default(d.MaxAttempts).Validate(httpclient.CheckMaxAttempts))
	c.Breaker.OpenFor = r.Get(env("PAYMENTS_BREAKER_OPEN_FOR").Parse(time.ParseDuration).Default(d.Breaker.OpenFor).Validate(httpclient.CheckBreakerOpenFor))
	return c
}).Read(ctx)
if err != nil {
	return nil, err // exit 78, naming every bad variable
}

readFile and trusting are the end user's helpers: trusting builds a *tls.Config whose RootCAs hold the PEM it read. The dependency is https, so PAYMENTS_CA_FILE has no .Default either.

Telemetry

The client takes everything from s.Telemetry and never touches OTel's globals:

otelhttp.NewTransport(breaker,
	otelhttp.WithTracerProvider(s.Telemetry.TracerProvider),
	otelhttp.WithMeterProvider(s.Telemetry.MeterProvider),
	otelhttp.WithPropagators(s.Telemetry.Propagator),
	otelhttp.WithSpanNameFormatter(func(_ string, r *http.Request) string { return r.Method }),
)

otelhttp's transport reads the global propagator and MeterProvider as defaults before it applies its options, and without a TracerProvider it falls back to the global one. All three are passed, so none of the globals is used.

• Spans. Each attempt is one span of kind CLIENT, a child of the span in the call's context. The HTTP semantic conventions make every resend a span of its own, and every span after the first carries http.request.resend_count. A redirect counts as a resend too, as the conventions say. otelhttp doesn't set that attribute, so the breaker layer sets it on the attempt's span, which otelhttp has put in the attempt's context by then. A span is named after the method, such as GET, as the conventions name a client span with no URL template: otelhttp's own default, HTTP GET, predates them. The span carries otelhttp's attributes, among them url.full, without user info, server.address, server.port and http.response.status_code. Its status is Error for a 4xx or 5xx and for a transport error.
• Rejections. An attempt the breaker rejects never reaches the transport, and still has a failed client span: otelhttp's, because otelhttp wraps the breaker. It is named and attributed as every attempt's span is, and ends at once with status Error and error.type set to *circuit.RejectedError. The call returns an error that errors.Is matches with circuit.ErrRejected, through http.Client's *url.Error.
• Metrics. otelhttp's http.client.request.duration and http.client.request.body.size, recorded for each attempt in otelhttp's scope, with error.type for a failed one. humus records no instrument of its own. Because otelhttp wraps the breaker, a rejected attempt is in http.client.request.duration too, with its error.type, which the gRPC client's rejections aren't.
• Logs. The client logs through s.Telemetry.LoggerProvider.Logger("github.com/z5labs/humus/httpclient"), with the OTel Logs API. It writes one record for each of the breaker's transitions, from circuit's observer, in the context of the call that made the change: circuit breaker open at WARN, and circuit breaker half-open and circuit breaker closed at INFO. Each has the attributes server.address and server.port of the dependency, circuit.previous_state, and the counts of the state it left: circuit.successes, circuit.failures, circuit.consecutive_successes, circuit.consecutive_failures and circuit.timed_out. OpenTelemetry has no convention for circuit breakers, so the circuit. names are humus's, the same in all three clients.
• Nothing reaches the real standard error. The client logs nothing else. An error is returned to service code, not logged.

Panics

The client recovers nothing. Nothing of the end user's runs inside a call but BreakerPolicy, and a panic there passes through circuit, which leaves the breaker consistent, to the goroutine that made the call. A panic in the client itself is humus's bug.

What wave 5 builds on this

Proposal: failure injection (wave 5) adds an injection point to every client's calls, and strips scenarios from calls to third parties. The seams in this package are:

• The injection point goes inside the attempt timeout, the breaker and the attempt's span: around the call to the http.Transport. An injected abort then reaches every layer as a transport error. It is counted by the breaker, retried by the retry layer, and recorded on the attempt's span, exactly as a real failure is. An injected delay counts against AttemptTimeout, as a slow dependency's does. A client has one dependency, known when NewClient runs, so a point's name can be fixed when the client is built, as fit's doc expects.
• Stripping goes in the retry layer, before otelhttp injects the propagator's baggage into an attempt's headers.

Nothing in this API is designed for failure injection, and the wave-5 proposal's stories make the change here.

Design decisions

• The package is httpclient. It mirrors httpserver, in the same module. http would clash with net/http, and client says nothing.
• NewClient returns *http.Client, not a type of its own. The gRPC client returns a type of its own, and the Kafka producer is NewProducer. HTTP differs because every Go HTTP library takes an *http.Client, generated clients and cloud SDKs among them, and a type of humus's own would need an adapter for each. The constructor keeps the name the other clients use. An http.RoundTripper alone would leave the end user to set Timeout and CheckRedirect correctly.
• One client per dependency. bedrock builds one breaker per dependency at startup and keeps no registry. A client bound to one origin is that, and its Settings are the dependency's: its own roots, timeouts and retry budget. fit expects the same of an injection point. The alternative, one client with a breaker for each origin it meets, is open question 1.
• Retries are on by default, for idempotent requests only. Three attempts within the budget is what gRPC's retry throttling and most SDKs default to. A request that isn't safe to send twice is never retried, so the default can't double a charge. MaxAttempts: 1 turns retries off.
• The breaker is always on. A BreakerPolicy whose Trip returns false never opens, for a dependency that must not be shed.
• The classification is fixed. See How an attempt's result counts, and open question 3.
• Each attempt is a span, and the call is not. That is the HTTP semantic conventions. A span for the whole call, retries included, is the end user's, such as their handler's.
• HTTP/2 is preferred. A humus server prefers it, and refuses HTTP/1.1 with HTTP2Only. HTTP/2 puts every call on one connection, so behind a Kubernetes ClusterIP Service, one client's calls reach one pod until that connection closes. A humus server's Shutdown sends a GOAWAY, so a rollout moves them. The docs page says so. Load balancing across a Service's pods is out of scope.
• No proxy, and no proxy setting. Proxy is nil, so the client never reads the proxy variables of the real environment. A proxy setting is added when users ask for one.
• The lint list bans net/http's default client. http.DefaultClient, http.DefaultTransport, http.Get, http.Head, http.Post and http.PostForm dial the real network and read the proxy variables. humus's library code never uses them, and forbidigo enforces that as it does the process boundary.

Testing

A prototype of this client passed the tests below, under go test -race -count=3, with bedrock's retry and circuit v0.24.0, otelhttp v0.72.0 and Go 1.27. Each test runs the end user's whole App inside a testing/synctest bubble, on a humustest Host, with a fake dependency served by httpserver on the same Host over TLS. The App is a job whose run calls the dependency. Certificates come from humustest.NewCA, so no key, certificate or password is fixed, and backoff and OpenFor waits take no real time. The prototype was in a scratchpad and is not kept, and the sketches below are trimmed from it.

The shared helpers: the fake dependency listens on :8443 and serves a certificate for payments, and the client's BaseURL is https://payments:8443:

// startPayments runs a fake dependency on h, at payments:8443, over TLS.
func startPayments(t *testing.T, h *humustest.Host, ca *humustest.CA, handler http.Handler) (stop func()) {
	cert := ca.Issue(t, "payments")
	p, status := humustest.Start(t, func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
		settings := httpserver.DefaultSettings()
		settings.Addr = ":8443"
		settings.TLS = &tls.Config{Certificates: []tls.Certificate{cert.TLS()}}
		svc, err := httpserver.NewService(s, settings, handler)
		if err != nil {
			return nil, err
		}
		return []humus.Service{svc}, nil
	}, humustest.WithApp(humus.WithHost(h.Host()), humus.WithTelemetry(telemetry.NoopBuilder)))
	<-h.Listening("payments:8443")
	return func() {
		p.Signal(syscall.SIGTERM)
		require.Zero(t, (<-status).Code())
	}
}

// paymentsSettings are the client's defaults, for https://payments:8443,
// trusting only the test's CA.
func paymentsSettings(t *testing.T, ca *humustest.CA) httpclient.Settings {
	u, err := url.Parse("https://payments:8443")
	require.NoError(t, err)
	settings := httpclient.DefaultSettings()
	settings.BaseURL = u
	settings.TLS = &tls.Config{RootCAs: ca.Pool()}
	return settings
}

// runJob runs the end user's App: a job whose run calls the dependency.
func runJob(t *testing.T, h *humustest.Host, tel *humustest.Telemetry, settings httpclient.Settings, run func(ctx context.Context, c *http.Client) error) {
	_, status := humustest.Start(t, func(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
		c, err := httpclient.NewClient(s, settings)
		if err != nil {
			return nil, err
		}
		return []humus.Service{job.NewService(s, func(ctx context.Context) error { return run(ctx, c) })}, nil
	}, humustest.WithApp(humus.WithHost(h.Host()), humus.WithTelemetry(tel.Builder())))
	require.Zero(t, (<-status).Code())
}

A 503 with Retry-After is retried after it, with a span for each attempt:

func TestClient_retries_a_503_after_its_Retry_After(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		ca := humustest.NewCA(t)
		var calls []time.Time
		stop := startPayments(t, h, ca, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			calls = append(calls, time.Now())
			if len(calls) == 1 {
				w.Header().Set("Retry-After", "2")
				w.WriteHeader(http.StatusServiceUnavailable)
				return
			}
			_, _ = io.WriteString(w, "ok")
		}))
		defer stop()

		tel := humustest.NewTelemetry()
		runJob(t, h, tel, paymentsSettings(t, ca), func(ctx context.Context, c *http.Client) error {
			resp, err := c.Get("https://payments:8443/charges/1")
			require.NoError(t, err)
			defer resp.Body.Close()
			require.Equal(t, http.StatusOK, resp.StatusCode)
			return nil
		})

		require.Len(t, calls, 2)
		require.GreaterOrEqual(t, calls[1].Sub(calls[0]), 2*time.Second)

		spans := clientSpans(tel) // the spans of kind CLIENT
		require.Len(t, spans, 2)
		require.Equal(t, "GET", spans[0].Name())
		require.Contains(t, spans[1].Attributes(), attribute.Int("http.request.resend_count", 1))
		require.True(t, spans[0].EndTime().Before(spans[1].StartTime())) // the first span ends before the wait
	})
}

The breaker opens, rejects without dialling, and closes again after OpenFor:

func TestClient_breaker_opens_rejects_and_closes(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		ca := humustest.NewCA(t)
		var calls atomic.Int32
		var healthy atomic.Bool
		stop := startPayments(t, h, ca, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			calls.Add(1)
			if !healthy.Load() {
				w.WriteHeader(http.StatusInternalServerError) // a breaker Failure, not retried
			}
		}))
		defer stop()

		tel := humustest.NewTelemetry()
		runJob(t, h, tel, paymentsSettings(t, ca), func(ctx context.Context, c *http.Client) error {
			for range 5 {
				resp, err := c.Get("https://payments:8443/")
				require.NoError(t, err)
				require.NoError(t, resp.Body.Close())
			}
			_, err := c.Get("https://payments:8443/")
			require.ErrorIs(t, err, circuit.ErrRejected)
			require.EqualValues(t, 5, calls.Load()) // the sixth call was never sent

			healthy.Store(true)
			time.Sleep(10 * time.Second) // OpenFor
			for range 3 {
				resp, err := c.Get("https://payments:8443/")
				require.NoError(t, err)
				require.NoError(t, resp.Body.Close())
			}
			return nil
		})

		records := logsOf(tel, "github.com/z5labs/humus/httpclient") // logsOf and bodies are small test helpers
		require.Equal(t, []string{"circuit breaker open", "circuit breaker half-open", "circuit breaker closed"}, bodies(records))
		require.Equal(t, log.SeverityWarn, records[0].Severity())
	})
}

The prototype's tests also showed the following:

• A POST isn't retried without an idempotency key. A POST answered 503 is sent once, and the same POST with Idempotency-Key is sent three times.
• A hung attempt is retried. A handler that waits on its context for the first request makes that attempt fail after AttemptTimeout's 5s. The second attempt succeeds, and the handler's context was cancelled, because the attempt's stream was reset.
• A Retry-After beyond BackoffMax isn't waited for. A 429 with Retry-After: 60 is returned at once, after one request.
• Timeout ends the retries. With Timeout: 6s and a 503 with Retry-After: 4 on every request, the call makes two attempts and returns the second 503 at 4s, rather than starting a wait that would end after the deadline.
• Redirects. A redirect within the origin is followed, and its span has http.request.resend_count 1. A redirect to another origin returns the 302 to service code.
• Another origin is refused. A request to https://ledger:8443/ fails with the message above, and nothing is dialled.
• HTTP/2. Every attempt to the httpserver dependency has network.protocol.version 2.0, as otelhttp writes it.
• No globals. With OTel's global TracerProvider set to a recorder of its own, a call records nothing in it.

The stories add the tests the prototype didn't have:

• The checks. Each accepts an in-range value with nil and rejects an out-of-range one without naming the field, and the budget's and breaker's checks accept exactly what retry.NewBudget and circuit.New accept. NewClient names each bad field and reports two together. It rejects BackoffMax below BackoffBase, a TLS that doesn't match the scheme, and a BreakerPolicy with a nil function. DefaultSettings() with an http BaseURL passes NewClient. The package never reads a bedrock.Environment and doesn't import bedrock/config.
• The transition records. The open record carries circuit.previous_state closed, and the counts that tripped it, such as circuit.consecutive_failures 5. The prototype checked these on the open record.
• The breaker counts a body. A response whose body fails mid-read counts as a Failure, and a caller that cancels while reading counts as Ignored.
• Idle connections close at OnStop. After the App exits, the dependency sees its connection from the client closed.
• A body that doesn't fit the buffer isn't retried. A 503 with a 1 MiB body is returned whole, after one attempt.
• A rejected attempt's span and metric. Its span has status Error and error.type *circuit.RejectedError, and so does its http.client.request.duration data point.

The package's example is an Example function or a package main with an app_test.go, as the charter requires, and the package is held to the 80% coverage gate.

Changes to other files

This proposal's stories make these, as it owns outbound HTTP:

• A docs page, "Calling HTTP services", at docs/content/http-client.md: one client per dependency, sourcing the settings, which requests are retried, the classification table, Retry-After, the breaker and its log records, the body rules, HTTP/2 behind a ClusterIP Service, and testing against a fake dependency on a humustest Host.
• The lifecycle page's Ambient leaks gains crypto/x509's system roots, read from SSL_CERT_FILE, SSL_CERT_DIR and the OS's files when a client's TLS has no RootCAs.
• The root package's "Being rebuilt" paragraph, and the docs site's index, name the package.
• The forbidigo list gains http.DefaultClient, http.DefaultTransport, http.Get, http.Head, http.Post and http.PostForm.
• The core module's go.mod gains github.com/z5labs/bedrock/retry and github.com/z5labs/bedrock/circuit v0.24.0.

httpserver, httpservertest, humustest and telemetry don't change.

Migration from today

Before the reset	With this proposal
&http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}, built in Init	httpclient.NewClient(s, settings)
one client shared by every dependency	one client per dependency, each with its own Settings, breaker and budget
the base URL in a config struct, read with bedrockconfig	Settings.BaseURL, sourced from whatever variable the end user names
otelhttp on OTel's global providers and propagator	otelhttp with s.Telemetry's, a span per attempt, named after the method
http.DefaultTransport: the real network, ProxyFromEnvironment	s.Host.Dial, and no proxy
no timeout	Timeout of 10s, and AttemptTimeout of 5s
no retries, no breaker	idempotent requests retried up to 3 attempts within a budget; a breaker per dependency

There was nothing else to migrate: humus had no outbound client.

Open questions

1. One client per dependency, or one for every origin? As proposed, a client is bound to BaseURL's origin and refuses others, and the end user builds one for each dependency. The alternative is one client that calls any origin, with a breaker and a budget made for each origin on its first call. That suits a client of URLs it doesn't know at startup, such as webhooks, but grows without bound, and puts every dependency on one set of settings. Recommendation: one client per dependency.
2. Plain text. As proposed, a nil TLS means plain text, with an http BaseURL, as kafka.Settings.TLS and telemetry.OTLPSettings.TLS do, and as the gRPC client proposes. A mesh sidecar, a node-local agent or a cloud metadata endpoint is often plain HTTP. Every humus server serves TLS, so a call to one is always https. Allow plain text, or require TLS and an https BaseURL, as the servers require TLS? Recommendation: allow it, and decide it the same way for all three clients.
3. Fixed classification, or a way to change it? As proposed, the table above is fixed. Some dependencies need another, such as an API that answers 409 to a retryable conflict or 200 with an error in its body. A field taking classifier functions would cover them, at the cost of an API that the end user has to get right for both retry and circuit. Recommendation: fixed, and a classifier field is added when users ask, as the Kafka consumer's extra settings are.
4. Metrics for the breaker and the budget? As proposed, humus records no instruments of its own. Rejections show in http.client.request.duration with their error.type, and transitions are WARN and INFO records. circuit and retry both expect a framework to export a breaker's state and a budget's tokens, as gauges read with Breaker.State and Budget.Tokens, which would let an alert fire on an open breaker without a log query. Add the two gauges now, or wait for the three client proposals to agree on their names? Recommendation: none now. The other two clients would need the same gauges, which makes it a decision for all three.

Out of scope

• Load balancing across a dependency's pods, and service discovery. The client dials the origin's host, and Kubernetes or a mesh balances connections.
• Proxies. The client never reads the proxy variables, and has no proxy setting.
• Credentials on outbound calls, such as a bearer token or an OAuth flow. Service code sets the header on each request.
• Hedging, rate limiting and bounding the concurrency of calls. bedrock's retry and circuit leave these out too.
• Caching, cookies and HTTP/3. The client's Jar is nil, and the end user may set one.
• Retrying a request that isn't safe to repeat, whatever its status.
• Injection points, and stripping scenarios before calling a third party: Proposal: failure injection (wave 5), at the seams above.
• Showing a service calling another end to end: Proposal: walkthroughs and a humus MIGRATING.md (wave 6).

References

• Proposal: humus on the bedrock process model, the charter, and its revision 7: the three client proposals and what keeps them in line
• Telemetry's revision 2: telemetry.SDK, no globals, and outbound instrumentation left out of scope
• The configuration revision: Settings, DefaultSettings and Check<Field>
• humustest's revision 2: the in-memory network and NewCA
• Proposal: HTTP server foundation: the server a test's fake dependency runs on, its HTTP/2 preference, and HTTP2Only
• bedrock v0.24.0's retry and circuit, and their discussions, z5labs/bedrock#606 and z5labs/bedrock#595; fit and z5labs/bedrock#578 for the seams of wave 5
• otelhttp v0.72.0, and OpenTelemetry's HTTP client spans and HTTP client metrics
• RFC 9110: idempotent methods (§9.2.2), 408 and 503, and Retry-After (§10.2.3); RFC 6585: 429
• net/http's Transport, on the requests it replays itself
• Before the reset: the v0.15.0 orders walkthrough's app/app.go

[Zaba505]

Revision 1: the open questions answered

The open questions:

1. One client per dependency, as recommended. It costs nothing in pooling. net/http keys its idle connections by scheme, host and port, so even one http.Transport shared by every dependency never lends one dependency's connection to another. A client per origin pools as a shared transport would. The client section now says how the two differ, and what does cost: a client built for each call.
2. Plain text is allowed, as recommended. humus makes a service as secure as it can, but it can't control what the service connects to: a mesh's sidecar, a node-local agent, a vendor's endpoint. A nil TLS with an http BaseURL stays plain text, and the two must still agree. The reason holds for every client. See The other clients.
3. The classification is a function from the start: Settings.Classify. APIs don't agree on what their answers mean. One answers 409 to a conflict that a retry resolves, another 200 with an error in its body, so nearly every client ends up classifying its own dependency. A field added later would leave each of them waiting for it. DefaultSettings().Classify is the table as proposed. An end user whose APIs follow one convention, such as a company's internal services, writes one function and sets it on every client. The client still decides what is about the call rather than the API: a rejection by the breaker, the caller giving up, the call's deadline, and which requests are safe to send twice.
4. No instruments of humus's own, as recommended. Whether humus should record any is wider than this client, and wider than the three clients, so it becomes a proposal of its own: Proposal: humus's own metrics, in wave 5, which the charter's revision 9 adds.

Each section below replaces or adds to the part of the same name in the post. Background, Shutdown, Timeouts, What wave 5 builds on this, Migration from today and References are unchanged.

Summary

The third bullet becomes:

• Every call goes through bedrock's retry and circuit. A request that is safe to repeat is retried, up to 3 attempts, with exponential backoff, a retry budget, and the server's Retry-After. Every attempt goes through one circuit breaker for the dependency. What a dependency's answers mean is a function, Settings.Classify. Its default follows RFC 9110, and the end user sets their own for an API that means something else.

Proposed API

Settings gains a field, and the package a type:

type Settings struct {
	// BaseURL, TLS, Timeout, AttemptTimeout, MaxAttempts, BackoffBase,
	// BackoffMax, RetryBudget, Breaker and BreakerPolicy, unchanged.

	// Classify decides how each attempt's result counts: whether another
	// attempt follows, and whether the result counts against the
	// dependency. It is set in code, not sourced, and one function may be
	// set on every client whose dependencies agree on what their answers
	// mean.
	//
	// Classify gets the attempt's response, with its status and headers
	// and its body unread, or the error the attempt failed with, never
	// both. resp.Request is the request the attempt sent. It isn't asked
	// about what the client decides itself: a rejection by the breaker, an
	// error once the call's context has ended, and a body that fails once
	// service code reads it.
	//
	// Classify may read resp.Body, to classify what the dependency sent,
	// such as a 200 that carries an error. It then closes the body it read,
	// and leaves in resp.Body what service code is to read, such as the
	// bytes it read.
	//
	// It is called on the goroutine that made the call, once for each
	// attempt, so it must be safe for concurrent use. A Classification
	// with a zero Outcome, or with retry.Retry and a negative RetryAfter,
	// panics, as bedrock's classifiers do.
	//
	// Default: the classification under "How an attempt's result counts".
	// Valid: set, which NewClient checks.
	Classify func(resp *http.Response, err error) Classification
}

// Classification is how one attempt's result counts, as Settings.Classify
// decides it.
type Classification struct {
	// Retry is whether another attempt follows. retry.Success and
	// retry.Stop end the call. retry.Retry asks for another attempt, which
	// the request, MaxAttempts, the budget and the call's deadline must
	// all allow.
	Retry retry.Outcome

	// RetryAfter is, with retry.Retry, the least time to wait before the
	// next attempt, such as the server's Retry-After, or 0. A RetryAfter
	// longer than BackoffMax isn't waited for: the call ends, and returns
	// the response as it came.
	RetryAfter time.Duration

	// Breaker is how the result counts in the dependency's circuit
	// breaker: circuit.Success, circuit.Failure, or circuit.Ignored for a
	// result that says nothing about the dependency.
	Breaker circuit.Outcome
}

NewClient's doc says it rejects a nil Classify, next to a BreakerPolicy with a nil function. DefaultSettings' doc is unchanged: Classify is among its defaults.

The client

These are added after "One client per dependency":

Two clients may share an origin. For example, two APIs behind one gateway, which the end user wants shed separately, get two clients. Each has its own BaseURL path, Settings, breaker and budget, as the gRPC client's end user builds two Clients for two services on one address.

Pooling. A client per dependency pools as one shared http.Transport would. net/http keys its idle connections by scheme, host and port, so a shared transport never lends one dependency's connection to another. The two differ in two ways, and neither is a cost in practice:

• Idle limits are per client. Each client keeps up to 100 idle connections for 90s, where one shared transport's MaxIdleConns would cap every dependency together. A process with many dependencies can hold more idle sockets: up to what each dependency's busiest burst opened, until they've been idle for 90s. In return, a burst to one dependency never evicts another's idle connections, which would cost each a new TLS handshake.
• Two clients of one origin don't share connections. Each opens its own. Over HTTP/2 that is one more connection for each client.

The cost that matters is a client built for each call. It would pool nothing, so every call would dial and handshake. Every call would also start with a fresh breaker and budget, which then never open or run down, and NewClient would register one more OnStop function each time. NewClient takes the *humus.Setup, so it belongs in Init, and the docs page says so.

The layers

Step 1 becomes:

1. Retry. It checks the origin, then ranges over Retrier.Attempts with the call's context. Before each attempt after the first, it rewinds the body with GetBody. After each, it passes Attempt.Done the attempt's result, with the Classification the breaker layer made of it.

Step 3 becomes:

3. The breaker. Breaker.Allow, a deferred Release, the attempt, and then Classify, once, unless the result is one the client decides itself. An error settles the Ticket at once. A response hands it to the body, which settles it. A rejected attempt returns the *circuit.RejectedError and sends nothing, and Classify never sees it.

Classify runs inside otelhttp's span, so a Classify that reads the body is part of the attempt's span, and the span ends when service code is done with the body it left.

Which requests are retried

This is added:

These rules hold whatever Classify returns. A Classify that returns retry.Retry for a POST without an idempotency key still gets one attempt. Whether a request is safe to send twice is about the request, not about what the dependency answered.

How an attempt's result counts

This section is replaced:

Every attempt's result is classified twice: for retry, whether another attempt follows, and for circuit, whether the result counts against the dependency. Some of that is about the call, and the client decides it the same way for every dependency. The rest is about what the dependency's answers mean, which differs from one API to another, and Settings.Classify decides it.

What the client decides. Classify is never asked about these:

An attempt's result	Retry	Breaker
rejected by the breaker	Stop	not admitted
an error, once the call's context was cancelled: the caller gave up	Stop	Ignored
an error, once the call's deadline passed: Timeout, or a deadline the caller set	Stop	Failure
reading the body fails, once service code has it	the call is service code's by then	Failure, or Ignored when the caller gave up

• A rejection is never retried at once, which would defeat the breaker, as the charter requires of every client.
• Every deadline counts against the dependency, as in bedrock's own example classifier: a dependency that hangs ends every call this way, and a breaker that ignored deadlines would never open. The deadline may be one the inbound request brought.
• A body that fails mid-read is a failure, whatever Classify said of its status: the dependency sent its headers and then failed.

And whatever Classify returns:

• a request that isn't safe to send twice, or can't be replayed, makes one attempt (Which requests are retried);
• a RetryAfter longer than BackoffMax ends the call, with the response as it came (Retry-After);
• a response whose body doesn't fit in 64 KiB isn't retried (The response body).

What DefaultSettings().Classify decides. Every response, and every error while the call's context is live:

An attempt's result	Retry	Breaker
a status below 500, other than 408 and 429	Success	Success
408 Request Timeout	Retry	Success
429 Too Many Requests	Retry, after Retry-After	Success
500 Internal Server Error	Stop	Failure
502 Bad Gateway, 504 Gateway Timeout	Retry	Failure
503 Service Unavailable	Retry, after Retry-After	Failure
any other 5xx, such as 501 Not Implemented	Stop	Success
a server certificate the client doesn't trust	Stop	Failure
any other error, such as refused, reset, a failed TLS handshake or AttemptTimeout	Retry	Failure

• A 4xx is the dependency working. It answered, and the request was wrong, so another attempt won't fix it, and the breaker counts the dependency healthy.
• 408 and 429 are retried, and don't trip the breaker. A 408 says the server gave up waiting for the request, which RFC 9110 says a client may repeat. A 429 is the server's own flow control: it is answering, and says when to come back. The budget bounds those retries, and Retry-After paces them.
• 500 trips the breaker but isn't retried. It is usually the server's bug, which a second attempt meets again, while a dependency returning many 500s should still be shed. 502, 503 and 504 say the dependency or what is in front of it is unavailable, which is what both retries and breakers are for.
• An attempt that runs past AttemptTimeout is retried. Its error wraps context.DeadlineExceeded while the call's context is live, so it reaches Classify, unlike the call's own deadline.
• The default reads only the status and headers. It never reads a body.

Changing it. The end user's Classify usually handles what their API means differently, and hands the rest to the default:

d := httpclient.DefaultSettings()
classify := d.Classify

settings.Classify = func(resp *http.Response, err error) httpclient.Classification {
	if resp != nil && resp.StatusCode == http.StatusConflict {
		// The ledger answers 409 while another write to the account is in
		// flight. That write is over by the next attempt.
		return httpclient.Classification{Retry: retry.Retry, Breaker: circuit.Success}
	}
	return classify(resp, err)
}

The function is the end user's own, so the same one can be set on every client whose dependencies share a convention. humus exports no classifier but the default in DefaultSettings, and no helpers to build one.

Retry-After

This section is replaced:

DefaultSettings().Classify reads Retry-After on the 429s and 503s it retries, and returns it as RetryAfter. Both forms of RFC 9110 are read: a number of seconds, and an HTTP-date, which is measured from now and clamped at 0. A malformed header is ignored. The wait is the longer of RetryAfter and the backoff. An end user's Classify returns whatever its API says, such as a RateLimit-Reset of its own.

A RetryAfter longer than BackoffMax, from any Classify, isn't waited for: the call ends, and the response goes to service code as it came. A server that asks for a minute is not one a request in flight should wait on, and that rule needs no setting of its own. A wait that would end after the call's deadline isn't started either: retry ends the call as Expired, and it returns the last response.

The response body

This is added at the start:

Classify may read it first. A Classify that reads the body closes it and leaves in resp.Body what service code is to read, such as the bytes it read. Both rules below apply to the body it leaves. Its read counts against Timeout, but not against AttemptTimeout, which ends with the headers.

The first sentence of "The breaker's ticket is settled by the body" becomes:

The breaker's ticket is settled by the body, as circuit's doc shows for an http.RoundTripper, with the Breaker outcome of the attempt's Classification.

Settings

This row is added to the table:

Field	Default	Valid	Check
Classify	the table under How an attempt's result counts	set	NewClient; set in code, not sourced

The bullet on TLS becomes:

• TLS is nil for plain text, as kafka.Settings.TLS and telemetry.OTLPSettings.TLS are. humus can't control what a service connects to, and a mesh's sidecar, a node-local agent or a cloud metadata endpoint is often plain HTTP. A URL names its scheme too, so NewClient requires the two to agree: an https BaseURL with a nil TLS, or an http one with a config, fails startup rather than quietly doing one of the two. Every humus server serves TLS, so a call to one is always https.

This bullet is added:

• Classify is set in code, like BreakerPolicy. What a dependency's answers mean is the end user's knowledge of their API, not an operator's choice. NewClient rejects a nil Classify, which would otherwise panic on the first call.

Telemetry

This is added to Metrics:

Whether humus should record instruments of its own, such as a gauge of each breaker's state and of each budget's tokens, is decided for every client at once by Proposal: humus's own metrics (wave 5).

Panics

This section is replaced:

The client recovers nothing. Nothing of the end user's runs inside a call but Classify and BreakerPolicy. A panic in either passes to the goroutine that made the call. circuit frees the attempt's slot and leaves the breaker consistent, retry holds nothing between attempts, and the client closes the attempt's response before the panic goes on, so its connection isn't held. A panic in the client itself is humus's bug.

Design decisions

The bullets "One client per dependency" and "The classification is fixed" become:

• One client per dependency. bedrock builds one breaker per dependency at startup and keeps no registry. A client bound to one origin is that, and its Settings are the dependency's: its own roots, timeouts, retry budget and Classify. fit expects the same of an injection point. It pools as a shared transport would, because net/http never shares a connection between origins. Rejected: one client for every origin, with a breaker and a budget made for each origin on its first call. It grows without bound, and puts every dependency on one set of settings.
• The classification is a function, Classify, from the start. APIs don't agree on what their answers mean, so nearly every client ends up classifying its own dependency. The default follows RFC 9110, and the end user's function wraps it. The client keeps what is about the call, not the API: the breaker's rejections, the caller giving up, deadlines and request safety, so a Classify can't defeat the breaker or double a charge. Rejected: a fixed table, with a field added when users ask. Every client of an API that differs would wait for it.

These are added:

• Plain text is allowed. humus can't control what a service connects to. TLS and the URL's scheme must agree, so plain text is always written out twice, never a default the operator fell into.
• No instruments of humus's own. otelhttp records every attempt, rejections included, and a breaker's transitions are log records. Whether humus adds instruments of its own is decided for every package at once, by Proposal: humus's own metrics (wave 5).

Testing

These bullets are added to what the stories add:

• An end user's Classify. A Classify that retries a 409 and hands the rest to the default makes a GET answered 409 and then 200 succeed, after two attempts. One that reads a 200's body, finds an error in it and returns retry.Retry makes a second attempt, and service code reads the second response's body whole.
• What the client decides stays the client's. A Classify that returns retry.Retry and circuit.Failure for everything still sends a POST without a key once. It is never asked about a rejection, so the open breaker's calls aren't retried. Calls whose caller cancels them leave the breaker closed.
• A RetryAfter beyond BackoffMax, returned by an end user's Classify, ends the call at once, after one request.
• NewClient rejects a nil Classify, naming the field, next to its other errors.
• A panic in Classify reaches the caller, and the attempt's response is closed.

Changes to other files

The docs page's list becomes: one client per dependency, built in Init and shared, and two clients of one origin; sourcing the settings; which requests are retried; the classification table and what the client decides itself; writing a Classify, with a 409 and with a body; Retry-After; the breaker and its log records; the body rules; HTTP/2 behind a ClusterIP Service; and testing against a fake dependency on a humustest Host.

The other clients

These answers reach past this proposal. Each other proposal records them when it is next revised:

• Plain text. The reason holds for every client. Outbound gRPC clients' open question 2 is the same question. kafka.Settings.TLS and RabbitMQ's Connection.TLS already mean plain text when nil.
• Classify. The reason holds for gRPC: a gRPC API picks its own codes too, such as Aborted for a conflict that a retry resolves. So outbound gRPC clients' revision gives grpcclient.Settings a Classify field of the same name, shaped for what a gRPC attempt ends with, its pushback included. It doesn't hold for producing to Kafka or for RabbitMQ's Publisher. They call a broker, whose protocol, not an API, fixes what its errors mean, so their classification stays fixed.
• Metrics. No wave-4 client records instruments of its own. Proposal: humus's own metrics (wave 5) decides it for all of them at once.

Open questions

All four are answered above. None are left.

Out of scope

These are added:

• Instruments of humus's own, such as gauges of the breaker's state and the budget's tokens: Proposal: humus's own metrics (wave 5).
• Classifiers for well-known APIs, and helpers to build one. A Classify is the end user's function.

[Zaba505]

Closing: outbound HTTP clients have landed

The revision's stories are done:

• story(httpclient): call one HTTP dependency through the Host with httpclient.NewClient #793 started httpclient, calling one HTTP dependency through the Host with httpclient.NewClient (story(httpclient): call one HTTP dependency through the Host with httpclient.NewClient #799);
• story(httpclient): trace and measure every attempt with the Setup's telemetry #794 traced and measured every attempt with the Setup's telemetry (story(httpclient): trace and measure every attempt with the Setup's telemetry #818);
• story(httpclient): retry a request that is safe to repeat, as the end user's Classify decides #795 retried requests that are safe to repeat, as the end user's Classify decides (story(httpclient): retry a request that is safe to repeat, as the end user's Classify decides #823);
• story(httpclient): a circuit breaker fails calls at once while the dependency keeps failing #796 added a circuit breaker that fails calls at once while the dependency keeps failing (story(httpclient): a circuit breaker fails calls at once while the dependency keeps failing #830);
• story(docs): document calling HTTP services #797 documented calling HTTP services (story(docs): document calling HTTP services #833).

One deviation: on a rejected attempt's http.client.request.duration data point, otelhttp v0.72.0 writes error.type as github.com/z5labs/bedrock/circuit.RejectedError rather than *circuit.RejectedError; the span carries *circuit.RejectedError. The package doc records the difference.

Revision 1 left no open questions.