Proposal: failure injection

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 5. It builds on bedrock's fit, on the wave-3 servers (HTTP server foundation, gRPC services, jobs, queue processing and the Kafka runtime), and on the wave-4 clients (outbound HTTP clients, outbound gRPC clients, producing to Kafka, RabbitMQ). It depends on no other wave-5 proposal. Whom a server trusts is decided without Proposal: authentication and authorization, so this proposal stays in wave 5, as the charter's revision 7 allows. It records no instrument, so it doesn't depend on Proposal: humus's own metrics either.

Summary

Failure injection lets one request fail on purpose, in production or in a test, without failing anyone else's. A request carries a failure scenario in its OpenTelemetry baggage. Each injection point the request passes through looks itself up in the scenario and learns whether to fail. bedrock's fit ships that model and leaves the conventions to humus. This proposal sets them:

• A new package, fault, builds an App's failure injection. The end user gives it a fit.Decoder, which owns the wire format, and a MaxDelay. The App builds it before Init, from a function the end user passes with humus.WithFaults, and hands it to every package in Setup.Faults. Without that option, failure injection is off and nothing changes.
• Every humus client is an injection point. httpclient, grpcclient, the Kafka Producer and the RabbitMQ Publisher evaluate a point named after their dependency, such as http:payments:8443 or kafka:orders. Each emulates an abort the way its protocol fails, inside the breaker and the attempt's deadline.
• A fault belongs to the request that carries it. A client doesn't retry an attempt that a fault ended, and its breaker doesn't count it. So a test request can't open a breaker that sheds real traffic, which a production scenario needs.
• Every humus server strips scenarios from callers it doesn't trust. That covers httpserver, grpcserver and the Kafka and RabbitMQ consumers. Each gets a TrustScenario func over its protocol's own request, set in code. The default trusts no caller.
• Every humus client strips the scenario before calling a dependency, unless ForwardScenario says that dependency takes part. Other baggage members always go through.
• Every evaluation is a fit.evaluation span event. Every strip is a fit.strip event. An aborted attempt fails with a *fault.InjectedError, whose error.type is the same in every client's spans and in otelhttp's metrics.
• An opt-in codec, fault/jsonscenario, carries a scenario as JSON in one baggage member whose key the end user names. Nothing makes an end user use it.

func main() {
	humus.Main(app.Init, humus.WithFaults(app.Faults))
}

// Faults is the service's failure injection.
func Faults(ctx context.Context, s *humus.Setup) (fault.Settings, error) {
	codec, err := jsonscenario.New("acme.fault")
	if err != nil {
		return fault.Settings{}, err
	}
	d := fault.DefaultSettings()
	return config.Struct(func(r *config.Resolver) fault.Settings {
		c := d
		c.Decoder = codec
		c.MaxDelay = r.Get(config.Env(s.Process.Environment, "FAULT_MAX_DELAY").
			Parse(time.ParseDuration).Default(d.MaxDelay).Validate(fault.CheckMaxDelay))
		return c
	}).Read(ctx)
}

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	payments := paymentsSettings // httpclient.Settings, sourced as usual
	payments.ForwardScenario = true // payments is ours, and honors acme.fault too
	client, err := httpclient.NewClient(s, payments)
	if err != nil {
		return nil, err
	}

	api := apiSettings // httpserver.Settings, sourced as usual
	api.TrustScenario = fromMesh // the end user's: a client certificate the mesh issued
	svc, err := httpserver.NewService(s, api, newAPI(client))
	if err != nil {
		return nil, err
	}
	return []humus.Service{svc}, nil
}

Background

• bedrock ships the model, and leaves the conventions to humus. fit v0.22.0 to v0.25.0 provide Scenario, Rule, Fault, the Decoder/Encoder split and NewEvaluator(decoder, maxDelay, observe). fit is unchanged between v0.24.0 and v0.25.0. Its doc gives a table of which call belongs at startup and which per request. The implementation comment on z5labs/bedrock#578 names what is left to humus: "A signed codec, the fit.evaluation observer, per-protocol injection points and Kafka header carriage." The review on #578 moved every wire convention into a caller-supplied codec. Its reason was that an organisation's edge, such as Envoy, Kong or Zuul, is usually owned by another team, and can't be made to produce bedrock's format.
• The charter scopes this proposal. Revision 7 asks for a codec, the observer, building the Evaluator from settings, injection point names and what an abort looks like per protocol, and stripping at every server and in every client. OpenTelemetry without globals left "stripping untrusted baggage, which Feature request: request-scoped failure injection (Netflix FIT) on OpenTelemetry baggage bedrock#578 calls for" to it.
• Every humus server accepts baggage from any caller today. The SDK's propagator is W3C trace context plus baggage (telemetry/sdk.go#L56). httpserver wraps every handler in otelhttp, and grpcserver uses otelgrpc's stats handler. Both extract it. The Kafka and RabbitMQ consumers extract it from each message's headers (kafka/service.go#L800, rabbitmq/service.go#L825). A job extracts it from BAGGAGE in its environment (job/service.go#L105). Every client writes it back out.
• Each landed package names its seam.
  • The servers' seams are the same in every case: after the extraction, and before the end user's code. For httpserver, that is inside the chain it wraps around the handler (service.go#L71). For grpcserver, it is after humus's recovery and before the end user's interceptors (internal/server/server.go#L106). For the consumers, it is between the extraction and Process (kafka/service.go#L817).
  • The clients' seams differ. In httpclient, it is around the call to the http.Transport, inside the attempt timeout (client.go#L119). In grpcclient, it is after Allow and before cc.Invoke or cc.NewStream (retry.go#L76, #L101). The Kafka Producer marks its seam before the breaker (producer.go#L294). The RabbitMQ Publisher marks its seam inside its attempt (publisher.go#L586).
  • Producing to Kafka left two things to this proposal: whether an injected abort counts against the breaker, and the point's name.
• Before the reset, humus had no failure injection. Its REST server registered the baggage propagator and accepted baggage from anyone (rest/rest.go#L256-L261), which is the exposure chore(deps): update module github.com/cenkalti/backoff/v5 to v7 #578's Trust section describes.

Proposed API

package fault // github.com/z5labs/humus/fault

// Settings configures an App's failure injection. Start from
// DefaultSettings: Decoder has no default, so it is always set.
type Settings struct {
	// Decoder reads the scenario a request carries from its baggage, and
	// strips it. It owns the wire convention: which members carry a
	// scenario, their grammar, size, expiry and trust. The edge that writes
	// scenarios and every service that reads them agree on it. It is set
	// in code, not sourced, and must be safe for concurrent use, as fit
	// requires.
	//
	// Default: none. Valid: set, which New checks.
	Decoder fit.Decoder

	// MaxDelay is the longest Delay any point waits. A longer Delay is
	// clamped to it, as fit.NewEvaluator does, so services with different
	// maximums still agree on the rest of a scenario.
	//
	// Default: 10s. Valid: positive. Check: CheckMaxDelay.
	MaxDelay time.Duration
}

// DefaultSettings returns the defaults. Its Decoder is nil, which New
// rejects.
func DefaultSettings() Settings

// CheckMaxDelay reports what is wrong with d without naming the field.
func CheckMaxDelay(d time.Duration) error

// Injector is an App's failure injection, which the App puts in
// Setup.Faults. Every humus server strips scenarios with it, and every
// humus client evaluates its point with it. The zero Injector is off:
// Eval returns the zero Fault and Strip returns its context unchanged. It
// is safe for concurrent use.
type Injector struct{ /* unexported */ }

// New checks settings, reporting every bad field in one error that names
// each one, and returns the Injector, whose fit.Evaluator is built from
// Decoder, MaxDelay and humus's observer.
func New(settings Settings) (Injector, error)

// Eval returns the Fault that the injection point named point emulates for
// the request ctx belongs to, as fit.Evaluator.Eval does. A request with
// no scenario gets the zero Fault. On a request with one, every
// evaluation, matched or not, is a fit.evaluation event on the span in
// ctx.
func (in Injector) Eval(ctx context.Context, point string) fit.Fault

// Strip returns ctx with the Decoder's members removed from its baggage,
// and whether there were any. Every other member stays.
func (in Injector) Strip(ctx context.Context) (context.Context, bool)

// InjectedError is the error of an attempt that a fault ended at Point.
// With Err nil, it is an abort, and it matches fit.ErrInjected with
// errors.Is. Otherwise it is a Delay that Err, the attempt's context's
// cause, cut short, and it matches Err. Its error.type is
// github.com/z5labs/humus/fault.InjectedError either way.
type InjectedError struct {
	Point string
	Err   error
}

func (e *InjectedError) Error() string
func (e *InjectedError) Unwrap() error
func (e *InjectedError) ErrorType() string

package humus

type App struct {
	// … Host, ShutdownGracePeriod, Drain, Telemetry, Init as today.

	// Faults returns the App's failure injection settings. Run calls it
	// after building telemetry and before Init, with the Setup that Init is
	// then given. Run builds the Injector with fault.New, and puts it in
	// Setup.Faults. An error from either fails startup the way an error
	// from Init does: ExitConfig for a *config.Error, ExitFailure
	// otherwise. nil, Main's default, means no failure injection.
	Faults func(ctx context.Context, s *Setup) (fault.Settings, error)
}

type Setup struct {
	// … Process, Host, Telemetry as today.

	// Faults is the App's failure injection, built from App.Faults before
	// Init. Every humus server and client takes it from the Setup it is
	// built with. When App.Faults is nil, it is the zero Injector, which
	// is off.
	Faults fault.Injector
}

// WithFaults sets App.Faults.
func WithFaults(f func(ctx context.Context, s *Setup) (fault.Settings, error)) Option

Each server gains a TrustScenario field, and each client gains a ForwardScenario field:

// httpserver.Settings
	// TrustScenario reports whether the failure scenario a request carries
	// reaches the handler. A request it doesn't trust has the scenario
	// stripped from its context first. It is asked only about a request
	// that carries a scenario, and only when the App has failure
	// injection. It is set in code, not sourced.
	//
	// Default: nil, which trusts no caller.
	TrustScenario func(r *http.Request) bool

// grpcserver.Settings: ctx holds the RPC's peer and incoming metadata.
	TrustScenario func(ctx context.Context, method string) bool

// kafka.Settings and rabbitmq.Settings, the consumers.
	TrustScenario func(m Message) bool

// httpclient.Settings, grpcclient.Settings, kafka.ProducerSettings and
// rabbitmq.PublisherSettings
	// ForwardScenario sends the failure scenario a call carries on to the
	// dependency, for a dependency that takes part in failure injection
	// with the same codec. When it is false, the scenario is stripped from
	// what the client sends, and the rest of the call's baggage still goes.
	//
	// Default: false. Valid: either.
	ForwardScenario bool

package jsonscenario // github.com/z5labs/humus/fault/jsonscenario

// Codec carries a fit.Scenario as JSON in one baggage member:
//
//	acme.fault={"rules":[{"point":"http:payments:8443","abort":true},{"point":"*","delay_ms":250}]}
//
// percent-encoded, as W3C Baggage requires. It is a fit.Decoder and a
// fit.Encoder, and is safe for concurrent use.
type Codec struct{ /* unexported: the key */ }

// New returns the Codec of the member named key, which must be a valid
// W3C baggage key. There is no default key.
func New(key string) (Codec, error)

func (c Codec) Decode(b baggage.Baggage) (fit.Scenario, bool, error)
func (c Codec) Strip(b baggage.Baggage) baggage.Baggage
func (c Codec) Encode(b baggage.Baggage, s fit.Scenario) (baggage.Baggage, error)

fault and fault/jsonscenario are in the core module. Its new dependency is github.com/z5labs/bedrock/fit v0.25.0, which needs only the OpenTelemetry API, already in the core module, and testify. The grpc, queue/kafka and queue/rabbitmq modules reach fault through the core module.

Building it

The end user's function sources the settings, as in the Summary, with whatever variable names the service uses. It is given the Setup, so a Decoder can load what it needs through s.Host, such as a registry of scenario IDs from a file, and register a refresher with s.OnStop.

Run calls it after telemetry and before Init, because every server and client copies s.Faults when it is built. If the field were set in Init, a server built before the line that set it would strip nothing. The function runs once, and its failure is reported as failure injection: …, through the lifecycle's ERROR record, like Init's.

Injection points

Each client evaluates one point per attempt, inside its breaker, so an attempt the breaker rejects evaluates nothing:

Client	Point	Where the fit.evaluation event goes	An abort	A delay
httpclient	http:<host>:<port> of BaseURL, with the scheme's port when the URL has none, such as http:payments:8443	the attempt's span, otelhttp's	a transport error: the call returns an *url.Error wrapping the *fault.InjectedError, with no response	waited inside AttemptTimeout, so it is in the attempt's span and in http.client.request.duration
grpcclient	grpc:<service>/<method>: the full method without its leading slash, as rpc.method writes it, such as grpc:payments.v1.Payments/Charge	the span in the call's context, the caller's: otelgrpc starts the attempt's span inside grpc-go	status Unavailable, wrapping the *fault.InjectedError. The attempt gets a span of the Client's own, named and attributed as a rejected attempt's is	waited before grpc-go is called, inside Timeout; cut short, it ends as grpc-go would, DeadlineExceeded or Canceled
Kafka Producer	kafka:<topic>, one evaluation per record	each record's span	every record of the call fails unsent. Each span records the error, and neither instrument counts it, as for a call the breaker rejected	the call waits the longest Delay of its records, inside DeliveryTimeout
RabbitMQ Publisher	rabbitmq:<messaging.destination.name>: <exchange>:<routing key>, leaving out what is empty, such as rabbitmq:orders:accepted	the Publish's span	the attempt fails with nothing sent: rabbitmq: publishing to …: fit: injected failure at …	waited inside Timeout

• A point names the dependency, from the attribute OpenTelemetry already records, as chore(deps): update module github.com/cenkalti/backoff/v5 to v7 #578 suggested. A rule can then target one dependency or all of them, such as http:payments:8443 or http:*, and the names line up with the trace. A gRPC point is per method, so grpc:payments.v1.Payments/* fails one service. A Kafka point is per topic. HTTP has no route template on the client side, so its point is the origin.
• A Kafka call is one unit. If any record's fault aborts, the whole call aborts, and every record fails with that point's *fault.InjectedError. This matches how Produce returns one error and how the breaker counts one call. The point moves from before the breaker, where Proposal: producing to Kafka #764 marked it, to just after Allow, so a Kafka abort is placed like every other client's.
• The end user's own code can be an injection point too. For example, a database call can do s.Faults.Eval(ctx, "postgresql:orders").Inject(ctx). The docs page suggests naming such a point <system>:<name>, after the protocol's *.system attribute.

A fault belongs to its request

An attempt that a fault ended is an abort, or a delay that the attempt's deadline or its caller cut short. Every client decides such an attempt itself, without asking the end user's Classify:

	Retry	Breaker
an attempt a fault ended	Stop	Ignored

So the call returns at once with the *fault.InjectedError, and the breaker's counts don't move. The retry budget isn't touched either, because a Stop outcome draws no token. An attempt that waited out its delay and then went to the dependency is classified as any other attempt is.

This is fit's model: "Only requests that carry a scenario are affected, so a failure can be injected into production traffic one request at a time." Under the default policy, five aborted attempts in a row would open a shared breaker and shed every other caller's requests to that dependency. The retries would also draw down the budget that real traffic shares. A scenario that should fail a dependency for everyone is a chaos experiment, not FIT, and it is out of scope.

The wave-4 proposals described the seam as one where "an injected abort is classified, counted and retried as a real failure would be". This proposal changes that, and its stories make the change in each client. Open question 1 asks whether that is right.

Stripping at servers

Each server strips a scenario right after it extracts the baggage, and before any of the end user's code runs, when the caller isn't trusted:

Server	Where	TrustScenario is given
httpserver	a layer between the panic layer and the handler, inside otelhttp, so a panic in TrustScenario is recorded as the handler's would be	the *http.Request
grpcserver	a unary and a stream interceptor, after humus's recovery and the streams interceptor, before the end user's interceptors	the RPC's context, with peer.FromContext and the incoming metadata, and its full method
Kafka consumer	after the record's span starts, before Process	the Message
RabbitMQ consumer	after the delivery's span starts, before Process	the Message

• TrustScenario is asked only about a request that carries a scenario, and only when the App has failure injection. It costs nothing on other requests. A stripped request's span gets a fit.strip event.

• nil trusts no caller. A server that the end user hasn't decided about is a trust boundary. So an API on the internet strips every scenario without any setting, and only a server the end user marks takes them.

• Trust needs nothing from authentication. Every protocol already gives the end user something to trust:

  • For mTLS, the verified client certificate: r.TLS.VerifiedChains, or credentials.TLSInfo from the gRPC peer.
  • A listener only internal callers reach.
  • A header that a gateway sets and strips from outside callers.
  • For Kafka, the topic, since the cluster's ACLs decide who can produce to it.
  • For RabbitMQ, Message.UserID. RabbitMQ validates user-id against the publishing connection's user, so it names the publisher.

  A trust func is the end user's code, and may call whatever the end user has, such as a verify function of their own. This proposal doesn't depend on Proposal: authentication and authorization. Its strip runs before any handler or interceptor of the end user's, so a principal which that proposal puts in the context isn't there yet when TrustScenario is asked.

• A job trusts its environment. The BAGGAGE that a job's run joins comes from its pod spec or whatever started it, and whoever sets that already sets every one of its settings. job gets no field and no code change, only a paragraph in its doc. A job started by a service hands on that service's baggage, which that service's own server has already stripped.

• The probes servers need nothing. httphealth's and grpchealth's own servers extract no baggage, as exposing health probes noted. An attached handler is behind its Service's strip, like every other request.

Stripping at clients

Each client strips the scenario from what it sends unless ForwardScenario is set. It evaluates its point first, because the point reads the scenario that is about to be stripped:

• httpclient strips in the injection layer. After otelhttp has written the attempt's headers, the layer clones the request, deletes the propagator's fields, and injects them again from the stripped context. That gives the attempt's span context and every baggage member except the scenario's. Proposal: outbound HTTP clients #762 put this strip in the retry layer, but there the point, which is inside, would no longer see the scenario.
• grpcclient strips the attempt's context before cc.Invoke or cc.NewStream, and otelgrpc injects from it.
• The Kafka Producer strips the context that each record's headers are injected from. The evaluation still uses the record's own context.
• The RabbitMQ Publisher strips the context its message's headers are injected from.

A stripped attempt's span gets a fit.strip event. Stripping is the default for the same reason servers trust no one by default. A dependency the end user hasn't decided about is a trust boundary. It may be a third party that would only log the scenario, or a service of another team with a different codec.

Telemetry

• The observer is a span event, modelled on OpenTelemetry's feature_flag.evaluation. It is added to trace.SpanFromContext(ctx), so the Injector needs no provider and touches no global. Every evaluation on a request that carries a scenario adds a fit.evaluation event:

  • always, fit.point;
  • when a rule matched, fit.rule (the rule's Point), fit.delay (the clamped Delay, in seconds) and fit.abort;
  • when the Decoder rejected the scenario, error.type and error.message. The request goes on unharmed, as fit requires.

  An unmatched evaluation still lists the point, which is how fit discovers the points a request passes through.

• A strip is a fit.strip event, with no attributes, on the span where it happens. The span's kind tells a server's strip from a client's.

• An aborted attempt's error.type is github.com/z5labs/humus/fault.InjectedError in every client, on spans and in otelhttp's http.client.request.duration. InjectedError implements ErrorType, because otelhttp's metrics name an error by its reflected type while semconv.ErrorType names a pointer *fault.InjectedError. The prototype found that mismatch. One value lets a query find every injected failure across protocols.

• No log records and no instruments. Evaluations happen only on requests under test, and each is on the request's trace. The failure an abort causes is already recorded by the client's own span, and by its metrics where a request was sent. An instrument of evaluations is out of scope.

• fit. is humus's namespace, as circuit. is for the breakers' records. OpenTelemetry has no convention for failure injection.

The codec

fault/jsonscenario is opt-in. Its grammar is {"rules":[{"point":…,"delay_ms":…,"abort":…}]}:

• point is required.
• delay_ms is a non-negative whole number of milliseconds, which any language can write.
• abort defaults to false.
• Unknown fields are rejected, so a typo such as "abrot" is an error the observer reports, not a rule that silently spares its point.
• Encode rejects a member over 2048 bytes percent-encoded, a quarter of W3C's 8192-byte guarantee. otel-go's baggage.Parse drops all of a header's baggage when it is over the limit, so an oversized scenario would wipe every other member downstream.
• Encode rejects a Delay that isn't whole milliseconds. Decode reads any size that arrives.
• It has no signature and no expiry. Trust comes from where a request came from, through TrustScenario and stripping at the edge. An end user who wants a signed format, a registry of scenario IDs, or Envoy's static two-member form writes a fit.Decoder, as bedrock's own examples do. The ones in fit's codec_test.go are each under 60 lines.

The edge, or a test, writes a scenario with fit.ContextWithScenario(ctx, codec, scenario), and the propagator carries it on. humus doesn't decide which requests are under test.

Panics

Nothing is recovered beyond what each package already recovers. A panic in TrustScenario is handled the way its server handles a panic of the end user's code. httpserver and grpcserver record it and fail that request, and the consumers recover nothing, as for a Processor. A panic in the Decoder passes through fit to the goroutine that evaluated it. A panic in fault itself is humus's bug.

Design decisions

• The package is fault. fit would clash with bedrock's package at every call site that uses both, and fit.Scenario and fit.Decoder are the types an end user writes. fault.Injector, fault.Settings and fault.InjectedError read plainly next to them. The charter calls the work "failure injection".
• The Injector is on Setup, like Telemetry. Every server must strip, and every client must evaluate. If the Injector were passed in each package's Settings, a server the end user forgot would strip nothing, and the scenario would reach clients that honor it. That hole can't be seen in review. On Setup, no package can miss it, and Apps that don't use failure injection pay nothing. It is built by a function given the Setup rather than by Init, so it exists before the first server is built. Open question 4.
• The Decoder is the end user's, and has no default. That is chore(deps): update module github.com/cenkalti/backoff/v5 to v7 #578's ruling: the organisation owns the wire format. jsonscenario is offered the way openapi/v3/jsoncodec is, as a ready-made choice that nothing requires.
• jsonscenario isn't jsoncodec. An edge service that serves an OpenAPI API and writes scenarios would import both, and two packages named jsoncodec would need an alias.
• Trust is a func per server, typed over the protocol's own request, and set in code like Classify and BreakerPolicy. One App-wide policy over a common caller type would be a closed set of protocols in the core module, and couldn't name gRPC's peer or a Kafka Message without importing those modules. A bool would trust every caller of a server or none of them, and one port often serves both a mesh and a gateway.
• Defaults fail safe on both sides. A server trusts no caller, and a client forwards to no dependency. An end user adopting failure injection marks each boundary they mean to open. A scenario aimed at a service's own clients needs neither field to be forwarded, only its own server's trust. Open question 3.
• A fault belongs to its request: see above, and open question 1.
• Points come from attributes OpenTelemetry already records, so a scenario's rules read like the trace they fail. Open question 5.
• The fit.evaluation event goes on the span in the point's context. For httpclient and the Kafka Producer, that is the span of the attempt or record that the fault fails. For grpcclient, it is the caller's, because grpc-go gives no hook between otelgrpc's span and the transport. The Client's own span for an aborted attempt still records the failure.
• InjectedError covers a delay cut short. The client needs to know that the fault ended the attempt, so the breaker can ignore it. The caller still sees the deadline it hit, through Unwrap. It isn't fit.ErrInjected, which fit defines for an abort.
• No setting turns failure injection on or off at run time. MaxDelay is the operator's knob. A Decoder that rejects every scenario, such as a registry with no entries, turns it off in effect without a rebuild.

Testing

A prototype of this proposal passed the tests below under go test -race -count=3, on Go 1.27.0 and fit v0.25.0, against main at 265d361. It covered fault, jsonscenario, the root package, httpserver, httpclient, grpcserver, grpcclient, the Kafka Producer and Service, and the RabbitMQ Publisher. Each test except RabbitMQ's runs whole Apps inside a testing/synctest bubble, on one humustest Host, over TLS from humustest.NewCA. Backoffs, delays and timeouts take no real time. Kafka's run against kfake, which franz-go keeps. RabbitMQ's run the Publisher on the package's unit-test stand-in, since humus keeps no broker. The RabbitMQ consumer's strip is the Kafka consumer's, and the prototype compiled it without running it, because that needs a container. Every existing unit test in the four modules still passed. The one exception was grpcserver's export hygiene test, which caught the prototype's wrapped stream type, so the stories put the interceptors in grpc/internal/server, beside the streams interceptor. The prototype was in a scratchpad and is not kept, and the sketches below are trimmed from it.

The service under test, orders, serves an API with httpserver and calls a fake payments dependency with httpclient. The test is the edge: it sends a request whose baggage carries a scenario and a tenant member.

// edge sends GET /orders/7/pay to orders, its baggage carrying s, if any,
// written with the codec, and tenant=acme.
func edge(t *testing.T, h *humustest.Host, ca *humustest.CA, s *fit.Scenario) int {
	m, err := baggage.NewMemberRaw("tenant", "acme")
	require.NoError(t, err)
	b, err := baggage.New(m)
	require.NoError(t, err)
	ctx := baggage.ContextWithBaggage(context.Background(), b)
	if s != nil {
		codec, err := jsonscenario.New("acme.fault")
		require.NoError(t, err)
		ctx, err = fit.ContextWithScenario(ctx, codec, *s)
		require.NoError(t, err)
	}
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://orders:9443/orders/7/pay", nil)
	require.NoError(t, err)
	propagation.Baggage{}.Inject(ctx, propagation.HeaderCarrier(req.Header))
	c := h.HTTPClient()
	c.Transport.(*http.Transport).TLSClientConfig = &tls.Config{RootCAs: ca.Pool()}
	resp, err := c.Do(req)
	require.NoError(t, err)
	require.NoError(t, resp.Body.Close())
	return resp.StatusCode
}

Ten aborted requests leave the breaker closed. A counted fault would have opened it after five:

func TestFaults_a_trusted_abort_fails_one_attempt_and_the_breaker_ignores_it(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		ca := humustest.NewCA(t)
		var payments seen
		defer startPayments(t, h, ca, &payments)()
		tel := humustest.NewTelemetry()
		var calls callResult
		stop := startOrders(t, h, ca, tel, ordersOpts{faults: true, trust: func(*http.Request) bool { return true }}, &calls)

		abort := &fit.Scenario{Rules: []fit.Rule{{Point: "http:payments:8443", Fault: fit.Fault{Abort: true}}}}
		for range 10 {
			require.Equal(t, http.StatusBadGateway, edge(t, h, ca, abort))
		}
		require.Equal(t, http.StatusOK, edge(t, h, ca, nil)) // no scenario: the breaker is closed
		stop()

		require.Equal(t, 1, payments.requests) // the aborted calls sent nothing
		var ie *fault.InjectedError
		require.ErrorAs(t, calls.errs[0], &ie)
		require.ErrorIs(t, calls.errs[0], fit.ErrInjected)
		require.Equal(t, "http:payments:8443", ie.Point)

		clients := spansOfKind(tel, trace.SpanKindClient)
		require.Len(t, clients, 11) // one attempt per aborted call: no retries
		ev := eventsNamed(clients[0], "fit.evaluation")[0]
		require.Contains(t, ev.Attributes, attribute.String("fit.point", "http:payments:8443"))
		require.Contains(t, ev.Attributes, attribute.Bool("fit.abort", true))
		require.Contains(t, clients[0].Attributes(), attribute.String("error.type", "github.com/z5labs/humus/fault.InjectedError"))
	})
}

With TrustScenario nil, the scenario never reaches the handler, and the dependency still sees the other members:

func TestFaults_an_untrusted_scenario_is_stripped_at_the_server(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		ca := humustest.NewCA(t)
		var payments seen
		defer startPayments(t, h, ca, &payments)()
		tel := humustest.NewTelemetry()
		var calls callResult
		stop := startOrders(t, h, ca, tel, ordersOpts{faults: true, forward: true}, &calls)
		require.Equal(t, http.StatusOK, edge(t, h, ca, abortPayments()))
		stop()

		require.Equal(t, []string{""}, payments.scenario)
		require.Equal(t, []string{"acme"}, payments.tenant)
		server := spansOfKind(tel, trace.SpanKindServer)[0]
		require.Len(t, eventsNamed(server, "fit.strip"), 1)
	})
}

The prototype's tests also showed the following:

• ForwardScenario. With it false, a trusted scenario that spares payments reaches orders' handler, but payments sees no acme.fault member, and the attempt's span has a fit.strip event. With it true, payments sees the member. Either way, it sees tenant=acme and a traceparent of the attempt's span.
• A delay. A 1s delay at http:payments:8443 makes the attempt's span at least 1s long, and the call succeeds. An 8s delay is cut short by AttemptTimeout at 5s, after one attempt. The handler gets an *fault.InjectedError that matches context.DeadlineExceeded, and not fit.ErrInjected.
• Off. With no humus.WithFaults, nothing is stripped, nothing is evaluated, and the scenario reaches payments as any baggage does.
• The metric. The ten aborted attempts are one http.client.request.duration data point with error.type github.com/z5labs/humus/fault.InjectedError and a count of 10.
• gRPC. front is a grpcserver whose handler calls backend through grpcclient.
  • An abort of grpc:grpc.testing.TestService/EmptyCall returns Unavailable to the edge, ten times, and a clean call still goes through.
  • Each aborted attempt has one client span with the error.type, and its fit.evaluation event is on front's server span.
  • An untrusted call is stripped, with fit.strip on the server span.
  • ForwardScenario works as for HTTP.
• Kafka.
  • A job's BAGGAGE, set with humustest.WithEnv, aborts ten Produce calls to kafka:orders. Nothing reaches the topic, each record's span has the evaluation and the error.type, and an eleventh call with no scenario is stored.
  • With ForwardScenario false, the stored record's baggage header has tenant and no acme.fault.
  • A consumer with TrustScenario nil hands Process a context without the scenario. With func(m kafka.Message) bool { return m.Topic == topic }, it hands over the scenario.
• RabbitMQ. On the stand-in, ten aborted Publish calls to rabbitmq:orders:accepted dial nothing, and an eleventh is confirmed. ForwardScenario decides whether the published headers carry the member.

The stories add the tests the prototype didn't have:

• fault. CheckMaxDelay accepts a positive value and rejects the rest without naming the field. New names each bad field, and reports a nil Decoder and a bad MaxDelay together. The zero Injector evaluates nothing and strips nothing. A rejected scenario's event has error.type and error.message, and its request proceeds. A Delay over MaxDelay is clamped, and its event shows the clamped value. An unmatched evaluation has fit.point alone. Strip keeps every other member.
• The App. Faults runs after telemetry and before Init, with the same Setup. A *config.Error from it exits 78, and any other error exits 1, with one ERROR record and the OnStop functions run. A nil Faults leaves the zero Injector.
• jsonscenario. It round-trips through propagation.Baggage. Unknown fields, a missing point, a negative delay_ms and trailing data are errors. Encode rejects a member over 2048 bytes and a Delay that isn't whole milliseconds. New rejects an invalid key. Strip leaves other members alone.
• gRPC streams. A stream's open is aborted like a unary attempt. A stripped stream's handler and the end user's stream interceptors see the stripped context.
• The RabbitMQ consumer, in the container job: an untrusted delivery is stripped, and a trusted one, by UserID, keeps its scenario.
• Every client's breaker ignores a delay cut short, as it ignores an abort.

Each new package has an Example, and is held to the 80% coverage gate.

Changes to other files

This proposal owns failure injection, so its stories make these:

• The root package. App.Faults, WithFaults and Setup.Faults. Run builds the Injector between telemetry and Init. Main's doc lists Faults: none among its defaults.
• The core module's go.mod gains github.com/z5labs/bedrock/fit v0.25.0, and so do the go.mod files of the grpc, queue/kafka and queue/rabbitmq modules, through the core module.
• httpserver: Settings.TrustScenario, and the strip layer.
• httpclient: Settings.ForwardScenario, the injection layer inside the attempt timeout, and the client deciding an attempt a fault ended.
• grpc/grpcserver and grpc/internal/server: Settings.TrustScenario, and the strip interceptors.
• grpc/grpcclient: Settings.ForwardScenario, the point in attempt and open, the aborted attempt's span, and the Client deciding an attempt a fault ended.
• queue/kafka: Settings.TrustScenario, ProducerSettings.ForwardScenario, the point moved after Allow, and the classifier ignoring an injected call. The marked seam's comment is updated.
• queue/rabbitmq: Settings.TrustScenario, PublisherSettings.ForwardScenario, the point in attempt, and both classifiers. The marked seam's comment is updated.
• job: a paragraph in its doc saying that the run's BAGGAGE is trusted, and why.
• A docs page, "Failure injection", at docs/content/failure-injection.md. It covers:
  • the model;
  • humus.WithFaults and sourcing MaxDelay;
  • writing or choosing a codec;
  • the points table, and naming the end user's own points;
  • trust at each server, with the mTLS, topic and UserID examples;
  • ForwardScenario;
  • why a fault doesn't move a breaker;
  • the events and the error.type;
  • testing a scenario through a humustest Host, a job's BAGGAGE, and a Kafka record's baggage header.
• Each package's docs page (HTTP server, HTTP client, gRPC, gRPC client, Kafka, RabbitMQ, jobs) gains a paragraph that links to it.
• The lifecycle page adds Faults to the startup order, after telemetry and before Init.
• The root package's "Being rebuilt" paragraph, the README and the docs site's index name the package.

telemetry, health, httphealth, grpchealth, openapi/v3 and humustest don't change. A test turns failure injection on with humustest.WithApp(humus.WithFaults(…)). A unit test of the end user's code that takes a Setup sets Setup.Faults from fault.New.

Migration from today

Before the reset	With this proposal
no failure injection	humus.WithFaults, a fit.Decoder, and MaxDelay
REST registered the baggage propagator and kept every caller's baggage	every server strips scenarios from callers its TrustScenario doesn't trust; other members are kept
otelhttp forwarded all baggage to every dependency	every client strips the scenario unless ForwardScenario is set
(none)	points in every humus client, named after their dependency; s.Faults.Eval for the end user's own
(none)	fit.evaluation and fit.strip span events, and one error.type for injected failures

There was nothing else to migrate.

Open questions

1. Should a fault move the breaker and the retries? As proposed, an attempt that a fault ended is Stop and Ignored. The call returns at once, the breaker's counts don't move, and no retry token is drawn. So a scenario affects only its own request, which is fit's promise and what production use needs. The alternative is what the wave-4 proposals sketched: retry and count it like a real failure. A scenario could then show retries and an opening breaker, but a few test requests would shed real traffic and drain the shared budget. A middle way retries and still ignores it in the breaker, which keeps the budget leak. Recommendation: Stop and Ignored. The clients' own tests already cover their retries and breakers against real failures.
2. Ship jsonscenario? As proposed, humus ships one opt-in codec: JSON in one member whose key the end user names, with no signature and no expiry, because trust comes from TrustScenario. The alternative ships no codec, leaving every end user to write a Decoder as bedrock's examples do, or ships a signed codec too, which every edge would then have to compute. Recommendation: ship jsonscenario alone. A service and its test harness need a shared format to start with, and a signature needs key distribution that humus doesn't own.
3. Fail-safe defaults on both sides? As proposed, a server with a nil TrustScenario trusts no caller, and a client with ForwardScenario false strips. Taking part across services means marking each server and each client on the path. The alternative trusts and forwards by default, which is less to set but exposes every internet-facing server that forgot. Recommendation: fail safe.
4. The Injector on Setup, built before Init? As proposed, App.Faults builds it and every package takes it from the Setup. The alternative passes it explicitly, in a field of every server's and client's Settings, with no change to the root package. That is more explicit, but a missed server is a silent hole. Recommendation: on Setup.
5. The point names. As proposed: http:<host>:<port>, grpc:<service>/<method>, kafka:<topic> and rabbitmq:<exchange>:<routing key>. HTTP could instead leave out the port when it is the scheme's default, which is shorter but gives one dependency two spellings. Recommendation: as proposed, always with the port.

Out of scope

• Deciding which requests are under test, by account, device or percentage, and writing scenarios at an edge. That is the end user's edge, with fit.ContextWithScenario and an Encoder.
• A control plane, or distributing scenarios or registries.
• Signed scenarios, expiry, and registries of scenario IDs. These are a Decoder of the end user's.
• Failing a dependency for every request, or opening a breaker for every caller. Those are chaos experiments, such as Envoy's fault filter, not request-scoped injection.
• Kinds of abort, such as a throttle as against an outage. fit has only Abort, and chore(deps): update module github.com/cenkalti/backoff/v5 to v7 #578 left error kinds out.
• Per-route HTTP points. The client knows no route template. The end user evaluates a point of their own where they need one.
• Points in the OTLP exporters, the probes, or grpcclient's Connected, none of which serves a request.
• Stripping baggage members other than a scenario's before calling a third party.
• An instrument of evaluations. Each is a span event on its request's trace.
• Trusting a caller because authentication says who it is, inside humus. A TrustScenario func may do it with whatever the end user has.
• Showing failure injection in a walkthrough, such as a Kafka processor's transient failures: Proposal: walkthroughs and a humus MIGRATING.md (wave 6).

References

• Proposal: humus on the bedrock process model, the charter, and its revision 7 (failure injection's scope, and why it doesn't depend on authentication) and revision 8 (the seams wave 3 hands on)
• bedrock's fit v0.25.0, and z5labs/bedrock#578: its review, its startup and per-request table, and what it leaves to humus
• OpenTelemetry without globals and its revision 2: the propagator, and stripping left to this proposal
• The seams: HTTP server foundation, gRPC services, jobs (TRACEPARENT and BAGGAGE through the SDK's propagator), queue processing and the Kafka runtime, outbound HTTP clients, outbound gRPC clients, producing to Kafka, RabbitMQ, exposing health probes
• Netflix, FIT: Failure Injection Testing (2014)
• W3C Baggage, and OpenTelemetry's feature_flag.evaluation event, which fit.evaluation is modelled on
• RabbitMQ's validated user-id

[Zaba505]

Revision 1: every name humus defines starts with humus.

Every telemetry name humus defines of its own now starts with humus.: instruments, attributes, and the names of span events. This is a standard for every proposal. Proposal: humus's own metrics' revision 1 set it, for its instruments and the breakers' log records, and gives the reasons. A name generic enough for another library to use too can't collide once it's prefixed. Most of these series end in Prometheus, where the instrumentation scope is only a label. OpenTelemetry's naming rules recommend a prefix for names it doesn't define, and libraries that define their own use their own name, as gRPC, Resilience4j and Polly do. Its revision 2 extends it to health.probe.

This proposal's names take the prefix:

In the post	Now
fit.evaluation, the span event	humus.fit.evaluation
fit.strip, the span event	humus.fit.strip
fit.point	humus.fit.point
fit.rule	humus.fit.rule
fit.delay	humus.fit.delay
fit.abort	humus.fit.abort

Wherever the post names one of them, in its prose, its sketches and its tests, read the new name. That includes the Summary, Telemetry, Injection points, Design decisions, Testing and Migration from today. Where Background quotes bedrock's "the fit.evaluation observer", it quotes bedrock, which left the name to humus.

These keep their names:

• OpenTelemetry's own, such as error.type, error.message and feature_flag.evaluation, which humus.fit.evaluation is modelled on.
• The end user's baggage member, acme.fault in the sketches. Its key is the end user's to name, and jsonscenario.New takes it.
• The point names, such as http:payments:8443. They are values, not names.

Nothing else changes. Every open question is still open.

[Zaba505]

Revision 2: the open questions answered, and a fault is a real failure to retries and breakers

The open questions:

1. A fault is a real failure. Retries and breakers know nothing of failure injection. The client's classifier decides an attempt that a fault ended, the end user's Classify included. It decides that attempt as it decides the failure the fault emulates. So the attempt is retried and counted as that failure would be. This reverses the recommendation. FIT is for watching how the whole service handles a failing dependency: its retries, its breaker and its fallbacks, not only the call that fails. The scenario still decides which requests carry a fault. What the fault then does to state those requests share with others, such as a breaker or the retry budget, is real. The wave-4 proposals' sketch stands: "an injected abort is classified, counted and retried as a real failure would be".

   Netflix worked this way. Its ChAP paper (Basiri et al., ICSE SEIP 2019) says FIT injects into Hystrix commands as well as RPC clients. Its stated motive is that "timeouts and fallbacks … are not exercised as frequently as the happy path". The resilience mechanisms reacted for real. 900 ms of latency injected into a Hystrix command exhausted its thread pool, "resulting in the Hystrix command serving a fallback. This leads to a short-circuiting behavior where the Hystrix command unconditionally serves fallbacks for a period of time." That is how a timeout set too high for its thread pool was found. ChAP contained the effect in its deployment, not in the libraries. It routes an experiment's users to a canary cluster of their own, 1% of the original cluster's size, beside a baseline cluster for the control group.

2. fault/jsonscenario ships, as recommended.

3. The defaults fail safe on both sides, as recommended.

4. The Injector is on Setup, built before Init, as recommended. Evaluating is stateless:

   • An Injector holds only its fit.Evaluator. That holds the Decoder, MaxDelay and humus's observer, all fixed when New builds it.
   • fit's Eval decodes the scenario from the request's baggage on every call, and keeps nothing between calls.
   • With jsonscenario, the whole scenario is in its baggage member, and a Codec holds only its key.

   A Decoder of the end user's may keep state of its own, such as a registry of scenario IDs. That is their choice, made in their Faults function, with s.Host and s.OnStop, as Building it says.

5. The point names are as proposed, always with the port.

These changes weren't prototyped: the prototype showed Stop and Ignored. The stories' tests, listed in Testing below, cover the new behaviour.

Each section below replaces or adds to the part of the same name in the post.

Summary

The third bullet becomes:

• A fault is a real failure to the client that meets it. The client's classifier decides an attempt that a fault ended as it decides the failure the fault emulates. So the attempt is retried, and the breaker counts it. A scenario then shows how the whole service handles a failing dependency. Containing what a fault does to a breaker other requests share is the deployment's job, not humus's.

Injection points

In the table, the RabbitMQ Publisher's abort becomes: the attempt fails with nothing sent, and is marked as a lost connection's is, so both classifiers decide it as one: rabbitmq: publishing to …: fit: injected failure at …. Marking an error lost changes neither its message nor its error.type. An abort emulates a dependency that can't be reached in every client: a transport error in httpclient, Unavailable in grpcclient, and a lost connection here.

The bullet on Kafka gains a sentence. The point moves to just after Allow so that the breaker counts an aborted call.

A fault is a real failure

This section replaces A fault belongs to its request.

Every client decides an attempt that a fault ended with its own classifier, as it decides any other attempt. The fault emulates a failure of the dependency, and the classifier sees that failure:

Client	An abort	A delay cut short
httpclient	a transport error: Retry, Failure, under the default Classify	Retry, Failure, as AttemptTimeout's own error is
grpcclient	Unavailable: Retry, Failure, under the default Classify	DeadlineExceeded: Stop, Failure. Canceled: Stop, Ignored
Kafka Producer	Failure. The Producer retries nothing of its own, and franz-go is never reached	by DeliveryTimeout: Failure, as records not stored in time are
RabbitMQ Publisher	as a lost connection: Retry, Failure	Retry, which ends as Expired. Failure when the deadline passed, Ignored when the caller cancelled

• A retried attempt meets the same fault. Each attempt evaluates the point again, and the scenario hasn't changed. So a call that a scenario aborts fails like a call to a dependency that is down. Its attempts run out, or the retry budget or the breaker stops them.
• The breaker opens for everyone. Under the default policy, five aborted attempts in a row open the client's breaker. It then sheds every caller's requests to that dependency until it probes again, not only the requests under test. The retries draw on the budget that real traffic shares. That is what a dependency that is down does to the service, and seeing it is the point.
• The end user can still spare their breaker. The end user's Classify, in httpclient and grpcclient, is asked about these attempts like any other. One that answers Ignored for a *fault.InjectedError keeps the breaker closed for that client. humus doesn't.
• Proposal: humus's own metrics's instruments count these attempts like any other.

No client's classifier changes. The stories add only the point, and the RabbitMQ abort's mark.

Design decisions

"A fault belongs to its request: see above, and open question 1" becomes:

• A fault is a real failure. Retries and breakers are oblivious to it: see A fault is a real failure. A scenario is there to show how the service handles a failing dependency, and its retries and its breaker are part of that. Netflix's FIT went through Hystrix the same way.

The bullet on InjectedError becomes:

• InjectedError covers a delay cut short, so every injected failure has one error.type, and a query can tell one from a real timeout. The caller and the classifier still see the deadline it hit, through Unwrap. It isn't fit.ErrInjected, which fit defines for an abort.

Testing

The prototype's first sketch, and its results for a delay, gRPC, Kafka and RabbitMQ, showed Stop and Ignored, which this revision drops. The stories' tests show the following instead, each with the default policy:

• HTTP.
  • An aborted GET is retried up to MaxAttempts, as a transport error is. Every attempt is aborted, and none reaches payments.
  • After the fifth aborted attempt in a row, the breaker opens, and the open breaker sheds a request with no scenario.
  • Each aborted attempt is a client span with the error.type, and http.client.request.duration counts every one.
• A delay. An 8s delay at http:payments:8443 is cut short by AttemptTimeout, retried, and cut short again, until Timeout ends the call. The breaker counts each cut attempt as a Failure. The handler gets an *fault.InjectedError that matches context.DeadlineExceeded.
• gRPC. An abort of grpc:grpc.testing.TestService/EmptyCall is retried as Unavailable is, and opens the breaker as it does.
• Kafka. Aborted Produce calls are Failures. Once the breaker opens, it rejects the next call, which carries no scenario.
• RabbitMQ. An aborted Publish is retried as a lost connection is, and sends nothing. Its attempts open the breaker as lost ones do.
• The end user's Classify. A Classify that answers Stop, Ignored for a *fault.InjectedError keeps httpclient's breaker closed.

"Every client's breaker ignores a delay cut short, as it ignores an abort" becomes: every client classifies an abort and a delay cut short as A fault is a real failure's table says.

Changes to other files

• httpclient: Settings.ForwardScenario, and the injection layer inside the attempt timeout. The client no longer decides an attempt that a fault ended itself.
• grpc/grpcclient: Settings.ForwardScenario, the point in attempt and open, and the aborted attempt's span. The client no longer decides an attempt that a fault ended itself.
• queue/kafka: Settings.TrustScenario, ProducerSettings.ForwardScenario, and the point moved after Allow. The marked seam's comment is updated. The classifier is unchanged.
• queue/rabbitmq: Settings.TrustScenario, PublisherSettings.ForwardScenario, the point in attempt, and the abort marked lost. The marked seam's comment is updated. The classifiers are unchanged.
• The docs page replaces "why a fault doesn't move a breaker" with two topics:
  • how a fault moves the retries and the breaker as a real failure does;
  • containing a scenario in production, by choosing the requests under test and where they run.

Out of scope

"Failing a dependency for every request, or opening a breaker for every caller" becomes:

• Failing a dependency for every request without a scenario, as Envoy's fault filter does. A scenario can still open a breaker that every caller shares, as a real failure would.
• Containing a scenario's effect on state it shares with other requests, such as a breaker or the retry budget. Routing requests under test to instances of their own, as ChAP does, is the end user's deployment.

References

Two references are added:

• Basiri et al., Automating chaos experiments in production (ICSE SEIP 2019): ChAP on FIT, Hystrix's fallbacks and short-circuiting under injected latency, and the baseline and canary clusters
• Alvaro et al., Automating Failure Testing Research at Internet Scale (SoCC 2016): FIT's failure scope, and request-level injection to control the blast radius

[Zaba505]

Revision 3: a delayed HTTP call that Timeout ends returns net/http's timeout error

The httpclient story, #894, found that revision 2's delay test can't pass. When an http.Client's Timeout fires, net/http's Client.do replaces the error the transport returned with a timeout error of its own, which keeps only the message text. That error matches context.DeadlineExceeded, but has no *fault.InjectedError in its chain. httpclient's Timeout is the http.Client's, so a call that Timeout ends returns net/http's error, whatever cut its last attempt short.

Timeout stays the http.Client's. Enforcing it in the client's own retry layer instead would rebuild what net/http already does for the whole exchange, reading the body included, and would change what Settings.Timeout is documented to be. The call did time out, and the error says so. The attempts still say a fault cut them short.

This wasn't prototyped: the prototype's delay ended after one attempt, which revision 2 dropped. #894's test covers it.

Each section below replaces or adds to the part of the same name in the post, as revision 2 left it.

Testing

Revision 2's "A delay" bullet becomes:

• A delay. An 8s delay at http:payments:8443 is cut short by AttemptTimeout, retried, and cut short again, until Timeout ends the call. The breaker counts each cut attempt as a Failure. Each cut attempt's span has the humus.fit.evaluation event and the error.type github.com/z5labs/humus/fault.InjectedError. The handler gets net/http's timeout error, which matches context.DeadlineExceeded, and not fit.ErrInjected.

Design decisions

Revision 2's bullet on InjectedError gains a sentence:

• InjectedError covers a delay cut short, so every injected failure has one error.type, and a query can tell one from a real timeout. The caller and the classifier still see the deadline it hit, through Unwrap. It isn't fit.ErrInjected, which fit defines for an abort. In httpclient, a call that Timeout ends returns net/http's timeout error instead, which matches context.DeadlineExceeded. Its attempts' spans still carry InjectedError's error.type, so a query still tells it from a real timeout.

Nothing else changes.

[Zaba505]

Closing: failure injection has landed

The revisions' stories are done:

• story(fault): build an App's failure injection with fault.New, and turn it on with humus.WithFaults #890 built an App's failure injection with fault.New, turned on with humus.WithFaults and handed to every package in Setup.Faults (story(fault): build an App's failure injection with fault.New, and turn it on with humus.WithFaults #901);
• story(jsonscenario): carry a failure scenario as JSON in one baggage member the end user names #891 carried a scenario as JSON in one baggage member the end user names, with fault/jsonscenario (story(jsonscenario): carry a failure scenario as JSON in one baggage member the end user names #903);
• story(httpserver): strip a failure scenario from every request whose caller TrustScenario doesn't trust #892 stripped a scenario from every request whose caller httpserver's TrustScenario doesn't trust (story(httpserver): strip a failure scenario from every request whose caller TrustScenario doesn't trust #906);
• story(grpcserver): strip a failure scenario from every RPC whose caller TrustScenario doesn't trust #893 did the same for every RPC in grpcserver, with interceptors in grpc/internal/server ahead of the end user's (story(grpcserver): strip a failure scenario from every RPC whose caller TrustScenario doesn't trust #921);
• story(httpclient): fail or delay an attempt that a failure scenario names, and strip the scenario unless ForwardScenario is set #894 failed or delayed an httpclient attempt at http:<host>:<port>, and stripped the scenario unless ForwardScenario is set. Under revision 3, a call that Timeout ends returns net/http's timeout error, and its attempts' spans carry InjectedError's error.type (story(httpclient): fail or delay an attempt that a failure scenario names, and strip the scenario unless ForwardScenario is set #928);
• story(grpcclient): fail or delay an attempt that a failure scenario names, and strip the scenario unless ForwardScenario is set #895 did the same in grpcclient at grpc:<service>/<method>. An aborted attempt and a delay cut short each get a span of the Client's own (story(grpcclient): fail or delay an attempt that a failure scenario names, and strip the scenario unless ForwardScenario is set #934);
• story(kafka): fail or delay a Produce call that a failure scenario names, and strip scenarios from untrusted records #896 failed or delayed a Kafka Produce call at kafka:<topic>, as one unit, and stripped scenarios from records the Service doesn't trust (story(kafka): fail or delay a Produce call that a failure scenario names, and strip scenarios from untrusted records #936);
• story(rabbitmq): fail or delay a Publish that a failure scenario names, and strip scenarios from untrusted deliveries #897 failed or delayed a RabbitMQ Publish at rabbitmq:<exchange>:<routing key>, with an abort marked as a lost connection's is, and stripped scenarios from deliveries the Service doesn't trust (story(rabbitmq): fail or delay a Publish that a failure scenario names, and strip scenarios from untrusted deliveries #938);
• story(docs): document failure injection, from turning it on to containing a scenario in production #898 documented failure injection, from turning it on to containing a scenario in production, on a new page and in every package's page (story(docs): document failure injection, from turning it on to containing a scenario in production #942).

The revisions left no open questions. A fault is a real failure to the retries and breakers of the client that meets it. fault/jsonscenario ships, opt-in. The defaults fail safe on both sides: a server trusts no caller, and a client strips. The Injector is on Setup, built before Init. The point names are as proposed, always with the port.

The rest is out of scope or belongs to other proposals, as "Out of scope" lists with revision 2's changes:

• Showing failure injection in a walkthrough is Proposal: walkthroughs and a humus MIGRATING.md (wave 6).
• Deciding which requests are under test, and writing scenarios at an edge, is the end user's edge, with fit.ContextWithScenario and an Encoder.
• Signed scenarios, expiry and registries of scenario IDs are a Decoder of the end user's.
• Containing a scenario's effect on a breaker or retry budget that other requests share is the end user's deployment.
• A control plane; failing a dependency for every request without a scenario; kinds of abort; per-route HTTP points; points in the OTLP exporters, the probes or Connected; stripping other baggage members; an instrument of evaluations; and trusting a caller by authentication inside humus stay out of scope.