Proposal: failure injection for Kubernetes controllers

[Zaba505]

Note

A new feature for humus. Like Proposal: Kubernetes controllers, it isn't part of the re-evaluation in Proposal: humus on the bedrock process model, so it has no wave. It builds on two proposals, both landed and closed Resolved:

• Proposal: failure injection, as its revision 1, revision 2 and revision 3 amend it;
• Proposal: Kubernetes controllers, as its revisions 1 to 5 amend it.

Everything it needs has landed, so its stories wait on nothing. Proposal: Kubernetes controller walkthrough depends on it.

Revision 1 brings it up to date with what landed. A delay that a rest.Config's Timeout cuts short may return net/http's timeout error, as #869's revision 3 found for httpclient. A panic in trust is recovered and never retried, as a Reconciler's is. The annotation is read before the controller adds or removes its finalizer. The injection layer sits between otelhttp and the token, and a watch's events land on the span in its context. The strip of an annotation that isn't baggage is named as the one strip with attributes. The open questions don't change.

Revision 2 answers all three open questions. The scenario comes from an annotation the end user names (1a), and points name the resource before the verb (2a). Forwarding, in Kubernetes, is carrying a scenario on to children (3): in a reconcile that carries a scenario, the Client sets the annotation in every server-side apply of an object whose controller owner reference is the reconciled object. So an operator of several controllers carries one experiment down through its resources. Every request's baggage is still stripped, and there is no ForwardScenario.

Summary

Proposal: failure injection gives every humus server and client a part in failure injection:

• a server trusts or strips the scenario a request carries;
• a client evaluates an injection point named after its dependency, and strips the scenario before calling a dependency that doesn't take part.

A Kubernetes controller has neither part yet. Its reconciles answer no request, so nothing carries a scenario into them. Its Client talks to the API server outside every injection point. This proposal gives kube both parts:

• An object's annotation carries a scenario into that object's reconciles.
  • kube.TrustScenario(annotation, trust) is a new Part. It reads the scenario from the annotation the end user names, for an object their trust func trusts, and puts it in the reconcile's context.
  • Everything the reconcile calls then sees the scenario: the Reconciler, a Finalize's f, the end user's own injection points, every humus client, and the controller's own writes.
  • Without the Part, a reconcile carries no scenario, so the default trusts no one, as a server's does.
• Every request through the kube.Client is an injection point, named after what it does to which resource, the way the API server's audit log names it: kube:deployments.apps:patch, kube:memcacheds.cache.example.com/status:patch.
  • An abort fails the request as an unreachable API server would.
  • A delay is waited before the request is sent.
  • Each is a real failure to the reconcile that meets it, as revision 2 decided for every client.
• The Client strips the scenario from every request. The API server takes no part in failure injection, so there is no ForwardScenario.
• Nothing else changes. The informers and leader election never carry a scenario. Telemetry is fault's humus.fit.evaluation and humus.fit.strip span events.

const scenarioAnnotation = "acme.example.com/failure-scenario"

ctrl, err := kube.NewController(s, client, settings, app.Memcached, reconciler,
	kube.Owns(appsv1.SchemeGroupVersion.WithResource("deployments")),
	// Only the staging namespace runs experiments. An admission policy lets
	// only the chaos harness's ServiceAccount set the annotation there.
	kube.TrustScenario(scenarioAnnotation, func(obj metav1.Object) bool {
		return obj.GetNamespace() == "staging"
	}),
)

A scenario that aborts every apply of a Memcached's Deployment, written by the test or tool that runs the experiment:

codec, _ := jsonscenario.New("acme.fault")
ctx, _ := fit.ContextWithScenario(ctx, codec, fit.Scenario{Rules: []fit.Rule{
	{Point: "kube:deployments.apps:patch", Fault: fit.Fault{Abort: true}},
}})
// The annotation's value is the W3C baggage it leaves, one acme.fault
// member, percent-encoded as W3C Baggage requires.
value := baggage.FromContext(ctx).String()

Background

• Failure injection is request-scoped. A scenario travels in a request's OpenTelemetry baggage. Each injection point evaluates it with s.Faults.Eval, and fit decodes it from the context's baggage on every call (#869's revision 2, answer 4). A server decides whether to trust a caller's scenario with TrustScenario, typed over its protocol's own request. A job trusts the BAGGAGE in its environment, because whoever sets that sets all of its settings.
• A reconcile is a root span with no request. Proposal: Kubernetes controllers starts a root span, reconcile <Kind>, for each key the work queue hands a worker. The context it hands the Reconciler has no baggage. So s.Faults.Eval in a Reconciler always returns the zero Fault. An httpclient the Reconciler calls never sees a scenario either.
• What triggers a reconcile is an object. A reconcile comes from a change to the object, a change to a resource it owns, After, a retry or the resync. The trigger isn't passed to the Reconciler. But the object always is, and its metadata is what Kubernetes gives for carrying data that isn't part of the spec: annotations.
• Every request of a controller goes through one transport. kube.NewClient builds the transport that dials through the Host and traces with otelhttp. Client.Config() hands it to every client-go client, the end user's clientsets included (#851's Connecting). It is the one seam that sees every request, as httpclient's injection layer is for HTTP.
• client-go doesn't retry a transport error, except for a GET. It retries a response with Retry-After. It also retries a GET whose connection was reset, ended early or lost its HTTP/2 connection (request.go#L1066-L1078). An injected abort is none of those, so client-go returns it to its caller at once.
• fit matches a point exactly, or by a prefix ending in * (fit.go#L64-L72). So a point name's parts should run from the broadest to the narrowest. Proposal: failure injection #869's names do: http:<host>:<port>, grpc:<service>/<method>.

Proposed API

One Part is added. Nothing else is exported, and no Settings field is added.

package kube

// TrustScenario carries the failure scenario that an object's annotation
// holds into each of the object's reconciles, when trust trusts the object.
// The annotation's value is W3C baggage that carries a scenario as the App's
// fault.Decoder reads it, such as a member of jsonscenario's. Only the
// Decoder's members join the reconcile's context. The reconcile's
// Reconciler, a Finalize's f, and every request and injection point they
// reach see the scenario.
//
// trust is asked only about an object whose annotation carries a scenario,
// and only when the App has failure injection. It is set in code, not
// sourced. Without TrustScenario, no reconcile carries a scenario.
func TrustScenario(annotation string, trust func(obj metav1.Object) bool) Part

TrustScenario works on NewController and NewKindController alike. NewController and NewKindController report these as mistakes in their parts:

• an annotation that isn't a valid annotation key;
• a nil trust;
• two TrustScenario parts.

Where a reconcile's scenario comes from

After the object is decoded, and before a finalizer or the Reconciler runs, the controller reads the annotation:

1. No annotation, or none of the Decoder's members in it: the reconcile carries no scenario, and nothing is recorded.
2. A value that isn't W3C baggage: the reconcile carries no scenario. Its span gets a humus.fit.strip event with error.type and error.message. The reconcile goes on unharmed, as fit requires of a scenario it can't read.
3. A scenario, on an object trust doesn't trust: the reconcile carries no scenario. Its span gets a humus.fit.strip event, as a server's request does.
4. A scenario, on an object trust trusts: the reconcile's context gets the Decoder's members, and no other member of the annotation.

• The scenario belongs to the object. A reconcile that a resource it owns triggered carries the owner's scenario, since it is the owner that is reconciled. The child's annotations aren't read.
• Adding or removing the annotation starts or ends an experiment at once. It changes the object, so the object is reconciled straight away.
• A scenario stays as long as its annotation. Every retry meets it again, so a scenario that aborts a write keeps the object backing off to RetryMaxDelay, as an outage would. Removing the annotation ends the experiment, and the reconcile that follows succeeds. A Decoder that knows expiry can expire it sooner, as Proposal: failure injection #869 left to the end user.
• Only the scenario's members join. Other members in the annotation are dropped, so an annotation can't put arbitrary baggage on every request a reconcile makes.

Injection points

Every request through the Client's transport evaluates one point, after otelhttp starts the request's span, so the humus.fit.evaluation event lands on it. A request whose context carries no scenario gets the zero Fault and no event, as #869 says of every point. The informers, leader election and the served check always run without one.

The point names what the request does to which resource, as the API server's audit log and authorizer name it:

kube:<resource>[.<group>][/<subresource>]:<verb>

• <resource> is the plural, and <group> is left out for the core group, as kubectl writes a resource: deployments.apps, secrets, memcacheds.cache.example.com.
• <verb> is the API server's: get, list, watch, create, update, patch, delete or deletecollection. A server-side apply is a patch.
• A request for a path that isn't a resource, such as discovery's /apis, is kube:<path>:<verb>.

A rule	Fails
kube:*	every request to the API server
kube:deployments.apps:*	every request about Deployments, but not their subresources
kube:deployments.apps:patch	every apply of a Deployment
kube:memcacheds.cache.example.com/status:patch	the controller's apply of a Memcached's status
kube:memcacheds.cache.example.com:patch	the controller's applies of a Memcached's finalizer

A point carries no namespace or object name. The scenario is already scoped to one object's reconciles, and #869's points name the dependency, not the request.

A fault is a real failure

As #869's revision 2 decided for every client, a fault emulates a failure of the dependency, and everything that meets it reacts as it would to that failure:

	An abort	A delay
The request	fails with nothing sent, as an unreachable API server's does. client-go returns an error that wraps the *fault.InjectedError	waits before the request is sent, inside the request's context. Cut short, it fails with an *fault.InjectedError that wraps the context's cause
client-go	returns it at once, since it retries only resets, early EOFs, lost HTTP/2 connections and Retry-After	sends the request once the delay is over
The reconcile	whatever the Reconciler does with the error. Returned, it is retried with backoff, and humus.kube.controller.reconcile.outcome is error	slower, as reconcile.duration shows

The controller's own writes are requests like any other. An abort at …/status:patch fails the status apply, which retries the object as any failed reconcile does. An abort at the resource's patch fails adding or removing a finalizer.

Stripping at the API server

The Client strips the Decoder's members from every request's baggage before it is sent, and the request's span gets a humus.fit.strip event. Every other member still goes.

There is no ForwardScenario. #869 gives every client one for a dependency that takes part in failure injection with the same codec. The API server never does: it evaluates no point of the end user's. A field that should never be set would be an API for nothing.

Telemetry

Nothing new. The events are fault's:

• humus.fit.evaluation on each request's span, and wherever the end user evaluates a point of their own;
• humus.fit.strip on a request's span, and on a reconcile's span for an annotation that is dropped.

An aborted request's span has the error.type github.com/z5labs/humus/fault.InjectedError, as every client's has. otelhttp's http.client.request.duration counts it, and humus.kube.controller.reconcile.duration counts the reconcile it fails with the outcome error.

Design decisions

• An annotation, because an object is what a reconcile is about. A request carries its scenario in its headers. An object carries what isn't spec in its metadata, and annotations are Kubernetes' place for data that tools write and controllers read. A label would be indexed and selectable, which a scenario doesn't need, and its value is limited to 63 characters.
• The end user names the annotation. humus imposes no baggage key on jsonscenario, and imposes no annotation key either. An organisation's tooling already has its own domain for annotations, and the edge that writes scenarios is the end user's.
• The value is W3C baggage, read by the App's own Decoder. The scenario's format stays the end user's, as chore(deps): update module github.com/cenkalti/backoff/v5 to v7 #578 ruled. A scenario written for an HTTP request's baggage header can be put on an object unchanged. One tool, fit.ContextWithScenario and an Encoder, writes both.
• Trust is a func of the object. It is the protocol's own request for a controller, as *http.Request is for httpserver. Who may set an annotation is decided by RBAC and admission, outside the process. trust reads what those decisions leave visible: the namespace, a label an admission policy guards, or the object's owner. nil isn't allowed: an end user who adds the Part has decided to trust someone.
• Without the Part, nothing is read, as a server's nil TrustScenario trusts no caller. A controller that never adds it can't be put under test by anyone who can annotate its objects.
• Every request is a point, and the controller's own writes are requests. Failing the status apply, or the finalizer's apply, is how an experiment shows that the controller retries them. A point of the controller's own, around Reconcile, would emulate a bug in the end user's code, which isn't a dependency's failure.
• Points name the resource before the verb. fit matches a prefix, so a rule can fail everything about one resource, kube:secrets:*. Failing one verb across every resource, such as every patch, needs one rule per resource. Experiments on a controller are about a resource it can't write far more often than a verb that fails everywhere. Open question 2.
• The Client always strips. See Stripping at the API server. Open question 3.

Testing

Nothing here was prototyped, because kube isn't built yet (#908 to #918). The stories' tests show:

• The Client's points, on humustest's Host with a plain HTTP handler, as kube's Client tests run:
  • a table of requests and their points: each verb, a core and a grouped resource, a subresource, a cluster-scoped resource, and a non-resource path;
  • an abort that never reaches the handler, and returns an error that errors.As finds an *fault.InjectedError in, from a clientset built from Config();
  • a GET aborted once, with no retry by client-go;
  • a delay waited before the request, and one cut short by the request's context;
  • the scenario stripped from the baggage header, with every other member kept, and the humus.fit.strip event;
  • no evaluation and no event for a request with no scenario.
• TrustScenario, inside synctest on client-go's fakes:
  • a trusted object's scenario reaching the Reconciler, a Finalize's f, and a point the test evaluates with s.Faults.Eval;
  • an untrusted object's scenario, and an annotation that isn't baggage, each stripped with its event, and trust never asked for an object without a scenario;
  • other members of the annotation dropped;
  • the owner's scenario on a reconcile that an owned resource triggered;
  • the mistakes, with several reported together;
  • nothing read when the App has no failure injection.
• Against k3s, a whole App with humus.WithFaults and jsonscenario. Annotating the Memcached with a scenario that aborts kube:deployments.apps:patch:
  • fails each reconcile, and the Memcached's status shows the Reconciler's failure condition, since the status apply isn't aborted;
  • retries the object with backoff;
  • stops once the annotation is removed, and the next reconcile applies the Deployment.

Changes to other files

• kube's package doc gains failure injection: TrustScenario, the point names, what an abort and a delay do, and the strip.
• The Kubernetes docs page gains a section on failure injection, with the example above.
• The failure injection docs page lists kube's points with the other clients', and the annotation with the servers' trust.

Migration from today

Nothing migrates. kube is new, and has no failure injection until this proposal's stories land.

Open questions

1. Where a reconcile's scenario comes from.

   • (a) An annotation the end user names, trusted per object, as proposed. It scopes an experiment to one object, starts and stops it at once, and needs nothing but RBAC to control.
   • (b) The process's environment, as a job's BAGGAGE. Every reconcile carries the scenario. That is the whole controller under one scenario for as long as the pod runs. It is closer to a chaos experiment, which Proposal: failure injection #869 left out of scope, and it changes only with a rollout.
   • (c) Both, with the annotation's scenario taking precedence.

   Recommendation: (a).

2. The point names.

   • (a) kube:<resource>.<group>[/<subresource>]:<verb>, as proposed, so a prefix fails one resource.
   • (b) kube:<verb>:<resource>.<group>[/<subresource>], so a prefix fails one verb across every resource, such as every write.
   • (c) kube:<host>:<port>, the API server's origin, as httpclient's points are. Every request is then one point, and an experiment can't fail one resource.

   Recommendation: (a).

3. A ForwardScenario for the Client.

   • (a) None: the Client always strips, as proposed.
   • (b) A field on Connection, for parity with Proposal: failure injection #869's clients, default false.

   Recommendation: (a). The API server takes no part.

Out of scope

• Deciding which objects are under test, and writing their annotations. That is the end user's tooling, with fit.ContextWithScenario and an Encoder, and their RBAC and admission policies.
• Faults in the controller's own machinery: the informers' lists and watches, leader election and the served check. They run on Serve's context, which never carries a scenario. Losing the Lease, or a watch that ends, is a different kind of experiment, run against the cluster.
• Emulating the API server's answers, such as a 409 conflict, a 429 throttle or an admission webhook's denial. fit has only Abort, and Proposal: failure injection #869 left kinds of abort out of scope.
• A point around Reconcile itself, and points per namespace or object.
• Forwarding a scenario to the API server, or to the admission webhooks it calls.
• A metric of experiments. Each evaluation is a span event on its request's trace, as Proposal: failure injection #869 decided.

References

• Proposal: failure injection, and its revisions 1 (the humus.fit. names) and 2 (a fault is a real failure, fail-safe defaults, the Injector on Setup)
• Proposal: Kubernetes controllers, and its revisions 1, 3, 4 and 5
• bedrock's fit v0.25.0: rule matching, and z5labs/bedrock#578
• client-go v0.37.1's retryable errors
• Kubernetes: annotations, request verbs, ValidatingAdmissionPolicy
• W3C Baggage

[Zaba505]

Revision 1: up to date with what landed

This proposal was posted before Proposal: failure injection's revision 3, and before any of kube was built. Both proposals have now landed. Four things change:

1. A Timeout on a rest.Config can replace an injected delay's error, as revision 3 found for httpclient's Timeout. Client.Config() sets none, so the controller's own requests are unaffected. An end user who sets one on their copy gets two timers from client-go, and net/http's can replace the error.
2. A panic in trust is recovered and never retried, as a panic in a Reconciler or a Finalize's f is (#851's revision 4). The post didn't say what a panic in trust does.
3. The scenario is read before the controller does anything with the object. The controller adds its finalizer before any decode, so "after the object is decoded" would leave the finalizer's apply out.
4. Where the layer sits and where its events land, in the Client as built. The Client gained a token layer and traces no watch, so the injection layer sits between otelhttp and the token, and a watch's events land on the span in its context.

It also names the one strip that has attributes. #869 gives a strip none, and the post gave the strip of an annotation that isn't baggage error.type and error.message without saying it differed.

The open questions don't change.

Each section below replaces or adds to the part of the same name in the post. The rest of the post stands.

Proposed API

TrustScenario's doc comment names the Decoder by its field, since fault has no Decoder type. Its first paragraph becomes:

// TrustScenario carries the failure scenario that an object's annotation
// holds into each of the object's reconciles, when trust trusts the object.
// The annotation's value is W3C baggage that carries a scenario as the
// App's Decoder, fault.Settings.Decoder, reads it, such as a member of
// jsonscenario's. Only the Decoder's members join the reconcile's context.
// The reconcile's Reconciler, a Finalize's f, the controller's own writes,
// and every request and injection point they reach see the scenario.
//
// trust is asked only about an object whose annotation carries a scenario,
// and only when the App has failure injection. It is set in code, not
// sourced. A panic in trust is recovered and recorded as a panic in the
// Reconciler is, and the object isn't retried. Without TrustScenario, no
// reconcile carries a scenario.

Where a reconcile's scenario comes from

The sentence before the list becomes:

When the reconcile's span starts, the controller reads the annotation from the object as the cache holds it. That is before it adds or removes its finalizer, calls a Finalize's f, or decodes the object for the Reconciler, so all of them see the scenario:

A bullet is added after the list:

• A panic in trust is recovered and recorded, as a panic in the Reconciler is: an ERROR record, reconcile panicked, and an exception event on the reconcile's span. The reconcile's outcome is panic, and its key is forgotten, not retried. Nothing else runs, so no finalizer and no status is applied. As with any panic, the object is reconciled again when it changes, when the resync comes, or when a new leader takes over.

Injection points

The first paragraph becomes:

Every request through the Client's transport evaluates one point. The injection layer sits between otelhttp and the token, as httpclient's sits just inside otelhttp. So a delay is inside the request's span and in http.client.request.duration, and an abort fails before Connection.Token is called. The humus.fit.evaluation event lands on the request's span. A watch has none, since the Client doesn't trace one. A watch's event lands on the span in its context, such as the reconcile's. A request whose context carries no scenario gets the zero Fault and no event, as #869 says of every point. The informers, leader election and the served check always run without one.

The table's status row gains a sentence:

A rule	Fails
kube:memcacheds.cache.example.com/status:patch	the controller's apply of a Memcached's status. A Kind's controller applies no status, so only a Resource's controller makes this request

A fault is a real failure

The table's "The request" row becomes:

	An abort	A delay
The request	fails with nothing sent, and with no call to Connection.Token, as an unreachable API server's does. client-go returns an error that wraps the *fault.InjectedError	waits before the request is sent, inside the request's context. Cut short, it fails with an *fault.InjectedError that wraps the context's cause, except when a rest.Config's Timeout cuts it short (below)

This paragraph is added after the table:

A delay that a rest.Config's Timeout cuts short. Client.Config() sets no Timeout, so the controller's own requests always fail as the table says. An end user who sets one on their copy gets two timers from client-go:

• a context deadline on each request (request.go#L151-L154, #L1060-L1064);
• the http.Client's own Timeout (transport.go#L39-L43).

The two go off within microseconds of each other, and at the same instant in a synctest bubble. If net/http's has gone off by the time the transport returns, Client.do replaces the transport's error with a timeout error of its own (client.go#L748-L750), as #869's revision 3 found. So such a request returns either the *fault.InjectedError or net/http's timeout error. net/http's error matches context.DeadlineExceeded and holds no *fault.InjectedError. Either way, the request did time out, and its span carries InjectedError's error.type.

Telemetry

The list becomes:

• humus.fit.evaluation on each request's span, and on the span in a watch's context, and wherever the end user evaluates a point of their own;
• humus.fit.strip on a request's span, and on a reconcile's span for an annotation that is dropped.

A sentence is added after the list:

A strip has no attributes, as #869 has it, with one exception: a reconcile's strip for an annotation that isn't W3C baggage has error.type and error.message.

The last paragraph gains a sentence:

A delay that a rest.Config's Timeout cuts short has that error.type on its span too, whichever error the request returns.

Design decisions

A bullet is added:

• The strip of an unreadable annotation has attributes. Proposal: failure injection #869's strips have none: a server's propagator drops baggage it can't parse before humus sees it, so nothing is left to explain. The controller parses the annotation itself, so it alone knows why a scenario didn't join. Without the error, an annotation that can't be read would look like an object that isn't trusted.

Testing

The Client's points gain three bullets:

• an abort that never calls Connection.Token;
• a delay that a rest.Config's Timeout cuts short, whose error matches context.DeadlineExceeded and whose span has InjectedError's error.type, whichever error it is;
• a watch's evaluation, on the span in its context.

TrustScenario's tests gain a bullet:

• a panic in trust, recovered and recorded, with the outcome panic, the key forgotten, and nothing applied.

The tests against k3s gain one. client-go's fakes never see a request's context, so only a real API server shows the finalizer's apply meeting the scenario:

• a controller with a Finalize, and a new Memcached annotated with a scenario that aborts kube:memcacheds.cache.example.com:patch. The finalizer isn't added while the annotation stays, and is added once it is removed.

Changes to other files

The second and third bullets become:

• The Kubernetes docs page, docs/content/kubernetes.md, gains a section on failure injection, with the example above.
• The failure injection docs page, docs/content/failure-injection.md, gains kube:
  • in each of its tables: the point in "Injection points", TrustScenario and the annotation in "Trust at each server", and what an abort and a delay do in "A fault is a real failure";
  • in the exception after "Injection points"' table, which gains the rest.Config's Timeout;
  • in "Telemetry", which names the strip with attributes;
  • in "Testing a scenario", which gains an object's annotation.

References

These are added:

• Proposal: failure injection's revision 3 (a delayed HTTP call that Timeout ends returns net/http's timeout error)
• client-go v0.37.1's request timeout and http.Client timeout
• net/http's timeout error

Nothing else changes.

[Zaba505]

Revision 2: the open questions answered, and a scenario goes on to the children a reconcile applies

The answers:

1. A reconcile's scenario comes from an annotation the end user names, trusted per object, option (a). The process's BAGGAGE isn't read.
2. Points name the resource before the verb, option (a): kube:<resource>[.<group>][/<subresource>]:<verb>.
3. In Kubernetes, forwarding a scenario means carrying it on to the reconciled object's children. The question asked about forwarding to the API server, which takes no part in failure injection. What can take part is another controller. An operator is often several controllers, each reconciling resources that the one above it applies. So in a reconcile that carries a scenario, the Client sets the annotation on every child the Reconciler applies. The child's controller then trusts it or not with its own TrustScenario. Nothing else is forwarded: every request's baggage is still stripped, and there is no ForwardScenario.

All three open questions are answered.

Each section below replaces or adds to the part of the same name in the post, as revision 1 left it. The rest of the post stands.

Summary

The bullet "The Client strips the scenario from every request" becomes:

• A scenario goes on to the reconciled object's children, and nowhere else.
  • In a reconcile that carries a scenario, the Client sets the annotation on every child the reconcile applies with server-side apply. A child is an object whose controller owner reference is the reconciled object. So one experiment on the top object of an operator of several controllers reaches every controller below it.
  • The Client strips the scenario from every request's baggage. The API server takes no part in failure injection, so there is no ForwardScenario.

Background

A bullet is added:

• An operator's controllers talk through the objects they apply. A controller applies the children its object declares, each with a controller owner reference to it, as the Memcached in docs/content/kubernetes.md applies its Deployment. In an operator of several controllers, a child is often a custom resource that another of its controllers reconciles. Server-side apply removes a field that a field manager stops applying, unless another manager also owns it.

Proposed API

The first paragraph of TrustScenario's doc comment gains a last sentence:

// TrustScenario carries the failure scenario that an object's annotation
// holds into each of the object's reconciles, when trust trusts the object.
// The annotation's value is W3C baggage that carries a scenario as the
// App's Decoder, fault.Settings.Decoder, reads it, such as a member of
// jsonscenario's. Only the Decoder's members join the reconcile's context.
// The reconcile's Reconciler, a Finalize's f, the controller's own writes,
// and every request and injection point they reach see the scenario. The
// Client carries it on to the reconciled object's children: it sets the
// annotation on every child the reconcile applies.

Stripping at the API server

A sentence is added at the end:

A scenario goes on to the reconciled object's children instead, in their annotation (see Carrying a scenario on to children).

Carrying a scenario on to children

This section is new.

In a reconcile that carries a scenario, the Client sets the annotation on each child the reconcile applies. It changes a request that is all of these:

• a server-side apply, a PATCH whose content type is application/apply-patch+yaml, to an object, not to one of its subresources;
• made with the reconcile's context, or a context made from it;
• of an object whose metadata.ownerReferences has a controller reference whose uid is the reconciled object's.

The Client sets metadata.annotations[<annotation>] in the request's body to the scenario's members as W3C baggage. That is the value the reconcile took from its object's annotation, without any other member. A value the body already gave the annotation is replaced. The Client does this after the point is evaluated, so an aborted apply sends nothing, and after the request's baggage is stripped. The body is sent on as JSON, which is YAML too. The controller marks a reconcile's context with its object's UID, unexported, for the Client to read.

What follows:

• The child's controller decides. A controller of the child's resource, with its own TrustScenario on the same annotation, carries the child's scenario into the child's reconciles if its trust trusts the child. Its own applies then carry the scenario on to the grandchildren. What makes a child's annotation worth trusting is an admission policy that lets only the operator's ServiceAccount set it on the children's resources.
• Ending the experiment ends it all the way down. Removing the annotation from the top object reconciles it at once, without a scenario. Its apply of each child no longer declares the annotation, so the API server removes it, since the apply's field manager owned it. That changes the child, whose controller reconciles it at once, without a scenario, and so on down.
• A child that no controller reads keeps an inert annotation, such as a Deployment. Kubernetes' own controllers ignore it, and a Deployment's annotations aren't its pods' template, so it starts no rollout.

What isn't changed:

• A create, an update, or any patch that isn't an apply. Only an apply takes the annotation off again. A child created or updated with it would keep it after the experiment ends, and keep failing.
• A body that isn't JSON or YAML. client-go sends JSON unless the end user sets the rest.Config's ContentType to protobuf, or turns on its ClientsAllowCBOR and ClientsPreferCBOR feature gates for CBOR. Such an apply is sent as it is.
• The reconciled object itself. The controller's applies of its status and its finalizer are of the object, which isn't its own child.
• An object that isn't the reconciled object's child, such as one whose controller is another object, or one that has only owner references that aren't a controller's.

Telemetry

A sentence is added:

Setting a child's annotation adds no event. The apply's span has its humus.fit.evaluation and humus.fit.strip events, and the child's reconciles show the scenario they carry.

Design decisions

"Points name the resource before the verb" drops its "Open question 2." The bullet "The Client always strips" is replaced by the last four bullets below, and the first is added:

• An annotation, not the environment. A scenario in the process's BAGGAGE would put the whole controller under it for as long as the pod runs. That is a chaos experiment, which Proposal: failure injection #869 left out of scope, and it would change only with a rollout.
• A scenario goes on to children, and to nothing else. The API server evaluates no point, and neither do the admission webhooks it calls, so a request's baggage is always stripped. A child is different: it is the input of the next controller in an operator, which can evaluate points. Carrying the scenario on lets one experiment on the top object exercise the whole operator, as Proposal: failure injection #869's ForwardScenario lets one exercise a chain of services. It needs no field. A child that no controller reads ignores the annotation, and a controller that reads it decides with its own trust.
• The Client sets the annotation, in the apply that writes the child. So the child has the scenario from its first reconcile, which is often the one an experiment is about, such as the one that provisions what the child declares. If the controller applied the annotation separately, after the reconcile, the child would be created without it. If the Reconciler had to set it, every write of a child would have to remember it.
• Only an apply, so the annotation always comes off. An apply's field manager owns what it declares, so its next apply without a scenario removes the annotation. A create or an update would leave it on the child for good.
• A child is named by its controller owner reference, as Kubernetes' garbage collector and Owns name one. The body declares the reference, so the Client needs only the reconciled object's UID, and the child's resource needs no Owns.

Testing

The Client's points gain these bullets:

• an apply of a child, made with a reconcile's context that carries a scenario. Its body reaches the handler with the annotation set to the scenario's members, replacing any value the body gave it, and its baggage header has no scenario;
• a YAML apply body, sent on as JSON with the annotation;
• an aborted apply of a child, which sends nothing;
• requests left as they are: an apply with no scenario; an apply of an object whose controller owner reference names another UID, or that has only owner references that aren't a controller's; an apply to a subresource; a create and an update of a child; and an apply with a CBOR or a protobuf body.

The tests against k3s gain two:

• a Memcached annotated with a scenario of no rules. Its Deployment gets the annotation through the Reconciler's apply, and loses it once the Memcached's annotation is removed;
• an operator of two controllers: a second controller in the same App, a NewKindController of Deployments with TrustScenario on the same annotation, whose Reconciler evaluates a point of its own. Its reconciles of the Memcached's Deployment carry the Memcached's scenario, and stop carrying it once the Memcached's annotation is removed.

Changes to other files

• kube's package doc also covers carrying a scenario on to children.
• The Kubernetes docs page's failure injection section also shows an operator of two controllers carrying a scenario down.
• The failure injection docs page's "Sending a scenario on: ForwardScenario" section gains kube: there is no ForwardScenario, and a reconcile's scenario goes on to the children it applies.

Out of scope

These are added:

• Carrying a scenario on through a create, an update or a patch that isn't an apply, or in a body that isn't JSON or YAML.
• Carrying a scenario on to an object that isn't the reconciled object's child.

Open questions

All three are answered, above.

Nothing else changes.

[Zaba505]

Closing: failure injection for Kubernetes controllers has landed

The revisions' stories are done:

• story(kube): fail or delay a request to the API server that a failure scenario names, and strip scenarios from every request #969 failed or delayed every Client request at kube:<resource>[.<group>][/<subresource>]:<verb>, between otelhttp and the token, and stripped the scenario from every request (story(kube): fail or delay a request to the API server that a failure scenario names, and strip scenarios from every request #973);
• story(kube): carry the failure scenario an object's annotation holds into its reconciles, with kube.TrustScenario #970 carried the scenario an object's annotation holds into that object's reconciles, with kube.TrustScenario. Baggage that won't parse, or an object trust doesn't trust, gets a humus.fit.strip event on the reconcile span, and a panic in trust is recovered and never retried (story(kube): carry the failure scenario an object's annotation holds into its reconciles, with kube.TrustScenario #996);
• story(kube): carry a reconcile's failure scenario on to the children it applies #971 carried a reconcile's scenario on to its children: the Client sets the annotation in every server-side apply, made with the reconcile's context, of an object whose controller owner reference is the reconciled object (story(kube): carry a reconcile's failure scenario on to the children it applies #1003);
• story(docs): document failure injection in Kubernetes controllers #972 documented failure injection in Kubernetes controllers, in a new section of the Kubernetes page and in every section of the failure injection page (story(docs): document failure injection in Kubernetes controllers #1025).

The revisions left no open questions. The scenario comes from an annotation the end user names, and only through TrustScenario. Points name the resource before the verb, and apply is patch. Every request's baggage is stripped, and there is no ForwardScenario. Forwarding, in Kubernetes, is the children's annotation. A fault is a real failure, and a delay that a rest.Config's Timeout cuts short may return net/http's timeout error.

The rest is out of scope or belongs to other proposals, as "Out of scope" lists:

• Showing failure injection in a controller walkthrough is Proposal: Kubernetes controller walkthrough, which no longer waits on this one.
• Deciding which objects are under test, and writing their annotations, is the end user's tooling, RBAC and admission policies.
• Faults in the controller's own machinery, emulating the API server's answers, a point around Reconcile or per object, forwarding a scenario to the API server or its webhooks, and a metric of experiments stay out of scope.