Proposal: Kubernetes controllers

[Zaba505]

Note

Shutdown is amended by revision 1. The controller no longer drains its work queue. On shutdown it stops taking keys, finishes the reconciles in flight, and leaves the backlog for the next leader. The work queue's goroutine outlives ShutDownWithDrain by up to 10s rather than leaking. The Proposed API doesn't change.

Revision 2 adds a fifth open question: whether a controller can reconcile one of Kubernetes' own kinds, such as a Namespace or a Secret, rather than a custom resource.

Revision 3 prefixes the names humus defines with humus.: kube.object.name becomes humus.kube.object.name, and open question 2's option (a) would put its instruments under humus.kube.controller., with humus.kube.controller.name.

Revision 4 answers open questions 1 to 4. A panic in a reconcile or a finalizer is recovered and recorded, and never retried. The controller defines six instruments under humus.kube.controller.. The end user fills the Connection, and tests a whole App against k3s, with no kubetest. Question 5 is still open.

Revision 5 answers open question 5: a controller can reconcile a built-in kind, such as a Namespace. kube.Kind[corev1.Namespace] names one, NewKindController reconciles it, and its KindReconciler returns only when to reconcile again. Controller and Finalize lose Spec and Status. All five open questions are answered.

Note

A new feature for humus. It isn't part of the re-evaluation in Proposal: humus on the bedrock process model, which covers what humus had before the reset, so it has no wave. It builds only on accepted proposals: the lifecycle, configuration, OpenTelemetry without globals, humustest, health monitors and probes, humus.Main and exposing health probes. Admission and conversion webhooks need nothing from it: the OpenAPI endpoint DSL serves them.

Summary

• kube is a new nested module for writing Kubernetes controllers as humus Services. It is built on client-go's informers, work queue and leader election, and uses no code generation, no comment markers and no controller-runtime.
• A custom resource is a Go value. kube.Resource[Spec, Status] names the resource, and its CRD is reflected from the Spec and Status types.
  • Constraints are typed values, attached to a field by pointer: kube.Field(&spec.Size).Minimum(1).
  • A misspelled constraint, or a value of the wrong type, doesn't compile. CRD() returns one error naming every other mistake.
  • The end user checks the CRD's YAML into their repo with a golden test, so it can't drift from the types.
• An object is a plain Go struct. kube.Object[Spec, Status] needs no DeepCopy, no scheme and no runtime.Object. The controller reads objects with client-go's dynamic client and decodes each into a fresh value, which is the end user's own copy.
• A reconcile is handed its object and returns its status: Reconcile(ctx, obj) (kube.Result[Status], error).
  • The controller applies the status, and its finalizer, with server-side apply, so it never fails with "the object has been modified".
  • An error retries the object with backoff, and Result.After reconciles it again later.
  • A reconcile error never ends the Service.
• The Controller is a humus Service.
  • Every replica keeps its caches warm, and only the Lease's holder reconciles. Synced is the readiness monitor.
  • On shutdown it lets the reconciles in flight finish, then releases the Lease.
  • Losing the Lease ends the App with exit 1.
• The API server is reached through the Host.
  • kube.NewClient(s, kube.Connection{Server, TLS, Token}) dials with s.Host.Dial, calls a rotating token's function at most once a minute, and traces every request with s.Telemetry.
  • The end user's Init sources the Connection. humus reads no KUBERNETES_SERVICE_HOST and no token file itself.
  • Client.Config() gives client-go's own clientsets the same transport, for the resources a controller owns.
• It is tested on client-go's fakes inside synctest, and against k3s in a container. A prototype passes both. Against k3s v1.37.1:
  • the reflected CRD is accepted, and its rules are enforced;
  • a whole App reconciles a resource into a Deployment it owns, writes the resource's status, runs its finalizer and exits 0.

func main() {
	humus.Main(app.Init, humus.WithTelemetry(app.BuildTelemetry))
}

app.Init sources a kube.Connection and kube.Settings, builds the Client and the end user's Reconciler, and returns the Service that kube.NewController(s, client, settings, app.Memcached, reconciler, kube.Owns(deployments)) builds.

Background

Why this proposal: controller-runtime

Nearly every Go controller is written with kubebuilder: controller-runtime at run time, and controller-gen for the code and YAML it generates from comment markers. This proposal exists because that stack can't be reconciled with humus, and because it is cumbersome to use. humus doesn't wrap controller-runtime, and doesn't depend on it.

It breaks humus's principles, some of them just by being imported. Every link is at v0.25.2, the latest release.

Principle	What controller-runtime does
3. No package-level state	metrics.Registry is a package-level Prometheus registry. init functions fill it, and set client-go's global work queue, client and leader election metrics providers (workqueue.go#L102-L111, client_go_adapter.go#L164-L174, leaderelection.go#L40-L42). A program that imports pkg/manager and does nothing else already has 4 metric families in the registry when main starts (checked).
3. No package-level state	It logs through log.Log, a global delegating logger. If SetLogger isn't called within 30s, it prints a warning and a stack trace to os.Stderr (log.go#L54-L72). Its cache logs through a package-level RuntimeLog, whatever Options.Logger says.
1. The process is the boundary	An init function adds --kubeconfig to flag.CommandLine (config.go#L43-L45), so importing the config package adds the flag (checked). GetConfig reads KUBECONFIG with os.Getenv and the home directory with user.Current (#L123, #L147). SetupSignalHandler calls signal.Notify, calls os.Exit(1) on a second signal, and panics if it is called twice (signal.go#L25-L41).
2. The host is passed in too	The health probe and pprof servers listen with net.Listen (manager.go#L474-L479). The metrics server binds :8080 by default (server.go#L45), a port humus would never impose.
4. Small and composable	manager.Options has 28 exported fields (manager.go#L107-L295). A Manager is the cache, the clients, the scheme, leader election, and the metrics, health, pprof and webhook servers, all in one: the large type that tries to anticipate every use.
6. Testable whole	envtest runs real kube-apiserver and etcd binaries with exec.Command, and finds them through KUBEBUILDER_ASSETS (process.go#L158, server.go#L49). Its fake client's own doc says "it's almost always better not to use this package" (fake/doc.go#L27).

Its telemetry is Prometheus only. The book tells end users to register custom metrics "with the global prometheus registry" (metrics). OpenTelemetry support is still an open issue (#305).

Its developer experience is cumbersome.

• Markers are comments, so nothing checks them.
  • controller-gen v0.18.0 silently skips a marker it doesn't know (collect.go#L115-L122). // +kubebuilder:validation:Minimun=1 produces a CRD with no minimum, and the run exits 0 with nothing on stderr (checked).
  • Only a known marker with a malformed argument, such as Minimum=one, is an error.
• Generated code and YAML have to be regenerated.
  • The book says: "Run make generate to create the DeepCopy implementations… Then, run make manifests to generate the CRD manifests" (getting started).
  • Under kubebuilder v4.16.0, kubebuilder init and one create api scaffold 54 files in 20 directories (counted).
• A reconcile is given a name, not an object.
  • reconcile.Request "does NOT contain information about any specific Event or the object contents itself" (reconcile.go#L62-L68).
  • So every Reconcile starts by fetching its object and handling not-found.
• Writing status conflicts.
  • The book says: "any update (including a status update) may change the resource version. If we continue reconciliation with a stale copy, subsequent updates may fail with a conflict such as: 'the object has been modified; please apply your changes to the latest version and try again'" (controller implementation).
  • Its fix is to fetch the object again after every status update.
• There are three ways to retry.
  • Result.Requeue, Result.RequeueAfter and a returned error all requeue, and an error makes the Result ignored (reconcile.go#L117-L127).
  • v0.21.0 deprecated Requeue because "it causes confusion" (reconcile.go#L31-L42, #3107).
• Reads aren't what they look like.
  • The default client reads from the cache and writes to the API server.
  • The first read of a type starts an informer for it, which watches every namespace unless the cache is told otherwise (manager.go#L121-L136, cache.go#L130-L137).
  • An unstructured object is read live unless Client.Cache.Unstructured is set (client.go#L92-L95).
• Every type must be registered in a scheme. An unregistered type fails at run time with "no kind is registered for the type" (error.go#L59).

Wrapping controller-runtime in humus's types would keep all of this. Its globals are set by init functions, so no wrapper can avoid them.

Why not the openapi package

The OpenAPI endpoint DSL (#760) was the first candidate. It serves the webhooks, and nothing else a controller needs:

• A controller is a client of the API server, and openapi is a server.

  • A controller's core loop never receives a request. It lists and watches, keeps caches and a work queue, and runs reconciles that apply status, finalizers and owner references. It also holds a Lease. Nothing in openapi has a place for that loop.
  • Admission and conversion webhooks are the exception. The API server POSTs JSON to them over TLS, which openapi, with jsoncodec, serves.

• openapi.SchemaOf writes the wrong dialect. It writes OpenAPI 3.2's JSON Schema 2020-12, in which a pointer field is type: ["null", …]. A CRD takes only OpenAPI 3.0's Schema Object, restricted to a structural subset:

  • type is one string, so a pointer is nullable: true;
  • exclusiveMinimum is a bool;
  • $ref and definitions are forbidden, so every schema is inlined;
  • x-kubernetes-* extensions mark int-or-string values, list and map types, and CEL validations.

  JSONSchemaProps.Type is a Go string (types_jsonschema.go#L45). k3s v1.37.1 rejects a CRD with a type array (checked): "json: cannot unmarshal array into Go struct field JSONSchemaProps…type of type string".

  The format won't change. Kubernetes closed "OpenAPI 3.1 Support" as not planned on 2026-10-17 (kubernetes#134684).

  kube-rs hit the same wall (kube#264). It runs schemars in OpenAPI 3.0 mode with every subschema inlined, then rewrites the output in three passes to make it structural (schema.rs).

Why not Metacontroller

Metacontroller v4.17.6 (2026-10-15) does the watching and applying itself. It calls a controller's sync and finalize hooks over HTTP, so openapi could serve a whole controller. But it has three costs:

• another component in every cluster, with broad RBAC of its own;
• a controller that declares its children instead of reconciling them;
• no help with the CRD.

That is a different product, which an end user can adopt today without anything from humus.

What client-go gives, and where it leaks

client-go v0.37.1 is Kubernetes' own client, released with each Kubernetes release. This proposal uses five of its parts directly:

• The dynamic and metadata clients, which read and apply any resource as unstructured data. Through them, a custom resource needs no generated clientset, no scheme and no DeepCopy.
• Informers, which list, watch, list again after a 410, and keep a cache. RunWithContext takes its logger from the context (shared_informer.go#L841).
• The typed rate-limiting work queue, which deduplicates keys and backs off per key.
• Leader election on a Lease.
• rest.Config, which takes a Transport of the caller's own (config.go#L108), so every connection can go through the Host.

It reaches around the process in six places, which this proposal avoids or documents:

1. In-cluster config reads the environment and the disk. rest.InClusterConfig reads KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT with os.Getenv, and the token from a hard-coded path with os.ReadFile (config.go#L546-L577). humus never calls it: the end user's Init sources the Connection.
2. A token file is re-read from the disk. A BearerTokenFile is read again with os.ReadFile every minute (token_source.go#L69-L81, #L135). The Connection's Token function replaces it.
3. The proxy comes from the environment. With no Proxy set, client-go uses http.ProxyFromEnvironment (cache.go#L142). The Client's transport has no proxy.
4. Its own transports read three variables. They read DISABLE_HTTP2, HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS with os.Getenv (http.go#L134-L170). The Client builds its own transport, with the same 30s and 15s.
5. Its feature gates read the environment. They read KUBE_FEATURE_* with os.LookupEnv on first use, and log through klog.Background() (envvar.go#L165-L169). The only replacement is features.ReplaceFeatureGates, which is global. This one stays an ambient leak.
6. klog writes to the real stderr. client-go logs through klog, and a code path with no logger in its context falls back to klog's global logger, which writes to the real stderr. The prototype's leader election did exactly that, until the controller put a logger in the context (checked).

Proposed API

package kube // import "github.com/z5labs/humus/kube"

// Connection is how a Client reaches an API server. Every field is a plain
// value.
type Connection struct {
	// Server is the API server's address, host:port. In a pod, it is
	// KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, joined.
	Server string

	// TLS is the config the API server is dialled with: the cluster's CA in
	// RootCAs, and a client certificate when the client authenticates with
	// one. ServerName defaults to Server's host.
	TLS *tls.Config

	// Token returns the bearer token every request carries, such as the
	// pod's service account token, read through the Host's FS. nil sends
	// none, for a client that authenticates with a certificate. The Client
	// calls it at most once a minute, and keeps its last token while a call
	// fails, so a token the kubelet rotates is picked up within a minute.
	Token func(ctx context.Context) (string, error)
}

func CheckServer(addr string) error
func CheckTLS(c *tls.Config) error

// Client reaches one API server through the Host.
type Client struct{ /* unexported */ }

// NewClient returns a Client that dials c.Server with s.Host.Dial, and
// traces and measures every request with s.Telemetry. It opens no
// connection.
func NewClient(s *humus.Setup, c Connection) (*Client, error)

// Config returns a rest.Config for client-go's clients, such as the
// clientset kubernetes.NewForConfig returns. They share the Client's
// transport: its dialler, its credentials and its telemetry.
func (c *Client) Config() *rest.Config

// Scope is whether a resource's objects are in a namespace.
type Scope string

const (
	Namespaced Scope = "Namespaced"
	Cluster    Scope = "Cluster"
)

// Resource declares a custom resource of one version, whose objects have a
// spec of type Spec and a status of type Status.
type Resource[Spec, Status any] struct {
	Group   string // such as cache.example.com
	Version string // such as v1alpha1
	Kind    string // such as Memcached
	Plural  string // such as memcacheds
	Scope   Scope

	// Rules constrain the fields of Spec, of Status, and of the types they
	// hold.
	Rules []TypeRules
}

// GVR is the resource's group, version and plural name.
func (r Resource[Spec, Status]) GVR() schema.GroupVersionResource

// CRD returns the resource's CustomResourceDefinition: one version, served
// and stored, with the status subresource, and a schema reflected from Spec
// and Status and constrained by Rules. It returns one error that names
// every mistake it finds.
func (r Resource[Spec, Status]) CRD() (*apiextensionsv1.CustomResourceDefinition, error)

// Object is one object of a Resource.
type Object[Spec, Status any] struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   Spec   `json:"spec"`
	Status Status `json:"status,omitempty"`
}

// TypeRules constrain the fields of one Go type, wherever the type appears
// in a schema.
type TypeRules struct{ /* unexported */ }

// Rules constrains the fields of T. v only names them: each of fields is a
// Field of a field of *v.
func Rules[T any](v *T, fields ...FieldRule) TypeRules

// Description and Validation describe and constrain the type itself.
// Validation is a CEL rule the API server checks, such as
// "self.minReplicas <= self.maxReplicas".
func (r TypeRules) Description(s string) TypeRules
func (r TypeRules) Validation(rule, message string) TypeRules

// Schema gives T's schema whole, for a type that encodes itself, whose
// schema can't be reflected.
func Schema[T any](s apiextensionsv1.JSONSchemaProps) TypeRules

// FieldRule is a FieldOf of any type.
type FieldRule interface{ /* unexported */ }

// FieldOf is a field of type T, and its constraints. Each method returns a
// copy with one more constraint. A constraint that doesn't fit the field's
// type, such as Pattern on a number, is an error from CRD.
type FieldOf[T any] struct{ /* unexported */ }

// Field names the field p points to.
func Field[T any](p *T) FieldOf[T]

func (f FieldOf[T]) Description(s string) FieldOf[T]
func (f FieldOf[T]) Default(v T) FieldOf[T] // and the field is optional
func (f FieldOf[T]) Enum(values ...T) FieldOf[T]
func (f FieldOf[T]) Minimum(v T) FieldOf[T] // numbers
func (f FieldOf[T]) Maximum(v T) FieldOf[T]
func (f FieldOf[T]) MinLength(n int) FieldOf[T] // strings
func (f FieldOf[T]) MaxLength(n int) FieldOf[T]
func (f FieldOf[T]) Pattern(re *regexp.Regexp) FieldOf[T]
func (f FieldOf[T]) MinItems(n int) FieldOf[T] // slices
func (f FieldOf[T]) MaxItems(n int) FieldOf[T]
func (f FieldOf[T]) ListSet() FieldOf[T]
func (f FieldOf[T]) ListMap(keys ...string) FieldOf[T]
func (f FieldOf[T]) Validation(rule, message string) FieldOf[T] // CEL

// Reconciler brings the cluster to what an object's spec declares.
type Reconciler[Spec, Status any] interface {
	// Reconcile is handed obj, its own copy of the object as the cache last
	// saw it, and returns what obj's status is now.
	Reconcile(ctx context.Context, obj Object[Spec, Status]) (Result[Status], error)
}

// ReconcilerFunc is a function that is a Reconciler.
type ReconcilerFunc[Spec, Status any] func(ctx context.Context, obj Object[Spec, Status]) (Result[Status], error)

func (f ReconcilerFunc[Spec, Status]) Reconcile(ctx context.Context, obj Object[Spec, Status]) (Result[Status], error)

// Result is what a reconcile leaves its object with.
type Result[Status any] struct {
	// Status is the object's status now. nil leaves it as it is. The
	// controller applies it when it differs from the object's, whether or
	// not Reconcile returned an error, so a failure can be a condition.
	Status *Status

	// After reconciles the object again after this long, unless it changes
	// first. Zero waits for a change, or for the resync.
	After time.Duration
}

// LeaderElection is the Lease a controller's replicas elect their leader
// with. The Lease is named after Settings.Name.
type LeaderElection struct {
	Namespace string // the Lease's namespace, such as the pod's
	Identity  string // this replica's name in the Lease, such as its pod's

	LeaseDuration time.Duration // how long a Lease lasts without renewal
	RenewDeadline time.Duration // how long the leader tries to renew
	RetryPeriod   time.Duration // how often a replica tries to take or renew
}

// Settings configures a Controller. Every field is a plain value: see the
// table under Settings for each one's default, range and check.
type Settings struct {
	// Name is the controller's field manager, and its Lease's name.
	Name string

	// Namespace limits the controller to one namespace. "" means every
	// namespace.
	Namespace string

	// Workers is how many objects are reconciled at once. One object is
	// never reconciled twice at once.
	Workers int

	// Resync reconciles every object again this often.
	Resync time.Duration

	// RetryBaseDelay and RetryMaxDelay bound the backoff of an object whose
	// reconcile failed: the delay doubles from the first to the second.
	RetryBaseDelay time.Duration
	RetryMaxDelay  time.Duration

	LeaderElection LeaderElection
}

func DefaultSettings() Settings

func CheckName(name string) error
func CheckWatchNamespace(ns string) error
func CheckNamespace(ns string) error
func CheckIdentity(id string) error
func CheckWorkers(n int) error
func CheckResync(d time.Duration) error
func CheckRetryDelay(d time.Duration) error
func CheckRetryPeriod(d time.Duration) error

// Part is a part of a controller beyond its Reconciler.
type Part interface{ /* unexported */ }

// Owns reconciles an object again when a resource of gvr that it controls
// changes: one whose controller owner reference names the object.
func Owns(gvr schema.GroupVersionResource) Part

// Finalize keeps every object until f has cleaned up after it. The
// controller adds the finalizer name to each object it reconciles. Once an
// object is being deleted, it calls f in place of the Reconciler until f
// returns nil, and then removes name. f's types must be the Controller's,
// which NewController checks.
func Finalize[Spec, Status any](name string, f func(ctx context.Context, obj Object[Spec, Status]) error) Part

// Controller reconciles the objects of one Resource. It is a humus.Service.
type Controller[Spec, Status any] struct{ /* unexported */ }

// NewController returns a Controller of r's objects. It opens no
// connection. It returns one error naming every mistake in settings, in r,
// as CRD reports them, and in parts.
func NewController[Spec, Status any](s *humus.Setup, c *Client, settings Settings, r Resource[Spec, Status], rec Reconciler[Spec, Status], parts ...Part) (*Controller[Spec, Status], error)

func (c *Controller[Spec, Status]) Serve(ctx context.Context) error
func (c *Controller[Spec, Status]) Shutdown(ctx context.Context) error

// Synced is healthy once the controller's caches have synced, on every
// replica, leader or not.
func (c *Controller[Spec, Status]) Synced() health.Monitor

// ErrLostLeadership is what Serve returns when the replica loses the Lease
// while it serves.
var ErrLostLeadership = errors.New("kube: lost the lease")

The module's dependencies are client-go, apimachinery and api v0.37.1, apiextensions-apiserver v0.37.1 for the CRD's types, and klog and logr, which client-go already brings. testcontainers' k3s module is used only behind the build tag.

Connecting

NewClient builds a net/http transport of its own, and hands client-go a rest.Config whose Transport is it:

• It dials s.Host.Dial, and runs TLS with a clone of Connection.TLS, whose ServerName defaults to Server's host.
• It has no proxy. client-go's default, http.ProxyFromEnvironment, would read the real environment.
• It sends HTTP/2 pings: after 30s without a frame, with 15s for the answer. These are the values client-go's own transports use, which they read from environment variables. A watch on a dead connection otherwise waits for TCP to give up.
• It adds the token. It calls Token at most once a minute, as client-go re-reads a token file. When a call fails, it keeps the last token, which a rotated service account token outlives.
• It is wrapped in otelhttp with s.Telemetry's providers and propagator.
• It doesn't limit the client's rate. client-go's default limit is 5 requests a second. The API server's priority and fairness limits clients itself.

So every client-go client built from Config() reaches the API server the same way: the dynamic and metadata clients the Controller uses, and the clientset of Kubernetes' own resources the end user's Reconciler applies its children with.

Settings

Field	Default	Valid	Check
Connection.Server	none	host:port, port 1 to 65535	CheckServer
Connection.TLS	none	not nil	CheckTLS
Connection.Token	nil, no token	any
Name	none	a DNS-1123 subdomain	CheckName
Namespace	"", every namespace	"" or a DNS-1123 label	CheckWatchNamespace
Workers	1	at least 1	CheckWorkers
Resync	10h	positive	CheckResync
RetryBaseDelay	5ms	positive	CheckRetryDelay
RetryMaxDelay	1000s	at least RetryBaseDelay	CheckRetryDelay, and NewController for the bound
LeaderElection.Namespace	none	a DNS-1123 label	CheckNamespace
LeaderElection.Identity	none	not empty	CheckIdentity
LeaderElection.LeaseDuration	15s	longer than RenewDeadline	NewController
LeaderElection.RenewDeadline	10s	longer than 1.2 RetryPeriods	NewController
LeaderElection.RetryPeriod	2s	positive	CheckRetryPeriod

• Server, TLS, Name and the Lease's Namespace and Identity are required. humus imposes no API server, no name and no namespace, as it imposes no port.
• The package reads no environment variable and defines no variable name, as httpserver and queue/rabbitmq don't. In a pod, the end user's Init sources Server from KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, which the kubelet sets. It reads the CA and the token from /var/run/secrets/kubernetes.io/serviceaccount/ through s.Host.FS, and takes the Lease's Namespace and Identity from the downward API (see In use).
• The retry delays are client-go's. They are the per-object backoff of DefaultTypedControllerRateLimiter (default_rate_limiters.go#L50-L56), whose overall limit of 10 retries a second, bursting to 100, the controller keeps as well. The Lease's durations are kube-controller-manager's defaults.

Resources, and their CRD

CRD() reflects Spec and Status into a structural schema:

Go	Schema
bool	type: boolean
int8 to int32, uint8, uint16	type: integer, format: int32
int, int64, uint, uint32, uint64	type: integer, format: int64
float32, float64	type: number
string	type: string
[]byte	type: string, format: byte
a slice or an array	type: array, with items
map[string]V	type: object, with additionalProperties
a struct	type: object, with a property for each exported field, named by its json tag; an embedded struct is inlined, as encoding/json does
*T	T's schema, with nullable: true unless the field is omitempty or omitzero
metav1.Time, metav1.MicroTime	type: string, format: date-time
metav1.Duration	type: string
resource.Quantity, intstr.IntOrString	x-kubernetes-int-or-string, anyOf integer and string
runtime.RawExtension	type: object, x-kubernetes-preserve-unknown-fields
metav1.ObjectMeta, in a template	type: object

• A field is required unless its json tag has omitempty or omitzero, or it has a Default. The encoding decides it: a field the encoder always writes is always there.
• Rules belong to a type, not a path.
  • Rules[T] constrains T's own fields, those promoted from embedded structs included, wherever T appears.
  • A field of a nested struct, such as &spec.Template.Port, is constrained by that struct's own Rules. A rule on it in the outer type's Rules is an error.
• CRD() returns one error naming every mistake it finds:
  • a missing Group, Version, Kind or Plural, or a Scope that is neither;
  • a type that implements json.Marshaler and has no Schema, since its fields aren't its encoding;
  • a map whose keys aren't strings, an interface, a channel or a function;
  • a type that contains itself, which a schema without $ref can't express;
  • a constraint that doesn't fit its field's type;
  • a rule that names no field, and two Rules for one type.
• These are compile errors, as the compiler reports them (checked):
  • kube.Field(&spec.Size).Minimun(1): "kube.FieldOf[int32] has no field or method Minimun".
  • kube.Field(&spec.Size).Minimum("one"): "cannot use "one" (untyped string constant) as int32 value".
  • kube.Field(&spec.Image).Default(3): "cannot use 3 (untyped int constant) as string value".
• The CRD's root has apiVersion, kind and metadata, a required spec, and status. It is named <Plural>.<Group>, and has one version, served and stored, with the status subresource.
• CRD() doesn't compile CEL. The API server compiles every rule when the CRD is applied, which the end user's container tests do (see Testing).

What Serve does

1. Checks the resource is served. It lists the resource once, with a limit of 1. An error, such as a CRD that isn't installed, ends Serve, naming the resource, so the App exits 1 (checked). Without this check, an informer's reflector would retry the list forever, and only log.
2. Starts the informers. There is one on the resource, through the dynamic client, and one for each Owns, through the metadata client, since an owned resource is only looked at for its owner references. All of them are limited to Settings.Namespace, and run with a logger in their context.
3. Waits for the caches to sync, then marks Synced healthy.
4. Campaigns for the Lease. Every replica runs steps 1 to 3, and keeps its caches warm. The Lease's holder runs Workers workers.
5. Reconciles. Each worker takes an object's key from the work queue and reconciles it (next section). The queue never hands one key to two workers at once.

Reconciling

For each key:

1. Look it up in the cache. An object that is gone needs nothing: its owned resources are garbage collected, through their owner references.
2. Decode it into a new Object. The decode is the copy, so nothing the Reconciler does changes the cache.
3. Handle the finalizer, when Finalize is given (see Finalizers).
4. Leave an object that is being deleted alone, when Finalize isn't given.
5. Call Reconcile. Its context carries the reconcile's span, and is cancelled only when Shutdown's context is done or the Lease is lost.
6. Apply the status. When Result.Status isn't nil and differs from the object's, compared by their JSON encodings, the controller applies it to the status subresource with server-side apply. The field manager is Settings.Name, and it forces, since the controller owns the status whole. This happens whether or not Reconcile returned an error.
7. Settle the key:
   • an error retries the object after a backoff of its own, from RetryBaseDelay, doubling to RetryMaxDelay. The error is recorded, and the Service goes on.
   • After reconciles it again after that long, unless it changes first.
   • otherwise the key is forgotten until the object, or a resource it owns, changes, or the resync comes.

Applying a status changes the object, so the controller reconciles it once more. That reconcile returns the same status, which isn't applied, and the object settles: two reconciles in all (checked).

Finalizers

Finalize(name, f):

• Adds the finalizer before the first reconcile. The controller applies metadata.finalizers: [name] with server-side apply, under its field manager.
• Finalizes instead of reconciling. Once the object is being deleted, the controller calls f, and not the Reconciler. An error retries, as a reconcile's does.
• Removes the finalizer once f returns nil. The controller applies the object's metadata without the finalizer. The field manager that set it is the only one that owns it, so the API server removes it, and the deletion completes (checked).

Finalize is generic in f, and a Part isn't, so NewController checks that f takes an Object of the Controller's types.

Owned resources

Owns(gvr) watches gvr's resources through a metadata informer. A change to one whose controller owner reference names the Resource's group and kind reconciles that owner. In the prototype, scaling an owned Deployment by hand was undone within the second (checked).

The end user sets the owner reference when they apply the child, with client-go's apply configurations (see In use).

Leader election

• Every controller elects a leader. There is no option to turn it off. One replica takes the Lease at once. Two replicas without it would both apply the same children and statuses.
• The Lease is released on shutdown, after the reconciles in flight have finished, as ReleaseOnCancel's doc requires: "you must ensure all code guarded by this lease has successfully completed prior to cancelling the context, or you may have two processes simultaneously acting on the critical path" (leaderelection.go#L153-L157). So the next replica of a rollout takes over without waiting out LeaseDuration (checked: the Lease has no holder after exit 0).
• A lost Lease ends the App. Renewal can fail for RenewDeadline, for example while the API server is unreachable, or another replica can take the Lease. Then the controller cancels the reconciles in flight, and Serve returns ErrLostLeadership. The App exits 1, and Kubernetes restarts the pod, which campaigns again.

Shutdown

• On Serve's context, the controller stops taking keys, lets the reconciles in flight finish, releases the Lease, stops its informers, and returns nil. The App exits 0 (checked).
• When Shutdown's context is done first, the controller cancels the reconciles in flight, waits for them to return, and returns an error that wraps the context's error.
• humus.Main's defaults fit a controller, as they fit a consumer: a 20s grace period and no drain. So there is no kube.Main.

Readiness

Synced() is unhealthy until the controller's caches have synced, and healthy from then until Serve returns. That holds on every replica, leader or not, so a rollout isn't held up by the replicas waiting for the Lease. Like rabbitmq's Consuming, it is for readiness only. The end user serves it with exposing health probes, as health.Readiness(ctrl.Synced()).

Telemetry

Under the scope github.com/z5labs/humus/kube, from s.Telemetry:

• Traces:
  • Every reconcile is a root span, reconcile <Kind>, with k8s.namespace.name and kube.object.name. An error sets its status to Error.
  • Every API request is a client span from otelhttp, except watches, which last as long as an informer runs.
• Logs:
  • A failed reconcile is one WARN record, reconcile failed, in the span's context, with the object and the error.
  • client-go's own records reach the LoggerProvider through the logger the controller puts in their context: the reflector's, the work queue's and leader election's. In the prototype, "Successfully acquired lease" arrived as a record, and nothing reached stderr (checked).
• Metrics: otelhttp's client instruments, for every API request. The work queue's own are open question 2.

In use

The kubebuilder book's Memcached operator, in full apart from its settings and its probes. The types, with no markers and no generated code:

package app

type MemcachedSpec struct {
	Size  int32  `json:"size"`
	Image string `json:"image"`
}

type MemcachedStatus struct {
	ReadyReplicas int32              `json:"readyReplicas,omitempty"`
	Conditions    []metav1.Condition `json:"conditions,omitempty"`
}

var (
	spec   MemcachedSpec
	status MemcachedStatus

	Memcached = kube.Resource[MemcachedSpec, MemcachedStatus]{
		Group:   "cache.example.com",
		Version: "v1alpha1",
		Kind:    "Memcached",
		Plural:  "memcacheds",
		Scope:   kube.Namespaced,
		Rules: []kube.TypeRules{
			kube.Rules(&spec,
				kube.Field(&spec.Size).Minimum(1).Maximum(3).Description("How many memcached pods run."),
				kube.Field(&spec.Image).Pattern(regexp.MustCompile(`^memcached:`)).Default("memcached:1.6.26-alpine3.19"),
			),
			kube.Rules(&status,
				kube.Field(&status.Conditions).ListMap("type"),
			),
		},
	}
)

The Reconciler applies a Deployment that the Memcached controls, and reports its ready replicas:

type Reconciler struct{ Kube kubernetes.Interface }

func (r Reconciler) Reconcile(ctx context.Context, m kube.Object[MemcachedSpec, MemcachedStatus]) (kube.Result[MemcachedStatus], error) {
	labels := map[string]string{"app": "memcached", "memcached": m.Name}
	d, err := r.Kube.AppsV1().Deployments(m.Namespace).Apply(ctx,
		appsv1ac.Deployment(m.Name, m.Namespace).
			WithOwnerReferences(metav1ac.OwnerReference().
				WithAPIVersion(m.APIVersion).WithKind(m.Kind).WithName(m.Name).WithUID(m.UID).
				WithController(true).WithBlockOwnerDeletion(true)).
			WithSpec(appsv1ac.DeploymentSpec().
				WithReplicas(m.Spec.Size).
				WithSelector(metav1ac.LabelSelector().WithMatchLabels(labels)).
				WithTemplate(corev1ac.PodTemplateSpec().
					WithLabels(labels).
					WithSpec(corev1ac.PodSpec().WithContainers(
						corev1ac.Container().WithName("memcached").WithImage(m.Spec.Image))))),
		metav1.ApplyOptions{FieldManager: "memcached-controller", Force: true},
	)
	st := m.Status
	if err != nil {
		meta.SetStatusCondition(&st.Conditions, metav1.Condition{
			Type: "Available", Status: metav1.ConditionUnknown, Reason: "ApplyFailed",
			Message: err.Error(), ObservedGeneration: m.Generation,
		})
		return kube.Result[MemcachedStatus]{Status: &st}, err
	}
	st.ReadyReplicas = d.Status.ReadyReplicas
	meta.SetStatusCondition(&st.Conditions, metav1.Condition{
		Type: "Available", Status: metav1.ConditionTrue, Reason: "DeploymentApplied",
		ObservedGeneration: m.Generation,
	})
	return kube.Result[MemcachedStatus]{Status: &st}, nil
}

Compared with the book's controller, there is no Get and no not-found, no Status().Update and no fetching again, and no conflict to retry. A failure is a condition and an error, together.

The end user's Init:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	conn, settings, err := readSettings(ctx, s) // a kube.Connection and kube.Settings, sourced with config.Struct
	if err != nil {
		return nil, err
	}
	client, err := kube.NewClient(s, conn)
	if err != nil {
		return nil, err
	}
	kc, err := kubernetes.NewForConfig(client.Config())
	if err != nil {
		return nil, err
	}
	ctrl, err := kube.NewController(s, client, settings, Memcached, Reconciler{Kube: kc},
		kube.Owns(appsv1.SchemeGroupVersion.WithResource("deployments")),
	)
	if err != nil {
		return nil, err
	}
	return []humus.Service{ctrl /* and the probes' Service, with health.Readiness(ctrl.Synced()) */}, nil
}

In a pod, readSettings sources the Connection from what the kubelet provides:

const serviceAccount = "/var/run/secrets/kubernetes.io/serviceaccount/"

ca, err := readFile(s.Host.FS, serviceAccount+"ca.crt") // the end user's helper, as in httpserver's docs
if err != nil {
	return kube.Connection{}, kube.Settings{}, err
}
roots := x509.NewCertPool()
roots.AppendCertsFromPEM(ca)

conn, err := config.Struct(func(r *config.Resolver) kube.Connection {
	return kube.Connection{
		// The kubelet sets both in every container. JoinHostPort brackets
		// an IPv6 host.
		Server: net.JoinHostPort(r.Get(env("KUBERNETES_SERVICE_HOST")), r.Get(env("KUBERNETES_SERVICE_PORT"))),
		TLS:    &tls.Config{RootCAs: roots},
		Token: func(context.Context) (string, error) {
			b, err := readFile(s.Host.FS, serviceAccount+"token")
			return string(bytes.TrimSpace(b)), err
		},
	}
}).Read(ctx)

readSettings sources the kube.Settings as queue/rabbitmq's docs show, with the Lease's Namespace and Identity from POD_NAMESPACE and POD_NAME, which the pod's spec sets from the downward API.

The CRD in the end user's repo. Whatever deploys the controller applies the CRD's YAML, such as a Helm chart or a kustomization. A golden test keeps that YAML in step with the Go types:

var update = flag.Bool("update", false, "rewrite the golden CRD")

func TestMemcachedCRD(t *testing.T) {
	crd, err := app.Memcached.CRD()
	require.NoError(t, err)
	got, err := yaml.Marshal(crd)
	require.NoError(t, err)
	const golden = "../deploy/crds/memcacheds.cache.example.com.yaml"
	if *update {
		require.NoError(t, os.WriteFile(golden, got, 0o644))
	}
	want, err := os.ReadFile(golden)
	require.NoError(t, err)
	require.Equal(t, string(want), string(got))
}

A change to the types that isn't in the YAML fails go test, in place of a make manifests that someone forgot to run.

Design decisions

• No controller-runtime, and no controller-gen, anywhere. Background says why. depguard denies both in every humus module, so neither can come back as a convenience.

• client-go's parts, without generated clientsets.

  • The dynamic and metadata clients, informers, the work queue and leader election are Kubernetes' own, kept with each release.
  • Reading the custom resource as unstructured data and decoding it into Object removes the scheme, runtime.Object and DeepCopy, which are why kubebuilder generates code at all.
  • A Reconciler applies Kubernetes' own resources with client-go's clientset and apply configurations. Kubernetes generates and publishes those, so no end user generates anything.

• The schema is reflected from the Go types, and constraints are typed values on fields. Four other ways were weighed:

  • struct tags such as kube:"minimum=1": a mini-language in strings, which nothing checks before CRD(), and whose values have no type;
  • methods on the types: the reason #760's revision 3 rejected body types with methods. They add methods to every type, and an embedded type's methods ignore the fields beside it;
  • a hand-built JSONSchemaProps: a second copy of every type, which drifts from the first;
  • field names as strings, such as kube.Field("size"): a typo is found by CRD() rather than the compiler, and a constraint's value has no type.

  A field pointer is resolved by its offset in the value given to Rules, and checked against the field's type. The compiler checks the field's name, the constraint's name and the value's type.

  The cost: a description is a value, not the field's doc comment, so kubectl explain shows only what Description gives. Rules also needs a value of the type to point into, usually a package-level variable. A closure, kube.Rules(func(s *MemcachedSpec) []kube.FieldRule {…}), would need no variable, and reads worse.

• Rules belong to a type. A type such as a port or a template is constrained once, and constrained alike wherever it appears.

• One version per Resource. Several versions need conversion between them, which is a webhook or a promise that the schemas match (see Out of scope).

• The controller never installs its CRD.

  • Installing it would give the controller's ServiceAccount write access to every CRD in the cluster.
  • A CRD outlives the controller's pods, and belongs to whatever deploys them.
  • The golden test keeps the YAML true to the types.
  • A missing CRD fails Serve loudly, rather than leaving a controller that waits forever.

• A reconcile is handed its object. The cache already has it, and the decode copies it. A name would make every Reconcile fetch it and handle not-found, as controller-runtime's do.

• A reconcile returns its status, and the controller applies it.

  • Server-side apply under one field manager has no conflict to retry, and needs no fetch after each write.
  • The status is the controller's whole, so it forces.
  • A nil status leaves it alone, so return kube.Result[S]{}, err can't wipe the conditions.
  • It is applied on an error too, so a failure can be a condition that kubectl get shows.

• One way to reconcile again for each reason: an error backs off, After waits, and nothing waits for a change. There is nothing to deprecate.

• A reconcile error never ends the Service, unlike queue's contract. A failed message comes back when the App restarts, so ending the Service is how queue retries. A controller is level-triggered: its queue retries the object, and a restart would only reconcile the same object again, after a crash loop's back-off.

• Finalizers are applied, not updated. Adding and removing one is an apply under the controller's field manager, so it never conflicts with other writers of the metadata.

• Leader election is always on. Apart from the Lease's RBAC, it costs one replica nothing. And two replicas without it would apply the same objects twice.

• Losing the Lease ends the App. Another replica may already be reconciling. client-go's own example exits when it stops leading (main.go#L150-L165), and the pod campaigns again when it restarts.

• Every replica keeps its caches warm. A new leader starts reconciling at once, and Synced can be every replica's readiness.

• The client has no rate limit of its own. client-go's default of 5 requests a second throttles a busy controller. The API server's priority and fairness protects it, and limits each client by its identity.

• The Client has its own transport. It is what dials through the Host, and it keeps client-go's environment variables out.

• Webhooks stay with openapi. An admission webhook is an HTTPS POST of JSON, which openapi serves. Note that a denial is a 200 with allowed: false, since any other status is a failure under the webhook's failurePolicy. CEL policies remove most webhooks anyway: ValidatingAdmissionPolicy is stable since v1.30, and MutatingAdmissionPolicy since v1.36.

• The package is kube, not kubernetes. client-go's clientset package is named kubernetes, and the end user imports it alongside.

• A nested module. client-go and apimachinery are large, and only a controller should bring them into a module graph.

Testing

The package's tests

• Unit tests run inside synctest, on client-go's own fakes.
  • Kubernetes keeps the fake dynamic, metadata and typed clients, as franz-go keeps kfake.
  • They don't serve HTTP, so they stand in for the Controller's clients, set behind an unexported seam, as queue/rabbitmq's SetOpen sets its channel.
  • The Client's transport, the token and the Host are unit tested on humustest's Host, with a plain HTTP handler that records what it receives.
• The fake dynamic client can't apply an unstructured object. It handles an apply as a strategic merge patch, which needs a typed struct, and fails with "unable to find api field in struct Unstructured for the json field "metadata"" (checked). So the unit tests apply through a reactor of their own. What server-side apply does is tested against k3s.
• Container tests run against k3s.
  • They use testcontainers' k3s module with rancher/k3s:v1.37.1-k3s1, started as server --disable-agent. That is the API server and the controller manager without a kubelet: no pod runs, but the garbage collector does.
  • Rootless Podman can't run the agent, since it doesn't delegate the cpuset controller.
  • They sit behind the testcontainers build tag, in a CI job of their own that isn't required, outside the coverage gate.
  • This is principle 6's exception as #629's revision 10 defines it. No Kubernetes API server runs on humustest's Host unless humus writes one, and humus doesn't write other projects' servers.

What the end user tests

• Their Reconciler, as a function. It is handed a plain Object, so a test builds one. Its children are applied through kubefake.NewClientset(), whose field-managed tracker applies typed objects with their managed fields (checked).
• Their CRD, with the golden test above.
• Their whole App, against k3s, on humus.OSHost() and humustest's process, as RabbitMQ's container tests run.

The prototype

A prototype was built in a scratch module, against:

• humus at d1cfa0bf;
• client-go, apimachinery and apiextensions-apiserver v0.37.1;
• testcontainers-go's k3s module v0.44.0;
• rancher/k3s:v1.37.1-k3s1.

It is about 1,000 lines: 343 for the schema, 100 for Resource and CRD, 120 for the Client and 461 for the Controller. It passes go test -race, the unit test with -count=5. Against k3s, it shows:

• The reflected CRD is accepted, with fieldValidation=Strict. The API server enforces its rules:

  • "spec.size in body should be greater than or equal to 1";
  • "spec.image in body should match '^memcached:'".

  It also defaults image when it is left out.

• A type array is rejected, as openapi.SchemaOf would write a pointer.

• A whole App, on humus.OSHost():

  • It applies a Deployment that a Memcached controls, and applies the Memcached's status. The object's managed fields are memcached-controller Apply and Apply on status.
  • It undoes a hand-made scale of the Deployment.
  • Deleting the Memcached runs its finalizer once, and removes it.
  • On SIGTERM it exits 0, and the Lease has no holder.

• A missing CRD ends the App with exit 1: "kube: listing cache.example.com/v1alpha1, Resource=absents: the server could not find the requested resource".

Inside synctest, on client-go's fakes, the informers, the work queue and leader election run through a minute of fake time:

• the object is reconciled twice, once for itself and once for its own status;
• After reconciles it again;
• the Lease is held, then released on shutdown;
• every reconcile is a span;
• client-go's records reach the LoggerProvider;
• no goroutine outlives the bubble.

It also found four things this proposal now accounts for:

1. The work queue leaks a goroutine when drained. ShutDownWithDrain doesn't stop the delaying queue's goroutine; only ShutDown does (delaying_queue.go#L240, queue.go#L327). So Serve calls ShutDown after draining. synctest's leak check caught it, and humustest.TestService's will.
2. The fake dynamic client can't apply, as above.
3. Without a logger in the context, client-go writes to stderr. Leader election wrote klog's lines to the real stderr until the controller put a logger in the context.
4. A status in Result needs to be a pointer. With a value, an error path that returns a zero Result would wipe the object's status.

These weren't prototyped:

• the instruments;
• Shutdown's deadline;
• panics;
• the token's cache across a rotation;
• the checks;
• TypeRules.Validation, Schema, ListSet, MinItems, MaxItems, MinLength and MaxLength;
• the filter that keeps watches out of traces.

Migration from today

humus has never had Kubernetes support, so nothing in humus changes. An end user moving from kubebuilder maps its pieces like this:

kubebuilder and controller-runtime	kube
kubebuilder init, create api: 54 files in 20 directories	the module, and the end user's own files
// +kubebuilder:validation:Minimum=1 and other markers	kube.Field(&spec.Size).Minimum(1), in Resource.Rules
doc comments as descriptions	.Description(…)
// +kubebuilder:object:root=true, zz_generated.deepcopy.go, make generate	nothing: kube.Object[Spec, Status]
make manifests	Resource.CRD(), and a golden test
AddToScheme, and a scheme in main	nothing
Reconcile(ctx, req), then r.Get and IsNotFound	Reconcile(ctx, obj)
r.Status().Update(ctx, obj), then r.Get again	kube.Result{Status: &st}
Result{Requeue: true}, RequeueAfter, a returned error	a returned error, or Result.After
controllerutil.AddFinalizer, r.Update, RemoveFinalizer	kube.Finalize(name, f)
Owns(&appsv1.Deployment{})	kube.Owns(appsv1.SchemeGroupVersion.WithResource("deployments"))
ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{…})	kube.NewClient and kube.NewController, in Init
ctrl.SetupSignalHandler(), mgr.Start	humus.Main
LeaderElection: true, LeaderElectionID	always, named Settings.Name
HealthProbeBindAddress, AddReadyzCheck	ctrl.Synced(), served with health probes
metrics.Registry.MustRegister	the MeterProvider in s.Telemetry
log.SetLogger(zap.New(…))	the LoggerProvider in s.Telemetry
envtest	k3s in a container
// +kubebuilder:rbac:…	the rules, written by the end user (see Out of scope)
// +kubebuilder:webhook:…, cert-manager	an openapi API on httpserver

Open questions

1. A panic in Reconcile or a finalizer.

   • (a) Recover it. The controller records the panic as openapi and grpcserver record a handler's: an exception event with the stack, the span's status set to Error, and one ERROR record, reconcile panicked. Then it retries the object with backoff. Like a request, a reconcile is about one object, and one object's bug doesn't stop the others.
   • (b) Crash, as queue does. The crash loop shows the bug. But it stops reconciling every object, and a restart doesn't help a level-triggered loop: the same object panics again after every restart.

   Recommendation: (a).

2. Instruments of the controller's own. OpenTelemetry has no conventions for controllers.

   • (a) Define them here, under kube.controller., with kube.controller.name:
     • the work queue's depth, adds, time in the queue, work duration, retries, unfinished work and longest running reconcile, through a work queue MetricsProvider of humus's own for each queue;
     • a reconcile duration with its outcome.
   • (b) None. The spans and records, and otelhttp's request instruments, are all there is.

   Recommendation: (a).

3. Local development against a cluster.

   • (a) The end user fills the Connection. The docs show filling it from a kubeconfig's embedded data, as kind writes it, with clientcmd.Load.
   • (b) kube.FromKubeconfig(fsys, path, context) (Connection, error). It would read the files a kubeconfig names through the FS, and reject exec plugins and auth providers. humus would then own a credential file's semantics, which grow with every cloud.

   Recommendation: (a).

4. The end user's whole-App tests.

   • (a) Against k3s, as proposed.
   • (b) A kubetest package that builds a Client on client-go's fakes, so a whole App runs inside synctest. But the fakes can't apply an unstructured object (checked), and have no admission, validation, defaulting or garbage collection. So a test passes where the cluster would reject.

   Recommendation: (a).

Out of scope

• Admission and conversion webhooks, which an openapi API serves on httpserver, and CEL admission policies, which need no server.

• Several versions of one resource, and conversion between them.

• CRD details: printer columns, short names, categories, and the scale subresource.

• Mapping resources the controller doesn't own to its objects, as controller-runtime's Watches does with a map function. Owns covers owned resources, and Result.After polls the rest.

• Generating RBAC. The docs list the rules a Controller needs itself:

  • get, list and watch on its resource, and patch on it and its status;
  • get, create and update on Leases, in the Lease's namespace;
  • list and watch on each owned resource.

  The end user adds their Reconciler's own.

• Installing CRDs.

• Exec credential plugins and auth providers. Token covers a cloud's tokens, from the cloud's own SDK in the end user's code.

• Typed clientsets, listers and informers for a custom resource, which client-gen generates.

• Aggregated API servers, and Metacontroller.

• Compiling CEL in CRD().

What the stories change outside the module:

• .github/workflows/build.yaml gets a "Kubernetes (testcontainers)" job, which isn't a required check, and a step that vets the tagged tests. .github/scripts/go-modules.sh finds the new module.
• .golangci.yaml enables depguard, and denies sigs.k8s.io/controller-runtime and sigs.k8s.io/controller-tools in every module. forbidigo gains:
  • rest.InClusterConfig, which reads the real environment and disk;
  • klog's SetLogger, SetLoggerWithOptions, SetOutput and InitFlags;
  • features.ReplaceFeatureGates, which is global.
• The docs site gets a new page, "Writing a Kubernetes controller". The lifecycle page's ambient leaks gain client-go's feature gates and klog's global logger. The root package doc names kube.

References

• controller-runtime v0.25.2 and controller-tools v0.18.0; the kubebuilder book's CRD validation markers
• client-go v0.37.1, and its fakes' tracker
• Kubernetes: CRDs and structural schemas, server-side apply, finalizers, owners and dependents, Leases, admission webhooks, ValidatingAdmissionPolicy, MutatingAdmissionPolicy
• testcontainers-go's k3s module v0.44.0
• humus: queue's contract, Proposal: RabbitMQ

[Zaba505]

Revision 1: shutdown leaves the backlog, and the work queue's goroutine lingers rather than leaks

Two of the post's claims about client-go's work queue were wrong, and the first hid a bug in the prototype:

1. Get keeps handing out queued keys after shutdown. Once ShutDown or ShutDownWithDrain is called, a worker's Get still returns each key left in the queue. It reports shutdown only once the queue is empty (queue.go#L265-L284). The prototype drained the queue with ShutDownWithDrain, so on SIGTERM it reconciled its whole backlog, not just the reconciles in flight that Shutdown described. A new test shows it: with three objects queued, and the first one's reconcile in flight at shutdown, it reconciled all three.
2. ShutDownWithDrain doesn't leak the delaying queue's goroutine. It delays the goroutine's exit. The post said only ShutDown stops it.
   • waitingLoop checks whether the queue is shutting down each time it wakes (delaying_queue.go#L291), and its heartbeat wakes it every 10s (#L273). So after ShutDownWithDrain, it exits within 10s, and nothing stops its heartbeat ticker.
   • ShutDown stops both at once (#L240-L246).
   • synctest reported the goroutine as blocked because the test's bubble ended first. In real time, it exited 10s after ShutDownWithDrain returned (checked).

So the controller no longer drains its queue. When shutdown begins, its workers stop taking keys themselves, and the queue is shut down with ShutDown. A key a worker takes after that is marked done, unreconciled, and left for the next leader. The prototype now does this. With the same backlog, it reconciles only the object in flight, waits for it, releases the Lease and exits 0. The Proposed API doesn't change, and neither do the open questions.

Each section below replaces or adds to the part of the same name. The rest of the post stands.

Shutdown

• On Serve's context, the controller stops taking keys:

  • each worker finishes the reconcile it has in flight;
  • a key a worker takes after shutdown has begun is marked done without being reconciled;
  • the work queue is shut down with ShutDown, which stops its delaying goroutine and heartbeat at once. It is never drained: ShutDownWithDrain would hand the workers the backlog.

  Then the controller releases the Lease, stops its informers, and returns nil. The App exits 0 (checked).

• When Shutdown's context is done first, the controller cancels the reconciles in flight, waits for them to return, and returns an error that wraps the context's error.

• humus.Main's defaults fit a controller, as they fit a consumer: a 20s grace period and no drain. So there is no kube.Main.

Leader election

Only the bullet "A lost Lease ends the App" changes:

• A lost Lease ends the App. Renewal can fail for RenewDeadline, for example while the API server is unreachable, or another replica can take the Lease. Then the controller stops taking keys, as it does on shutdown, and cancels the reconciles in flight. Serve returns ErrLostLeadership. The App exits 1, and Kubernetes restarts the pod, which campaigns again.

Design decisions

These are added:

• Shutdown leaves the backlog. A controller is level-triggered: the next leader lists every object when its informer starts, and reconciles each one.
  • Reconciling the backlog during shutdown would spend the grace period on objects another replica reconciles anyway.
  • It would also hold the Lease, which delays that replica's takeover.
  • Only a reconcile in flight is finished, so a cancelled context never leaves an object half-applied.
• The work queue is never drained.
  • ShutDownWithDrain's doc says it "waits until all items in the queue have been processed", but it waits only for the keys in flight (queue.go#L321-L342).
  • The open PR kubernetes#141079 would make it wait for the queued keys too.
  • Neither is what the controller wants, so it doesn't call it, and stops its workers itself.

The prototype

The first of the four things the prototype found is replaced, and the Controller is now 476 lines:

1. Get hands out the backlog after shutdown, as above. So Serve stops its workers itself, and shuts the queue down with ShutDown. A test queues three objects with one worker, and holds the first one's reconcile in flight at shutdown (checked, -race -count=5):

   • Against the previous Serve, all three are reconciled.
   • Now only the first is. Serve waits for it, and returns nil with the Lease released.

   The delaying queue's goroutine outlived ShutDownWithDrain by up to 10s, which synctest's leak check caught. It doesn't outlive ShutDown.

The container tests against k3s pass unchanged with the new shutdown.

[Zaba505]

Revision 2: a fifth open question, on controllers of built-in kinds

Every Controller in the post reconciles a custom resource. NewController takes a Resource, whose CRD is reflected from its types, and the Reconciler is handed an Object with a spec and a status. The post doesn't say whether a controller can reconcile one of Kubernetes' own kinds instead, and Out of scope doesn't rule it out.

Open questions

Questions 1–4 are as in the post. Question 5 is new:

5. A controller of a built-in kind. Some controllers reconcile one of Kubernetes' own kinds, not a custom resource: one that gives every Namespace a default NetworkPolicy, or one that copies a Secret into every namespace. controller-runtime reconciles them with For(&corev1.Namespace{}). Two things in the post don't fit them:

   • Object assumes a spec and a status. A ConfigMap, a Secret and a Role have neither.
   • A built-in's status is written by Kubernetes' own controllers: a Deployment's by the deployment controller, and a Pod's by the kubelet. A controller of a built-in kind has no status of its own to return.

   The options:

   • (a) Out of scope. kube reconciles custom resources only. Such a controller is written on client-go's informers, work queue and leader election directly, with Client.Config() for the transport. It gets none of the Controller's Lease handling, shutdown, telemetry or Synced.
   • (b) A Controller of a client-go type. Its Reconciler is handed client-go's own struct, such as a corev1.Namespace, decoded from the cache as an Object is, and returns only After. Owns, Finalize, the Lease, shutdown, telemetry and Synced work as they do for a custom resource. The Proposed API gains a second constructor, Reconciler and Result for it, or Controller, Reconciler and Finalize become generic over the object's type. A revision would choose between the two.
   • (c) A Resource can name a built-in kind, reconciled as an Object of client-go's spec and status types, such as Object[appsv1.DeploymentSpec, appsv1.DeploymentStatus]. It fits only a kind with a spec and a status. There is no CRD to reflect for one, yet NewController checks r as CRD() does. And a Result.Status that isn't nil would be applied, with force, over the status another controller writes.

   Recommendation: (b). Leaving these controllers out sends their authors back to wiring client-go by hand, where each has to find what the prototype found: leader election writes to the real stderr until a logger is in its context. An end user moving a controller of Pods off controller-runtime would also have nothing to move it to.

[Zaba505]

Revision 3: every name humus defines starts with humus.

Every telemetry name humus defines of its own now starts with humus.: instruments, attributes, and the names of span events. This is a standard for every proposal. Proposal: humus's own metrics' revision 1 set it, for its instruments and the breakers' log records, and gives the reasons. A name generic enough for another library to use too can't collide once it's prefixed. Most of these series end in Prometheus, where the instrumentation scope is only a label. OpenTelemetry's naming rules recommend a prefix for names it doesn't define, and libraries that define their own use their own name, as gRPC, Resilience4j and Polly do. Its revision 2 extends it to health.probe.

This proposal's names take the prefix:

In the post	Now
kube.object.name, on every reconcile's span	humus.kube.object.name
open question 2's option (a): instruments "under kube.controller."	under humus.kube.controller.
open question 2's option (a): kube.controller.name	humus.kube.controller.name

Wherever the post names one of them, read the new name. These keep their names:

• OpenTelemetry's own, such as k8s.namespace.name on the reconcile's span, and otelhttp's client instruments.
• The span's name, reconcile <Kind>, and the records' bodies, such as reconcile failed. They aren't attribute or instrument names.

Nothing else changes. Open question 2 is still open: this changes only the names that its option (a) would use. Revisions 1 and 2 use no telemetry names.

[Zaba505]

Revision 4: open questions 1 to 4 answered

The answers:

1. A panic in Reconcile or a finalizer is recovered, and never retried. This is option (a) without its retry. A panic is a fundamental mistake in the controller, such as a nil pointer, an index out of range or a division by zero. It shows that the developer's assumptions about how the system works are wrong, not that the system is briefly unavailable. A retry runs the same code on the same object, and panics the same way.
2. The controller defines instruments of its own, option (a), under humus.kube.controller., as revision 3 renamed them. Instruments, below, defines them. Two of the work queue's metrics that option (a) listed aren't recorded: the work duration and the retries.
3. The end user fills the Connection, option (a), so a test fills it with whatever its cluster needs, such as the k3s container's kubeconfig. There is no kube.FromKubeconfig.
4. The end user's whole-App tests run against k3s, option (a), as queue/kafka's and queue/rabbitmq's run against their brokers. There is no kubetest.

Question 5 is still open.

Each section below replaces or adds to the part of the same name. The rest of the post, with revisions 1 to 3, stands.

Summary

This bullet is added under "A reconcile is handed its object and returns its status":

• A panic is recovered and recorded, and never retried.

Reconciling

Step 7 gains a bullet:

• a panic, in Reconcile or a finalizer's f, is recovered and recorded (see Telemetry). The key is forgotten, as a success without After is. Nothing retries it, and no status is applied, since there is no Result. The Service goes on, and so do the other objects' reconciles.

A panicked object is reconciled again only as any settled object is: when it, or a resource it owns, changes, when the resync comes, or when a new leader takes over. A change while it ran queues it again once it's done, as after any reconcile. So the rollout that ships the fix reconciles it with the fixed code, without anyone touching the object. Until then, it panics once per resync, not once per backoff.

Finalizers

The second bullet becomes:

• Finalizes instead of reconciling. Once the object is being deleted, the controller calls f, and not the Reconciler. An error retries, as a reconcile's does. A panic isn't retried, as a reconcile's isn't: the object keeps its finalizer, and its deletion waits for its next reconcile, such as the one the fix's rollout makes.

Telemetry

The Logs and Metrics bullets become:

• Logs:
  • A failed reconcile is one WARN record, reconcile failed, in the span's context, with the object and the error.
  • A panic is one ERROR record, reconcile panicked, in the span's context, with the object and OpenTelemetry's exception.type, exception.message and exception.stacktrace. The span gets an exception event with the same three, and its status is set to Error. This is how openapi records an operation's panic (openapi/v3/serve.go#L364-L379).
  • client-go's own records reach the LoggerProvider through the logger the controller puts in their context: the reflector's, the work queue's and leader election's. In the prototype, "Successfully acquired lease" arrived as a record, and nothing reached stderr (checked).
• Metrics: otelhttp's client instruments, for every API request, and the controller's own (see Instruments).

Instruments

This section is new. NewController creates the instruments from s.Telemetry.MeterProvider.Meter("github.com/z5labs/humus/kube"), and an error creating one is among the errors it returns.

humus.kube.controller.queue.depth                            Int64UpDownCounter  {object}
    "Objects waiting in the work queue for a worker."
humus.kube.controller.queue.adds                             Int64Counter        {object}
    "Objects added to the work queue, not counting one already waiting."
humus.kube.controller.queue.duration                         Float64Histogram    s
    "How long an object waited in the work queue for a worker."
humus.kube.controller.reconcile.duration                     Float64Histogram    s
    "How long each reconcile took, by its outcome."
humus.kube.controller.reconcile.unfinished_duration          Float64Gauge        s
    "How long the reconciles in flight have run, added together."
humus.kube.controller.reconcile.longest_unfinished_duration  Float64Gauge        s
    "How long the longest reconcile in flight has run."

Instrument	Recorded
humus.kube.controller.queue.depth	+1 when a key is added, and −1 when a worker takes it
humus.kube.controller.queue.adds	when a key is added, unless it is already waiting
humus.kube.controller.queue.duration	when a worker takes a key: the time since it was added. The wait of After or of a backoff comes before the key is added, so it isn't counted
humus.kube.controller.reconcile.duration	at the end of each reconcile <Kind> span, over the same time
humus.kube.controller.reconcile.unfinished_duration and humus.kube.controller.reconcile.longest_unfinished_duration	every 500ms, by the work queue, while it runs

Attribute	Values	On
humus.kube.controller.name	Settings.Name	every data point
humus.kube.controller.reconcile.outcome	success, error, panic	humus.kube.controller.reconcile.duration

The outcomes follow step 7 of Reconciling: error is a key retried with backoff, panic is a recovered panic, and success is everything else, After included.

• Every instrument but reconcile.duration comes from client-go's work queue, through a workqueue.MetricsProvider of the controller's own, set in the queue's config. The queue is named Settings.Name. client-go hands the name to the provider, which records it as humus.kube.controller.name. The queue updates the two gauges from a goroutine of its own, which runs only when the queue has a provider, and stops with the queue (queue.go#L175-L184).
• Two of the queue's metrics aren't recorded. The provider returns a no-op for each:
  • the work duration times what reconcile.duration times, from a worker taking a key to settling it, without the outcome;
  • the retries are counted on every AddAfter (delaying_queue.go#L255), so every Result.After would count as a retry. The error outcome counts the retries exactly.
• Both histograms take OpenTelemetry's advice for a duration in seconds, [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10], as http.server.request.duration does. The SDK's default boundaries are for milliseconds.
• A standby records them too. It runs no workers, so nothing leaves its queue. Its queue.depth holds a key for every object. Until it takes the Lease, its reconcile.duration records nothing, and its two gauges read 0. Read queue.depth per replica, by its resource's attributes: a sum over the replicas counts the standbys' keys too.
• Nothing per object. No data point has an object's namespace or name. An object's reconciles are its spans.

In Prometheus, they are humus_kube_controller_queue_depth, humus_kube_controller_queue_adds_total, humus_kube_controller_queue_duration_seconds, humus_kube_controller_reconcile_duration_seconds, humus_kube_controller_reconcile_unfinished_duration_seconds and humus_kube_controller_reconcile_longest_unfinished_duration_seconds.

Design decisions

These are added:

• A panic is recovered, and never retried. A panic means the controller's assumptions about the system are wrong, and a retry runs the same code on the same object, so it panics again. Backoff would only repeat the same stack trace, up to every RetryMaxDelay. An error is how a Reconciler asks to be tried again, and a panic never asks. Recovering it keeps one object's bug from stopping every other object's reconciles, and the fix's rollout reconciles the object anyway. controller-runtime differs: by default, it recovers a panic as the error panic: <value> [recovered], which retries the object with backoff (controller.go#L196-L213, #L486-L491).
• The queue's instruments come through a provider of the controller's own. client-go's workqueue.SetProvider sets a package-level provider, and only its first call has an effect (metrics.go#L249-L255). A provider in the queue's config belongs to one controller, and records into the MeterProvider in s.Telemetry.
• The end user fills the Connection. A kubeconfig's semantics grow with every cloud's credentials, and a Connection is three fields that a test sets directly.
• Whole-App tests run against k3s, with no kubetest. client-go's fakes can't apply an unstructured object, and have no admission, validation, defaulting or garbage collection, so a test on them passes where a cluster would reject.

Testing

The package's tests gain:

• A panic, in a Reconciler and in a finalizer's f, inside synctest. The Service goes on. The span has an exception event and an Error status, and there is one ERROR record, reconcile panicked, with the stack. The object isn't reconciled again until it changes, and another object is still reconciled.
• The instruments, read with humustest.Telemetry.Metrics: each one's name, kind and unit, humus.kube.controller.name on every data point, and reconcile.duration's outcome for a success, an error and a panic.

Out of scope

These are added:

• Reading a kubeconfig. The end user fills the Connection. The docs show filling it from a kubeconfig's embedded data with clientcmd.Load.
• A kubetest package. The end user's whole-App tests run against k3s.

What the stories change outside the module gains:

• forbidigo denies client-go's package-level metrics setters: workqueue.SetProvider, leaderelection.SetProvider and metrics.Register in k8s.io/client-go/tools/metrics.
• The docs page "Writing a Kubernetes controller" lists the instruments, and says that a panic isn't retried.

Open questions

Questions 1 to 4 are answered above. Question 5, a controller of a built-in kind, is still open.

[Zaba505]

Revision 5: open question 5 answered, so a controller can reconcile a built-in kind

A controller can reconcile one of Kubernetes' own kinds, such as a Namespace or a Secret. This is option (b) of revision 2. It is easy to add, since everything but the Reconciler is the same as for a custom resource. And it is an easy request to predict: an end user moving a controller of Pods or Namespaces off controller-runtime would ask for it first.

Revision 2 left one choice to a revision: a second constructor and Reconciler, or a Controller, Reconciler and Finalize that are generic over the object's type. This revision takes both halves where each fits:

• A second constructor and Reconciler. A Reconciler of a built-in kind has no status of its own to return. Kubernetes' own controllers write a built-in kind's status, and option (c) was set aside for applying one over theirs. A Reconciler generic over the object's type would still carry a status type, which a built-in kind would have to fill with something meaningless. So NewKindController takes a KindReconciler, which returns only when to reconcile again.
• Controller and Finalize lose Spec and Status. No method of Controller uses them, so both constructors return one *Controller. Finalize becomes generic over the object's type, so one Finalize serves both.

All five open questions are now answered.

Each section below replaces or adds to the part of the same name. The rest of the post, with revisions 1 to 4, stands.

Summary

This bullet is added after "A reconcile is handed its object and returns its status":

• A built-in kind can be reconciled too. kube.Kind[corev1.Namespace] names one, and NewKindController reconciles it. Its Reconciler is handed client-go's own struct, and returns only when to reconcile it again. It applies no status. Owns, Finalize, the Lease, shutdown, telemetry and Synced work as they do for a custom resource.

Proposed API

These are added:

// Kind names a kind whose Go type is defined elsewhere, such as one of
// Kubernetes' own, corev1.Namespace. Its objects decode into T. Its status
// belongs to whoever defines the kind, so a controller of a Kind applies no
// status.
type Kind[T any] struct {
	GVR   schema.GroupVersionResource // such as corev1.SchemeGroupVersion.WithResource("namespaces")
	Kind  string                      // such as Namespace
	Scope Scope
}

// KindReconciler brings the cluster to what an object of a Kind declares.
type KindReconciler[T any] interface {
	// Reconcile is handed obj, its own copy of the object as the cache last
	// saw it. A positive after reconciles obj again after that long, unless
	// it changes first. Zero waits for a change, or for the resync.
	Reconcile(ctx context.Context, obj T) (after time.Duration, err error)
}

// KindReconcilerFunc is a function that is a KindReconciler.
type KindReconcilerFunc[T any] func(ctx context.Context, obj T) (time.Duration, error)

func (f KindReconcilerFunc[T]) Reconcile(ctx context.Context, obj T) (time.Duration, error)

// NewKindController returns a Controller of k's objects. It opens no
// connection. It returns one error naming every mistake in settings, in k
// and in parts. *T must be a metav1.Object, as every client-go type's is.
func NewKindController[T any](s *humus.Setup, c *Client, settings Settings, k Kind[T], rec KindReconciler[T], parts ...Part) (*Controller, error)

These replace their namesakes:

// Finalize keeps every object until f has cleaned up after it. The
// controller adds the finalizer name to each object it reconciles. Once an
// object is being deleted, it calls f in place of the Reconciler until f
// returns nil, and then removes name. T is the Controller's object type:
// Object[Spec, Status] for a Resource, and T for a Kind[T]. NewController
// and NewKindController check it.
func Finalize[T any](name string, f func(ctx context.Context, obj T) error) Part

// Controller reconciles the objects of one Resource or one Kind. It is a
// humus.Service.
type Controller struct{ /* unexported */ }

// NewController returns a Controller of r's objects. It opens no
// connection. It returns one error naming every mistake in settings, in r,
// as CRD reports them, and in parts.
func NewController[Spec, Status any](s *humus.Setup, c *Client, settings Settings, r Resource[Spec, Status], rec Reconciler[Spec, Status], parts ...Part) (*Controller, error)

func (c *Controller) Serve(ctx context.Context) error
func (c *Controller) Shutdown(ctx context.Context) error

// Synced is healthy once the controller's caches have synced, on every
// replica, leader or not.
func (c *Controller) Synced() health.Monitor

A call such as kube.Finalize("cache.example.com/finalizer", func(ctx context.Context, m kube.Object[MemcachedSpec, MemcachedStatus]) error {…}) reads as before, since T is inferred from f.

Kinds

This section is new, after "Resources, and their CRD".

A Kind is reconciled as a Resource is (see What Serve does, and Reconciling), except:

• The object is client-go's own struct, decoded from the cache into a new T, as an Object is.
• No status is applied. Step 6 of Reconciling is skipped, and step 7 settles the key with the Reconciler's after in place of Result.After.
• There is no CRD. A Kind has no CRD() and no Rules. The Kind's own definition is the API server's.
• Owns matches the Kind. A change to a resource whose controller owner reference names the Kind's group, GVR.Group, and its Kind reconciles that owner. A namespaced resource can name a cluster-scoped owner, such as its Namespace.

NewKindController returns one error naming every mistake it finds:

• a GVR without a Version or a Resource. Its Group is "" for the core group;
• a missing Kind, or a Scope that is neither;
• a T whose pointer isn't a metav1.Object.

A cluster-scoped Kind or Resource can't be limited to a namespace. NewKindController, and NewController for a Resource, return an error when Scope is Cluster and Settings.Namespace isn't "".

In use

This is added after the Memcached example. A controller gives every Namespace a NetworkPolicy that denies ingress, and puts it back when it's changed or deleted:

var Namespaces = kube.Kind[corev1.Namespace]{
	GVR:   corev1.SchemeGroupVersion.WithResource("namespaces"),
	Kind:  "Namespace",
	Scope: kube.Cluster,
}

type DefaultDeny struct{ Kube kubernetes.Interface }

func (r DefaultDeny) Reconcile(ctx context.Context, ns corev1.Namespace) (time.Duration, error) {
	_, err := r.Kube.NetworkingV1().NetworkPolicies(ns.Name).Apply(ctx,
		networkingv1ac.NetworkPolicy("default-deny", ns.Name).
			WithOwnerReferences(metav1ac.OwnerReference().
				WithAPIVersion("v1").WithKind("Namespace").WithName(ns.Name).WithUID(ns.UID).
				WithController(true)).
			WithSpec(networkingv1ac.NetworkPolicySpec().
				WithPodSelector(metav1ac.LabelSelector()).
				WithPolicyTypes(networkingv1.PolicyTypeIngress)),
		metav1.ApplyOptions{FieldManager: "default-deny", Force: true},
	)
	return 0, err
}

In the end user's Init:

ctrl, err := kube.NewKindController(s, client, settings, Namespaces, DefaultDeny{Kube: kc},
	kube.Owns(networkingv1.SchemeGroupVersion.WithResource("networkpolicies")),
)

settings.Namespace is "", since a Namespace is cluster-scoped.

Testing

The package's tests gain:

• A controller of a built-in kind, inside synctest on client-go's fakes: its Reconciler is handed client-go's struct, its after reconciles the object again, a change to a resource it owns reconciles it, Finalize runs on it, and no status is applied.
• The checks of a Kind, and of a cluster-scoped Kind or Resource with a Settings.Namespace.

The container tests gain:

• A whole App with a controller of Namespaces, against k3s. It applies a resource that each Namespace owns, such as a ConfigMap, puts it back when it's deleted, and exits 0 on SIGTERM. k3s runs without its agent, so the owned resource can't be one that needs a pod.

The prototype

These weren't prototyped either: a controller of a built-in kind, and the checks of a Kind.

Migration from today

These rows are added:

kubebuilder and controller-runtime	kube
ctrl.NewControllerManagedBy(mgr).For(&corev1.Namespace{})	kube.NewKindController, with a kube.Kind[corev1.Namespace]
a Reconciler of a built-in kind, Reconcile(ctx, req), then r.Get	Reconcile(ctx, ns corev1.Namespace) (time.Duration, error)

Out of scope

The RBAC that the docs list for a Controller gains, for a controller of a Kind:

• get, list and watch on its kind, and patch on it when Finalize is given.

Open questions

All five are answered: questions 1 to 4 in revision 4, and question 5 above.