Proposal: Kubernetes controller walkthrough

[Zaba505]

Important

Revision 1 brings this proposal in line with the job and RabbitMQ processor walkthroughs that landed, and with #629's revisions 13 to 17. The walkthrough is a module of its own, example/databases. The Walkthroughs section and the docs test's rule already exist. Its code has no comments, and its pages are code-first steps. It ships Grafana alert rules and a dashboard, with no Prometheus. Its kind overlay runs the LGTM stack, brought up by up.sh. Its probes are served over TLS. Its own points are postgresql:provision and postgresql:drop. #851 and #922 are done, so nothing blocks its stories. Open questions 2 to 4 take #629's answers. Revision 2 answers question 1: the PostgreSQL database provisioner. Every question is answered.

Note

A new feature for humus, with no wave. Proposal: humus on the bedrock process model plans walkthroughs in its wave 6, for features old humus had: a Kafka processor (#317) and a RabbitMQ processor (#318). This walkthrough is of a feature old humus never had, so it isn't part of that wave, and it needs no revision of the charter. It is the first walkthrough proposed.

It depends on two proposals, and its stories wait for theirs:

• Proposal: Kubernetes controllers, with its revisions 1 to 5 (stories story(kube): reach a Kubernetes API server through the Host with kube.NewClient #908 to story(docs): document writing a Kubernetes controller #918);
• Proposal: failure injection for Kubernetes controllers, which builds on Proposal: failure injection.

Summary

A walkthrough is a production-ready application, kept in this repository, with pages that walk a reader through how it was built and why. Feature docs show one package at a time. A walkthrough shows them put together: settings, telemetry, probes, failure handling, tests, deployment and failure injection, in one service an end user could run. It is built, tested and deployed in CI, so it can't drift from humus.

This proposal sets out the first one, a Kubernetes controller:

• The controller provisions PostgreSQL databases.

  • A Database custom resource names a database. The controller creates the database and a role that owns it in a PostgreSQL server, and writes the role's credentials to a Secret that the Database owns.
  • It reports progress in the Ready condition.
  • On delete, it drops the database and the role, or keeps them, as spec.deletionPolicy says.

  It is the kind of controller platform teams write, and it uses nearly every part of kube:

  • Resource and typed Rules, including a CEL rule that makes the name immutable;
  • a status of conditions;
  • Owns for the Secret, and Finalize for the drop;
  • leader election, Synced, the instruments and the recovery of panics;
  • TrustScenario, beside an injection point of its own for PostgreSQL.

• It is production-ready by a stated bar. Every setting is sourced and checked, and a bad one exits 78. It runs two replicas with leader election and probes, under least-privilege RBAC, and sends traces, logs and metrics over OTLP. Its failure modes are shown, not just claimed.

• It is tested at three levels, with the cases written once:

  • its Reconciler as a function;
  • the whole App against k3s and PostgreSQL in containers;
  • the deployed image against a kind cluster in CI.

  The second and third run the same black-box test cases, which also run against an end user's staging cluster.

• Failure injection is part of the tests. Annotating a Database with a scenario makes PostgreSQL unavailable, slow, or the Secret's apply fail. The cases show that the Ready condition says why, that the controller backs off, and that it recovers once the annotation is gone.

• It ships with its manifests and its image: a kustomization with the CRD, RBAC, the Deployment and a PodDisruptionBudget, and a Dockerfile.

• Its pages are a new Walkthroughs section of the docs site. Every code sample on them is quoted from the walkthrough's files, and a test checks it.

apiVersion: postgres.example.com/v1alpha1
kind: Database
metadata:
  name: orders
  namespace: shop
spec:
  name: orders
  deletionPolicy: Delete
status:
  secretName: orders-postgres
  conditions:
  - type: Ready
    status: "True"
    reason: Provisioned
    observedGeneration: 1

Background

• Feature docs leave the assembly to the reader. Each page in the docs site covers one package: its API, its settings, its telemetry. What an end user needs next is how those choices fit together in a service that someone operates: which settings an operator sets, what the probes are, what the alerts are, how the tests are split, and what the manifests hold. story(docs): create walkthrough documentation for prod ready kafka processor #317 and story(docs): create walkthrough documentation for a prod ready rabbitmq processor #318 asked for exactly that, for queue processors. The reset closed them, and Proposal: humus on the bedrock process model #629 kept them as wave 6.
• The shoppingcart is the reference service, not a walkthrough. #842 rebuilt it on openapi/v3 and PostgreSQL, with container tests. It has no deployment, no failure injection and no pages that explain it.
• kubebuilder's tutorials are the comparison. The CronJob tutorial builds a controller step by step. Its tests run against envtest, an API server and etcd, without the controller's image or any pods, and none of them makes a dependency fail. Its Memcached tutorial is the example Proposal: Kubernetes controllers #851 used.
• k3s in a container runs no pods here, but CI can. kube's container tests start k3s as server --disable-agent, because rootless Podman doesn't delegate the cpuset controller (#851's Testing). A controller whose children are pods can't be shown working there. GitHub's runners have Docker, where kind runs a whole cluster, pods included.
• The docs test accepts a sample only from a _test.go file. TestDocs_code_samples requires every Go sample on the site to appear, line for line, in a _test.go file of the repository (docs_test.go#L23-L28). A walkthrough quotes the application itself.
• Test cases can be written once and run against both harnesses. The pattern agreed for wave 6's walkthroughs is cases in a package of their own, which imports neither the application nor humustest, and takes a Target. The same cases run in-process in the container tests, and against a deployed environment as functional tests. Plain poll loops work in both.

Proposed API

A walkthrough adds no API to humus. What it adds is an application, its manifests, its tests and its pages.

The application

It lives in the example module, beside the shoppingcart, at example/databases. The module gains github.com/z5labs/humus/kube, replaced with ../kube as the core module is.

example/databases/
├── main.go                    humus.Main(app.Init, WithTelemetry, WithFaults)
├── app/                       Init, the settings, Faults
├── api/v1alpha1/              DatabaseSpec, DatabaseStatus, the Resource and its Rules
├── controller/                the Reconciler and the finalizer
├── postgres/                  the admin operations, behind an interface, with their injection point
├── cases/                     the shared test cases, and their Target
├── functional/                the cases against a deployed controller (build tag functional)
├── deploy/                    the kustomization
└── Dockerfile

The resource:

type DatabaseSpec struct {
	// Name is the database's name in the server, and its owning role's.
	Name string `json:"name"`

	// DeletionPolicy is what deleting the Database does to the database:
	// Retain keeps it and its role, and Delete drops both.
	DeletionPolicy DeletionPolicy `json:"deletionPolicy,omitempty"`
}

type DatabaseStatus struct {
	SecretName string             `json:"secretName,omitempty"`
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

Its rules:

• Name matches ^[a-z_][a-z0-9_]{0,62}$, PostgreSQL's limit for an unquoted identifier, and is immutable through a CEL rule, self == oldSelf;
• DeletionPolicy is Retain or Delete, and defaults to Retain, so deleting by mistake loses no data;
• Conditions is a list map keyed by type.

Reconciling, level-triggered, as the Kubernetes docs page describes:

1. Ensure the role exists, with the password from the Secret when there is one, and a new random password when there isn't.

2. Ensure the database exists, owned by the role.

3. Apply the Secret, which the Database controls, with server-side apply: host, port, database, username and password.

4. Set Ready:

   • True, reason Provisioned;
   • or False, with a reason that says which step failed (PostgresUnavailable or SecretApplyFailed) and a stable message.

   An error is also returned, so the object backs off.

Deleting the Secret makes a new password, which Owns brings about within a reconcile. That doubles as the walkthrough's rotation.

Finalizing: with Delete, f drops the database, forcing out its connections, then the role. With Retain, it drops nothing. Either way, the finalizer is removed once f succeeds. A failed drop keeps the Database, and its finalizer, until it succeeds.

PostgreSQL:

• The connection. The admin connection's DSN and password are read through s.Host.FS, from a Secret mounted as files, never from the environment.
• The injection point. Each admin operation evaluates the end user's own point, postgresql:<host>:<port>, as Proposal: failure injection #869's docs suggest naming one, before it calls the server.
• The Secret. No credential is logged or put on a span. The generated password comes from crypto/rand.

Init:

• sources the kube.Connection from the service account, as Proposal: Kubernetes controllers #851's In use does, and the kube.Settings, with the Lease's namespace and identity from the downward API;
• sources the PostgreSQL settings;
• builds the Client and a clientset, the admin pool and the Reconciler;
• returns the Controller and an httphealth Service of its probes, with Synced as readiness.

Faults builds failure injection with jsonscenario. TrustScenario trusts a Database in a namespace whose name starts with a prefix that a setting names. The prefix is empty by default, so production trusts none. The container tests and the kind overlay set it to the cases' prefix.

Production-ready, as a checklist

The walkthrough's pages close with this list, and the application meets each item:

	How
Every setting sourced and checked; a bad one exits 78, naming its variable	config.Struct and the packages' checks
No secret in the environment, the logs or the spans	files through s.Host.FS; a password only in the Secret
Two replicas, one reconciling	leader election; a PodDisruptionBudget of minAvailable: 1
Ready only once its caches have synced	Synced as readiness
A rollout loses nothing	SIGTERM finishes the reconciles in flight and releases the Lease. terminationGracePeriodSeconds exceeds humus's 20s grace
Least privilege	a ClusterRole for databases, their status and Secrets, and a Role for Leases in the controller's namespace. A non-root, read-only container
Observable	traces, logs and metrics over OTLP. The pages suggest alerts on queue.depth, the error and panic outcomes, and reconcile.longest_unfinished_duration
Failures say why	the Ready condition's reason and message
Failures are rehearsed	the failure injection cases, below
Safe by default	deletionPolicy: Retain, and an immutable name

The tests

Level	Where it runs	What it covers
The Reconciler as a function	go test, in the Build job	each step and each condition, with the Reconciler handed a plain Object, the Secret applied through kubefake.NewClientset(), and PostgreSQL behind the walkthrough's own interface, faked in its test. Finalize's f with each policy
The CRD	go test	the golden YAML in deploy/, as #851's In use shows
The shared cases, in-process	testcontainers tag, in a CI job that isn't required	the whole App on humus.OSHost() and humustest's process, against k3s and PostgreSQL in containers
The shared cases, deployed	functional tag, in a CI job that isn't required, and against an end user's staging cluster	the image built from the Dockerfile, deployed with the kustomization to kind, with PostgreSQL beside it

The shared cases are in cases. It imports neither the application nor humustest, so the compiler keeps them black-box. Each takes a Target: a client-go config, a namespace prefix, the scenario annotation and codec key, a PostgreSQL DSN to verify with, and a timeout. They poll with plain loops, which work in both harnesses. The cases:

• Provisioning. A Database becomes Ready. Its Secret's credentials connect to its database, as its owner.
• Rotation. Deleting the Secret brings a new one, whose new password connects, and whose old one doesn't.
• Deletion. With Delete, the database and role are gone. With Retain, both remain, and the Database is gone either way.
• Rules. The API server rejects an invalid name and a changed name.
• Failure injection, each under a scenario annotation:
  • aborting postgresql:*: Ready is False with PostgresUnavailable, and becomes True once the annotation is removed;
  • aborting kube:secrets:patch: Ready is False with SecretApplyFailed, then recovers;
  • delaying postgresql:* by 2s: Ready still becomes True;
  • aborting postgresql:* while a Delete policy's Database is deleted: it stays, with its finalizer, until the annotation is removed.

Every case creates its own namespace under the Target's prefix, and deletes it, so the cases can run in parallel and against a shared staging cluster. They mutate, so they never run against production.

The deployment

deploy/ is a kustomization:

• the CRD, the golden file;
• a Namespace, a ServiceAccount, the ClusterRole and Role, and their bindings;
• the Deployment, with two replicas, the probes, the downward API, resources and a security context;
• the PodDisruptionBudget;
• the admin Secret's reference.

An overlay for the kind job adds PostgreSQL, and sets the trusted prefix to the cases'. The Dockerfile builds a static binary onto a distroless, non-root base.

The pages

A new section, docs/content/walkthroughs/, opens with what a walkthrough is. This walkthrough's pages follow the order it was built in:

1. What we're building, and why — the design, and the choices against alternatives, such as one PostgreSQL server per controller.
2. The Database resource — the types, the rules and the golden CRD.
3. Talking to PostgreSQL — the admin interface, the injection point, and keeping credentials out of telemetry.
4. Reconciling — the steps, the conditions, idempotency and rotation.
5. Cleaning up — the finalizer and deletionPolicy.
6. Wiring the App — Init, the settings, the probes, telemetry and Faults.
7. Deploying — the image, RBAC, the kustomization, leader election, rollouts and upgrading the CRD.
8. Observing — the traces, the records, the instruments, and suggested alerts.
9. Testing — the three levels, and the shared cases.
10. Breaking it on purpose — the failure injection cases, and running them against staging.
11. Production-ready — the checklist.

Changes to other files

• docs_test.go: a Go sample on a walkthrough's page may also quote any .go file under its walkthrough's directory, and a YAML sample any file under its deploy/. Both are checked line for line, as samples are today.
• .github/workflows/build.yaml:
  • a "Databases walkthrough (testcontainers)" job and a "Databases walkthrough (kind)" job, neither required;
  • the Build job's vet step vets both tags.
• example/go.mod: the kube module, client-go and the k3s module.
• The docs site's menu: the Walkthroughs section, after the feature pages.
• The Kubernetes docs page: links to the walkthrough.

Design decisions

• A database provisioner, not Memcached. Its children aren't pods, so the whole App can be shown working against k3s without its agent. It also needs everything a real controller does: an external system, a Secret to own, cleanup outside the cluster that only a finalizer can do, and failures worth reporting. Memcached's controller is a Deployment with a replica count. Open question 1.
• One PostgreSQL server per controller. The admin connection is a setting, like any dependency's. A PostgresServer resource with credentials of its own would double the reconciling, and teach nothing new about kube.
• The Secret is the password's source of truth. Reconciling reads the password from the Secret, and makes a new one only when the Secret is gone. So every reconcile is idempotent, and rotation needs no API of its own.
• Retain by default. A controller that drops data on a mistaken delete isn't production-ready.
• Cases written once, run against both harnesses. Most functional tests repeat in-process ones. Here one set of cases checks the App in a container test and the deployed image, through a Target and nothing else.
• A kind job proves the deployment. The container tests prove the App. Only a real cluster proves the image, the RBAC, the probes and the manifests. CI's runners can run kind, though rootless Podman can't run k3s's agent locally. Open question 4.
• Kustomize, not Helm. kubectl applies a kustomization without another tool, and an overlay is enough for the kind job and for staging. Open question 2.
• The walkthrough's code is quoted, never copied. Every sample on its pages appears in its files, which the docs test checks. So the pages can't drift from the code that CI runs.
• In the example module. It already has pgx and PostgreSQL's testcontainers module, and isn't released. Open question 3.

Testing

The walkthrough is its own test: see The tests. Nothing here was prototyped, since kube isn't built yet.

Migration from today

Nothing migrates. The site has no Walkthroughs section yet.

Open questions

1. Which controller.

   • (a) A PostgreSQL database provisioner, as proposed.
   • (b) Memcached, as kubebuilder's tutorial and Proposal: Kubernetes controllers #851's examples do. It is familiar, and the closest comparison with kubebuilder. But its children are pods, so the container tests can't show it working, and it has no cleanup outside the cluster.
   • (c) A CronJob controller, as kubebuilder's second tutorial. It is rich in scheduling, but its children are Jobs, which are pods again.

   Recommendation: (a).

2. How it's packaged.

   • (a) A kustomization, as proposed.
   • (b) A Helm chart.
   • (c) Both.

   Recommendation: (a).

3. Where the code lives.

   • (a) In the example module, as proposed.
   • (b) A module of its own, example/databases, which keeps client-go out of the shoppingcart's module graph.

   Recommendation: (a).

4. The kind job.

   • (a) A CI job that builds the image, deploys it to kind, and runs the shared cases, as proposed.
   • (b) No deployed test in CI. The pages show running the cases against staging, and the container tests are all CI runs.

   Recommendation: (a).

Out of scope

• The wave-6 walkthroughs and the MIGRATING.md, which Proposal: humus on the bedrock process model #629 plans for the Kafka and RabbitMQ processors.
• More than one PostgreSQL server, schemas and grants beyond the owner, extensions, and backups.
• A second version of the resource, and conversion.
• Publishing the image, or a release of the walkthrough.
• Admission webhooks. The rules are the CRD's own, and CEL covers the immutable name.
• ko, Helm and OLM, unless open question 2 says otherwise.

References

• Proposal: humus on the bedrock process model, its wave 6, and story(docs): create walkthrough documentation for prod ready kafka processor #317 and story(docs): create walkthrough documentation for a prod ready rabbitmq processor #318
• Proposal: Kubernetes controllers, and its revisions 1, 3, 4 and 5
• Proposal: failure injection for Kubernetes controllers, and Proposal: failure injection
• story(example): rebuild the shoppingcart as the reference service on openapi/v3 and Postgres #842, the shoppingcart, and the docs test, docs_test.go
• kubebuilder's CronJob tutorial and getting started
• kind, and Kubernetes' PodDisruptionBudget

[Zaba505]

Revision 1: in line with the walkthroughs that landed

This proposal was posted before #629's revisions 13 to 17 and before any walkthrough was built. Since then, the job walkthrough and the RabbitMQ processor walkthrough have landed, at example/billing and example/mailer. #851 and #922 have landed too.

Revision 14 says #629's points don't bind this proposal, since it is outside the waves. This revision takes them anyway, so this walkthrough is built, tested and read like the two that landed. Nine things change:

1. A module of its own, github.com/z5labs/humus/example/databases. #975 removed the example module, so open question 3 has no option (a) left.
2. The Walkthroughs section and the docs test's rule already exist. #974 and #1090 added them, so this walkthrough changes neither.
3. No comments in its code, except license headers and one-line package comments.
4. Code-first pages, in few words.
5. Grafana's alerts and a dashboard, and no Prometheus.
6. The kind overlay runs the LGTM stack, and up.sh brings it up.
7. Its probes are served over TLS, with a certificate that is read again when it's renewed.
8. Its own injection points are named <system>:<name>, as the failure injection page says. postgresql:<host>:<port> wasn't what Proposal: failure injection #869's docs suggest.
9. Names from what landed: the humus.kube.controller.* instruments, the scenario key acme.fault, and the Secret carrying the scenario annotation.

Each section below replaces or adds to the part of the same name in the post. The rest of the post stands.

The note at the top

The second paragraph becomes:

It depends on Proposal: Kubernetes controllers and Proposal: failure injection for Kubernetes controllers. Both are done, so nothing blocks its stories.

"It is the first walkthrough proposed" is dropped.

Background

The bullet "The docs test accepts a sample only from a _test.go file" is dropped. A Go, YAML, JSON, Dockerfile or protobuf sample on a page under docs/content/walkthroughs/databases/ must now appear, line for line, in a file under example/databases/. Its _test.go files count too.

The application

example/databases/
├── go.mod
├── main.go
├── app/
├── api/v1alpha1/
├── controller/
├── postgres/
├── internal/certfile/
├── cases/
├── functional/
├── deploy/
│   └── kind/
├── Dockerfile
└── Dockerfile.dockerignore

• go.mod replaces github.com/z5labs/humus with ../.. and github.com/z5labs/humus/kube with ../../kube. The module isn't released.
• main.go is humus.Main(app.Init, ...) with WithTelemetry, WithFaults and a shutdown grace period, as the mailer's is.
• internal/certfile is the walkthrough's own copy of the helper that reads a certificate again when it changes, as the mailer has. A walkthrough imports no other walkthrough's code.
• functional/ holds the cases against kind and the Grafana check.
• deploy/ holds the kustomization, alerts.yaml and dashboard.json. deploy/kind/ holds the overlay, up.sh and a Database to apply.

The resource is shown bare:

type DatabaseSpec struct {
	Name           string         `json:"name"`
	DeletionPolicy DeletionPolicy `json:"deletionPolicy,omitempty"`
}

type DatabaseStatus struct {
	SecretName string             `json:"secretName,omitempty"`
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

Name is the name of the database in the server, and of the role that owns it. DeletionPolicy says what deleting the Database does: Retain keeps the database and the role, and Delete drops both. The rules don't change.

Reconciling gains a step 3 sentence: while the Database carries a scenario annotation, so does its Secret, because the Client sets the annotation on a child's server-side apply (#922's revision 2). Nothing reads it there.

PostgreSQL: the injection point bullet becomes two points of the end user's own, named as the job walkthrough's postgresql:issue and postgresql:settle are:

• postgresql:provision, evaluated before the role and the database are ensured;
• postgresql:drop, evaluated before they're dropped.

Init's last bullet becomes: it returns the Controller and an httphealth Service of its probes, with Synced as readiness. The probes are served over TLS, with a certificate from a mounted Secret that internal/certfile reads again when it changes.

Faults builds failure injection with jsonscenario, whose key is acme.fault, as it is in the other walkthroughs.

Production-ready, as a checklist

These rows change, and the others stand:

	How
Certificates are files, read again when they're renewed	the probes' certificate, through internal/certfile
Observable	traces, logs and metrics over OTLP, Grafana alert rules on its instruments, humus's own included, and a dashboard
Safe by default	deletionPolicy: Retain, an immutable name, and no scenario trusted

The tests

The deployed level's row becomes:

Level	Where it runs	What it covers
The shared cases, deployed	functional tag, in the kind job, and against an end user's staging cluster	the image built from the Dockerfile, deployed to kind by up.sh. Also a check that Grafana loaded every alert rule and the dashboard, that each rule evaluates without error, and that Mimir holds every series the rules read

The in-process level's k3s and PostgreSQL get passwords generated on each run.

In the Target, the codec key is acme.fault. In the last failure injection case, the rule aborts postgresql:drop, not postgresql:*.

The alerts and the dashboard

This section is added.

deploy/alerts.yaml is a Grafana alerting provisioning file, and deploy/dashboard.json is a Grafana dashboard. The kind overlay's Grafana provisions both. The rules query Mimir, using the names the instruments have there:

Alert	Reads
DatabasesReconcileFailing	the count of humus.kube.controller.reconcile.duration with humus.kube.controller.reconcile.outcome error
DatabasesReconcilePanicked	the same, with outcome panic
DatabasesReconcileStuck	humus.kube.controller.reconcile.longest_unfinished_duration
DatabasesBacklogged	humus.kube.controller.queue.depth
DatabasesRestarting	k8s.container.restarts, from the Collector's Kubernetes cluster receiver, since a controller that crashes can't report

Nothing is Prometheus's: there is no server, no exporter and no promtool.

The deployment

deploy/ also holds alerts.yaml and dashboard.json.

The overlay, deploy/kind/:

• adds PostgreSQL;
• adds the LGTM stack: an OpenTelemetry Collector with the Kubernetes cluster receiver, which forwards OTLP to Loki, Tempo and Mimir, and Grafana, which reads all three;
• sets the trusted prefix to the cases' prefix.

up.sh, beside the overlay:

1. creates the kind cluster if there is none;
2. generates every password, key and certificate: PostgreSQL's admin password, the probes' certificate and Grafana's admin password;
3. loads the image;
4. applies the overlay.

The kind job runs it, and so does a reader. There is no compose file, and no secret is kept in the repository. The manifests, the script and the Dockerfile have no comments.

The pages

They are replaced. Each page is a list of steps: a sentence or two, then the code it arrives at. There is no history, there are no alternatives and there are no asides. The pages are in docs/content/walkthroughs/databases/:

1. Databases: the Database resource, its names, and the files the walkthrough ends with.
2. The Database resource: the types, the rules and the golden CRD.
3. Talking to PostgreSQL: the admin interface, its injection points, and keeping credentials out of telemetry.
4. Reconciling: the steps, the conditions, the Secret and rotation.
5. Cleaning up: the finalizer and deletionPolicy.
6. Init: the settings, the secrets as files, the Connection, the Settings and the Lease, TrustScenario, the Controller, the probes, BuildTelemetry, Faults, and main.
7. Testing: the Reconciler as a unit, the golden CRD, the shared cases in process and on kind, and the failure injection cases.
8. Deploying: the image, the kustomization, RBAC, the Deployment, the disruption budget, the grace period, the kind overlay, running it on kind, and applying a Database.
9. Observing: in Grafana on kind, a reconcile's trace, its log records, its metrics, the alerts and the dashboard.
10. Production-ready: the checklist.

The pages also add the walkthrough to the Walkthroughs section's index, as the billing and mailer pages did.

Changes to other files

The list becomes:

• .github/workflows/build.yaml: a "Databases walkthrough (testcontainers)" job and a "Databases walkthrough (kind)" job, neither required. The kind job builds the image, runs up.sh, and runs the shared cases and the Grafana check. The vet jobs already cover every module under example/.
• The Walkthroughs section's index: a line for this walkthrough.
• The Kubernetes docs page: links to the walkthrough.

docs_test.go, example/go.mod and the menu are dropped.

Out of scope

The first bullet becomes the wave-6 walkthroughs, which #629 plans. #629's revision 13 put the MIGRATING.md out of scope. The last bullet becomes ko, Helm and OLM. One bullet is added: running it in production, with runbooks, as for every walkthrough.

Open questions

1. Which controller: still open. Recommendation: (a).
2. How it's packaged: a kustomization, as #629's revision 15 answered for every walkthrough.
3. Where the code lives: a module of its own. Revision 15 made every example a module, and story(example): make every example a module of its own #975 built that.
4. The kind job: a kind job, as revision 15 answered. kind is the only maintained way to run a walkthrough's whole stack.

References

These are added:

• The billing and mailer walkthroughs, and story(docs): add the Walkthroughs section, and check walkthrough samples against their files #974, story(example): make every example a module of its own #975 and story(docs): let a walkthrough's samples quote its own tests #1090
• Grafana: provisioning alert rules from files, and provisioning dashboards
• The OpenTelemetry Collector's Kubernetes cluster receiver

[Zaba505]

Revision 2: every question answered

Question 1 is answered with (a): the controller provisions PostgreSQL databases, as proposed. Revision 1 answered questions 2 to 4 with #629's answers. So every question is answered.

Nothing else changes. The stories go in the milestone Kubernetes controller walkthrough.

[Zaba505]

Closing: the Kubernetes controller walkthrough has landed

The revisions' stories are done. example/databases is a PostgreSQL database provisioner, a module of its own:

• story(example): define the Database resource, its rules and its golden CRD #1126 defined the Database resource, its rules and its golden CRD (story(example): define the Database resource, its rules and its golden CRD #1146);
• story(example): provision and drop a database and its role in PostgreSQL #1127 provisioned and dropped a database and its role in PostgreSQL, sending a SCRAM-SHA-256 verifier, never the password (story(example): provision and drop a database and its role in PostgreSQL #1150);
• story(example): reconcile a Database into its role, database, Secret and Ready condition #1128 reconciled a Database into its role, database, Secret and Ready condition (story(example): reconcile a Database into its role, database, Secret and Ready condition #1154), and story(example): clean up a deleted Database as its deletionPolicy says #1129 cleaned up a deleted one as its deletionPolicy says (story(example): clean up a deleted Database as its deletionPolicy says #1156);
• story(example): serve the controller's probes certificate from its files, read again when renewed #1130 served the probes' certificate from its files (story(example): serve the controller's probes certificate from its files, read again when renewed #1158), and story(example): wire the controller's App: settings, secrets as files, the Controller, probes and main #1131 wired the App: settings, secrets as files, the Controller, probes and main (story(example): wire the controller's App: settings, secrets as files, the Controller, probes and main #1159);
• story(example): run the controller's shared cases in process, against k3s and PostgreSQL in containers #1132 ran the shared cases in process, against k3s and PostgreSQL in containers (story(example): run the controller's shared cases in process, against k3s and PostgreSQL in containers #1161);
• story(example): rehearse failures from a Database's scenario annotation, trusted by namespace prefix #1133 rehearsed failures from a Database's scenario annotation, trusted by namespace prefix (story(example): rehearse failures from a Database's scenario annotation, trusted by namespace prefix #1162), and story(example): rehearse failures in the controller's shared cases #1135 added failure injection to the shared cases (story(example): rehearse failures in the controller's shared cases #1165);
• story(example): trace, log and measure each reconcile, without a credential #1134 traced, logged and measured each reconcile, without a credential (story(example): trace, log and measure each reconcile, without a credential #1164);
• story(example): build the controller's image and its manifests #1136 built the image and its manifests (story(example): build the controller's image and its manifests #1167), story(example): bring the controller up on kind with one script #1137 brought it up on kind with up.sh (story(example): bring the controller up on kind with one script #1168), and story(example): run the controller's shared cases against kind #1138 ran the shared cases against kind (story(example): run the controller's shared cases against kind #1170);
• story(example): send the controller's telemetry to the LGTM stack in kind #1139 sent its telemetry to the LGTM stack in kind (story(example): send the controller's telemetry to the LGTM stack in kind #1171), and story(example): alert on the controller in Grafana, and give it a dashboard #1140 gave it Grafana alerts and a dashboard (story(example): alert on the controller in Grafana, and give it a dashboard #1173);
• story(docs): walk through building the databases controller #1141, story(docs): walk through testing and deploying the databases controller #1142 and story(docs): walk through observing the databases controller, and its production-ready checklist #1143 walked through building, testing, deploying and observing it, and its production-ready checklist (story(docs): walk through building the databases controller #1182, story(docs): walk through testing and deploying the databases controller #1183, story(docs): walk through observing the databases controller, and its production-ready checklist #1185).

The revisions left no open questions.

The rest stays out of scope, as "Out of scope" lists: more than one PostgreSQL server, schemas and grants beyond the owner, extensions and backups; a second version of the resource; publishing the image or releasing the walkthrough; admission webhooks; and ko, Helm and OLM.