Proposal: Kafka processor walkthrough

[Zaba505]

Important

Revision 3 answers open question 4, the last one. The projection keeps every event it applied, and an order's status is folded from them in the order of their at. A redelivery is found by its offset and skipped. A late event takes its place in the history, and orders.status gets its change and every later one again, so a consumer that keeps the latest status may see it flicker. It's a decision for now, without a real product's business case.

Important

Revision 2 answers open questions 1 and 3, and adds question 4. It is an order-status projection. The consumer never stops on a record: one that fails for a reason that isn't its own goes to orders.retry, with its attempt in a header and a wait derived from it, and to orders.dlq once its attempts run out, so the abort cases are shared. An event out of order is retried, not trusted. Operating it is out of scope. The topics and their ACLs are created with Kafka's CLI, and a small Go CLI produces events by hand. Question 4, the guard against redelivery, is open.

Important

Revision 1 answers open question 2 and corrects question 1. There is no redrive job or runbook: whether a dead letter can be replayed depends on why it was dead-lettered, which each end user learns by running their system, so the pages tell the reader to plan for it from their business requirements. Question 1's option (b) would run its in-process cases against WireMock in a container. The pages are code-first steps. Questions 1 and 3 are still open.

Note

Part of the humus re-evaluation: wave 6 of Proposal: humus on the bedrock process model, which plans one proposal per walkthrough. This is the Kafka processor that #317 asked for. It keeps the points every walkthrough keeps, from revision 13, and doesn't decide them again.

It builds on proposals that are done:

• queue processing and the Kafka runtime and producing to Kafka, which hands on a processor that produces dead letters, and the committed offset as its tests' barrier;
• exposing health probes, failure injection, which names a Kafka processor's transient failures, and humus's own metrics, which hands on alert rules;
• jobs, for its redrive job, humus.Main, and wave 1.

It depends on no other wave-6 proposal. Its pages wait for the story of revision 13 that adds the Walkthroughs section.

Summary

A walkthrough is a production-ready application, kept in this repository, with pages that walk a reader through how it was built and why. This one is a Kafka processor of an event stream, the thing Kafka is for:

• It projects orders' events into their statuses. Other services write each order's lifecycle events, placed, paid, shipped, delivered and cancelled, to orders.events, keyed by the order's ID. The processor keeps each order's status in PostgreSQL, and produces each change of status to orders.status, for services such as notifications to consume.
  • Order per key matters. An order's events are applied in the order they were written, through a state machine, so shipped before paid is refused.
  • Every record is processed at least once, and processing one twice changes nothing: the projection keeps the offset of the last record it applied.
  • A record it can't apply is a dead letter. One that isn't an event, of a type it doesn't know, or out of order, goes to orders.dlq with its reason, and the partition moves on.
  • A failure that isn't the record's stops it. PostgreSQL that stays down, or a cluster that won't store its status event, is retried and then stops the consumer, loudly, rather than dead-lettering every record of an outage.
• It is production-ready by a stated bar, that of revision 13. Every setting is sourced and checked, and a bad one exits 78. It runs three replicas on twelve partitions with readiness on joining the group, speaks mTLS to Kafka and TLS to PostgreSQL with secrets read as files, and sends traces, logs and metrics over OTLP. Its alerts are a rule file that CI tests.
• It shows what its operators do: watch the group's lag, roll out without losing or repeating anything, get past a record that stops the consumer, replay dead letters with a redrive job, and rebuild the projection by resetting the group's offsets.
• It is tested at three levels, with the cases written once. The cases check only what the processor writes to Kafka, behind the group's committed offset. They run inside synctest against kfake, and against the deployed image in kind.
• Failure injection is part of the tests. A record whose baggage header carries a scenario slows PostgreSQL or the status topic. Its status is still projected, once, and the scenario doesn't go on to orders.status.

orders.events   key: 7f3c…  {"type":"paid","at":"2026-10-18T14:03:00Z"}
                   │
                   ▼  projector: placed → paid
orders.status   key: 7f3c…  {"order_id":"7f3c…","status":"paid","previous":"placed","at":"2026-10-18T14:03:00Z"}
order_status    7f3c… | paid | partition 4 | offset 1932

Background

• story(docs): create walkthrough documentation for prod ready kafka processor #317 asked for a production-ready Kafka processor walkthrough. The reset closed it, and #629 kept it as wave 6.
• v0.15.0 had three Kafka examples, kafka-at-least-once, kafka-at-most-once and kafka-mtls-at-least-once. Each one ran with podman-compose beside a Grafana stack and a dashboard, and was tested by an integration_test/main.go against localhost:9092. The reset deleted them. They showed the API, but not a service someone operates:
  • their idempotency was a map in memory (processor.go), which a restart or a second replica forgets;
  • nothing was deployed, nothing failed on purpose, and the integration tests weren't run in CI.
• The Kafka page documents each part, one at a time. It covers what each Delivery commits, why an at-least-once failure stops the Service, dead letters stored before the commit, retries that are the Processor's, readiness and testing on kfake. What it leaves to the reader is putting them together: where the state lives, which failures are dead letters and which stop the consumer, how many replicas, what to alert on, and what to do at 3am.
• Producing to Kafka handed this walkthrough its barrier. #764 left dead letters to the Processor, stored before Process returns. So once the group has committed past a record, the dead-letter topic holds its dead letter. This walkthrough also produces its status events before it returns, so the same barrier covers them.
• A record's scenario can't be taken back. Failure injection trusts a Kafka record by its topic, and a fault is a real failure. A request's scenario ends with the request, but a record keeps its headers, and every redelivery carries the same scenario. An abort that the processor can't get past stops the consumer on every restart, as a poison record would. That is the first thing a Kafka consumer's failure injection has to teach.
• Consumer lag is measured outside the consumer. None of humus's, kotel's or the semantic conventions' instruments is the group's lag, and a consumer that has crashed reports nothing. Lag comes from whatever watches the cluster, such as kafka_exporter's kafka_consumergroup_lag.
• Test cases can be written once and run against both harnesses. This is the pattern revision 13 sets out. The cases live in a package of their own that imports neither the application nor humustest, and take a Target. The same cases run in process, and against a deployed environment as functional tests. They poll with plain loops, which work in both.

Proposed API

A walkthrough adds no API to humus. What it adds is an application, its manifests, its tests and its pages.

The application

It lives in the example module, at example/orderstatus. The module gains github.com/z5labs/humus/queue/kafka, replaced with ../queue/kafka as the core module is, and franz-go's kadm and kfake for its tests. It already has pgx and PostgreSQL's testcontainers module.

example/orderstatus/
├── main.go           humus.Main(app.Init, WithTelemetry, WithFaults)
├── app/              Init, the settings, BuildTelemetry, Faults
├── projector/        the Processor: decoding, the transitions, dead letters
├── store/            the projection in PostgreSQL, its schema and its injection point
├── redrive/          the redrive job's run
├── cmd/redrive/      its main
├── cases/            the shared test cases, and their Target
├── functional/       the cases against a deployed processor (build tag functional)
├── deploy/           the kustomization, the kind overlay, and the alert rules
└── Dockerfile        both binaries, on a distroless, non-root base

The topics and their records

Topic	Key	Value	Written by
orders.events	the order's ID	{"type": "placed" | "paid" | "shipped" | "delivered" | "cancelled", "at": RFC 3339}	the end user's other services
orders.status	the order's ID	{"order_id", "status", "previous", "at"}	the processor, for each change of status
orders.dlq	the order's ID	the consumed record's value, unchanged	the processor, for each record it can't apply

• Keys keep an order on one partition. Every producer keys by the order's ID, so an order's events are processed in the order they were written. The pages say why orders.events never gains partitions: adding some moves keys to other partitions, which breaks both the order and the projection's offset guard. A topic that needs more partitions is a new topic.
• The topics are the cluster owner's to create, with their ACLs, as kafka.Settings leaves them. The pages list what each needs: partitions, min.insync.replicas, and retention. orders.events keeps its records as long as the projection might need rebuilding from them. The kind overlay creates its own topics.
• Twelve partitions, three replicas. A replica processes each of its partitions at once, so the partitions bound the concurrency, and the replicas never outnumber them.

Projecting an event

The projector is the queue.Processor[kafka.Message] that kafka.NewService runs, under queue.AtLeastOnce. For each record:

1. Decode it. A record with no key, a value that isn't an event, or a missing at is a dead letter, malformed. An unknown type is a dead letter, unknown-type, so that a producer that ships a new type before the processor doesn't stop it.
2. Read the order's projection: its status, and the partition and offset of the record it was last projected from. A record at or before that offset, on the same partition, was applied already, and is skipped. That is what makes a redelivery harmless.
3. Check the transition. placed comes first. paid follows placed, shipped follows paid, delivered follows shipped, and cancelled follows placed or paid. An event that repeats the current status changes nothing. Any other event is a dead letter, invalid-transition.
4. Produce the change to orders.status, and only then store the projection, with the record's offset. A crash between the two produces the status event again on redelivery, and never loses one. The consumers of orders.status tell duplicates apart by the order and the status, as any consumer under at-least-once must.

Dead letters are produced synchronously, before Process returns nil, as the Kafka page's Dead letters requires. They keep the record's key, value and headers, and add headers of their own: dlq.reason, dlq.error, dlq.topic, dlq.partition and dlq.offset. The Producer replaces the trace context, so a dead letter's send span is in the consumed record's trace.

Failures that aren't the record's are retried, and then stop the consumer:

• The store's calls are retried with bedrock's retry: 5 attempts, backoff from 100ms to 2s, and a 2s timeout per attempt.
• Producing is not retried by the projector. franz-go already retries within DeliveryTimeout, and the Kafka page explains why a retry around Produce would duplicate and reorder records.
• Every record's work has a deadline of 15s, under humus.Main's 20s grace period, so a rollout's SIGTERM never finds a record in flight that can't finish.
• What's left is Process's error. The Service stops, and the App exits 1 with the record named. The partition's later records wait until the outage ends, and the lag alert fires. Dead-lettering instead would send an outage's every record to orders.dlq, and apply each order's later events without its earlier ones.

The projection in PostgreSQL

CREATE TABLE order_status (
    order_id         text        PRIMARY KEY,
    status           text        NOT NULL,
    at               timestamptz NOT NULL,
    source_partition integer     NOT NULL,
    source_offset    bigint      NOT NULL
);

• One upsert stores a change, and only moves forward. It updates the row only when the record's offset is past the stored one, so two replicas racing for a partition during a rebalance can't move an order back.
• The schema is applied by the deployment, not by the processor. store/schema.sql is the schema. The kind overlay applies it before the processor starts, and production uses the end user's migration tool, as the API walkthrough shows for the shoppingcart's. The processor's role needs SELECT, INSERT and UPDATE on order_status and nothing else.
• The connection is TLS, with its secrets read as files. Init builds the pool's config from POSTGRES_URL, which holds no password. The password comes from POSTGRES_PASSWORD_FILE, and the CA from POSTGRES_TLS_CA_FILE, both read through s.Host.FS. The pool dials through s.Host.Dial, and nothing is left for pgx to read from the real environment or filesystem, as the shoppingcart's app doc explains.
• The injection point. Each call evaluates the end user's own point, postgresql:orders, named <system>:<database> as Points of your own suggests. It wraps the Store interface rather than living in the PostgreSQL code, so the in-memory store the in-process tests use has the point too.

Init

• It sources every setting with config.Struct, and a bad one exits 78, naming it:

  Variable
  KAFKA_BROKERS, KAFKA_GROUP_ID	required
  KAFKA_EVENTS_TOPIC, KAFKA_STATUS_TOPIC, KAFKA_DEAD_LETTER_TOPIC	required
  KAFKA_TLS_CA_FILE, KAFKA_TLS_CERT_FILE	required: the brokers' CA, and the client's certificate and key, as on the Kafka page's TLS and mTLS
  KAFKA_SESSION_TIMEOUT, KAFKA_DELIVERY_TIMEOUT, KAFKA_BREAKER_OPEN_FOR	kafka.DefaultSettings and kafka.DefaultProducerSettings by default
  POSTGRES_URL, POSTGRES_PASSWORD_FILE, POSTGRES_TLS_CA_FILE	required
  PROBES_ADDR, PROBES_TLS_PEM_FILE	required, as httphealth.Settings
  FAULT_MAX_DELAY, FAULT_TRUSTED_TOPICS	fault.DefaultSettings's 10s, and no topic
  OTEL_SERVICE_NAME, OTEL_*_EXPORTER, OTEL_EXPORTER_OTLP_ENDPOINT	read by BuildTelemetry, as on the Telemetry page

• It opens the pool, and registers its close with s.OnStop.

• It builds the Producer, for the consumer's own cluster, with the same brokers and TLS.

• It builds the projector on the store, wrapped by its injection point, and the Service, with TrustScenario set to trust the topics FAULT_TRUSTED_TOPICS names.

• It returns the Service and a probes server, httphealth.NewService(s, probes, health.Readiness(consumer.Joined())).

main is humus.Main(app.Init, humus.WithTelemetry(app.BuildTelemetry), humus.WithFaults(app.Faults)). The process has no server that a load balancer routes to, so it stays on humus.Main's defaults, as the Kafka page's Which Main says.

Faults builds failure injection with jsonscenario, under the edge's key, acme.fault, the Failure injection page's key, which every walkthrough uses. FAULT_TRUSTED_TOPICS is empty by default, so production trusts no record's scenario. The in-process tests and the kind overlay set it to orders.events. The Producer keeps ForwardScenario off, so a scenario never goes on to orders.status or orders.dlq, whose consumers are other teams' trust boundaries.

Its own instrument is orderstatus.dead_letters, a counter of dead letters by orderstatus.reason. Everything else it shows comes from humus, kotel and the semantic conventions.

Operating it

The pages give each task a runbook, and the tests run what can be run in process:

• Lag. Alert on the group's lag growing, and on lag with no commits. Then look at messaging.process.duration, PostgreSQL, and the Producer's breaker. More replicas help only up to the partition count.

• A record that stops the consumer. The App's last log record names the topic, partition and offset, and so does each restart's. Either fix forward and deploy, or skip the record:

  1. scale the Deployment to zero;
  2. run kafka-consumer-groups.sh --reset-offsets --to-offset <offset+1> --topic orders.events:<partition> --execute;
  3. scale it back.

  The skipped record stays in orders.events until its retention ends. A scenario that stops the consumer is recovered from by removing its topic from FAULT_TRUSTED_TOPICS. The rollout then strips the scenario, and the record is processed.

• Replaying dead letters. Once a fix has shipped, such as support for a new event type, an operator runs the redrive job, a Kubernetes Job from deploy/redrive/:

  • It is a job.NewService run on humus.Main, in the same image.
  • It reads orders.dlq from REDRIVE_SINCE up to the end offsets it finds at start, by partition, with a franz-go client that dials s.Host.Dial. A finite read is a job's, not queue/kafka's Service.
  • It produces each dead letter whose dlq.reason is in REDRIVE_REASONS back to its dlq.topic, with its key, value and headers, minus the dlq. ones, through a kafka.Producer.

  A record redriven twice is a duplicate, which the projector handles. One that is still invalid returns to orders.dlq.

• Rebuilding the projection.

  1. Scale to zero.
  2. Truncate order_status.
  3. Reset the group's offsets to the earliest.
  4. Scale back.

  The status events are produced again, and their consumers dedupe them. Without the truncate, the replay changes nothing, which is how the offset guard is tested.

Production-ready, as a checklist

The walkthrough's pages close with this list, and the application meets each item:

	How
Every setting sourced and checked; a bad one exits 78, naming its variable	config.Struct and the packages' checks
No secret in the environment, the logs or the spans	the client certificate, PostgreSQL's password and the probes' key are files read through s.Host.FS. The record's key, an order's ID, isn't recorded, as queue/kafka doesn't record it
Certificates renewed without a restart	the Kafka client certificate is served through GetClientCertificate, and the probes' through GetCertificate, by a small helper of the walkthrough's own that reads their files again when they change
Concurrency within the partitions	three replicas on twelve partitions; a PodDisruptionBudget of maxUnavailable: 1
Ready only once it has joined its group	Joined as readiness, on an HTTPS probes port
A rollout loses and repeats nothing	the revoke commits the record in flight. Every record's work fits in 15s, under the 20s grace period, and terminationGracePeriodSeconds is 30, as the Kafka page's budget shows. The rollout case checks it
Redelivery is harmless	the offset guard in order_status
A record it can't apply doesn't stop it	dead letters, stored before the commit, with their reason
An outage doesn't lose order or records	retries, then a loud stop with the record named
Least privilege	mTLS with ACLs that let it read orders.events and write the other two. A role with three privileges on one table. No service account token. A non-root, read-only container
Observable	traces from producer to orders.status, logs and metrics over OTLP, and rules for lag, a stalled group, a crash loop, dead letters, an open breaker and slow processing
Failures are rehearsed	the failure injection cases, below
Safe by default	no record's scenario trusted; ForwardScenario off

The tests

Level	Where it runs	What it covers
The projector	go test, in the Build job, inside synctest	decoding, the transitions, the offset guard, produce-then-store, dead letters' headers, and retries and the deadline. Its Store and producer are the walkthrough's own interfaces, faked in the test
The store	testcontainers tag, in a CI job that isn't required	store against PostgreSQL in a container: the forward-only upsert, a race between two writers, TLS, and the injection point
The shared cases, in process	go test, in the Build job	the whole App, Init and Faults included, on humustest's Host inside synctest, against kfake over mTLS with certificates from humustest.NewCA. Its store is in memory, as the shoppingcart's unit tests swap theirs in
The App, in process	go test, in the Build job	what only a test that controls the App can show: an abort that stops the consumer, and the recovery once the topic is untrusted; skipping a record by committing past it; a replay that changes nothing; a grace period that runs out; bad settings; the spans, the dead-letter counter and the breaker's instruments; the redrive job against the same cluster
The alert rules	promtool test rules, in the kind job	each alert fires on the series that should fire it, and on no other
The shared cases, deployed	functional tag, in a CI job that isn't required, and against an end user's staging cluster	the image, deployed with the kustomization to kind, beside Kafka, PostgreSQL and a collector

The shared cases are in cases, and take a Target:

• franz-go options to reach the cluster, with its TLS;
• the three topics and the group;
• the scenario codec and its key;
• a timeout;
• an optional Rollout func, which replaces the processor's replicas and returns once the old ones are gone.

Each case keys its records by order IDs from crypto/rand, so it only ever reads its own. Its barrier is the group's committed offset: it polls kadm's FetchOffsets until the group has committed past its last record. Then it reads orders.status and orders.dlq by partition, from a minute before it began, which allows for clock skew, up to the end offsets. After the barrier, what it finds, and what it doesn't, is settled. The cases:

• Projection. placed, paid and shipped give three status events, in order, and no dead letter.
• Redelivery. An event repeated with a new offset changes nothing, and produces nothing.
• Dead letters. A record that isn't JSON, one of an unknown type, and one out of order each become a dead letter with its reason and its source offset. The partition moves on, so the next event of the same partition is projected.
• Rollout. Records produced before, during and after Rollout are each projected once. In process, Rollout starts a second App on the same cluster, waits for the group to hold both, and stops the first with SIGTERM. Deployed, it is kubectl rollout restart and kubectl rollout status. With no Rollout, the case is skipped.
• Failure injection, each a record whose baggage header carries a scenario:
  • delaying postgresql:orders by 1s, under the store's 2s attempt timeout: the status is projected once, with no dead letter, and the status event's baggage doesn't carry the scenario;
  • delaying kafka:orders.status by 1s: the status is projected once.

Every case mutates, so the cases never run against production. A shared staging topic is fine: the cases read only their own keys.

Aborts stay in process. An abort on postgresql:orders stops the consumer, and the record still carries its scenario after every restart. Against a shared cluster, it would hold that partition for everyone until an operator untrusted the topic. So the abort, the crash it causes, the records before it committed and the one it stopped on not committed, and the recovery after the topic is untrusted, are tests of the App in process. Each of those tests has a cluster of its own. Open question 3.

The deployment

deploy/ is a kustomization:

• a ServiceAccount, with no token mounted, since the processor calls no Kubernetes API;
• a ConfigMap of the settings, and references to three Secrets mounted as files: the Kafka client certificate, PostgreSQL's password and CA, and the probes' certificate;
• the Deployment, with three replicas, maxSurge: 1 and maxUnavailable: 0, the HTTPS probes, resources, terminationGracePeriodSeconds: 30 and a security context;
• the PodDisruptionBudget;
• alerts.yaml, the Prometheus rule file;
• redrive/, the redrive Job, which nothing applies until an operator does.

The kind overlay adds:

• Kafka, apache/kafka-native as the Kafka module's container tests run it, with an mTLS listener;
• a Job that creates the three topics;
• PostgreSQL over TLS, with schema.sql applied before the processor starts;
• an OpenTelemetry Collector the App exports to;
• the trusted topic.

The kind job generates a CA and every certificate and password on each run, so none is kept in the repository. The Dockerfile builds orderstatus and redrive as static binaries, onto a distroless, non-root base.

The alert rules

deploy/alerts.yaml holds six rules, each explained on the Observing page:

Alert	Fires when	From
OrderStatusLagHigh	the group's lag is over a threshold for 10m	kafka_exporter's kafka_consumergroup_lag
OrderStatusStalled	there is lag, and no commit in 5m	the lag, and messaging_client_operation_duration_seconds_count{messaging_operation_name="commit"}
OrderStatusCrashLooping	a pod restarted more than 3 times in 15m	kube-state-metrics' kube_pod_container_status_restarts_total
OrderStatusDeadLetters	any dead letter in 15m, as a ticket rather than a page	orderstatus_dead_letters_total, by reason
OrderStatusBreakerOpen	the Producer's breaker is open	humus_circuit_state{humus_circuit_state="open"}
OrderStatusSlowProcessing	p99 of processing over 5s for 10m	messaging_process_duration_seconds

The first and third read metrics from outside the App, because a consumer that has crashed reports nothing. The pages say which exporter each assumes, and how to change it.

The pages

They are docs/content/walkthroughs/orderstatus/, in the order the processor was built:

1. What we're building, and why: the projection, and the choices against the alternatives, such as an outbox or Kafka's transactions.
2. The events and their topics: keys, partitions, retention, ACLs, and why partitions are never added.
3. Projecting an event: the rules, the transitions, the offset guard, and producing before storing.
4. Dead letters: the reasons, the headers, and storing them before the commit.
5. The projection in PostgreSQL: the schema, the forward-only upsert, retries and the deadline, the injection point, and the credentials.
6. Wiring the App: Init, the settings, mTLS, the probes, telemetry and Faults.
7. Deploying: the image, replicas against partitions, rollouts, the disruption budget and the grace period.
8. Observing: one trace across three topics, the records, the instruments, and the alert rules.
9. Testing: the levels, the shared cases, and the barrier.
10. Breaking it on purpose: the delay cases, why aborts stay in process, and untrusting a topic.
11. Operating it: lag, a record that stops the consumer, redriving, and rebuilding.
12. Production-ready: the checklist.

Changes to other files

• .github/workflows/build.yaml:
  • an "Order status walkthrough (testcontainers)" job and an "Order status walkthrough (kind)" job, neither required;
  • the Build job's vet step vets both tags.
• example/go.mod: the queue/kafka module, replaced with ../queue/kafka, and franz-go's kadm and kfake.
• The Kafka docs page: a link to the walkthrough, from Dead letters and from Testing.

Design decisions

• A projection, not a stateless transformer. Most Kafka consumers keep state, and what's hard about one only shows with state: idempotency, order per key, replaying, and rebuilding. Open question 1.
• The offset is stored with the state. It's one column, written in the statement that stores the change. So a redelivery is detected without a table of seen events, and a replay changes nothing. Kafka's transactions were rejected: kafka.Producer doesn't offer them, and they can't reach PostgreSQL anyway.
• The status event is produced before the projection is stored. Under at-least-once, the choice is between duplicates and losses, and duplicates are the consumers' to handle. An outbox table and a relay would remove the duplicates, at the cost of a second process, and are out of scope.
• Dead letters for the record's faults, a stop for everyone else's. A record that is malformed, unknown or out of order will fail on every delivery, so it goes to orders.dlq. A failure of PostgreSQL or the cluster would dead-letter every record of an outage, and leave orders half applied. So it is retried, and then stops the consumer, which is the loud failure queue's contract chose.
• No retry around Produce. franz-go retries inside DeliveryTimeout, and a retry around it would store records twice and out of order.
• The in-process cases use an in-memory store. PostgreSQL can't run on humustest's Host, and humus writes no fake of another project's server. The cases check only the topics, so they hold for any store. The PostgreSQL store is proven by its own container test, and the whole composition by the kind job. Running the whole App inside synctest keeps the cases fast and in the required Build job, as revision 13's second level asks.
• Only delays are shared. A record's scenario can't be taken back, so an abort in a shared environment is a poison record. The abort and its recovery are shown in process, where each test has its own cluster. Open question 3.
• Lag comes from outside. A consumer that is down can't report its lag. The rule file reads the exporter most clusters run, and the pages say how to point it at another.
• mTLS to Kafka, TLS to PostgreSQL, secrets as files. The ACLs that make topic trust meaningful need an identity, and a client certificate is one with no password to rotate. Its renewal needs no restart either: the certificate is read again when its file changes, as revision 13's bar asks.
• An operator's tools are tested like the processor. A redrive that only exists as a runbook is first run during an incident. Open question 2.

Testing

The walkthrough is its own test: see The tests.

A prototype checked the claims the cases depend on, at main 1bb3bb53, in a module of its own that replaces humus and humus/queue/kafka with this repository. It has a projector over an in-memory store with the postgresql:orders point, the real kafka.Service and kafka.Producer, a cases package that imports neither, and an app_test.go running them inside synctest on humustest's Host against kfake. go test -race -count=3 ./... passes:

• The barrier. Polling kadm's FetchOffsets with plain sleeps, inside the bubble, returns once the group has committed past the case's record. orders.status and orders.dlq, read by ListOffsetsAfterMilli, ListEndOffsets and ConsumePartitions, then hold exactly the case's records.
• Dead letters. A record that isn't JSON is a dead letter with dlq.reason and its source offset, and the next record of its key is projected.
• Trust by a setting. A delay scenario in a record's baggage header reaches postgresql:orders when FAULT_TRUSTED_TOPICS names the topic, and the record is still projected once. The status event's baggage carries no acme.fault.
• An abort. It stops the App with exit 1, and the cause names the record: serve: kafka: processing orders.events, partition 2, offset 1: reading order …: fit: injected failure at postgresql:orders. Nothing is produced for that order. A second App with FAULT_TRUSTED_TOPICS unset processes the same record.
• A rollout. A second App joins, the first stops on SIGTERM, and 30 orders produced before, during and after are each projected once.

Not prototyped: the PostgreSQL store, mTLS on kfake (which the Kafka page's tests already cover), the redrive job, the rule tests and the kind job.

Migration from today

Nothing migrates. The reset deleted v0.15.0's Kafka examples, and the site has no Walkthroughs section yet.

Open questions

1. Which processor.

   • (a) An order-status projection on PostgreSQL, as proposed.
   • (b) A stateless enricher. It consumes orders, prices each one through an HTTP service of the end user's own, and produces priced orders. The whole App, its dependency's stand-in included, runs inside synctest with nothing swapped. But it has no state, so there is no idempotency, replay or rebuild to show, and order per key barely matters.
   • (c) A clickstream counter under queue.AtMostOnce. It shows the other Delivery, but most processors need at-least-once, and its aggregation across rebalances would be the harder half of the walkthrough.

   Recommendation: (a).

2. The operator's tools.

   • (a) A redrive job in the walkthrough, tested in process, plus runbooks with Kafka's own CLI for offsets, as proposed.
   • (b) Runbooks only, with Kafka's CLI and kcat for redriving too. That is less code, but a redrive that strips the dlq. headers and keeps each key is then a shell pipeline, first run during an incident.

   Recommendation: (a).

3. Aborts in the shared cases.

   • (a) Delays only in the shared cases. Aborts are tested in process, as proposed.
   • (b) Aborts shared too, with a Target hook that untrusts the topic by rolling the processor out with FAULT_TRUSTED_TOPICS unset. Staging would then see the crash loop that production would, but every abort case would cost a rollout, and would hold its partition for everyone else's traffic until the hook ran.

   Recommendation: (a).

Out of scope

• Kafka's transactions, exactly-once processing, and the outbox pattern.
• Schemas beyond JSON, such as a schema registry, Avro or Protobuf.
• A read API over the projection. An API is the API walkthrough's.
• Creating topics and ACLs, and schema migrations in production. The kind overlay does its own.
• Adding partitions to a live topic, and moving to a new one.
• Static group membership, and a lag gauge recorded by the processor itself.
• Publishing the image, or a release of the walkthrough.
• Dashboards, which revision 13's point 8 leaves out of every walkthrough.

References

• Proposal: humus on the bedrock process model, its revision 13, and story(docs): create walkthrough documentation for prod ready kafka processor #317
• Proposal: queue processing and the Kafka runtime, Proposal: producing to Kafka, Proposal: exposing health probes, Proposal: failure injection, Proposal: humus's own metrics and Proposal: jobs
• Consuming and producing Kafka, Failure injection and Health
• v0.15.0's example/queue
• kafka_exporter, kube-state-metrics, and Prometheus's unit tests for rules
• kind, and Kafka's kafka-consumer-groups.sh

[Zaba505]

Revision 1: no redrive job, and code-first pages

Open question 2: no redrive job, and no redrive runbook. Whether a dead letter can be replayed depends on why it was dead-lettered. That is learned only by running the system and looking at what fills orders.dlq, and why, so it is each end user's own. The pages leave the reader a note: plan how to handle dead letters from your business requirements.

Two more changes:

• Option (b) of open question 1 is corrected. Since the charter's revision 14, WireMock stands in for an HTTP API a walkthrough invents. WireMock can't run on humustest's Host, so the enricher's whole App couldn't run inside synctest.
• The pages are code-first steps, as revision 14's point 9 says. The post's first page weighed the projection against alternatives, which point 9 leaves out.

The charter's revisions 14 and 15 change this walkthrough as their tables say: a module of its own, up.sh, the LGTM stack, Grafana alert rules and a dashboard, Collector receivers in place of exporters, and no comments. Its stories make those changes, so this revision doesn't repeat them.

Each section below replaces or adds to the part of the same name in the post. Everything else is unchanged.

Summary

The bullet on operators becomes:

• It shows what its operators do: watch the group's lag, roll out without losing or repeating anything, get past a record that stops the consumer, and rebuild the projection by resetting the group's offsets. What to do with dead letters is the reader's to plan.

The application

redrive/ and cmd/redrive/ are dropped. The Dockerfile builds only orderstatus.

Operating it

"Replaying dead letters" is replaced by:

• Dead letters. OrderStatusDeadLetters says when they arrive, and by reason. Each one's dlq. headers name the record it came from. Whether any can be replayed depends on why it was dead-lettered, so the pages tell the reader to plan for orders.dlq from their business requirements, and the walkthrough ships no tool for it.

The tests

In the row "The App, in process", "the redrive job against the same cluster" is dropped.

The deployment

redrive/ is dropped from the kustomization. The Dockerfile builds orderstatus as a static binary, onto a distroless, non-root base.

The pages

This section is replaced.

They are under docs/content/walkthroughs/orderstatus/. Each page is a list of steps. A step is a sentence or two, then the code it arrives at, quoted from the walkthrough's files. A reason is given only where the code would be wrong without it, and in one sentence, such as why the status event is produced before the projection is stored, or why the upsert only moves forward. There is no history, no alternatives and no asides.

1. Overview: what the processor does, its three topics, and the files it ends with.
2. The topics: keys, partitions, retention and ACLs.
3. The projection: schema.sql, and the store's upsert.
4. Projecting an event: decoding, the transitions, the offset guard, and producing before storing.
5. Dead letters: the reasons, the headers, and producing them before Process returns.
6. Init: the settings, the secrets as files, mTLS, the pool, the Producer, the probes, BuildTelemetry, Faults and main.
7. Testing: a unit test of the projector, the shared cases with their Target and barrier, and the failure injection cases.
8. Deploying: the Dockerfile, the kustomization, and the kind overlay. Running it on kind with up.sh.
9. Observing: a record's trace across the topics in Grafana, its logs and metrics, the alerts and the dashboard.
10. Operating it: lag, skipping a record that stops the consumer, untrusting a topic, rebuilding the projection, and the note on dead letters.
11. Production-ready: the checklist.

Design decisions

The bullet "An operator's tools are tested like the processor" is replaced by:

• No redrive. Whether a dead letter can be replayed depends on why it was dead-lettered, and that is learned by running the system. So it is each end user's to plan, and the walkthrough ships no tool for it.

Testing

"The redrive job" is dropped from what wasn't prototyped.

Open questions

Question 2 is answered above. In question 1, option (b) becomes:

• (b) A stateless enricher. It consumes orders, prices each one through an HTTP service of the end user's own, and produces priced orders. WireMock stands in for that service, so its in-process cases run against containers on humus.OSHost(), as the job walkthrough's do. It has no state, so there is no idempotency, replay or rebuild to show, and order per key barely matters.

The recommendation stays (a). Questions 1 and 3 are still open.

Out of scope

This is added:

• Replaying dead letters, by a job or a runbook. What can be replayed depends on why records are dead-lettered in each end user's system.

References

Proposal: jobs is dropped, here and from what the walkthrough builds on.

[Zaba505]

Revision 2: a retry topic, order enforced, and no operating

Open question 1: an order-status projection, as recommended.

Open question 3 no longer arises. The consumer never stops on a record. A record that fails for a reason that isn't its own goes to a retry topic, orders.retry, and to orders.dlq once its attempts run out. An abort then holds nothing up, so the abort cases are shared like the delays.

Three more changes:

• Order is still enforced. An event out of order is retried, and then dead-lettered, because the event before it may be in orders.retry. The service owns order status, so it trusts no producer's order.
• Operating it is out of scope. A walkthrough is a production-ready application you can run. Running one like it in production is for the end user who ships it. The runbooks go, and so does their page.
• Setting it up and trying it. The topics and their ACLs are created with Kafka's CLI. A small Go CLI produces events by hand.

Each section below replaces or adds to the part of the same name in the post, as revision 1 left it. Everything else is unchanged.

Summary

The bullet on operators is dropped. The bullets on dead letters, failures and failure injection become:

• A record it can't apply is a dead letter. One that isn't an event, or is of a type it doesn't know, goes to orders.dlq at once, with its reason.
• No record stops it. An event out of order, PostgreSQL that is down, or a cluster that won't store the status event sends the record to orders.retry. It is tried again after a wait that grows with its attempts, and goes to orders.dlq once they run out. Neither topic is blocked.
• Failure injection is part of the tests. A record whose baggage header carries a scenario slows or fails PostgreSQL or the status topic. A slowed record is still projected once. A failed one goes through its attempts to orders.dlq. The scenario goes on to neither orders.status nor orders.dlq.

Background

The bullet "A record's scenario can't be taken back" becomes:

• A record's scenario follows it through its retries. Failure injection trusts a record by its topic, and a record keeps its headers. The producer to orders.retry forwards the scenario, so every attempt fails the same way, and the dead letter is stored without it. An abort in a shared environment ends in orders.dlq, and holds up nothing else.

The application

cmd/orderevents/ is added: a small CLI that produces an order's events to orders.events, keyed by its ID, to try the processor by hand. The Dockerfile builds it beside orderstatus.

The topics and their records

This row is added:

Topic	Key	Value	Written by
orders.retry	the order's ID	the consumed record's value, unchanged	the processor, for each record it will try again

• orders.retry has twelve partitions, as orders.events does.
• The topics and their ACLs are created with Kafka's CLI, kafka-topics.sh and kafka-acls.sh, before the processor starts. The kind overlay runs them, and the pages show them.

Projecting an event

Steps 1 to 3 become:

1. Decode it. A record with no key, a value that isn't an event, or a missing at is a dead letter, malformed. An unknown type is a dead letter, unknown-type. Neither is retried, since no attempt would change it.
2. Read the order's projection, as the post says. A record from orders.retry is guarded by the partition and offset it came from on orders.events, which its headers carry. Open question 4.
3. Check the transition, by the post's rules. An event that repeats the current status changes nothing. Any other event is retried, invalid-transition.

"Failures that aren't the record's" is replaced by:

Retries go through orders.retry. One Process handles both topics, and keeps no state of its own:

• A record's attempt is a header, retry.attempt. A record from orders.events has none, and is attempt 0.
• A retried record waits until its timestamp plus a backoff derived from its attempt: RETRY_MIN_DELAY, doubled each attempt, up to RETRY_MAX_DELAY. It waits as long as that is. A record also waits for those ahead of it on its partition, so a retry may come later than its backoff, but never sooner.
• A record that fails for a reason that isn't its own is produced to orders.retry with its key, value and headers, retry.attempt one higher, retry.reason and retry.error, and, on its first retry, retry.partition and retry.offset. The reasons are invalid-transition and unavailable: the store's retries ran out, or the cluster wouldn't store the status event. Once RETRY_ATTEMPTS have failed, it is a dead letter instead, with the last attempt's reason.
• The store's calls are still retried in process first, as the post says, and producing is still not retried by the projector.
• Every record's work has a deadline of 15s, counted after its wait.
• The Service stops only on a record that can go nowhere, because the cluster won't store it in orders.retry or orders.dlq. Process returns the error, the App exits 1, and the record is processed again after the restart rather than lost. Every Kafka consumer has to be ready for that.
• A wait during shutdown or a revoke. A record that is waiting when SIGTERM comes holds the shutdown until the grace period runs out, and the App exits 1. A record that is waiting through a revoke holds it up to RebalanceTimeout, and then the member is fenced out. Either way, the partition's next owner processes the record again, and nothing is lost. In a healthy system, waits are short. How a Processor's context should end is for #1004 to work out.

Dead letters add dlq.attempts. Their dlq.topic, dlq.partition and dlq.offset name the record on orders.events, even when it came from orders.retry.

Init

These settings are added:

Variable
KAFKA_RETRY_TOPIC	required
RETRY_ATTEMPTS, RETRY_MIN_DELAY, RETRY_MAX_DELAY	10, 1s and 5m by default

• The Service consumes orders.events and orders.retry, in one group.
• It builds two Producers. The one for orders.retry sets ForwardScenario, so a retried record keeps its scenario. The one for orders.status and orders.dlq doesn't, so a scenario never reaches other teams' consumers.
• The in-process tests and the kind overlay set FAULT_TRUSTED_TOPICS to orders.events and orders.retry.

Its own instruments are orderstatus.dead_letters and orderstatus.retries, counters by orderstatus.reason.

Operating it

This section is dropped. Operating a processor like this one in production is the business of the end user who ships it. The note on dead letters moves to the page on retries and dead letters.

Production-ready, as a checklist

These rows change:

	How
A rollout loses nothing	the revoke commits the record in flight. A record's work fits in 15s, under the 20s grace period, and terminationGracePeriodSeconds is 30. A retried record still waiting at SIGTERM is cut short when the grace period runs out, and processed again by its partition's next owner. The rollout case checks it
No record stops it	records that are malformed or of an unknown type are dead letters at once. The rest go through orders.retry, and then to orders.dlq
An outage blocks neither topic, and keeps order	its records wait in orders.retry. An order's later events follow its earlier ones there, since an event out of order is retried too
Least privilege	mTLS with ACLs that let it read orders.events and orders.retry, and write orders.status, orders.retry and orders.dlq. The rest is unchanged
Safe by default	no record's scenario trusted. Only the producer to orders.retry forwards one

The rows "A record it can't apply doesn't stop it" and "An outage doesn't lose order or records" are replaced by the second and third above.

The tests

These rows change:

Level	Where it runs	What it covers
The projector	go test, in the Build job, inside synctest	decoding, the transitions, the guard, produce-then-store, the retry headers and the backoff, dead letters after the last attempt, and the deadline. Its Store and producers are the walkthrough's own interfaces, faked in the test
The App, in process	go test, in the Build job	what only a test that controls the App can show: a retried record waiting at SIGTERM, cut short when the grace period runs out and processed again; a record that can go to neither orders.retry nor orders.dlq stopping the App, and processed after the restart; bad settings; the spans, the instruments and the breaker's

The Target adds the retry topic, and the deployment's retry settings, so a case knows how long a record's attempts can take. The barrier covers orders.retry too: a case polls the group's committed offsets on both topics until neither holds a record of its own that isn't committed. The kind overlay and the in-process tests set short retry delays, so a record's attempts take seconds.

The cases change:

• Out of order is added. placed, shipped, then paid: shipped is retried, and the order ends shipped, with three status events in order and no dead letter.
• Dead letters becomes: a record that isn't JSON, and one of an unknown type, are dead letters at once. An event that never becomes valid, such as delivered for an order that was cancelled, is a dead letter, invalid-transition, after RETRY_ATTEMPTS. The partition moves on throughout.
• Failure injection adds aborting postgresql:orders. The record goes through its attempts to orders.dlq, unavailable, with no scenario. Nothing is projected for its order, the next event on its partition is projected, and the consumer never stops.

"Aborts stay in process" is dropped. The in-process tests of an abort that stops the consumer, and of untrusting the topic, go with it.

The deployment

• The ConfigMap adds the retry settings.
• The kind overlay's Job creates the four topics and their ACLs with kafka-topics.sh and kafka-acls.sh.
• The kind overlay sets short retry delays, and trusts orders.events and orders.retry.

The alert rules

• OrderStatusRetrying is added: records have been retried for 10m, from orderstatus.retries, as a ticket.
• The lag rules read orders.events only, since records in orders.retry wait on purpose.

The pages

This section is replaced. The rules for a page are revision 1's.

1. Overview: what the processor does, its topics, and the files it ends with.
2. The topics: creating them and their ACLs with Kafka's CLI, keys, partitions and retention.
3. The projection: schema.sql, and the store's upsert.
4. Projecting an event: decoding, the transitions, the guard, and producing before storing.
5. Retries and dead letters: the retry topic, the attempt header and the backoff, dead letters, and a note that what to do with orders.dlq is the reader's to plan, from their business requirements.
6. Init: the settings, the secrets as files, mTLS, the pool, the Producers, the probes, BuildTelemetry, Faults and main.
7. Testing: a unit test of the projector, the shared cases with their Target and barrier, and the failure injection cases.
8. Deploying: the Dockerfile, the kustomization, and the kind overlay. Running it on kind with up.sh, and producing events with orderevents.
9. Observing: a record's trace across the topics in Grafana, its logs and metrics, the alerts and the dashboard.
10. Production-ready: the checklist.

Design decisions

"Dead letters for the record's faults, a stop for everyone else's" and "Only delays are shared" are dropped. These are added:

• A consumer never stops on a record. Records that are malformed or of an unknown type are dead letters at once. Everything else is retried through orders.retry and then dead-lettered, so neither topic is blocked. Only a record that can't be stored anywhere stops it.
• Order is enforced, not trusted. The service owns order status, so it checks every transition, whatever its producers promise. An event out of order is retried rather than dead-lettered, since the event before it may be in orders.retry.
• Retries keep no state. The attempt is a header and the wait derives from it, so one Process handles both topics, and any replica can take any record.

"No retry around Produce" adds: a status event that fails sends the record to orders.retry, as any outage does.

Testing

This is added: the prototype's abort stopped the App, as the post designed. The retry topic, its wait and the guard on retried records weren't prototyped.

Open questions

Questions 1 and 3 are answered above. This one is added:

4. The guard against redelivery. It keeps the partition and offset of the last event applied, and skips any record at or before them. With a retry topic, one case gets past the transition check unseen. paid waits in orders.retry, then cancelled arrives, and placed → cancelled is valid, so it's applied. When paid comes back, it is below the guard and skipped, so orders.status never says the order was paid.

   • (a) Keep every event applied, by its partition and offset on orders.events, rather than only the last. A late paid is then seen as never applied, fails cancelled → paid, and goes through its attempts to orders.dlq.
   • (b) Keep only the last, as the post does, and accept the case.

   Recommendation: (a).

Out of scope

This is added:

• Operating it in production: runbooks for lag, a record that won't process, or rebuilding the projection. They're for an end user who ships a processor like this one.

References

These are added:

• queue: when should a Processor's context be cancelled?
• Kafka's topic and ACL tools

[Zaba505]

Revision 3: an order's status is folded from its events

Open question 4: the projection keeps every event it applied, and an order's status is folded from them. A late event takes its place in the order's history rather than being skipped or dead-lettered. Without a real product's business case, this is a decision made for now.

Each section below replaces or adds to the part of the same name in the post, as revision 2 left it. Everything else is unchanged.

Summary

The bullets on order and on redelivery become:

• Order is enforced. An order's status is folded from its events, in the order of their at, through a state machine. A history the state machine refuses, such as shipped with no paid before it, is retried, and then dead-lettered.
• Every record is processed at least once, and processing one twice changes nothing: the projection keeps every event it applied.

Projecting an event

Steps 2 to 4 become:

2. Skip it if it was applied, found by its partition and offset on orders.events. A record from orders.retry is found by the ones its headers carry.
3. Fold the order's history with it. The order's events, and this one, are sorted by at, then by offset, and run through the post's transitions. An event that repeats the status before it changes nothing. A history the transitions refuse is retried, invalid-transition, since an event before it may still be coming.
4. Produce the changes to orders.status, and only then store the event and the folded status. The changes run from the event's place in the history to its end. An event at the end produces its own change. A late one also produces each later change again. A late paid, after cancelled, produces paid, previous placed, and then cancelled again, previous paid. A consumer that keeps the latest status sees it flicker back to paid before it settles on cancelled. One that orders by at doesn't.

The projection in PostgreSQL

The schema becomes:

CREATE TABLE order_events (
    source_partition integer     NOT NULL,
    source_offset    bigint      NOT NULL,
    order_id         text        NOT NULL,
    type             text        NOT NULL,
    at               timestamptz NOT NULL,
    PRIMARY KEY (source_partition, source_offset)
);

CREATE INDEX order_events_by_order ON order_events (order_id, at);

CREATE TABLE order_status (
    order_id text        PRIMARY KEY,
    status   text        NOT NULL,
    at       timestamptz NOT NULL
);

"One upsert stores a change, and only moves forward" is replaced by:

• One transaction stores an event: it locks the order's status row, inserts the event and stores the folded status. Two replicas racing for a partition during a rebalance can't both apply one event, since the second insert conflicts and its transaction rolls back.
• order_events grows by a row per event. An order's status is folded from all of its events, so none is deleted.

The role's privileges become SELECT, INSERT and UPDATE on both tables.

Production-ready, as a checklist

This row changes:

	How
Redelivery is harmless	every event applied is kept, by its partition and offset, and skipped when it comes again

The tests

The projector's row covers the fold in place of the guard: an event at the end, a late event and its changes, a refused history, and a skipped redelivery.

This case is added:

• A late event. placed, cancelled, then paid with an at between them. The order ends cancelled. orders.status has placed, cancelled, then paid and cancelled again, and there is no dead letter.

The pages

Pages 3 and 4 become:

3. The projection: schema.sql, the events, and the folded status.
4. Projecting an event: decoding, skipping what was applied, folding the history, and producing before storing.

Design decisions

"The offset is stored with the state" is replaced by:

• The status is folded from its events. Keeping only the last offset can't tell a late event from a redelivered one, because one partition carries many orders. So a late event would be skipped, and its change lost. Keeping every event finds a redelivery by its offset, and puts a late event where it belongs.
• The fold orders by at. An order's events are on one partition, but a producer that writes them out of order still gives each the time it happened. Folding by offset would refuse shipped written before paid, even when their ats are right. A history whose ats break the transitions is still refused.

Open questions

None. Question 4 is answered above.

Out of scope

These are added:

• Pruning order_events.
• What a consumer of orders.status does with a late change, such as a refund for an order paid and then cancelled. That depends on the product.