Proposal: RabbitMQ processor walkthrough

[Zaba505]

Important

Revision 1 answers every open question, and takes the Kafka processor walkthrough's changes. It is a mailer, its topology is a definitions file, and failures wait in a delay queue with one fixed delay. There is no replay: the pages tell the reader to plan for mail.send.dead from their business requirements. The worker never stops on a request, and an event it can't publish parks the request. Operating it is out of scope. The logins are created with rabbitmqctl, and a small Go CLI publishes requests by hand. WireMock replaces the stand-in, and the cases read its request journal. #953 is fixed, so no case waits for it.

Note

Part of the humus re-evaluation: wave 6 of Proposal: humus on the bedrock process model, which plans one proposal per walkthrough, each keeping the points of its revision 13. This is the RabbitMQ processor that #318 asked for. The reset closed #318.

It builds on proposals that have all landed:

• RabbitMQ, with its revision 1;
• outbound HTTP clients;
• exposing health probes;
• failure injection;
• humus's own metrics;
• waves 1 to 3.

It found one gap in humus: a humus Publisher can't set a message's user-id, so no consumer can trust a scenario that a humus service forwards. Revision 13's first point makes a gap a bug report, not a workaround, so it is reported as #953, and the one case that needs its fix waits for it. See A gap in humus.

Summary

A walkthrough is a production-ready application, kept in this repository, with pages that walk a reader through how it was built and why. This one is a RabbitMQ work queue: a mailer, which sends transactional email through an email provider's HTTP API.

• Other services ask for an email by publishing a request to the mail exchange. The mailer consumes the mail.send queue, calls the provider once per request, and publishes the result to mail.events: mail.sent with the provider's ID, or mail.refused with the provider's reason.

• It is what RabbitMQ is for. Replicas compete for one quorum queue, and each message is acked once its result is the broker's. Transient failures wait in a delay queue and come back. A request that can never succeed goes to a parking-lot queue, where an operator inspects it and replays it. The queue's delivery limit is the backstop for a message that crashes the worker. The Kafka processor walkthrough, beside this one in wave 6, is the event stream, ordered per key. This one is the opposite case: no order, and work per message.

• It uses most of queue/rabbitmq, httpclient and fault:

  • the Service at AtLeastOnce, with Prefetch and Concurrency tuned for I/O-bound work;
  • two Publishers, one per trust boundary;
  • Consuming as readiness;
  • httpclient's retries for a POST with an Idempotency-Key, a Classify of the provider's own, Retry-After and the breaker;
  • TrustScenario by UserID, ForwardScenario, and injection points in both clients.

• It owns its topology, because the Service declares nothing. A RabbitMQ definitions file of exchanges, quorum queues, bindings and policies is shipped for the broker's team to import. The tests and the kind job import the same file.

• It is tested at three levels, with the cases written once:

  • its Processor and its provider client as units, inside synctest;
  • the whole App against RabbitMQ in a container, on humus.OSHost();
  • the deployed image in kind.

  RabbitMQ has no committed offset to wait on, so each case's barrier is its request's outcome: an event, or a dead letter.

• Failure injection is part of the tests. A scenario that aborts the provider shows the outage end to end: retries, delays, giving up, the parking lot and the replay. A scenario that cuts the worker off from the broker shows a lost connection and a redelivery that sends nothing twice. A scenario that fails the event's publish shows the crash loop that the delivery limit ends.

{"to": "ada@example.com", "subject": "Your order shipped", "text": "Order 1234 is on its way."}

Published to mail with the routing key mail.send, a MessageId (its idempotency key) and an optional CorrelationId, it becomes a mail.sent event with the same IDs and a body of {"provider_id":"msg_…"}.

Background

• story(docs): create walkthrough documentation for a prod ready rabbitmq processor #318 asked for a production-ready RabbitMQ processor. Proposal: RabbitMQ #765 built queue/rabbitmq and left the walkthrough to wave 6. It said this walkthrough's App tests run against RabbitMQ in a container, on humus.OSHost(), rather than inside synctest, and that its test cases would be written once and run against a deployed environment too.

• humus leaves the hard parts of a work queue to the end user, on purpose. The RabbitMQ page says so in three places:

  • Topology: the Service and the Publisher declare nothing;
  • Dead letters: there is no dead-letter exchange, retry queue or skip policy of humus's own;
  • A processing error: under AtLeastOnce, a processing error stops the Service and exits 1.

  So a worker whose dependency has an outage of a minute must not return an error for it, or every replica crash-loops. Where the message waits instead, how long, and when to give up, are what a walkthrough has to show.

• An HTTP dependency fits httpclient when it is one origin. One client per dependency is httpclient's rule. An email provider is one origin, with idempotency keys, rate limits, Retry-After, and refusals that aren't outages, so it uses all of the client.

• Trust in a RabbitMQ message comes from its user-id. RabbitMQ validates it against the publishing connection's user, so failure injection trusts a delivery by UserID. But rabbitmq.Publishing has no UserID field, and its conversion to amqp091's sets no UserId (publisher.go#L42-L118). So a message that a humus service publishes arrives with an empty UserID, and a consumer strips any scenario it forwards (faults.go#L34-L44). The prototype below confirms it.

• v0.15.0 had no RabbitMQ example. Its walkthroughs were an orders REST API and a 1BRC job, and they ran under podman-compose with no deployment. The reset deleted them.

• Proposal: Kubernetes controller walkthrough is the first walkthrough proposed. It is outside the waves, and this proposal shares its shape: the production-ready checklist, the three levels and the shared cases.

Proposed API

A walkthrough adds no API to humus. What it adds is an application, its topology, its manifests, its tests and its pages.

The application

It lives in the example module, at example/mailer. The module gains github.com/z5labs/humus/queue/rabbitmq, replaced with ../queue/rabbitmq as the core module is.

example/mailer/
├── main.go               humus.Main(app.Init, WithTelemetry, WithFaults, WithShutdownGracePeriod(25s))
├── app/                  Init, the settings, BuildTelemetry, Faults
├── mail/                 the request and the events: the mailer's contract with other services, and its topology's names
├── worker/               the Processor: decode, send, decide, publish
├── provider/             the provider's client, over httpclient's *http.Client, and its Classify
├── standin/              the stand-in provider: an http.Handler, and a main for the kind overlay
├── cases/                the shared test cases, and their Target
├── functional/           the cases against a deployed mailer (build tag functional)
├── deploy/
│   ├── base/             the kustomization: Deployment, ServiceAccount, PodDisruptionBudget
│   ├── rabbitmq/         definitions.json, the topology for the broker's team
│   ├── alerts.yaml       the Prometheus rules
│   └── overlays/kind/    RabbitMQ, the stand-in, an OpenTelemetry Collector, the logins
└── Dockerfile

The contract

A request is published to the mail exchange with the routing key mail.send:

• its MessageId is required, and is the provider's idempotency key, so a publisher that publishes a request twice gets one email;
• its CorrelationId is optional, and the events carry it back;
• its body is JSON, to, subject and text, up to 256 KiB.

The sender's address isn't in the request. It is the mailer's setting, so no publisher can send as anyone else.

An event is published to the mail.events topic exchange, with the request's MessageId and CorrelationId:

• mail.sent, with the provider's message ID;
• mail.refused, with the provider's reason, for a request that the provider will never accept, such as an address that doesn't exist.

A service that wants the results binds a queue of its own. An event that no queue is bound to receive is returned as rabbitmq.ErrUnroutable, which the mailer treats as no error, because nobody subscribes to it yet.

The topology

deploy/rabbitmq/definitions.json is a RabbitMQ definitions file. It holds no users and no passwords:

Name	Kind	What it does
mail	direct exchange	where requests are published
mail.send	quorum queue, bound to mail by mail.send	the work queue
mail.retry	direct exchange	where the worker parks a request that failed for now
mail.send.retry	quorum queue, bound to mail.retry	holds each parked request for the delay, then dead-letters it back to mail with mail.send
mail.dead	fanout exchange	dead letters, from the worker and from the queue's delivery limit
mail.send.dead	quorum queue, bound to mail.dead	the parking lot, which operators inspect and replay
mail.events	topic exchange	mail.sent and mail.refused

The behaviour is in policies, not queue arguments, so an operator can change a delay or a limit without deleting a queue:

• mail.send: delivery-limit 5, dead-letter-exchange mail.dead, dead-letter-strategy at-least-once, and overflow reject-publish, which at-least-once dead-lettering requires.
• mail.send.retry: message-ttl 5 minutes, dead-letter-exchange mail, dead-letter-routing-key mail.send, and the same strategy and overflow.

The broker's team imports the file with rabbitmqctl import_definitions, or the management API. The kind overlay and the container tests import it the same way, into a running broker, and add a policy of their own, of higher priority, with a delay of 1s and a delivery limit of 2. The prototype found that importing at boot, with load_definitions, needs the vhost in the file and skips creating the default user. The pages say so.

Least privilege. The mailer's RabbitMQ user is granted:

• configure: none (^$), since it declares nothing;
• write: ^mail\.(retry|dead|events)$;
• read: ^mail\.send$.

Processing a request

The Processor runs under queue.AtLeastOnce, so a request is acked only once its result is the broker's. It decides on each delivery like this:

The request	The provider	The worker publishes	Then
malformed: no MessageId, not JSON, a field missing, over 256 KiB	not called	a dead letter to mail.dead, reason malformed	acked once the broker confirms it
valid	2xx	mail.sent	acked once confirmed, or at once when it's unroutable
valid	a 4xx other than 408, 409 and 429	mail.refused	the same
valid, below MAIL_MAX_ATTEMPTS	unreachable, a 5xx, a 408 or 429 the client gave up on, or the breaker open	the request again, to mail.retry, with x-mail-attempt one higher	acked once confirmed. It comes back after the delay.
valid, at MAIL_MAX_ATTEMPTS	the same	a dead letter, reason gave-up, with the last error	acked once confirmed
any		a publish that fails	Process returns the error. The Service stops, the pod restarts, and the delivery is redelivered and counted toward the delivery limit.

• A dead letter is the request as first published, plus why it died: x-mail-dead-reason, x-mail-dead-attempts and x-mail-dead-error, without x-mail-attempt. So a replay starts counting again.
• The worker keeps no state. The provider's idempotency key makes every redelivery safe: one that arrives after the provider sent the email is answered with the first send's ID, and the event is published again with it.
• No address or body reaches the telemetry. The process span never records a body. The worker logs and sets attributes from MessageId, the outcome and the attempt only.

Calling the provider

• One httpclient client for PROVIDER_URL, trusting PROVIDER_CA_FILE or the system's roots.
• Every send is a POST /v1/messages with an Idempotency-Key, the request's MessageId. Its body is a *bytes.Reader, so the client may retry it (Which requests are retried).
• The provider's API key is read from PROVIDER_API_KEY_FILE, through s.Host.FS, and set as Authorization: Bearer by the provider client. Outbound credentials are service code's, as Proposal: authentication and authorization #870 left them.
• Its Classify is the default, plus one answer. The provider answers 409 while another request with the same key is in flight. That happens when a lost connection redelivers a request to a second replica while the first is still sending it. A 409 is then retried, and counts for the provider rather than against it, as in Writing a Classify.
• A Retry-After longer than BackoffMax ends the call, as the client does. The request then waits in the delay queue instead, so a rate limit holds no worker.

The stand-in implements the provider's API, which the walkthrough defines, since the provider is fictional:

• POST /v1/messages: 202 with an ID; the same key again gets the same ID; 409 while that key is in flight; 422 for an address in .invalid, a domain that can't exist (RFC 2606);
• GET /v1/messages?idempotency_key=…: what it accepted for a key, which is how the cases check that an email was sent once.

The stand-in is an http.Handler. The unit tests serve it on humustest's Host, the container tests on the OS Host with httpserver, and the kind overlay as a Deployment of its own, from the same image. It is the walkthrough's own stand-in for a dependency it invents, not a fake of another project's server.

Init

• It sources every setting with config.Struct. A bad one exits 78, naming its variable:

  Variable
  RABBITMQ_ADDR, RABBITMQ_USERNAME, RABBITMQ_PASSWORD_FILE	required
  RABBITMQ_VHOST, RABBITMQ_CA_FILE	/; TLS when the CA file is set
  RABBITMQ_PREFETCH, RABBITMQ_CONCURRENCY	16 and 8: the work is I/O-bound and needs no order
  MAIL_FROM	required
  MAIL_MAX_ATTEMPTS	5
  MAIL_TRUSTED_PUBLISHERS	the RabbitMQ users whose scenarios are trusted; empty
  PROVIDER_URL, PROVIDER_API_KEY_FILE	required
  PROVIDER_CA_FILE, PROVIDER_TIMEOUT, PROVIDER_ATTEMPT_TIMEOUT, PROVIDER_MAX_ATTEMPTS	the system's roots, then httpclient.DefaultSettings
  PROBES_ADDR, PROBES_TLS_PEM_FILE	required
  FAULT_MAX_DELAY	10s

  Passwords and keys are read from files, mounted from Secrets, never from the environment.

• It builds two Publishers, one per trust boundary. ForwardScenario is a Publisher's setting, and the two kinds of destination differ:

  • the worker's own: mail.retry, with ForwardScenario set, so a parked request keeps its scenario;
  • everyone else's: mail.events, whose consumers are other teams, and mail.dead, whose consumers are operators and their tools. These strip the scenario.

• TrustScenario trusts a delivery whose UserID is in MAIL_TRUSTED_PUBLISHERS, or is the mailer's own user. The mailer's own messages carry a scenario only when a trusted publisher's did, since an untrusted one was stripped first. With the default, an empty list, production trusts no publisher. Trusting its own user needs the fix for the gap.

• It returns the Service and an httphealth probes server, with health.Readiness(svc.Consuming()). It has no liveness monitor: the Service reconnects on its own.

Faults builds failure injection with jsonscenario, under the key acme.fault, the Failure injection page's key, which every walkthrough uses.

main sets a grace period of 25s with humus.WithShutdownGracePeriod. A delivery in flight at SIGTERM may need the provider's Timeout and a publish's Timeout, 10s each, and humus.Main's 20s would cut it short. The manifest's terminationGracePeriodSeconds is 35: the grace period, the telemetry's 5s shutdown, and a margin.

Production-ready, as a checklist

The pages close with this list, and the mailer meets each item:

	How
Every setting sourced and checked; a bad one exits 78, naming its variable	config.Struct and the packages' checks
No secret in the environment, the logs or the spans	the RabbitMQ password, the provider's key and the probes' certificate are files read through s.Host.FS
Certificates renewed without a restart	the probes' certificate is served through GetCertificate, by a small helper of the walkthrough's own that reads its files again when they change
No personal data in the telemetry	addresses and bodies are never logged or put on a span
Two replicas, competing for one queue	a PodDisruptionBudget of minAvailable: 1
Ready only once it consumes	Consuming as readiness, on a probes server
A rollout loses nothing, and sends nothing twice	SIGTERM cancels the consumer, finishes the deliveries in flight within 25s, and nacks the rest, which counts nothing against them. terminationGracePeriodSeconds: 35
A broker's rolling restart costs no restart	the Service reconnects. The deliveries that were in flight are redelivered, and the idempotency key sends them once
A dependency's outage is ridden out	the client's retries, then the delay queue, then the parking lot after MAIL_MAX_ATTEMPTS. The breaker sheds calls while the provider keeps failing
A message that can never succeed doesn't block the queue	malformed requests go to the parking lot at once. A message that crashes the worker goes there after the delivery limit
Least privilege	no configure permission on the broker; write and read on its own exchanges and queue only. No service account token. A non-root, read-only container
Observable	traces, logs and metrics over OTLP, and the alerts in deploy/alerts.yaml
Failures say why	each dead letter's reason headers, and the mail.refused event
Failures are rehearsed	the failure injection cases, below
Safe by default	TrustScenario trusts no publisher; the sender's address is a setting

Operating it

The pages give operators a runbook:

• Queue depth and age. The backlog of mail.send, and how Concurrency and replicas drain it. Autoscaling on queue depth is out of scope.

• Redeliveries. What raises them: a lost connection, a rollout cut short, a crash. Each counts toward the delivery limit.

• The parking lot. How to read a dead letter's reason headers, and how to replay mail.send.dead back to mail, with a dynamic shovel or the management UI's Move messages. The prototype found three things the pages state:

  • a replayed message gets the whole delivery limit again;
  • its first delivery carries the dead letter's stale x-delivery-count;
  • its user-id is kept.

  A replay moves every dead letter, so one that can't succeed comes straight back. A replay later than the provider's idempotency window, 24 hours for the stand-in, can send an email the provider had already accepted. The pages say to replay within it.

• A broker's rolling restart. Every connection is closed once per node, with 320 CONNECTION_FORCED. The workers log lost the connection and reconnect, and nothing restarts.

• Rollouts. What the drain does, and what HUMUS_SHUTDOWN_GRACE_PERIOD changes.

Observing it

• Spans: each request's process mail.send, a child of its publisher's send mail span, with the provider's attempts and the send span of its result under it. The worker sets mail.attempt and mail.outcome on the process span.
• Its own instrument: mail.requests, a counter of requests by mail.outcome: sent, refused, retried, gave_up or malformed. It is the end user's own instrument, named for its domain.
• humus's: the messaging instruments, and humus.circuit.* and humus.retry.decisions for the provider's client and both Publishers.
• RabbitMQ's own: its Prometheus plugin's per-queue metrics.

deploy/alerts.yaml holds the rules, each explained on the pages:

Alert	When
MailBacklog	mail.send has had more than a threshold of ready messages for 10 minutes
MailNotConsuming	mail.send has had no consumer for 5 minutes
MailParked	mail.send.dead isn't empty
MailGivingUp	mail.requests{mail.outcome="gave_up"} rose in the last 15 minutes
MailProviderBreakerOpen	the provider client's humus.circuit.state has been open for 5 minutes
MailRetriesThrottled	humus.retry.decisions was throttled in the last 5 minutes
MailCrashLooping	a mailer container restarted more than 3 times in 15 minutes, which is what a message hitting the delivery limit looks like

The tests

Level	Where it runs	What it covers
The Processor as a function	go test, inside synctest, in the Build job	every row of the decision table, with the provider client and both Publishers behind the walkthrough's own interfaces, faked in its tests
The provider client	go test, inside synctest	the client against the stand-in on humustest's Host: the idempotency key, the 409 retried, Retry-After, the breaker opening, and the API key
Init	go test, inside synctest	every setting's exit 78, the files read through the Host, and with nothing listening, waiting for the broker and exiting 0 on SIGTERM
The topology	go test	definitions.json declares every exchange and queue that mail names, with the policy keys the design relies on
The alerts	promtool check rules and promtool test rules, in the kind job	each rule fires on the series its test gives it, and only then
The shared cases, in-process	testcontainers tag, in the "Mailer walkthrough (testcontainers)" job, not required	the whole App on humus.OSHost() and humustest's process, against RabbitMQ in a container with the definitions imported, and the stand-in on the OS Host
The shared cases, deployed	functional tag, in the "Mailer walkthrough (kind)" job, not required, and against an end user's staging	the image built from the Dockerfile, deployed with the kind overlay

The shared cases are in cases. It imports neither the application nor humustest, so the compiler keeps them black-box. Each case is a function of a testing.TB and a Target:

Target field	What it is
the broker's address, vhost, TLS and login	the cases' own login, a trusted publisher of the deployment under test
the stand-in's URL and an HTTP client	to read what the provider accepted for a key
ProviderPoint	the provider's injection point as the mailer names it, such as http:standin.mail.svc:8443, which a port-forward's address differs from
ScenarioKey	the baggage member the mailer's codec reads
CloseConnections	closes every connection to the broker, as a node's shutdown does. nil skips the cases that need it
Replay	moves the parking lot back to mail with a shovel, as the runbook does. nil skips the case that needs it
Timeout	bounds every wait. It exceeds MAIL_MAX_ATTEMPTS times the delay

The barrier is the request's outcome. RabbitMQ has no committed offset to wait on. So before it publishes, each case binds queues of its own, exclusive and server-named, to mail.events and to mail.dead. It then waits for the event or the dead letter with its MessageId.

• The worker publishes a result only after the provider has answered, and acks only after the broker has confirmed it. So once the outcome arrives, what the provider holds for the key is final, and an assertion that something didn't happen holds.
• A redelivery may publish a second event, with the same IDs, so a case asserts on the first.
• mail.dead is a fanout exchange, so a case's own queue gets a copy of every dead letter, the delivery limit's included, and the cases run in parallel against one broker.

The cases:

• Run in parallel:
  • Sent. A request becomes mail.sent, with its MessageId and CorrelationId, and the provider holds one email for its key.
  • Sent once. The same request published twice gets one email, and every event names the same provider ID.
  • Refused. A recipient in .invalid becomes mail.refused, at the first attempt, and nothing is dead-lettered.
  • Malformed. A body that isn't a request is dead-lettered with malformed, no event follows, and the provider holds nothing for its key.
  • A slow provider. Under a scenario that delays ProviderPoint by 2s, under AttemptTimeout, the request is still sent at the first attempt.
  • An untrusted scenario. The abort scenario below, published by a login that isn't trusted, is stripped, and the request is sent at the first attempt.
• Run alone, one at a time, after the rest. Each opens a breaker or cuts connections, which slows every message in flight:
  • The provider down, then replayed. Under a scenario that aborts ProviderPoint, each processing fails and parks the request, scenario and all. After MAIL_MAX_ATTEMPTS processings, it is dead-lettered with gave-up and without its scenario, and the provider holds nothing for its key. Replay then moves it back, and it is sent at the first attempt.
  • A lost connection. Under a scenario that delays the provider by 3s, CloseConnections closes every connection while the request is in flight. Its ack fails, it is redelivered, and the provider is called again with the same key. The provider holds one email, the event arrives, and a request published after it is sent too.
  • A result that can't be published. Under a scenario that aborts rabbitmq:mail.events:mail.sent, the event's publish fails, so Process returns its error, and the worker exits 1 on each delivery. After the delivery limit, the request is dead-lettered with x-first-death-reason: delivery_limit, and the provider still holds one email. The in-process harness restarts the App after each exit, as the kubelet does.

The cases mutate the broker and the provider, so they never run against production. In staging, the alone cases are an outage of the mailer that runs them. The pages say to run them against instances of their own, as Containing a scenario in production describes.

The deployment

deploy/base is a kustomization:

• the Deployment: two replicas, the probes on the probes port with scheme: HTTPS, resources, terminationGracePeriodSeconds: 35, and a non-root, read-only security context;
• a ServiceAccount with automountServiceAccountToken: false, since the mailer never calls the Kubernetes API;
• the PodDisruptionBudget;
• references to the Secrets of the RabbitMQ password, the provider's key and the probes' certificate.

The kind overlay adds:

• a single-node RabbitMQ with its management, Prometheus and shovel plugins, serving AMQP over TLS;
• the definitions, imported with the overlay's policy;
• the stand-in;
• an OpenTelemetry Collector that the mailer exports to;
• the logins, generated by the job: the mailer's, with the permissions above, and the cases', which is in MAIL_TRUSTED_PUBLISHERS.

The Dockerfile builds static binaries of the mailer and the stand-in onto a distroless, non-root base.

The pages

docs/content/walkthroughs/mailer/, in the order the mailer was built:

1. What we're building, and why: a work queue against an event stream, and the contract with other services.
2. The topology: the exchanges and quorum queues, the policies, the delay queue, the parking lot, least-privilege permissions, and importing the definitions.
3. Calling the provider: httpclient, idempotency keys, Classify, Retry-After, the breaker, the API key, and the stand-in.
4. Processing a request: the decision table, when a message is acked, publishing with confirms, what ErrUnroutable means for each destination, and idempotency.
5. Wiring the App: Init, the settings, secrets from files, the two Publishers, the probes, telemetry, Faults and TrustScenario, and the grace period.
6. Deploying: the image, the kustomization, the disruption budget, terminationGracePeriodSeconds, and handing the definitions to the broker's team.
7. Operating it: the runbook above.
8. Observing it: the spans, the records, the instruments, and the alerts.
9. Testing: the levels, the shared cases, the Target, and the barrier.
10. Breaking it on purpose: the failure injection cases, trust by UserID, and running them in staging.
11. Production-ready: the checklist.

A gap in humus

RabbitMQ validates a message's user-id against the publishing connection's user, which is why failure injection trusts a delivery by UserID. But a humus Publisher can't set it: rabbitmq.Publishing has no UserID field, and its conversion to amqp091's sets no UserId (publisher.go#L42-L118). So every message a humus service publishes arrives with an empty UserID, and a consumer strips any scenario it carries (faults.go#L34-L44). The prototype confirms it.

That breaks more than this walkthrough. The failure injection page's own example trusts the end user's services by their UserID, which no humus service can send today. Revision 13's first point makes such a gap a bug report, not a workaround:

• It is reported as #953, with a fix such as a UserID on Publishing, which RabbitMQ then validates against the Publisher's login.
• One case waits for the fix: the provider down, then replayed. Its story is natively blocked by the fix's. Until then, the worker's retries strip the scenario, so a parked request comes back clean.
• Nothing else does. Every other case, and every other part of the mailer, needs no user-id of its own.

Changes to other files

• example/go.mod: the queue/rabbitmq module, replaced with ../queue/rabbitmq; amqp091-go; and testcontainers' RabbitMQ module.
• .github/workflows/build.yaml:
  • a "Mailer walkthrough (testcontainers)" job and a "Mailer walkthrough (kind)" job, neither required;
  • the Build job vets the testcontainers and functional tags of example/mailer.
• The RabbitMQ page: links to the walkthrough from Topology, Dead letters and Testing.

The Walkthroughs section and the docs test's rule for its samples come from #629's revision 13. This walkthrough's pages wait for them.

Design decisions

• A mailer. It is the work RabbitMQ is usually given: independent tasks, a slow third-party API, failures of two kinds that need telling apart, and results that other services want. It needs no database, since the provider's idempotency key makes every redelivery safe, so the walkthrough stays about RabbitMQ. Open question 1.
• A delay queue, and a count the worker keeps. A transient failure can't be an error, because an error stops the Service, and every replica would crash-loop through an outage. It can't be a requeue in place either: humus has none, and an immediate requeue would spin. So the worker parks the request in mail.send.retry, whose TTL is the delay, and counts attempts in a header of its own. One fixed delay, not one queue per delay or a TTL per message: a quorum queue expires messages only from its head, so a long TTL would hold shorter ones behind it. Open question 3.
• The delivery limit is a backstop, not the retry. It counts what crashes the worker, or a result it can't publish. Those restart a pod per delivery, so 5 is enough. The quorum queue's default of 20 would crash-loop for an hour.
• Dead letters lose their attempt count. A replay then starts over, and the dead letter says how many attempts it took.
• Two Publishers. ForwardScenario is per Publisher, and the trust boundary differs per destination: the worker's own retry queue takes part in the scenario, but other teams' consumers and the operators' parking lot don't. Two Publishers are two connections, which RabbitMQ handles easily.
• Policies, not queue arguments. RabbitMQ refuses to redeclare a queue with different arguments, so an argument can only change by deleting the queue. A policy changes in place. Only the queue type is an argument.
• The topology is a definitions file that the broker's team imports. It works with any broker, needs nothing in the cluster, and keeps the mailer without a configure permission. Open question 2.
• The mailer trusts its own user. It can't originate a scenario: one reaches it only from a trusted publisher, and an untrusted one is stripped first. So trusting its own retries widens nothing.
• The stand-in implements an API the walkthrough defines. The provider is fictional, so its stand-in is no fake of another project's server. A real provider's sandbox takes its place in staging.
• 25s of grace. A delivery in flight needs up to the provider's 10s and a publish's 10s. Cutting it short redelivers it, which counts toward the delivery limit.
• Cases written once, against both harnesses. The barrier is each request's own outcome, read from queues the case binds, so cases run in parallel against one broker and against staging. The cases that open breakers or cut connections run alone.

Testing

The walkthrough is its own test: see The tests.

Prototyped against RabbitMQ 4.3.6 in a container, with humus at 1bb3bb53, under go test -race -count=3:

• The definitions. A definitions file with these exchanges, quorum queues, bindings and policies imports into a running broker with rabbitmqctl import_definitions, and list_queues shows each policy applied. Imported at boot, with load_definitions, the file needs its vhost, and the broker then creates no default user, so the testcontainers login is refused.
• The delay hop. A request published to mail.retry came back to mail.send after the 1s TTL (1.03s), with x-death reason expired.
• The delivery limit, and the barrier. With a limit of 2, a message rejected with requeue was delivered 3 times, with x-delivery-count 1 and 2 on the redeliveries. It was then dead-lettered through the fanout mail.dead to both the parking lot and a case's own exclusive, server-named queue, with x-first-death-reason: delivery_limit and its user-id kept.
• The replay. A dynamic shovel from mail.send.dead to mail moved the dead letter back, with its user-id kept. Its first delivery carried the stale x-delivery-count 3, and it was then delivered 3 more times before it was dead-lettered again, so a replay gets the whole limit back.
• Trust, and the gap. A humus App, with rabbitmq.NewService, an httpclient client to a stand-in on the OS Host, humus.WithFaults and a TrustScenario on UserID, was given messages carrying a scenario that aborts the stand-in's point:
  • a message whose user-id was trusted reached the Processor with its scenario, and its call failed with *fault.InjectedError;
  • an untrusted one was stripped, and its call reached the stand-in;
  • the trusted message's copy, which a Publisher with ForwardScenario set published again, carried the scenario in its baggage header but arrived with an empty UserID, so it was stripped.

Migration from today

Nothing migrates. humus had no RabbitMQ example before the reset, and the site has no Walkthroughs section yet.

Open questions

1. Which worker.

   • (a) A mailer, as proposed.
   • (b) A thumbnail renderer, with images in object storage, in a MinIO container. It shows CPU-bound work: Prefetch, Concurrency and resources tuned for it. But it calls no API through httpclient, and an S3 SDK brings retries of its own.
   • (c) A webhook dispatcher. It is familiar, but each subscriber is an origin of its own, and httpclient is one client per origin, built in Init. So it would teach working around the client.

   Recommendation: (a).

2. How the topology reaches the broker.

   • (a) A definitions file that the broker's team imports, as proposed.
   • (b) The Messaging Topology Operator's resources, Exchange, Queue, Binding and Policy, applied with the mailer's kustomization. They are Kubernetes-native, but need RabbitMQ's Cluster Operator and Topology Operator in the cluster, and in the kind overlay.
   • (c) A Job that declares the topology with amqp091 before each rollout. It needs a login with configure permission, and RabbitMQ refuses to redeclare a queue whose arguments changed.

   Recommendation: (a).

3. How transient failures wait.

   • (a) A delay queue and the worker's own count, as proposed.
   • (b) No delay: Process returns the error, and the delivery limit bounds it. Each failure restarts a pod, so an outage of the provider crash-loops every replica, and readiness can't stop it.
   • (c) The delayed-message exchange plugin, which delays each message by a header. It is a community plugin, not in the official image, and holds delayed messages on one node only, outside any quorum queue.

   Recommendation: (a).

Out of scope

• Templates, HTML bodies and attachments. The request carries plain text.
• Bounce and complaint webhooks from the provider. They are an HTTP API, which is another walkthrough's subject.
• More than one provider, or failing over between them.
• Autoscaling on queue depth, such as with KEDA.
• Operating RabbitMQ itself: clustering, upgrades, backups and quorum queue membership. The broker is the broker team's. The overlay's single node is for CI.
• Order per recipient, which would need a single-active-consumer queue and Concurrency 1.
• Exactly-once sending. It is impossible. The idempotency key makes a redelivery safe within the provider's window.
• Publishing the image, or a release of the walkthrough.
• Dashboards, which revision 13's point 8 leaves out of every walkthrough.

References

• Proposal: humus on the bedrock process model, its wave 6 and revision 13, and story(docs): create walkthrough documentation for a prod ready rabbitmq processor #318
• Proposal: RabbitMQ and its revision 1
• Proposal: outbound HTTP clients, Proposal: exposing health probes, Proposal: failure injection, Proposal: humus's own metrics
• Proposal: Kubernetes controller walkthrough
• The docs site's pages: RabbitMQ, Calling HTTP services, Failure injection, Health
• publisher.go#L42-L118 and faults.go#L34-L44
• RabbitMQ: quorum queues, dead lettering, policies, definitions, validated user-id, access control, shovel

[Zaba505]

Revision 1: no replay, a worker that never stops, and WireMock

Open questions 1 to 3 are answered as recommended:

1. A mailer.
2. A definitions file that the broker's team imports with rabbitmqctl import_definitions.
3. A delay queue and the worker's own count, with one fixed delay. A message waiting in a queue holds no worker and never holds up a shutdown, and one policy changes the delay in place.

The Kafka processor walkthrough's revisions 1 and 2 made choices that hold here too:

• No replay. Whether a dead letter can be replayed depends on why it was dead-lettered, and each end user learns that by running their own system. The parking lot stays. The shovel, the runbook, Target.Replay and the replay half of a case go. The pages tell the reader to plan for mail.send.dead from their business requirements.
• The worker never stops on a request. An event that can't be published parks the request, as an outage does. Only a request that can go to neither mail.retry nor mail.dead stops the worker.
• Operating it is out of scope. A walkthrough is a production-ready application you can run. Running one like it in production is for the end user who ships it. The runbook goes, and so does its page.
• Setting it up and trying it. The logins and their permissions are created with rabbitmqctl, just as the definitions are imported with it. A small Go CLI publishes requests by hand.
• The pages are code-first steps, as the charter's revision 14's point 9 says.

Two more changes:

• #953 is fixed. #962 added UserID to rabbitmq.Publishing, so no case waits for it.
• WireMock replaces the stand-in, as the charter's revision 14 says. The stand-in kept what it had accepted for each key. WireMock keeps only its request journal, so the cases change, and this revision says how.

The charter's revisions 14 and 15 change the rest as their tables say: a module of its own, up.sh, the LGTM stack, Grafana alert rules and a dashboard, the Collector's RabbitMQ and Kubernetes cluster receivers, and no comments. Its stories make those changes, so this revision doesn't repeat them.

Each section below replaces or adds to the part of the same name in the post. Everything else is unchanged.

Summary

In the bullet "It is what RabbitMQ is for", the parking lot's sentence becomes: "A request that can never succeed goes to a parking-lot queue, with why. What to do with it is the reader's to plan."

In the bullet on the three levels, the second becomes "the whole App against RabbitMQ and WireMock in containers, on humus.OSHost()".

The bullet on failure injection becomes:

• Failure injection is part of the tests. A scenario that aborts the provider shows the outage end to end: retries, delays, giving up and the parking lot. A scenario that cuts the worker off from the broker shows a lost connection, and a redelivery that sends the same idempotency key. A scenario that fails the event's publish shows the request parked and tried again, and the worker never stopping.

Background

The bullet "Trust in a RabbitMQ message comes from its user-id" ends: "#953 reported that rabbitmq.Publishing had no UserID, and #962 added it."

The application

• standin/ is dropped.
• cmd/mailrequest/ is added: a small CLI that publishes a request to mail with the routing key mail.send, to try the mailer by hand. It takes to, subject and text. It makes a random MessageId unless it is given one, and prints it. It reaches the broker with a login that may write mail and nothing else, its password read from a file.
• The Dockerfile builds mailer and mailrequest.

The topology

• mail.send.dead's row says "the parking lot, where dead letters wait with why they died".
• "so an operator can change a delay or a limit" becomes "so a delay or a limit can change".
• The logins and their permissions are created with rabbitmqctl, add_user and set_permissions, with the patterns of "Least privilege". The broker's team runs them, as it imports the definitions. up.sh runs them in kind, and the pages show them.

Processing a request

These rows of the table change:

The request	The provider	The worker publishes	Then
valid, below MAIL_MAX_ATTEMPTS	unreachable, a 5xx, a 408 or 429 the client gave up on, or the breaker open; or it answered, and the event's publish failed	the request again, to mail.retry, with x-mail-attempt one higher	acked once confirmed. It comes back after the delay, and calls the provider again with the same key
valid, at MAIL_MAX_ATTEMPTS	the same	a dead letter, reason gave-up, with the last error	acked once confirmed
any		a publish to mail.retry or mail.dead that fails	Process returns the error. The Service stops, the pod restarts, and the delivery is redelivered and counted toward the delivery limit

In the dead letters' bullet, "So a replay starts counting again" is dropped. A dead letter still leaves out x-mail-attempt, since x-mail-dead-attempts says the same.

Calling the provider

"The stand-in" is replaced by:

WireMock stands in for the provider. The provider is fictional, so the walkthrough defines its API, and owns WireMock's mappings. They are in the kind overlay, and the container tests load the same files:

• POST /v1/messages with no bearer token: 401;
• POST /v1/messages to an address in .invalid, a domain that can't exist (RFC 2606): 422, with a reason;
• any other POST /v1/messages: 202, with the ID msg_ and its Idempotency-Key, by response templating, so the same key always gets the same ID.

WireMock keeps no state between requests, so it never answers 409. The provider client's unit tests serve handlers of their own on humustest's Host, which answer as each test needs: 409 while a key is in flight, 429 with Retry-After, and 5xx.

Init

• In the Publishers' bullet, mail.dead is described as "whose dead letters are read outside the mailer".
• In the TrustScenario bullet, "Trusting its own user needs the fix for the gap" becomes: "The retry Publisher sets Publishing.UserID to the mailer's login, which RabbitMQ validates, so a parked request keeps a trusted scenario."

Production-ready, as a checklist

These rows change:

	How
No request stops the worker	malformed requests go to the parking lot at once. The rest are parked in mail.send.retry, an event that can't be published included, and dead-lettered after MAIL_MAX_ATTEMPTS. Only a request that can go to neither mail.retry nor mail.dead stops the worker, and the delivery limit dead-letters it after 5 deliveries
Least privilege	no configure permission on the broker; write and read on its own exchanges and queue only, granted with rabbitmqctl set_permissions. No service account token. A non-root, read-only container

The row "A message that can never succeed doesn't block the queue" is replaced by the first.

Operating it

This section is dropped. Running a mailer like this one in production is the business of the end user who ships it. The note on dead letters moves to the page on retries and dead letters, and what a rollout does moves to the page on deploying.

Observing it

"RabbitMQ's own" becomes: each queue's depth and consumers, from the Collector's RabbitMQ receiver, which reads the management API with a login tagged monitoring.

The alerts change:

Alert	When
MailRetrying	requests have been parked for 10 minutes, from mail.requests{mail.outcome="retried"}, as a ticket
MailCrashLooping	a mailer container restarted more than 3 times in 15 minutes, which is what a request that can go nowhere looks like

MailRetrying is added. MailBacklog reads mail.send only, since the messages in mail.send.retry wait on purpose.

The tests

These rows change, and the last is added:

Level	Where it runs	What it covers
The provider client	go test, inside synctest	the client against handlers of the test's own on humustest's Host: the idempotency key, the 409 retried, Retry-After, the breaker opening, and the API key
The shared cases, in-process	testcontainers tag, in the "Mailer walkthrough (testcontainers)" job, not required	the whole App on humus.OSHost() and humustest's process, against RabbitMQ in a container with the definitions imported, and WireMock in a container with the walkthrough's mappings
The App, in process	testcontainers tag, in the same job	a request that can go to neither mail.retry nor mail.dead, under a scenario that aborts the provider and the publish to mail.retry. The App exits 1 on each delivery, and the test restarts it after each exit, as the kubelet does. After the delivery limit, the request is dead-lettered with x-first-death-reason: delivery_limit. In a deployed environment this is a crash loop of every replica that takes it, so it isn't a shared case

The Target:

• "the stand-in's URL and an HTTP client" becomes "WireMock's URL and an HTTP client, to read its request journal";
• ProviderPoint's example becomes http:wiremock.mail.svc:8443;
• Replay is dropped.

What the provider was sent. Each case sends to an address of its own. It finds what the provider was sent in WireMock's request journal, POST /__admin/requests/find, by that address. Under "The barrier is the request's outcome", "what the provider holds for the key is final" becomes "what WireMock's journal holds for the address is final".

The cases change:

• Sent: WireMock was sent one request, with the MessageId as its Idempotency-Key, and a bearer token.
• Sent once: every request WireMock was sent carries the same Idempotency-Key, and every event names the same provider ID.
• Malformed: WireMock was sent nothing.
• The provider down replaces "The provider down, then replayed". Under a scenario that aborts ProviderPoint, each processing fails and parks the request, scenario and all. After MAIL_MAX_ATTEMPTS processings, it is dead-lettered with gave-up and without its scenario, and WireMock was sent nothing.
• A lost connection: every request WireMock was sent carries the same Idempotency-Key, the event names the ID that key gets, and a request published after it is sent too.
• A result that can't be published: under a scenario that aborts rabbitmq:mail.events:mail.sent, each processing calls the provider, fails to publish the event, and parks the request, scenario and all. After MAIL_MAX_ATTEMPTS processings, it is dead-lettered with gave-up and without its scenario. Every request WireMock was sent carries the same Idempotency-Key, no mail.sent arrives, and the worker never exits. It still runs alone, since it opens the events Publisher's breaker.

The deployment

• The Dockerfile builds static binaries of mailer and mailrequest.
• The kind overlay's RabbitMQ has its management plugin only.
• WireMock, with the walkthrough's mappings, replaces the stand-in.
• up.sh generates the logins and creates them with rabbitmqctl: the mailer's; the cases', which is in MAIL_TRUSTED_PUBLISHERS; mailrequest's, which may write mail only; and the Collector's, tagged monitoring.

The pages

This section is replaced.

They are under docs/content/walkthroughs/mailer/. Each page is a list of steps. A step is a sentence or two, then the code it arrives at, quoted from the walkthrough's files. A reason is given only where the code would be wrong without it, and in one sentence. For example, a reason explains why a transient failure is parked rather than returned. There is no history, no alternatives and no asides.

1. Overview: what the mailer does, its contract, its exchanges and queues, and the files it ends with.
2. The topology: definitions.json and its policies, importing it, and creating the logins and their permissions with rabbitmqctl.
3. Calling the provider: the httpclient client, the Idempotency-Key, Classify, Retry-After, the API key, and WireMock's mappings.
4. Processing a request: decoding, the decision table, publishing with confirms, ErrUnroutable, and acking.
5. Retries and dead letters: the delay queue, the attempt header, the dead letters and their headers, the delivery limit, and a note that what to do with mail.send.dead is the reader's to plan, from their business requirements.
6. Init: the settings, the secrets as files, the two Publishers and UserID, TrustScenario, the probes, BuildTelemetry, Faults, and main's grace period.
7. Testing: a unit test of the Processor, the shared cases with their Target and barrier, and the failure injection cases.
8. Deploying: the Dockerfile, the kustomization, the disruption budget, the grace period, and the kind overlay. Running it on kind with up.sh, and publishing requests with mailrequest.
9. Observing: a request's trace in Grafana, its logs and metrics, the alerts and the dashboard.
10. Production-ready: the checklist.

A gap in humus

This section is replaced.

The gap is closed. #953 reported that a humus Publisher couldn't set a message's user-id. Its fix, #962, added UserID to rabbitmq.Publishing, which RabbitMQ validates against the Publisher's login. The retry Publisher sets it to the mailer's login, so a parked request keeps a trusted scenario. No case waits for it any longer.

Design decisions

These are replaced:

• "A delay queue, and a count the worker keeps" ends: "One fixed delay, rather than one that grows with each attempt: a message waiting in a queue holds no worker and never holds up a shutdown."
• "The delivery limit is a backstop, not the retry" becomes: it counts what crashes the worker, which is a request that can go to neither mail.retry nor mail.dead, or a pod killed while it works. Those restart a pod per delivery, so 5 is enough. The quorum queue's default of 20 would crash-loop for an hour.
• "Two Publishers" ends: "the worker's own retry queue takes part in the scenario, but other teams' consumers and the parking lot don't."
• "The stand-in implements an API the walkthrough defines" becomes: WireMock, with mappings the walkthrough owns. The provider is fictional, so WireMock stands in for no other project's server. A real provider's sandbox takes its place in staging.

"Dead letters lose their attempt count" is dropped. These are added:

• A worker never stops on a request. A request that can never succeed is dead-lettered at once. Everything else is parked, an event that can't be published included, and dead-lettered after its attempts. Only a request that can go to neither mail.retry nor mail.dead stops it.
• No replay. Whether a dead letter can be replayed depends on why it was dead-lettered, and that is learned by running the system. So it is each end user's to plan, and the walkthrough ships no tool for it.

Testing

"The replay" is dropped. This is added: the prototype ran before #962. It didn't try a retry Publisher that sets UserID, or WireMock.

Open questions

None. Questions 1 to 3 are answered above.

Out of scope

These are added:

• Replaying dead letters, with a shovel, the management UI or a runbook. What can be replayed depends on why requests are dead-lettered in each end user's system.
• Operating it in production: runbooks for queue depth, redeliveries, the parking lot, a broker's rolling restart, or rollouts. They're for an end user who ships a mailer like this one.

"Dashboards" is dropped, since the charter's revision 15 gives every walkthrough one.

References

The shovel is dropped. These are added:

• Proposal: Kafka processor walkthrough, its revisions 1 and 2
• #953 and its fix, #962
• RabbitMQ's rabbitmqctl
• WireMock's response templating and verifying
• The OpenTelemetry Collector's RabbitMQ receiver