Proposal: job walkthrough

[Zaba505]

Important

Revision 1 answers the open questions and changes four things. It is a billing job, run daily, against an invented payments API, as recommended. WireMock stands in for the API, so paymentsapi is dropped, and there is no compose file: the kind overlay is how to run the job with WireMock. The overlay's OpenTelemetry Collector forwards to the LGTM stack (Loki, Grafana, Tempo and Mimir), and Mimir's ruler loads the alerts. The code has no comments except its license headers and package comments, and the pages are code-first steps, in few words.

Note

Part of the humus re-evaluation: wave 6 of Proposal: humus on the bedrock process model. It keeps the points that revision 13 sets for every walkthrough, so it doesn't ask them again: no API of its own, standing alone, a stated production-ready bar, three levels of tests sharing one set of cases, failure injection in the tests, an image and manifests, CI jobs that aren't required, alerts as rules, pages under the Walkthroughs section, and a home in the example module.

It builds on proposals that are all done: jobs, outbound HTTP clients, failure injection and humus's own metrics, with humus.Main, the HTTP server foundation and wave 1. It depends on no other walkthrough.

Summary

A walkthrough is a production-ready application, kept in this repository, with pages that walk a reader through how it was built and why. This proposal sets out the job's: a billing job that charges subscriptions once a month.

• It bills a month of subscriptions. Each run issues an invoice for every active subscription for the month that just ended, and charges every invoice that is still pending through a payments API. A charge is declined or made, and the invoice records which.
• It is a job in every way a job differs from a service:
  • it runs to completion, and its exit status says whether anything is left to do;
  • Kubernetes acts on that status: a bad setting fails the Job at once, an interrupted run is retried without counting as a failure, and a run that left invoices pending is retried;
  • it is safe to run twice, to run again after any interruption, and to overlap with itself. The invoice's ID is the payments API's idempotency key, so no invoice is charged twice, whatever is retried;
  • a CronJob runs it daily, so a run that fails is caught up by the next one, with no catch-up code;
  • it joins the trace of whatever started it, such as a backfill started by a CI step;
  • a scenario in its BAGGAGE rehearses the payments API or PostgreSQL failing, mid-run.
• It is tested at three levels, with the cases written once: the run as a unit, inside synctest; the whole App against PostgreSQL in a container; and the deployed image, run as Kubernetes Jobs in a kind cluster in CI. The second and third run the same black-box cases.
• It ships with its manifests and its image: a CronJob with a pod failure policy, its alert rules, and a Dockerfile. The payments API is the walkthrough's own invention, and a stand-in of it, built with httpserver, runs beside the job in the tests and in kind.
• Its pages are a section of the Walkthroughs. Every code sample on them is quoted from the walkthrough's files.

$ kubectl -n billing get jobs
NAME                COMPLETIONS   DURATION   AGE
billing-29338140    1/1           41s        2d
billing-29339580    1/1           12s        1d
billing-29341020    0/1           3m10s      3m10s   ← the payments API is down: exit 1, retried

Background

• Feature docs leave the assembly to the reader. Jobs documents each piece: the run, its exit statuses, a pod failure policy and the trace a run joins. It ends with what job leaves to you: retries inside a run, and idempotency across Kubernetes' retries. Those are the hard parts of a real job, and no page shows them done.
• v0.15.0 had a job walkthrough. It solved the One Billion Row Challenge, reading measurements from MinIO and writing results back (1brc-job, example/job/1brc-walkthrough). It ran once under podman-compose, beside an LGTM stack. It read its MinIO keys from an embedded YAML file, and it had no tests, no manifests, and no story for a retry, a resume or a schedule. The reset deleted it.
• A CronJob runs a job about once per schedule, not exactly once. Kubernetes' CronJob docs say that a CronJob may create two Jobs for one time, or none, so jobs should be idempotent. A Job then retries a failed pod under backoffLimit, from the start, in a new process. A job that charges money has to be safe under all of that.
• Payment APIs make a retried charge safe with an idempotency key. The client sends a key with each charge, and the API answers a repeated key with the first answer instead of charging again, as Stripe does, and as the IETF's Idempotency-Key header draft standardizes. httpclient retries a POST that carries an Idempotency-Key, and makes one attempt of a POST without one.
• Earlier proposals left this walkthrough two things. Failure injection left showing failure injection in a walkthrough. Jobs and failure injection left the one case where a scenario is trusted with no TrustScenario: a job trusts its BAGGAGE.
• Test cases can be written once and run against both harnesses. Revision 13 sets the pattern for every walkthrough: cases in a package of their own, which imports neither the application nor humustest, and takes a Target. For a service, a case sends requests. For a job, a case seeds the job's state, runs the job, and reads what it did. So this walkthrough's Target runs the job.

Proposed API

A walkthrough adds no API to humus. What it adds is an application, its manifests, its tests and its pages.

The application

It lives in the example module at example/billing, beside the shoppingcart. It needs nothing beyond the core module and pgx, which the module already has.

example/billing/
├── main.go          humus.Main(app.Init, app.Options()...)
├── app/             Init, the settings, Faults, the telemetry Builder, Options
├── billing/         the run: issuing, charging, settling, and the exit it ends with
├── postgres/        the run's Invoices on PostgreSQL, its injection points, schema.sql
├── payments/        the payments API's client: the request, the answers, Classify
├── paymentsapi/     the stand-in payments API, and its main in paymentsapi/cmd
├── cases/           the shared test cases, and their Target
├── functional/      the cases against kind, and the checks only kind can make (build tag functional)
├── deploy/          the kustomization, its kind overlay, and alerts.yaml
└── Dockerfile       two targets: billing, and the stand-in

app.Options() returns what main passes to humus.Main, the telemetry Builder and humus.WithFaults(Faults), so the tests run the App that main runs, as Sharing main's options describes.

The data

Two tables, in postgres/schema.sql:

• subscriptions: an account, its plan's amount in cents and currency, and whether it is active. The end user's other services keep it. The job only reads it.

• invoices: one per account and month, UNIQUE (account_id, period). Each has an ID, a UUID, which is also its idempotency key; an amount and currency; and a state:

  • pending until it is charged;
  • charged, with the payments API's charge ID;
  • declined, with the API's decline code;
  • failed, with the answer that made it fail, for a person to look at.

  charged, declined and failed are final: nothing changes them.

The job applies no migrations. The schema is applied before the job runs: by the container tests, by the kind overlay, and in production by the end user's migration tool, as the API walkthrough shows for the shoppingcart's.

The run

billing.Run is the run that job.NewService is given. It does three things, in order:

1. Issue. One statement inserts an invoice for every active subscription for the run's period, ON CONFLICT (account_id, period) DO NOTHING. Running it again issues nothing.
2. Charge. It reads every invoice that is still pending, whatever its period, oldest first, and charges each through the payments API, with up to BILLING_CONCURRENCY charges in flight.
3. Settle. It records each answer with UPDATE … WHERE id = $1 AND state = 'pending', so a final state is never overwritten.

The run's period is BILLING_PERIOD, or, without it, the month before the run started, in UTC. Every run charges whatever is pending, from any period. So a run that left invoices pending, for any reason, is caught up by the next one, even across a month's end.

The run stops early and returns an error in three cases, without handing out more invoices. It waits for the charges already in flight first.

• The payments client's breaker rejects a call. The API is down, and every further charge would fail at once.
• The API rejects the job's key, with a 401 or a 403.
• A statement fails. If a charge can't be recorded, the next charge's record would fail too.

The run ends by returning:

• nil, when no invoice is left pending;
• an error that names how many invoices are left pending, and why;
• context.Cause(ctx), when a signal stops it. The charges in flight end with the run's context, and their invoices stay pending.

Charging, and what each answer means

The payments API is the walkthrough's own invention, kept small:

• POST /v1/charges, with Authorization: Bearer <key> and Idempotency-Key: <invoice ID>, and the invoice's account, amount and currency as JSON.
• GET /v1/charges?account=<id>, the account's charges, which the test cases read.

The payments package sends the charge, with its body in a *bytes.Reader, so httpclient can replay it. It reads the answer:

The answer	Retried	The breaker	The invoice
201 Created, or 200 with an earlier answer for the key	no	success	charged
402 Payment Required	no	success	declined
409 Conflict: a request with this key is still in flight	yes, by the end user's Classify	success	still pending, if the attempts run out
401, 403	no	success	still pending, and the run stops
422, or another 4xx	no	success	failed: the job's bug or bad data, which a retry meets again
429, 502, 503, 504, a transport error	yes, by the default Classify	429 success, the rest failure	still pending, if the attempts run out
500, another 5xx	no	500 failure	still pending
rejected by the open breaker	no		still pending, and the run stops

The Classify is the default with one addition, the 409, as Writing a Classify shows for a ledger. Retrying a POST is safe only because it carries the key, and the pages say so where the request is built.

The key makes every unknown outcome safe. The answer to a charge can be lost after the API has made it: through a timeout, a reset, a SIGTERM, or a record that fails to save. The invoice then stays pending. The next attempt, in the same call or in the next run, sends the same key, and the API answers with the charge it already made.

Exit statuses, and what Kubernetes does with each

The run	Exit	The CronJob's Job
leaves nothing pending	0	completes
leaves invoices pending: the API down, the key rejected, or a statement failed	1	retries the pod, after a back-off, under backoffLimit
is stopped by SIGTERM: an eviction, a deletion, or activeDeadlineSeconds	1, the signal in its cause	retries an eviction without counting it. At activeDeadlineSeconds the Job fails, and the next day's run catches up
reads a bad setting	78	fails at once, with no retry
panics	2	retries, and counts it

Init

• It sources its settings with config.Struct. A bad one exits 78 and names its variable.
• It reads the database URL and the API key from files, through s.Host.FS. Both are mounted from Secrets.
• It opens its pgx pool through the Host, as the shoppingcart does (openPostgres), with its close registered with s.OnStop.
• It builds the payments client with httpclient.NewClient.
• It returns job.NewService(s, run.Run), and nothing else: a Job's pod has no probes.

Every variable it reads:

Variable	What it is	Default
DATABASE_URL_FILE	a file holding the PostgreSQL URL, password included, which sets every field pgx uses	required
PAYMENTS_URL	the payments API's base URL, https	required
PAYMENTS_API_KEY_FILE	a file holding the API key	required
PAYMENTS_CA_FILE	the CA certificates the API is trusted through, PEM	the system's roots
PAYMENTS_TIMEOUT, PAYMENTS_ATTEMPT_TIMEOUT, PAYMENTS_MAX_ATTEMPTS	httpclient.Settings' fields	httpclient.DefaultSettings()'
BILLING_PERIOD	the month to issue, YYYY-MM. A month that hasn't ended is a bad setting	the month before the run started, in UTC
BILLING_CONCURRENCY	charges in flight, 1 to 32	4
FAULT_MAX_DELAY	fault.Settings.MaxDelay	10s
OTEL_SERVICE_NAME, OTEL_*_EXPORTER, OTEL_EXPORTER_OTLP_ENDPOINT	the telemetry Builder's, as the Telemetry page's	billing, and telemetry.DefaultSettings()'

HUMUS_SHUTDOWN_GRACE_PERIOD is humus's own, and keeps its 20s.

Failure injection

Faults builds the App's fault.Injector with jsonscenario, under the member acme.fault, the Failure injection page's key, which every walkthrough uses. The job has three injection points:

• http:<host>:<port> of PAYMENTS_URL, in the payments client, which every humus client has;
• postgresql:issue and postgresql:settle, of the job's own, before each statement, named as Points of your own suggests. An abort fails the statement with a *fault.InjectedError.

A job trusts its BAGGAGE, so there is nothing to set:

• The CronJob's template sets no BAGGAGE, so its runs carry no scenario.
• A rehearsal is a Job that someone creates from the CronJob's template, with BAGGAGE set. Anyone who can do that can already change every setting, so the trust boundary is the RBAC on jobs in the job's namespace.
• An abort sends nothing, and a delay only waits. So the worst that a rehearsal does, even against production's payments API, is leave invoices pending for the next run.

The payments client keeps ForwardScenario false, so a scenario never reaches the payments API.

Telemetry

• Spans. job's span is the root of each run, or a child of the trace in TRACEPARENT. A backfill started by a CI step, or a workflow engine, sets it. Under it: an issue span, a charge span for each invoice with its ID, and httpclient's span for each attempt.
• Records. A WARN record for each invoice that fails, with its ID and the API's answer. An INFO record at the end of the run, with its counts. The API key is a header, which otelhttp doesn't record, and the database URL is never logged.
• The job's own instruments, in its own scope:
  • billing.invoices.issued, a Counter;
  • billing.invoices.settled, a Counter, by billing.invoice.state: charged, declined or failed;
  • billing.invoices.pending, a Gauge recorded once, as the run ends.
• humus's own instruments, from the payments client: humus.circuit.state, humus.circuit.transitions, humus.circuit.calls and humus.retry.decisions.

A job exports its metrics at its exit, when the telemetry shuts down. Each run is a new process, whose counters start from zero, so a series holds a run's own totals, often in a single sample. The alerts read those values with max_over_time, rather than increase, which needs two samples of one process.

Alerts

deploy/alerts.yaml is a Prometheus rule file:

Alert	Severity	Fires when	Reads
BillingNotSucceeded	critical	no run has succeeded for 26h	kube_cronjob_status_last_successful_time, from kube-state-metrics
BillingInvoicesPending	warning	the last run left invoices pending	billing_invoices_pending
BillingInvoicesFailed	warning	an invoice failed in the last day, for a person to look at	billing_invoices_settled_total{billing_invoice_state="failed"}
BillingPaymentsBreakerOpened	info	the payments client's breaker opened in the last day	humus_circuit_transitions_total{humus_circuit_state="open"}

CI checks the file with promtool, as every walkthrough's. This one also unit-tests its alerts with promtool test rules, against series shaped as a job exports them. That is where increase would silently never fire.

The stand-in payments API

paymentsapi is an httpserver Service that implements the two endpoints, kept in memory:

• It charges once per key, and answers a repeated key with the first answer.
• It answers 422 to a key reused with another body, and 401 to any key but its own.
• It declines every account whose ID starts with declined-.

It is the walkthrough's own: the payments API is invented, so this is no fake of another project's server. It runs as a second App on the unit tests' humustest Host, on the OS Host in the container tests, and as a Deployment in the kind overlay. It never runs in production.

Production-ready, as a checklist

The walkthrough's pages close with this list, and the application meets each item:

	How
Every setting sourced and checked. A bad one exits 78, naming its variable, and isn't retried	config.Struct and the packages' checks, and a pod failure policy that fails the Job on 78
No secret in the environment, the logs or the spans	the database URL and the API key are files from Secrets, read through s.Host.FS. The key is a header, which otelhttp doesn't record
Never charges twice	the invoice's ID is the idempotency key, and a final state is never overwritten
Safe to run again, or twice at once	issuing is ON CONFLICT DO NOTHING. concurrencyPolicy: Forbid keeps scheduled runs apart, podReplacementPolicy: Failed keeps a retry from overlapping a pod that is still stopping, and the key makes any overlap harmless
Resumes after any interruption	every run charges whatever is pending, from any period
An interruption isn't a failure	SIGTERM ends the run with exit 1, which is retried. The pod failure policy ignores DisruptionTarget, so an eviction isn't counted against backoffLimit
Bounded	activeDeadlineSeconds per run, startingDeadlineSeconds per schedule, httpclient's Timeout per call, and the breaker per outage
Least privilege	no Kubernetes API (automountServiceAccountToken: false), non-root, a read-only root filesystem, no capabilities
Observable	traces, logs and metrics over OTLP, and the alerts in deploy/alerts.yaml
Failures say why	the exit cause names what is left pending and why, and a failed invoice keeps the answer that failed it
Failures are rehearsed	the failure injection cases, below
Safe by default	the CronJob's runs carry no scenario, a declined card is final, and a month that hasn't ended can't be issued

The job has no probes and no PodDisruptionBudget. Nothing routes to a Job's pod, and a budget would only hold up a node's drain, which the pod failure policy already absorbs.

The tests

Level	Where it runs	What it covers
The run as a unit	go test, in the Build job, inside synctest	the run against an in-memory Invoices of the test's own, and the stand-in on the same humustest Host. Each answer in the table above. A retried POST. SIGTERM mid-charge, and the next run replaying its key. The breaker opening and the run stopping. BILLING_CONCURRENCY. TRACEPARENT. Each bad setting exiting 78
postgres against PostgreSQL	testcontainers tag	issuing twice issues once, settling never overwrites a final state, and pending invoices come oldest first
The shared cases, in-process	testcontainers tag, in the "Billing walkthrough (testcontainers)" job, which isn't required	the whole App, Init and app.Options(), on humus.OSHost() with humustest's process, against PostgreSQL in a container, and the stand-in on the OS Host
The shared cases, deployed	functional tag, in the "Billing walkthrough (kind)" job, which isn't required, and against an end user's staging	the image built from the Dockerfile, run as Jobs made from the deployed CronJob's template, in kind
The CronJob as deployed	functional tag, kind only	a Job made from the template as it is, with a bad BILLING_PERIOD, fails after one pod, with the reason PodFailurePolicy
The alerts	promtool	check rules, and test rules

The shared cases are in cases. It imports neither the application nor humustest, so the compiler keeps them black-box. Each takes a Target:

// Target is where the cases run the billing job, and what they read its
// work back from.
type Target struct {
	// Postgres is a URL of the PostgreSQL server, as a role that may create
	// databases. Each case creates one of its own, applies Schema to it,
	// and drops it.
	Postgres string
	Schema   string

	// Payments is the payments API's base URL, Client reaches it, and Key
	// is accepted by it. The cases read an account's charges with them.
	Payments *url.URL
	Client   *http.Client
	Key      string

	// Run runs the job once, to completion, with its database at
	// databaseURL and env added to its environment, and returns its exit
	// code.
	Run func(ctx context.Context, databaseURL string, env map[string]string) (int, error)

	// Timeout bounds each case.
	Timeout time.Duration
}

• In-process, Run writes the database URL into a file, and runs the App with humustest.Start on humus.OSHost(), with app.Options().
• Deployed, Run puts the database URL in a Secret of its own. It then creates a Job from the CronJob's template, with that Secret mounted in place of the database's, env added, and backoffLimit: 0, so a case waits for no retries. It returns the container's exit code. The overlay suspends the CronJob, so only the cases' Jobs run.

Every case creates a database of its own, so the cases run in parallel, and against a shared staging server. They never touch another database. The cases:

• Billing a month. Three active subscriptions and one inactive one: exit 0, three invoices charged, and one charge for each at the API. The inactive account has none.
• Running again. The same month again: exit 0, and no new invoice or charge.
• A declined card. A declined- account: exit 0 and declined. Run again, it isn't charged again.
• A bad setting. BILLING_PERIOD of a month that hasn't ended, and one that isn't a month: exit 78 and nothing issued.
• Failure injection, each under a scenario in BAGGAGE:
  • The payments API is down: aborting http:* exits 1, with every invoice pending and nothing sent. A run without the scenario then exits 0, and charges each invoice once.
  • The payments API is slow: delaying http:* by 2s still exits 0, with everything charged.
  • Charged, but not recorded: aborting postgresql:settle exits 1. The API has made the first charge, and the invoice is still pending. A run without the scenario exits 0, and the API still holds one charge for each invoice: the key was replayed.
  • The month can't be issued: aborting postgresql:issue exits 1, having charged nothing. A run without the scenario bills the month.

They mutate, so they never run against production.

The deployment

deploy/ is a kustomization:

• a Namespace and a ServiceAccount with no bindings;
• the CronJob, below;
• the references to the database's and the API key's Secrets, which the operator creates;
• alerts.yaml.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: billing
spec:
  schedule: "0 3 * * *"
  timeZone: Etc/UTC
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 3600
  jobTemplate:
    spec:
      backoffLimit: 6
      activeDeadlineSeconds: 3600
      podReplacementPolicy: Failed
      podFailurePolicy:
        rules:
          - action: FailJob
            onExitCodes:
              containerName: billing
              operator: In
              values: [78]
          - action: Ignore
            onPodConditions:
              - type: DisruptionTarget
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: billing
          automountServiceAccountToken: false
          containers:
            - name: billing
              image: billing

The pages give the CronJob's every field, the security contexts, the environment and the Secrets' volumes. Kubernetes allows only podReplacementPolicy: Failed with a pod failure policy. It is written out anyway, because it is what keeps a retry from overlapping a pod that is still stopping.

The kind overlay:

• suspends the CronJob;
• adds PostgreSQL with the schema, the stand-in payments API, and an OpenTelemetry Collector that the job exports to;
• points the job's settings at them.

The kind job generates the database's password, the API key, and the stand-in's CA and certificate before it deploys, so no secret is in the repository. The Dockerfile builds static binaries onto a distroless, non-root base, with one target for the job and one for the stand-in.

The pages

They are under docs/content/walkthroughs/billing/, in the order the job was built:

1. What we're building, and why: the design, and the alternatives, such as a job per invoice, or a queue of charges.
2. The invoices: the schema, and issuing a month idempotently.
3. Charging: the payments API, the idempotency key, Classify and what each answer means, and the breaker that stops a run.
4. Running to completion: the run's three ends, its exit statuses, SIGTERM, and what a lost answer costs.
5. Wiring the App: Init, the settings, secrets as files, Faults, the telemetry Builder, and Options.
6. Scheduling it: the CronJob and the Job's every field. Retries and the pod failure policy. Deadlines. Why daily. Backfilling a month.
7. Observing: the spans, joining a backfill's trace, the records, the instruments, and the alerts, with why max_over_time.
8. Testing: the three levels, and the shared cases and their Target.
9. Breaking it on purpose: the failure injection cases, and rehearsing in staging, and in production.
10. Production-ready: the checklist.

Changes to other files

• .github/workflows/build.yaml:
  • a "Billing walkthrough (testcontainers)" job and a "Billing walkthrough (kind)" job, neither required;
  • the Build job vets the walkthrough's testcontainers and functional tags.
• The Jobs docs page: a link to the walkthrough from "What job leaves to you".

Design decisions

• Billing, not retention or an export. Charging money is where a job's properties matter most. A retried charge must not charge twice, and an interrupted run must resume. Its dependency can fail mid-run, and its exit status has to say what is left. A retention job deletes, which is idempotent by nature, so it shows none of that. An export to object storage needs an SDK that retries on its own, and that the charter's rule forbids pairing with humus's retries. Open question 1.
• The invoice's ID is the idempotency key. The key is stored before the first attempt, so every attempt and every run sends the same one. A key made per attempt, or per run, would charge twice after a lost answer.
• No lock. concurrencyPolicy: Forbid and podReplacementPolicy: Failed keep runs apart. When a CronJob creates two Jobs anyway, the key makes the overlap harmless: both runs send the same keys, and the API charges each once. A database lock would add a failure mode and protect nothing more.
• Daily, and level-triggered. Each run issues one month and charges whatever is pending. So the first run of a month bills it, and the runs after it catch up on whatever a failure left, with no code of their own for that. Open question 2.
• The breaker ends the run. While the payments API is down, every charge would be rejected at once. Stopping, and exiting 1, hands the retry to the Job's back-off, which waits longer than OpenFor and costs nothing while it waits.
• A declined card is final. Retrying a declined card on later days, which is dunning, is a product decision with rules of its own, and out of scope.
• The stand-in is the walkthrough's. The payments API is invented, so no third party's sandbox, credential or network is needed in CI, and the walkthrough doesn't become one about a particular provider. Open question 3.
• Secrets are files, as revision 13's bar asks of every walkthrough. The database URL and the API key are mounted from Secrets and read through s.Host.FS. They aren't in the environment, which a process hands to its children and a crash report may dump. Each run is a new process, so a rotated secret, or a renewed CA in PAYMENTS_CA_FILE, is read by the next run, with nothing to re-read during one.
• A database per case. A job's state is its database, and the run charges every pending invoice in it. Cases that shared one would charge each other's invoices. With a database each, they run in parallel, and against staging without touching its data.
• The deployed cases run Jobs made from the CronJob's template. They test the image, the Secrets' mounts and the security contexts as deployed. Their backoffLimit: 0 keeps a failing case quick. The policy itself is checked once, separately, from the template as it is.
• Points of its own on PostgreSQL. "Charged, but not recorded" is the failure that idempotency is for. Aborting postgresql:settle causes it on demand, with no change to the stand-in.

Testing

The walkthrough is its own test: see The tests.

What was prototyped. A prototype of the run passed under go test -race -count=3, against main at 1bb3bb5. It used an in-memory store, with the stand-in as a second App on humustest's Host, inside synctest, and the run built with job.NewService, httpclient and humus.WithFaults with jsonscenario. It showed that:

• a POST with an Idempotency-Key is retried through two 503s, and charged once. Without the key, it makes one attempt, and the run exits 1 with the invoice pending;
• a 402 is final, and the run exits 0;
• SIGTERM while a charge's answer is lost exits 1, with serve: terminated signal received. The next run sends the same key, and the API makes no second charge;
• a BAGGAGE scenario aborting http:payments:8443 sends nothing. The first two invoices' calls spend their attempts, the breaker opens, and the third is rejected. The run stops with all eight invoices pending, and errors.Is(cause, circuit.ErrRejected) holds through the exit status's cause;
• a 2s delay still charges everything, in 6s of fake time for three charges. A 6s delay, past AttemptTimeout, ends the call at Timeout, with nothing sent, and exits 1;
• aborting a point of the job's own, postgresql:settle, through s.Faults.Eval with the run's context, leaves the first charge made but not recorded. The next run replays its key: four requests, three charges.

Neither PostgreSQL nor kind was prototyped.

Migration from today

Nothing migrates. v0.15.0's 1BRC walkthrough isn't brought back.

Open questions

1. Which job.

   • (a) A billing job, as proposed.
   • (b) A data-retention job that deletes expired rows in batches. It is simpler, but a delete is idempotent by nature, so it shows no key, no resume and no dependency that fails mid-run.
   • (c) An export to object storage, as v0.15.0's 1BRC walkthrough was. It needs MinIO and an S3 SDK, whose own retries the charter says not to pair with humus's, and its resume would be the SDK's multipart upload, not humus's.

   Recommendation: (a).

2. How often it runs.

   • (a) Daily, issuing the month that ended and charging whatever is pending, as proposed.
   • (b) Monthly, with an alert and a manual backfill after a run that fails. It runs less often, but a failure waits for a person.

   Recommendation: (a).

3. The payments API.

   • (a) Invented, with a stand-in of the walkthrough's own, as proposed.
   • (b) A real provider's test mode, such as Stripe's. It is closer to production, but CI needs its credentials and its network, and the walkthrough becomes one about that provider.

   Recommendation: (a).

Out of scope

• Dunning: retrying declined cards on later days.
• Proration, taxes, refunds, currency conversion, and invoices as documents or emails.
• The payments API's webhooks, and charges that settle asynchronously.
• A real payments provider.
• Schema migrations. The schema is applied before the job runs. The API walkthrough shows applying the shoppingcart's.
• Splitting a run across pods, with parallelism or an Indexed Job.
• A Pushgateway. Metrics leave by OTLP, as every walkthrough's do.
• Publishing the image, or a release of the walkthrough.

References

• Proposal: humus on the bedrock process model, and its revision 13
• Proposal: jobs, and the Jobs page
• Proposal: outbound HTTP clients, and which requests are retried
• Proposal: failure injection, and a job trusts its BAGGAGE
• Proposal: humus's own metrics
• v0.15.0's 1BRC job walkthrough
• Kubernetes: CronJob, Job, and its pod failure policy and delayed creation of replacement pods
• Stripe's idempotent requests, and the IETF draft The Idempotency-Key HTTP Header Field
• Prometheus: unit testing rules

[Zaba505]

Revision 1: the open questions answered, WireMock, the LGTM stack, and code-first pages

The open questions:

1. A billing job, as recommended.
2. Daily, as recommended.
3. An invented payments API, as recommended. The walkthrough is about the job, not the API it calls.

Review changes four more things:

• WireMock stands in for the payments API. It is what an end user would use, rather than writing a payments API of their own. paymentsapi is dropped.
• No compose file. The kind overlay is how to run the job with WireMock, in CI and on a laptop.
• The LGTM stack receives the telemetry. In the kind overlay, an OpenTelemetry Collector forwards OTLP to Loki, Tempo and Mimir, and Grafana reads all three. Mimir's ruler loads alerts.yaml.
• The code has no comments except its license headers and one-line package comments, and the pages are code-first steps, in few words.

Each section below replaces or adds to the part of the same name in the post. Background, The data, The run, Exit statuses, Init, Failure injection, Telemetry, Production-ready and Migration from today are unchanged.

Summary

The bullets on tests, manifests and pages become:

• It is tested at three levels, with the cases written once: the run as a unit, inside synctest; the whole App against PostgreSQL and WireMock in containers; and the deployed image, run as Kubernetes Jobs in a kind cluster in CI. The second and third run the same black-box cases.
• It ships with its manifests and its image: a CronJob with a pod failure policy, its alert rules, and a Dockerfile. The kind overlay adds PostgreSQL, WireMock in place of the payments API, and the LGTM stack. One script brings it up, in CI and for a reader.
• Its pages are code-first steps in the Walkthroughs section. Every code sample is quoted from the walkthrough's files, and the code has no comments to read past.

The application

example/billing/
├── main.go          humus.Main(app.Init, app.Options()...)
├── app/             Init, the settings, Faults, the telemetry Builder, Options
├── billing/         the run: issuing, charging, settling, and the exit it ends with
├── postgres/        the run's Invoices on PostgreSQL, its injection points, schema.sql
├── payments/        the payments API's client: the request, the answers, Classify
├── cases/           the shared test cases, and their Target
├── functional/      the cases against kind, and the checks only kind can make (build tag functional)
├── deploy/          the kustomization, alerts.yaml, and the kind overlay with its script and WireMock's mappings
└── Dockerfile       the job

The code

This section is added.

• Every Go file, tests included, starts with the license header. Each package has a one-line package comment.
• There are no other comments. The names say what the code does, and the pages say why.
• Build constraints and //go:embed are directives, and stay.
• The manifests, the mappings, the script and the Dockerfile have no comments either.

Charging, and what each answer means

GET /v1/charges is dropped. The cases read what the job sent from WireMock's request journal instead, so the API is one endpoint, POST /v1/charges. The table of answers is unchanged.

WireMock in place of the payments API

This section replaces The stand-in payments API.

WireMock serves the payments API from the mappings in deploy/kind/wiremock/:

The charge	WireMock answers
has no Authorization: Bearer header	401
is for an account whose ID starts with declined-	402, {"code":"card_declined"}
anything else	201, {"id":"ch_<Idempotency-Key>"}

• The charge's ID is made from the key, so a repeated key gets the same answer back, as from an idempotent API.
• WireMock serves HTTPS, with a CA and a PKCS build(deps): bump go.opentelemetry.io/otel from 1.30.0 to 1.31.0 #12 keystore generated on each run.
• It runs in a container in the container tests, which mount the same mappings, and as a Deployment in the kind overlay.
• Every other answer in the table under Charging is covered by a unit test.

Alerts

This is added:

In the kind overlay, Mimir's ruler loads alerts.yaml, so its alerts are evaluated against the job's own metrics and listed in Grafana. BillingNotSucceeded has no series there, because the overlay runs no kube-state-metrics, so promtool's tests are what cover it.

The tests

The table becomes:

Level	Where it runs	What it covers
The run as a unit	go test, in the Build job, inside synctest	the run against an in-memory Invoices of the test's own, and an HTTPS handler of the test's own on the same humustest Host, answering as the test needs. Each answer in the table under Charging. A retried POST. SIGTERM mid-charge, and the next run replaying its key. The breaker opening and the run stopping. BILLING_CONCURRENCY. TRACEPARENT. Each bad setting exiting 78
postgres against PostgreSQL	testcontainers tag	issuing twice issues once, settling never overwrites a final state, and pending invoices come oldest first
The shared cases, in-process	testcontainers tag, in the "Billing walkthrough (testcontainers)" job, which isn't required	the whole App, Init and app.Options(), on humus.OSHost() with humustest's process, against PostgreSQL and WireMock in containers
The shared cases, deployed	functional tag, in the "Billing walkthrough (kind)" job, which isn't required, and against an end user's staging where WireMock stands in for the provider	the image built from the Dockerfile, run as Jobs made from the deployed CronJob's template, in kind
The CronJob as deployed	functional tag, kind only	a Job made from the template as it is, with a bad BILLING_PERIOD, fails after one pod, with the reason PodFailurePolicy
The telemetry as deployed	functional tag, kind only	after a run, Mimir holds every series alerts.yaml reads, and its ruler has loaded the rules
The alerts	promtool	check rules, and test rules

The Target becomes:

type Target struct {
	Postgres string
	Schema   string
	WireMock *url.URL
	Client   *http.Client
	Key      string
	Run      func(ctx context.Context, databaseURL string, env map[string]string) (int, error)
	Timeout  time.Duration
}

• Postgres is a URL of a role that may create databases. Each case creates a database of its own, applies Schema to it, and drops it.
• WireMock is WireMock's base URL, and Client reaches it. A case finds the job's requests with POST /__admin/requests/find, matching its own accounts, and checks that each one carried Bearer and Key.
• Run runs the job once, to completion, with its database at databaseURL and env added to its environment, and returns its exit code. In-process and deployed, it runs as the post says.
• Timeout bounds each case.

A case's account IDs start with its database's name. So the cases share one WireMock, run in parallel, and each reads only its own requests.

The cases become:

• Billing a month. Three active subscriptions and one inactive one. The run exits 0, and the three invoices are charged with the IDs WireMock returned. Each invoice was sent once, with its ID as the Idempotency-Key. The inactive account was never sent.
• Running again. The same month again exits 0, and issues and sends nothing new.
• A declined card. A declined- account exits 0, and its invoice is declined. Run again, it isn't sent again.
• A bad setting. BILLING_PERIOD of a month that hasn't ended, and one that isn't a month, each exit 78, with nothing issued or sent.
• Failure injection, each under a scenario in BAGGAGE:
  • The payments API is down: aborting http:* exits 1, with every invoice pending and nothing sent. A run without the scenario exits 0, and sends each invoice once.
  • The payments API is slow: delaying http:* by 2s still exits 0, with everything charged.
  • Charged, but not recorded: aborting postgresql:settle exits 1, with one invoice sent and still pending. A run without the scenario exits 0. That invoice was sent twice, both times with the same key, and every invoice is charged.
  • The month can't be issued: aborting postgresql:issue exits 1, with nothing sent. A run without the scenario bills the month.

The deployment

The kind overlay becomes:

• It suspends the CronJob.
• It adds PostgreSQL with the schema, and WireMock with the mappings.
• It adds the LGTM stack. An OpenTelemetry Collector takes the job's OTLP and forwards it to Loki for logs, Tempo for traces and Mimir for metrics. Grafana has all three as data sources. Loki, Tempo and Mimir each run as one process, storing to the pod's disk.
• It points the job's settings at them.

deploy/kind/up.sh generates the database's password, the API key, WireMock's CA and keystore, and Grafana's admin password. It then loads the image into kind and applies the overlay. The kind job runs it, and so does a reader who wants to run the job. No secret is kept in the repository. The Dockerfile has one target, the job.

The pages

This section is replaced.

They are under docs/content/walkthroughs/billing/. Each page is a list of steps. A step is a sentence or two, then the code it arrives at, quoted from the walkthrough's files. A reason is given only where the code would be wrong without it, and in one sentence, such as why the key is the invoice's ID, or why the alerts use max_over_time. There is no history, no alternatives and no asides.

1. Overview: what the job does, and the files it ends with.
2. The invoices: schema.sql, and issuing a month.
3. Charging: the request and its key, the answers, and Classify.
4. The run: issue, charge, settle, and what it returns.
5. Init: the settings, the secrets as files, the pool, the client, Faults, Options and main.
6. Testing: a unit test of the run, the shared cases and their Target, and the failure injection cases.
7. Deploying: the Dockerfile, the CronJob, and the kind overlay with WireMock and the LGTM stack. Running it on kind, a backfill, and a rehearsal.
8. Observing: a run's trace, records and metrics in Grafana, and the alerts.
9. Production-ready: the checklist.

Changes to other files

This is added:

• docs_test.go: the rule for a walkthrough's samples accepts JSON too, for the WireMock mappings.

Design decisions

"Open question 1." and "Open question 2." are dropped from their bullets. The bullet "The stand-in is the walkthrough's" is replaced by the first of these, and the others are added:

• The payments API is invented, and WireMock stands in for it. The walkthrough is about the job, not a provider. WireMock is what an end user reaches for to stand in for an HTTP dependency, so nobody writes a server to do it. Its mappings answer what the shared cases need, and the unit tests cover every other answer.
• The cases read WireMock's journal. They check what the job sent, and with which key. That is the job's half of idempotency.
• Failures come from the scenario, not from WireMock. The cases share one WireMock, so changing its stubs for one case would break the others. A scenario also rehearses the same failure in staging and production, against whatever API the job calls.
• No compose file. The kind overlay already runs the job, PostgreSQL, WireMock and the LGTM stack. A compose file would be a second way to run them, kept in step with the first.
• The LGTM stack. Loki, Tempo and Mimir each take OTLP, so the Collector forwards what the job sends without converting it. Mimir's ruler evaluates the walkthrough's own rule file.
• No comments in the code. A reader checks their code against the walkthrough's. Comments would be a second explanation to read past and keep true, and the pages are already the explanation.

Testing

This is added:

The prototype ran the stand-in that this revision drops. Its results stand. The unit tests now serve a handler of their own in its place.

Open questions

This section is replaced.

None. The three above are answered.

Out of scope

These are added:

• A compose file. The kind overlay runs everything.
• A dashboard, as the charter's revision 13 proposes. The pages use Grafana's Explore.

References

These are added:

• WireMock: running standalone, and verifying requests
• Loki's OTLP ingestion, and Mimir's