Proposal: API walkthrough

[Zaba505]

Note

Part of the humus re-evaluation: wave 6 of Proposal: humus on the bedrock process model, which plans one proposal per walkthrough. This walkthrough keeps the points every walkthrough keeps, set by its revision 13, and doesn't ask them again: how it's packaged, where its deployed tests run, where its code lives and how its alerts are written are that revision's to decide.

It builds on proposals that have all landed: OpenAPI endpoint DSL, whose shoppingcart it turns into the walkthrough, authentication and authorization, gRPC services, outbound gRPC clients, outbound HTTP clients, exposing health probes, failure injection and humus's own metrics. It depends on no other walkthrough.

Summary

A walkthrough is a production-ready application, kept in this repository, with pages that walk a reader through how it was built and why. This one is an API: the shoppingcart, made production-ready, deployed, and explained.

#842 and #886 built the shoppingcart as humus's reference service: six typed operations on openapi/v3, on PostgreSQL, behind JWT bearer tokens. #760 handed it to the walkthroughs. What it lacks is everything between a service that passes its tests and one that someone runs:

• A client sets its own price. addCartItem stores the unitPrice its body carries.
• Any token can read or change any cart. Carts have no owner.
• Nothing applies its schema in production. schema.sql says so.
• No probes, no failure injection, no deployment, and telemetry that exports nothing.

This proposal fixes exactly those, and takes three things #762, #763 and #870 handed to the walkthroughs: a service calling another end to end, a service that calls gRPC, and authentication in a production-ready service.

• Prices come from a catalog, over gRPC. The walkthrough adds a second, small humus service of its own, the catalog, served with grpcserver. The cart asks it for a product's price with grpcclient, over mTLS, and keeps the price in the cart. The catalog's writes need a token with catalog:write, through grpcauth.
• A cart is its owner's. A cart belongs to the subject of the token that created it. Any other subject gets a 404.
• Migrations run before the new code serves. Each service's schema is a set of versioned migrations, applied by an init container under an advisory lock, as a role that can change the schema. The services connect as roles that can only read and write.
• It is production-ready by the bar revision 13 sets. Every setting is sourced and checked, and every secret and certificate is a file, re-read as cert-manager renews it. Two replicas of each service run behind a PodDisruptionBudget, ready only once they've reached their dependencies. They drain on SIGTERM and send traces, logs and metrics over OTLP.
• Failure injection crosses the two services. A trusted request to the cart can fail the cart's call to the catalog, or, forwarded, the catalog's own query to PostgreSQL. The cases show the cart answering with a 503 that says why, and recovering.
• It is tested at three levels, with the cases written once. The same black-box cases run against both services in-process inside synctest, against PostgreSQL in containers, and against the deployed image in kind.

client ──HTTPS + bearer──▶ cart (httpserver, openapi/v3) ──gRPC, mTLS──▶ catalog (grpcserver)
                               │                                           │
                               ▼                                           ▼
                          PostgreSQL: carts                         PostgreSQL: catalog

Background

• The shoppingcart is the reference service, not yet a walkthrough. It shows the pieces of openapi/v3 and auth together, with whole-App tests and a container test (app/app.go). It has no deployment, no failure injection and no pages of its own. OpenAPI endpoint DSL said the walkthroughs would build on it.

• Its gaps are real ones:

  • AddItemRequest takes unitPrice from the client.
  • Store.GetCart takes only a cart ID, so a token with carts:write can delete anyone's cart.
  • schema.sql: "The service applies no migrations".
  • BuildTelemetry keeps telemetry.DefaultSettings(), so it exports no traces or metrics.
  • DATABASE_URL and TLS_PKCS12_PASSWORD carry secrets in the environment.

• v0.15.0 had an orders REST walkthrough, which the reset deleted (its index, example/rest/orders-walkthrough). It orchestrated three backend services that Wiremock mocked, and ran an observability stack with podman-compose. It was never deployed, and its backends were never real.

• Waves 4 and 5 handed this walkthrough four things:

  • a service calling another end to end, from Proposal: outbound HTTP clients #762;
  • a service that calls gRPC, from Proposal: outbound gRPC clients #763;
  • authentication in a production-ready service, from Proposal: authentication and authorization #870;
  • failure injection shown in a walkthrough, from Proposal: failure injection #869.

  Probes, from Proposal: exposing health probes #761, and alert rules on humus's own instruments, from Proposal: humus's own metrics #871, go to every walkthrough.

• Failure injection across services needs a service of the end user's on the other side. ForwardScenario sends a scenario on only to a dependency that honors it with the same codec (Failure injection). A dependency that has points of its own is the only way to show a scenario reaching them.

Proposed API

A walkthrough adds no API to humus. What it adds is an application, its manifests, its tests and its pages.

The application

It stays at example/shoppingcart, in the example module. The module gains github.com/z5labs/humus/grpc, replaced with ../grpc as the core module is, and github.com/jackc/tern/v2.

example/shoppingcart/
├── main.go                 the cart: httpserver.Main(app.Init, WithTelemetry, WithFaults)
├── app/                    the cart's Init, settings, Faults, trust and readiness
├── endpoint/               the API's operations, on a Store and a Catalog
├── database/               the Store on PostgreSQL, its injection point, migrations/
├── catalog/
│   ├── main.go             the catalog: grpcserver.Main(app.Init, WithTelemetry, WithFaults)
│   ├── app/                its Init, settings, Faults, grpcauth rules and probes
│   ├── server/             the catalog.v1.Catalog service
│   ├── database/           its Store on PostgreSQL, its injection point, migrations/
│   ├── proto/              catalog/v1/catalog.proto, and buf's configuration
│   └── catalogpb/          the generated code, checked in
├── migrate/                the migrations' init container: humus.Main and a job
├── internal/certfile/      certificates read from files, and re-read when they're renewed
├── cases/                  the shared test cases, and their Target
├── functional/             the cases against a deployment (build tag functional)
├── deploy/                 the kustomization, its kind overlay, and alerts.yaml
└── Dockerfile              one image, with the three programs

The cart's API

The six operations stay, with their paths, codecs and answers. Three things change.

Prices come from the catalog. AddItemRequest is productId and quantity. addCartItem asks the catalog for the product with GetProduct, and stores its price with the item, so a later change of price doesn't change a cart. A product the catalog doesn't have is a 422 Invalid.

A cart is its owner's. Claims gains Subject, the token's sub. Every Store method takes the owner, and a cart whose owner isn't the caller is ErrCartNotFound: a 404, so the API never says whose carts exist.

A dependency that is down is a 503, never a 500. A new answer, Unavailable, is a problem detail at 503 that says to try again, given with openapi.Error:

• for the catalog: Unavailable, DeadlineExceeded, or a call its breaker rejected;
• for PostgreSQL: an error the Store wraps as database.ErrUnavailable, which it does for a connection that failed, and for its injection point's abort.

The catch-all still answers anything else with a 500, and records it.

The API's handler is wrapped in http.MaxBytesHandler at 64 KiB, so a body over it is the API's 413. A cart's bodies are a few dozen bytes.

The catalog

A gRPC service, catalog.v1.Catalog, with two methods:

service Catalog {
  // GetProduct returns a product and its unit price, or NOT_FOUND.
  rpc GetProduct(GetProductRequest) returns (Product);
  // PutProduct creates or replaces a product. It needs catalog:write.
  rpc PutProduct(PutProductRequest) returns (Product);
}

• It is served by grpcserver, over TLS, with ClientAuth: tls.VerifyClientCertIfGiven and the internal CA as ClientCAs.
• grpcauth decides who may call what:
  • GetProduct requires auth.MutualTLS: a caller with a certificate the internal CA issued, which is the cart.
  • PutProduct requires a bearer token with catalog:write, verified by auth/jwt against the same identity provider as the cart.
• Its probes are grpchealth.NewService, on a plaintext port of its own. The Service takes client certificates, and the kubelet's grpc probes use plain text on most clusters (Health).
• Its products are a table in a database of its own. Each service owns its data, and neither reads the other's.
• Its own injection point is postgresql:catalog, evaluated before each query, as the docs suggest naming one. An abort fails the query as an unreachable database does, and GetProduct ends Unavailable.

The generated code is checked in, made by buf generate from catalog/proto, whose buf.gen.yaml pins protoc-gen-go and protoc-gen-go-grpc. The walkthrough's container job generates it again, and fails on a diff.

Calling the catalog

The cart builds one grpcclient.Client for the catalog, from CATALOG_ADDR and CATALOG_CA_FILE, with its client certificate for mTLS. It keeps DefaultSettings()' retries, budget, breaker and Classify:

• GetProduct is a read, so retrying it on Unavailable is safe;
• NotFound is the catalog working, so it counts as a success to the breaker.

It sets ForwardScenario, because the catalog is the walkthrough's own and reads the same codec.

Migrations

Each service's schema is a directory of versioned SQL migrations, embedded in its database package. They replace schema.sql.

• migrate is a program of its own: a humus job.
  • It opens one pgx.Conn through the Host.
  • It runs tern's Migrator on the migrations its MIGRATIONS setting names, carts or catalog.
  • It grants the service's role, which its APP_ROLE setting names, read and write on the schema's tables, and nothing more.
  • It connects as a role that owns the schema, from a file of its own.
• It runs as each Deployment's init container, so a pod's service starts only once its schema is current.
• Two pods that start at once both run it. tern takes an advisory lock, so one migrates and the other waits, then finds nothing to do. A rollout with no new migration skips the lock.
• A migration is backwards-compatible with the code it replaces, expand then contract, because the old pods keep serving during a rollout. The pages show how a change that renames a column is split across two releases.

The settings

Every setting is sourced with config.Struct and checked, and a bad one exits 78, naming its variable. Every secret is a file, read through s.Host.FS:

• The database's URL is a file, DATABASE_URL_FILE. The URL still sets every field pgx uses, so pgx finds nothing in the real environment or ~/.pgpass, as the shoppingcart's package doc explains today. The deployment connects with sslmode=verify-full.
• Certificates are PEM files, TLS_CERT_FILE and TLS_KEY_FILE, which is what cert-manager writes to a Secret. They replace TLS_PKCS12_FILE and its password. internal/certfile serves them through tls.Config.GetCertificate, and the cart's client certificate through GetClientCertificate. It reads them again when their files change, and checks at most every 30s. So a renewed certificate is served without a restart. httpserver.CheckTLS and grpcserver.CheckTLS already accept a config with GetCertificate alone.
• Telemetry is sourced too. Each service's Builder reads the specification's OTEL_* variables that telemetry.Settings names, such as OTEL_TRACES_EXPORTER and OTEL_EXPORTER_OTLP_ENDPOINT, as Telemetry suggests. Its resource takes the pod's name and namespace from the downward API.
• FAULT_TRUST_SCOPE names the scope a token needs for the cart to trust its scenario. It is empty by default, so production trusts no one.

Each app package's doc lists every variable its service reads, as today's does.

Readiness

The cart's probes are attached to its own port, with httphealth.NewHandler at /health/, beside the API. The kubelet probes it over HTTPS, and nothing wraps it, so it stays open while the API requires tokens. Its readiness is health.And of two monitors:

• PostgreSQL, reached once: a monitor of the walkthrough's own that pings the pool until a ping first succeeds, then stays healthy;
• the catalog, connected once: the catalog Client's Connected.

Both latch, as Connected does, for the reason Calling gRPC services gives:

• A release that can't reach a dependency never turns ready. A wrong address, a CA that doesn't trust the server, or a bad password stops the rollout at its first pod, while the old pods keep serving.
• An outage after that keeps every pod ready, so callers get the cart's own 503, and reads that don't need the catalog still work.

The catalog's readiness latches on its own PostgreSQL the same way.

Failure injection

Both services pass humus.WithFaults(Faults), with jsonscenario under the baggage member acme.fault, the Failure injection page's key, which every walkthrough uses.

	The cart	The catalog
Trusts a scenario from	a bearer token, verified with the cart's own Verifier, whose scopes hold FAULT_TRUST_SCOPE	a caller whose client certificate the internal CA issued: the cart
Its points	postgresql:carts, of its own; grpc:catalog.v1.Catalog/GetProduct, its grpcclient; http:<key set host>:<port>, its key set's httpclient	postgresql:catalog, of its own
Sends a scenario on	to the catalog, with ForwardScenario	nowhere: it calls nothing

• The cart's trust is a scope, not a certificate, because the cart is the edge: a request reaches it through an ingress, which a client certificate doesn't cross, and a token does. Its TrustScenario verifies the token itself, only for a request that carries a scenario. Open question 3.
• A scenario aimed at the catalog's database goes through the cart. The cart trusts it, forwards it, and the catalog trusts the cart. The catalog's query fails, GetProduct ends Unavailable, and the cart's client retries it, meets the same fault, and gives up. The cart answers 503. The trace shows every attempt, and the catalog's humus.fit.evaluation events.

Production-ready, as a checklist

The walkthrough's pages close with this list, and the application meets each item:

	How
Every setting sourced and checked; a bad one exits 78, naming its variable	config.Struct and the packages' checks, in all three programs
No secret in the environment, the logs or the spans	the database URL and the keys are files, through s.Host.FS; no handler logs a token
Certificates renewed without a restart	internal/certfile, on cert-manager's Secrets
Two replicas of each service	a PodDisruptionBudget of minAvailable: 1, and a rolling update of maxUnavailable: 0
Ready only once it has reached its dependencies, and still ready through their outages	the latching monitors
A rollout loses no request	httpserver.Main's and grpcserver.Main's 5s drain, and terminationGracePeriodSeconds above the drain, the grace period and the telemetry's shutdown
The schema is current before the code that needs it serves	the migrate init container, under tern's lock, with backwards-compatible migrations
Least privilege	the services' roles read and write their tables and change no schema; a non-root, read-only container; no service account token mounted
A caller sees only their own carts	the owner, as a 404 for anyone else
Prices can't be set by a client	the catalog, over mTLS
Observable	traces across both services, logs and metrics over OTLP, and deploy/alerts.yaml
Failures say why	a 503 for a dependency that is down, a 422 for an unknown product, and problem details throughout
Failures are rehearsed	the failure-injection cases, in CI and against staging
Safe by default	FAULT_TRUST_SCOPE empty; the OpenAPI document the only open path besides the probes

The tests

Level	Where it runs	What it covers
Units	go test, in the Build job	the operations on a fake Store and Catalog; the catalog's methods; internal/certfile on a humustest Host, rewriting its files; the cart's TrustScenario; each Store's injection point
The stores and the migrations	testcontainers tag	each Store against PostgreSQL in a container, with its schema applied by migrate's own Init; two migrate runs at once
The shared cases, inside synctest	go test, in the Build job	both services' whole Apps on one humustest Host, beside the tests' identity provider, with in-memory stores that keep the injection points
The shared cases, in-process	testcontainers tag, in a CI job that isn't required	both services on humus.OSHost(), against PostgreSQL in a container, migrated by migrate
The shared cases, deployed	functional tag, in a CI job that isn't required, and against an end user's staging	the image built from the Dockerfile, deployed with the kind overlay

The whole-App level runs twice. The cart's state is a Store the walkthrough already fakes in memory, so its cases can run inside synctest in the Build job too, in a few milliseconds, besides against PostgreSQL. Other whole-App tests stay in app: a bad setting exits 78, readiness waits for both dependencies, and the telemetry carries the evaluations and strips, read with humustest.Telemetry.

The shared cases are in cases. It imports neither the application nor humustest, so the compiler keeps them black-box. It does import catalogpb, the catalog's published contract, as any client of it would. Each case takes a Target:

• the cart's base URL, and an *http.Client that trusts it;
• a catalogpb.CatalogClient;
• a function that returns a token for a subject and scopes;
• the scenario's baggage key, the trust scope, and a timeout.

They poll with plain loops, which work in both harnesses. Each case makes its own subjects and products, named with crypto/rand, so the cases can run in parallel and against a shared staging environment. They mutate, so they never run against production. The cases:

• A cart's life. Create a cart, add an item, read it, change its quantity, remove it, delete the cart, and get a 404.
• Prices. An item's price is the catalog's. After PutProduct changes it, the item keeps its price and a new item gets the new one.
• An unknown product is a 422, and adds nothing.
• Ownership. Another subject's token gets a 404 for every operation on the cart.
• Scopes. A token without carts:write gets a 403 on a change. No token gets a 401, except for /openapi.json. PutProduct without catalog:write is PermissionDenied.
• Failure injection, each under a scenario from a token with the trust scope:
  • aborting grpc:catalog.v1.Catalog/GetProduct: adding an item is a 503, the cart is unchanged, and the next add without the scenario succeeds, polled until the breaker, if it opened, lets it through;
  • aborting postgresql:catalog, forwarded to the catalog: adding an item is a 503;
  • delaying grpc:catalog.v1.Catalog/GetProduct by 2s: the add still succeeds;
  • aborting postgresql:carts: reading the cart is a 503 that says to try again, and the next read succeeds;
  • the first scenario from a token without the trust scope is stripped, and the add succeeds.

The deployment

deploy/ is a kustomization:

• a Namespace, and a ServiceAccount for each service, with automountServiceAccountToken: false;
• for each service, a Deployment with two replicas, the migrate init container, probes, a startup probe, resources, a non-root, read-only security context, and the downward API;
• a Service for each, and a PodDisruptionBudget for each;
• the names of the Secrets each Deployment mounts, which an operator provides: certificates as kubernetes.io/tls Secrets, the database URLs, and the CA bundles;
• alerts.yaml, the Prometheus rules.

The kind overlay adds what a cluster would provide:

• cert-manager, with a self-signed issuer and an internal CA, issuing every certificate;
• PostgreSQL, with TLS, its two databases, and four roles whose passwords the kind job generates for each run;
• a static key set, served over TLS: the kind job generates a signing key for each run, publishes its public half as a JSON Web Key Set, and hands the private half to the cases, which sign their own tokens;
• an OpenTelemetry Collector that both services export to;
• FAULT_TRUST_SCOPE set to the cases' scope.

The kind job builds the image, loads it into the cluster, applies the overlay, waits for both rollouts, and runs the cases through NodePorts that the cluster's config maps to the runner.

The Dockerfile builds the three programs as static binaries, onto a distroless, non-root base. Each Deployment runs its program as the container's command.

Alerts

deploy/alerts.yaml holds the services' alert rules, which CI checks with promtool. The pages explain each one, and what to look at first:

Alert	Reads
The cart answers 5xx to over 1% of requests for 10 minutes	http.server.request.duration's count, by status
The cart's p99 latency is over 1s for 10 minutes	http.server.request.duration
The catalog fails over 1% of calls for 10 minutes	rpc.server.call.duration's count, by status code
A breaker is open: the catalog's, or the key set's	humus.circuit.state
A breaker opened in the last 5 minutes, however briefly	humus.circuit.transitions
The retry budget refused retries	humus.retry.decisions, throttled

The pages

A walkthrough's pages go in the Walkthroughs section that revision 13's story adds. This one's are docs/content/walkthroughs/shoppingcart/, in the order it was built:

1. What we're building, and why: the two services, and the choices against the alternatives, such as a price in the request.
2. The cart's API: the operations, the answers, and the document.
3. Who may do what: bearer tokens, scopes, and a cart's owner.
4. The catalog, over gRPC: the proto, grpcserver, mTLS and grpcauth, and its probes.
5. Calling the catalog: grpcclient, its retries and breaker, and turning its failures into answers.
6. The schema: migrations, the init container, the roles, and changing a schema under a rollout.
7. Wiring the Apps: Init, the settings, secrets and certificates as files, readiness, telemetry and Faults.
8. Deploying: the image, the kustomization, cert-manager, rollouts and the drain.
9. Observing: one trace across both services, the records, the instruments, and the alerts.
10. Testing: the levels, and the shared cases.
11. Breaking it on purpose: the scenarios, whom each service trusts, forwarding, and running the cases against staging.
12. Production-ready: the checklist.

Changes to other files

• docs/content/openapi.md: its section on the shoppingcart links to the walkthrough, and keeps only what is about openapi/v3. The samples it quotes follow the code.
• .github/workflows/build.yaml:
  • "Shoppingcart (testcontainers)" becomes "Shoppingcart walkthrough (testcontainers)", and generates catalogpb again, failing on a diff;
  • a new "Shoppingcart walkthrough (kind)" job;
  • neither is required, and the Build job vets the functional tag too.
• example/go.mod: the grpc module, replaced with ../grpc, and tern.
• The Walkthroughs section's index lists this walkthrough.

Design decisions

• The shoppingcart becomes the walkthrough, rather than a new API beside it. One reference API is easier to keep right than two, and Proposal: OpenAPI endpoint DSL #760 named it for the walkthroughs. Its pages explain the code that the openapi page already points to.
• The catalog is a real service, not a stand-in. It is small, but it is a humus gRPC service with its own data, probes, authentication and point, deployed beside the cart. A stand-in would be test-only code that the kind overlay deploys, and couldn't show a scenario reaching a dependency's own point. Open question 1.
• The price is the catalog's, not the client's. A cart that takes a price from its client lets anyone buy at their own price.
• Another subject's cart is a 404, not a 403. A 403 would confirm that the cart exists.
• Migrations run in an init container, under tern's lock, as their own role. The schema is current before the code that needs it starts, on every rollout, with no step outside kubectl apply. The services can't change their schemas. tern is pgx's author's migrator: it takes a pgx.Conn, which dials through the Host as the pool does, and reads its migrations from an fs.FS, so they're embedded. Open question 2.
• Readiness latches. It gates a rollout on reaching each dependency, and never takes every pod out of service during an outage.
• Secrets and certificates are files, and certificates are re-read. An environment variable is visible to every process in the container and to anyone who can read the pod spec. cert-manager renews a certificate before it expires, so a service that reads it once has an outage when it does.
• A dependency's outage is a 503. A 500 says the cart is broken. A 503 says to try again, which is the truth, and what an alert on 5xx should still count.
• The cart trusts a scope; the catalog trusts the cart's certificate. Each trusts what reaches it: a token crosses an ingress, and a client certificate is what the catalog already verifies. Open question 3.
• The kind overlay serves a static key set, not an identity provider. The cart fetches it over HTTPS, exactly as it fetches a provider's. The cases sign their own tokens with a key generated for each run, so no credential is stored, and no identity provider is configured for scopes.
• One image, three programs. The three ship together and are built from one module, so one build and one scan cover them. Each Deployment names its program.
• The shared cases also run inside synctest. The cart's in-memory Store already exists for its unit tests. Running the cases there too gives the Build job every case in milliseconds, with fake time for the breaker's wait.

Testing

The walkthrough is its own test: see The tests. Two of the claims it rests on were prototyped, in a module that replaces humus with this repository at 1bb3bb5:

• Failure injection across the two services, inside synctest, on one humustest Host, under go test -race -count=3. An httpserver cart called a grpcserver catalog with grpcclient over mTLS, with grpc-go's interop service standing in for catalogpb.
  • A scenario from a token whose scopes held the trust scope, verified by a jwt.Verifier inside TrustScenario, reached the catalog. The catalog's own postgresql:catalog point aborted each of the 3 attempts that DefaultSettings().Classify made, and the cart answered 503.
  • The next request without the scenario succeeded: 3 failures in a row leave the default breaker closed.
  • A token without the scope, and no token, were stripped at the cart. With ForwardScenario off, the scenario was stripped at the client. Aborting the cart's own grpc:… point sent the catalog nothing.
  • Readiness of health.And with Connected, attached with httphealth.NewHandler to the cart's port, was 503 until the catalog listened, then 200.
  • One finding: a probe with no deadline, while the dependency is down, waits as long as the dependency does, because Connected's check waits for the connection. In synctest that is a test that never ends, as grpc-go's reconnect backoff keeps fake time moving. The kubelet bounds every probe with timeoutSeconds, and the tests bound theirs the same way.
• Two migrators at once, tern v2.4.3 against PostgreSQL 18 in a container, each on a pgx.Conn that dialed through a DialFunc of the test's. Both succeeded, each migration ran once, and a third run did nothing, under -race -count=3.

Migration from today

The shoppingcart changes, and has no clients outside this repository:

• addCartItem's body loses unitPrice; the golden document changes with it.
• Carts gain an owner, so the same token is needed to read a cart again.
• TLS_PKCS12_FILE and TLS_PKCS12_PASSWORD become TLS_CERT_FILE and TLS_KEY_FILE, and DATABASE_URL becomes DATABASE_URL_FILE. It gains the catalog's, the telemetry's and the faults' settings.
• schema.sql becomes the cart's first migration.
• Its container test becomes the shared cases.

Open questions

1. The gRPC dependency.

   • (a) A catalog service of the walkthrough's own, deployed beside the cart, as proposed.
   • (b) A stand-in catalog, a gRPC server that only the tests and the kind overlay run. It is smaller, but it's test-only code deployed as though it were a service, and a scenario can't reach a point of its own.
   • (c) No gRPC: the catalog over HTTP, with httpclient. That leaves Proposal: outbound gRPC clients #763's handoff to no walkthrough.

   Recommendation: (a).

2. How migrations are applied.

   • (a) An init container in each Deployment, as proposed.
   • (b) A Job applied before the Deployment, which needs a second kubectl apply, or a tool that orders them, such as Argo CD's sync waves.
   • (c) In each service's Init, under the same lock. It needs no second program, but the service's role must be able to change its schema.

   Recommendation: (a).

3. Whom the cart trusts with a scenario.

   • (a) A token whose scopes hold FAULT_TRUST_SCOPE, as proposed.
   • (b) A client certificate from a CA that a setting names. It needs no token, but doesn't cross an ingress that terminates TLS, so it works only inside the cluster.
   • (c) Either.

   Recommendation: (a).

Out of scope

• Checkout, orders, payments, stock, and carts for anonymous users.
• Idempotency keys on createCart and addCartItem.
• Rate limiting, which is an ingress's or a gateway's.
• Provisioning PostgreSQL: the server, its databases, its roles, its backups and its failover are the platform's. The kind overlay creates them only for the tests.
• Network policies, which depend on the cluster's network plugin.
• A Horizontal Pod Autoscaler.
• An identity provider in the overlay. A real one is the platform's.
• A second version of the schema in the walkthrough's code. The pages describe expand then contract instead.
• Credentials on the cart's call to the catalog beyond its client certificate, which Proposal: authentication and authorization #870 left to service code and opts.
• Publishing the image, or a release of the walkthrough.

References

• Proposal: humus on the bedrock process model and its revision 13
• Proposal: OpenAPI endpoint DSL, and the shoppingcart's stories story(example): rebuild the shoppingcart as the reference service on openapi/v3 and Postgres #842 and story(example): require bearer tokens in the shoppingcart with auth/jwt #886
• Proposal: authentication and authorization, Proposal: gRPC services, Proposal: outbound gRPC clients, Proposal: outbound HTTP clients, Proposal: exposing health probes, Proposal: failure injection and Proposal: humus's own metrics
• Proposal: Kubernetes controller walkthrough, the first walkthrough proposed
• v0.15.0's orders REST walkthrough
• tern, buf, cert-manager and kind

[Zaba505]

Revision 1: the charter's revisions 14 and 15, and what the job and mailer walkthroughs found

The charter's revisions 14 and 15 change this walkthrough as their tables say: the module revision 15's story made, up.sh and the LGTM stack in the kind overlay, Grafana alert rules and a dashboard in place of Prometheus rules and promtool, no comments, and code-first pages. The post's sections on these would contradict the charter as they stand, so this revision rewrites them.

The job and RabbitMQ processor walkthroughs have since landed, and their code found more that holds here:

• The cases reach kind through kubectl port-forward. up.sh creates its cluster with no config, so no NodePort is mapped to the runner.
• up.sh issues every certificate, as the charter's revision 14 says, from a CA it generates. The kind overlay runs no cert-manager.
• The Collector says which pod sent what. humus's telemetry keeps OTEL_RESOURCE_ATTRIBUTES out of its resource (#1085). The mailer's two replicas are told apart by the Collector's k8s_attributes processor, whose attributes Mimir promotes to labels.
• deploy/ is a base and an overlay, deploy/base/ and deploy/kind/, as the mailer's is. kustomize refuses a base that contains its overlay (#1080).
• The cases that fail a shared breaker run alone. The mailer splits its cases into Parallel and Alone, because a case that opens the provider's breaker fails every other case's call on that replica (#1078).
• The kind check makes the series its rules read. A counter such as humus.circuit.transitions has no series until it is first recorded. The billing and mailer checks each run a case that opens a breaker before they look in Mimir.
• An App that is down can't alert on itself. The mailer alerts on its restarts from the Collector's Kubernetes cluster receiver.
• No walkthrough writes a server for a dependency. WireMock serves the key set.

The Kafka processor walkthrough's revisions 1 and 2, and the RabbitMQ processor walkthrough's revision 1, made two choices that hold here too:

• Operating it is out of scope. A walkthrough is a production-ready application you can run. Running one like it in production, such as splitting a schema change across releases, is for the end user who ships it.
• Trying it by hand. A small Go CLI signs a token, so a reader can call the cart on kind.

Open questions 1 to 3 are still open. This revision adds none.

Each section below replaces or adds to the part of the same name in the post. Everything else is unchanged.

Summary

In the bullet "It is production-ready by the bar revision 13 sets", its last sentence becomes: "They drain on SIGTERM, send traces, logs and metrics over OTLP, and ship Grafana alert rules and a dashboard."

The bullet on the three levels becomes:

• It is tested at three levels, with the cases written once. The same black-box cases run against both services in-process inside synctest, against PostgreSQL and WireMock in containers, and against the deployed image in kind.

This bullet is added:

• It runs on kind with one script, up.sh, in CI and for a reader, beside PostgreSQL, WireMock serving the key set, and the LGTM stack. carttoken signs a token to call the cart with.

The application

The tree becomes:

example/shoppingcart/
├── go.mod                  github.com/z5labs/humus/example/shoppingcart
├── main.go                 the cart: httpserver.Main(app.Init, WithTelemetry, WithFaults)
├── app/                    the cart's Init, settings, Faults, trust and readiness
├── endpoint/               the API's operations, on a Store and a Catalog
├── database/               the Store on PostgreSQL, its injection point, migrations/
├── catalog/
│   ├── main.go             the catalog: grpcserver.Main(app.Init, WithTelemetry, WithFaults)
│   ├── app/                its Init, settings, Faults, grpcauth rules and probes
│   ├── server/             the catalog.v1.Catalog service
│   ├── database/           its Store on PostgreSQL, its injection point, migrations/
│   ├── proto/              catalog/v1/catalog.proto, and buf's configuration
│   └── catalogpb/          the generated code, checked in
├── migrate/                the migrations' init container: humus.Main and a job
├── cmd/carttoken/          a CLI that makes the key set and signs tokens, to call the cart by hand
├── internal/certfile/      certificates read from files, and re-read when they're renewed
├── cases/                  the shared test cases, their Target, Parallel and Alone
├── functional/             the cases against kind, and the checks only kind can make (build tag functional)
├── deploy/
│   ├── base/               the Namespace, Deployments, Services, PodDisruptionBudgets and ServiceAccounts
│   ├── kind/               the kind overlay, up.sh, and WireMock's mapping for the key set
│   ├── alerts.yaml         Grafana's alert rules
│   └── dashboard.json      Grafana's dashboard
└── Dockerfile              one image, with the cart, the catalog and migrate

The module is github.com/z5labs/humus/example/shoppingcart, which the charter's revision 15 made. It already replaces the core module with ../... It gains github.com/z5labs/humus/grpc, replaced with ../../grpc, and github.com/jackc/tern/v2.

The code

This section is added.

• Every Go file, tests included, starts with the license header. Each package has a one-line package comment.
• There are no other comments. The names say what the code does, and the pages say why. The shoppingcart's comments go too, its package docs included.
• Build constraints and //go:embed are directives, and stay. So does the Code generated ... DO NOT EDIT. header that buf generate writes on catalogpb's files, which tools read to skip generated code.
• catalog.proto, the manifests, the mapping, the script and the Dockerfile have no comments either.

The cart's API

The list under "A dependency that is down is a 503, never a 500" ends:

Unavailable's detail says which: the catalog failed, or the cart's breaker shed the call without making it.

The catalog

catalog.proto loses its comments. The pages say that GetProduct answers NOT_FOUND, and that PutProduct needs catalog:write.

Migrations

The last bullet becomes:

• A migration is backwards-compatible with the code it replaces, because the old pods keep serving during a rollout. The pages say so in one sentence. How to split a change across releases, such as renaming a column, is the end user's, with operating it.

The settings

In the bullet "Telemetry is sourced too", "Its resource takes the pod's name and namespace from the downward API" becomes: "Its resource is service.name. The kind overlay's Collector adds the pod's attributes."

"Each app package's doc lists every variable its service reads, as today's does" becomes: "The pages list every variable each service reads. Each package has a one-line comment, as the charter's point 11 says."

Trying it by hand

This section is added.

cmd/carttoken is a small CLI with two commands:

• jwks prints the public half of a private key, as a JSON Web Key Set. Its kid is the key's thumbprint.
• sign signs a token with the same key and kid, for a subject, scopes, an issuer and an audience, and prints it.

It reads the private key from a PEM file. up.sh runs jwks to make the key set WireMock serves. A reader runs sign with the key from the Secret up.sh made, port-forwards to the cart, and calls it with curl, trusting the overlay's CA. It runs on the reader's machine, so the image doesn't carry it.

Production-ready, as a checklist

The checklist gains a third column, Where, linking each item to the page and section that shows it, as the job's and the mailer's do. These rows change:

	How
Certificates renewed without a restart	internal/certfile, on the kubernetes.io/tls Secrets that cert-manager renews. Its unit tests rewrite the files under a running server
Observable	traces across both services, logs and metrics over OTLP, the alerts in deploy/alerts.yaml, deploy/dashboard.json, and alerts on each Deployment from the Collector's Kubernetes cluster receiver

The tests

These rows change, and the last two are added:

Level	Where it runs	What it covers
Units	go test, in the Build job	as the post says, and carttoken's tokens verified by auth/jwt against its key set
The shared cases, in-process	testcontainers tag, in "Shoppingcart walkthrough (testcontainers)", which isn't required	both services on humus.OSHost(), against PostgreSQL in a container, migrated by migrate, and WireMock in a container, serving a key set the test makes with the kind overlay's mapping
The shared cases, deployed	functional tag, in "Shoppingcart walkthrough (kind)", which isn't required, and against an end user's staging	the image built from the Dockerfile, brought up by up.sh, and reached through kubectl port-forward
The telemetry as deployed	functional tag, kind only	one add's trace in Tempo holds spans of both services. Its records are in Loki, with its trace ID. Both services' metrics are in Mimir, and Grafana has all three as data sources
The alerts as deployed	functional tag, kind only	Grafana loaded every rule and the dashboard. After "The catalog's breaker opens", Mimir holds every series the rules read, and each rule has evaluated since without error

Parallel and Alone. The cases are two lists, as the mailer's are. Parallel runs at once. Alone then runs one case at a time, after the rest, in every harness. A case is alone when it fails calls to the catalog, since a replica's breaker and retry budget are shared by every request it serves. With DefaultSettings(), 5 failures in a row open the breaker for 10s, and two aborted adds at once make 6.

• Alone, in order: aborting grpc:catalog.v1.Catalog/GetProduct; aborting postgresql:catalog, forwarded; and the catalog's breaker opens. The last runs last because it leaves a breaker open.
• Parallel: every other case. Delaying GetProduct and aborting postgresql:carts fail nothing the breaker counts.

The abort cases also spend the retry budget, which a success refills by a tenth of a token. So a later case may see fewer attempts than MaxAttempts, and the cases check the answer, not the number of attempts.

Against a shared staging environment, the last Alone case opens the breaker of the replica it reaches. Every caller's adds there are shed until the breaker closes.

This case is added, to Alone:

• The catalog's breaker opens. Under a scenario that aborts grpc:catalog.v1.Catalog/GetProduct, the case adds an item until a 503's detail says the breaker shed the call. Adds without the scenario are then polled until one succeeds. It records the breaker's transitions and the retry budget's throttled decisions, which the alerts check needs.

The deployment

This section is replaced.

deploy/base/ is a kustomization:

• a Namespace, and a ServiceAccount for each service, with automountServiceAccountToken: false;
• for each service, a Deployment with two replicas, the migrate init container, probes, a startup probe, resources, and a non-root, read-only security context;
• a Service for each, and a PodDisruptionBudget for each;
• the names of the Secrets each Deployment mounts, which an operator provides: certificates as kubernetes.io/tls Secrets, which is how cert-manager writes them, the database URLs, and the CA bundles.

The kind overlay, deploy/kind/, adds to the base:

• PostgreSQL, with TLS, its two databases, and four roles with the passwords up.sh generates;
• WireMock, serving the key set over HTTPS, from the mapping deploy/kind/wiremock/jwks.json and the key set up.sh makes;
• the LGTM stack: an OpenTelemetry Collector that both services export to, which forwards OTLP to Loki, Tempo and Mimir, and Grafana, with all three as data sources and the alert rules and dashboard provisioned. Loki, Tempo and Mimir each run as one process, storing to the pod's disk;
• in the Collector, the k8s_attributes processor, which adds each pod's name, namespace and Deployment, and the Kubernetes cluster receiver, which reports the Deployments' replicas and the containers' restarts. A ClusterRole lets it read both. Mimir promotes the pod's attributes, and scope metadata, to labels;
• FAULT_TRUST_SCOPE set to the cases' scope.

deploy/kind/up.sh creates a kind cluster if there is none. Then it:

• generates a CA, and with it every certificate: the cart's, the cart's client certificate for the catalog, the catalog's, PostgreSQL's, and WireMock's keystore;
• generates the four roles' passwords, Grafana's admin password, and the signing key, whose key set it makes with carttoken jwks;
• loads the image into kind, applies the overlay with the Secrets it made, and waits for every rollout.

The kind job runs it, and so does a reader. No secret is kept in the repository.

The kind job builds the image, runs up.sh, and runs the shared cases and the kind-only checks through kubectl port-forward, as the billing and mailer jobs do.

The Dockerfile builds the cart, the catalog and migrate as static binaries, onto a distroless, non-root base. Each Deployment runs its program as the container's command.

Alerts

This section is replaced.

deploy/alerts.yaml is a Grafana alerting provisioning file, and deploy/dashboard.json a Grafana dashboard. The rules query Mimir, on the names the instruments have there. The kind overlay's Grafana provisions both. The pages explain each alert, and show the dashboard.

Alert	When	Reads
CartErrors	the cart answers 5xx to over 1% of requests for 10 minutes	http.server.request.duration's count, by status
CartSlow	the cart's p99 latency is over 1s for 10 minutes	http.server.request.duration
CatalogErrors	the catalog fails over 1% of calls for 10 minutes	rpc.server.call.duration's count, by status code
CartBreakerOpen	a breaker is open: the catalog's, or the key set's	humus.circuit.state
CartBreakerOpened	a breaker opened in the last 5 minutes, however briefly	humus.circuit.transitions
CartRetriesThrottled	the retry budget refused retries	humus.retry.decisions, throttled
CartUnavailable, CatalogUnavailable	the Deployment has had no available replica for 5 minutes	k8s.deployment.available, from the Collector's Kubernetes cluster receiver
CartCrashLooping, CatalogCrashLooping	a container restarted more than 3 times in 15 minutes	k8s.container.restarts, from the same receiver

• The breakers are told apart by their instrumentation scope. The catalog's client is grpcclient's and the key set's is httpclient's, and the cart has one of each. So CartBreakerOpen reads otel_scope_name, as the mailer's provider rule does.
• "In the last 5 minutes" counts a series that is new in the window. A replica's first sample of a counter already holds its count, which increase alone misses. So CartBreakerOpened and CartRetriesThrottled add x unless x offset 5m to increase(x[5m]), as the mailer's rules do (#1089).
• The last four rows read what watches the services, since an App that is down reports nothing.
• The dashboard shows the cart's requests by status and its latency, the catalog's calls by code, both breakers' state and transitions, the retry decisions, and each Deployment's available replicas and restarts.

The pages

This section is replaced.

They are under docs/content/walkthroughs/shoppingcart/. Each page is a list of steps. A step is a sentence or two, then the code it arrives at, quoted from the walkthrough's files. A reason is given only where the code would be wrong without it, and in one sentence, such as why a price comes from the catalog, or why another subject's cart is a 404. There is no history, no alternatives and no asides. The post's first page weighed choices against alternatives, which point 9 leaves out.

1. Overview: what the two services do, how a request crosses them, and the files it ends with.
2. The cart's API: the operations, the answers, and the document.
3. Who may do what: bearer tokens, scopes, and a cart's owner.
4. The schema: the migrations, migrate, the init container and the roles.
5. The catalog: the proto and buf generate, grpcserver, mTLS and grpcauth, its injection point and its probes.
6. Calling the catalog: grpcclient, its retries and breaker, ForwardScenario, and turning its failures into answers.
7. Init: each service's settings, the secrets and certificates as files, readiness, BuildTelemetry, Faults and TrustScenario, and main.
8. Testing: the unit tests, the shared cases with their Target, Parallel and Alone, and the failure injection cases.
9. Deploying: the Dockerfile, the kustomization, the disruption budgets and the drain, and the kind overlay. Running it on kind with up.sh, and calling the cart with a token from carttoken.
10. Observing: one trace across both services in Grafana, the logs and metrics, the alerts and the dashboard.
11. Production-ready: the checklist.

Changes to other files

This section is replaced.

• docs/content/openapi.md:
  • Its section on the shoppingcart links to the walkthrough, and keeps only what is about openapi/v3.
  • It no longer says that the package doc lists every variable or explains authentication, since the code has no such docs.
  • Its samples from app_test.go, such as the start helper, are quoted again from the code, which loses its comments.
  • Its section on pgx's ambient leaks names DATABASE_URL_FILE.
• docs/content/auth.md: its paragraph on the shoppingcart's tests names the shared cases in place of its container test, and links to the walkthrough's page on testing.
• .github/workflows/build.yaml:
  • "Shoppingcart (testcontainers)" becomes "Shoppingcart walkthrough (testcontainers)", and generates catalogpb again, failing on a diff;
  • a new "Shoppingcart walkthrough (kind)" job;
  • neither is required. The vet jobs already vet every example module with both tags.
• example/shoppingcart/go.mod: the grpc module, replaced with ../../grpc, and tern.
• The Walkthroughs section's index lists this walkthrough.

docs_test.go needs no change. Its rule for a walkthrough's samples already accepts protobuf, JSON and the walkthrough's own tests.

Design decisions

The bullet "The kind overlay serves a static key set, not an identity provider" becomes:

• WireMock serves a static key set, not an identity provider. The cart fetches it over HTTPS, exactly as it fetches a provider's. No walkthrough writes a server to stand in for a dependency, so WireMock serves it, from a mapping the walkthrough owns. The cases sign their own tokens with a key generated for each run, so no credential is stored, and no identity provider is configured for scopes.

These are added:

• up.sh issues every certificate. One CA it generates signs every certificate the run needs, and the run throws them away. cert-manager would make kind depend on another project's release for them. The base still mounts kubernetes.io/tls Secrets, which cert-manager writes, so a cluster that runs it issues them unchanged.
• The Collector says which pod sent what. humus's telemetry keeps OTEL_RESOURCE_ATTRIBUTES out of its resource, and a pod's name belongs to where the App runs, not to the App. The Collector's k8s_attributes processor adds it, so the two replicas' series are told apart.
• The cases that fail the catalog run alone. A replica's breaker and retry budget are shared by every request it serves, so one case's failures would fail another case's add.
• A CLI makes the key set and signs the tokens. A reader can try ownership and scopes as the cases do. up.sh makes the key set with the same CLI, so no script turns a key into a JSON Web Key by hand.
• Operating it is out of scope. A walkthrough is a production-ready application you can run. Running one like it in production is for the end user who ships it.

Testing

This is added:

The prototype ran before the charter's revisions 14 and 15. It didn't try WireMock, up.sh, the LGTM stack or carttoken. Its results stand.

Open questions

Questions 1 to 3 are still open.

Out of scope

"A second version of the schema in the walkthrough's code. The pages describe expand then contract instead." becomes:

• A second version of the schema in the walkthrough's code.

These are added:

• Operating it in production: runbooks, and splitting a schema change across releases. They're for an end user who ships a service like this one.
• cert-manager in the kind overlay. up.sh issues the certificates, and the base mounts the Secrets cert-manager would write.

References

These are added:

• The charter's revisions 14 and 15
• Proposal: job walkthrough and its revision 1; Proposal: RabbitMQ processor walkthrough and its revision 1; Proposal: Kafka processor walkthrough and its revisions 1 and 2
• What the job and mailer walkthroughs' stories found: #1078, #1080, #1085 and #1089
• Grafana: provisioning alert rules from files, and provisioning dashboards
• The OpenTelemetry Collector's Kubernetes cluster receiver and Kubernetes attributes processor
• WireMock: stubbing, including a response body from a file

[Zaba505]

Revision 2: the open questions answered, and a catalog over HTTP

The open questions:

1. No gRPC: the catalog over HTTP, with httpclient. A gRPC client is better shown in a gRPC walkthrough, which a later proposal plans. There, gRPC and protoc are already needed for the server. Here, they would be needed only for the catalog. So Proposal: outbound gRPC clients #763's handoff, a service that calls gRPC, goes to that walkthrough. This one drops grpcserver, grpcclient, grpcauth, grpchealth, the proto and buf.
2. An init container in each Deployment, as recommended.
3. A token whose scopes hold FAULT_TRUST_SCOPE, as recommended.

The catalog is still a service of the walkthrough's own, as option (c) had it, not WireMock. It is deployed from deploy/base/ beside the cart and ships with it, so it is half of the application, not a stand-in for a dependency. It has its own data, probes, authentication and injection point, and a scenario forwarded to it reaches that point. WireMock has no point to reach. The catalog is written as the cart is, with openapi/v3 on httpserver, so both services show the same way to write an API.

Each section below replaces or adds to the part of the same name in the post, as revision 1 left it. Everything else is unchanged.

Summary

"This proposal fixes exactly those, and takes three things #762, #763 and #870 handed to the walkthroughs" becomes: "This proposal fixes exactly those, and takes two things #762 and #870 handed to the walkthroughs: a service calling another end to end, and authentication in a production-ready service."

The bullet "Prices come from a catalog, over gRPC" becomes:

• Prices come from a catalog, over HTTP. The walkthrough adds a second, small humus service of its own, the catalog, an openapi/v3 API served with httpserver. The cart asks it for a product's price with httpclient, over mTLS, and keeps the price in the cart. The catalog's writes need a token with catalog:write.

The diagram becomes:

client ──HTTPS + bearer──▶ cart (httpserver, openapi/v3) ──HTTPS, mTLS──▶ catalog (httpserver, openapi/v3)
                               │                                           │
                               ▼                                           ▼
                          PostgreSQL: carts                         PostgreSQL: catalog

Background

The walkthrough takes three of the four things waves 4 and 5 handed it: a service calling another end to end, from #762; authentication in a production-ready service, from #870; and failure injection shown in a walkthrough, from #869. #763's, a service that calls gRPC, goes to a gRPC walkthrough.

The application

In the tree, catalog/ becomes the following, and this line is added under endpoint/:

├── catalogclient/          the cart's client of the catalog: the request, and what each answer means
├── catalog/
│   ├── main.go             the catalog: httpserver.Main(app.Init, WithTelemetry, WithFaults)
│   ├── app/                its Init, settings, Faults, trust and probes
│   ├── endpoint/           its operations, on a Store
│   └── database/           its Store on PostgreSQL, its injection point, migrations/

proto/ and catalogpb/ are dropped. Nothing is generated.

The module gains only github.com/jackc/tern/v2. It doesn't need the grpc module.

The code

The sentence on the Code generated ... DO NOT EDIT. header is dropped, since nothing is generated. "catalog.proto, the manifests, the mapping, the script and the Dockerfile have no comments either" becomes: "The manifests, the mapping, the script and the Dockerfile have no comments either."

The cart's API

"addCartItem asks the catalog for the product with GetProduct" becomes "addCartItem asks the catalog for the product with GET /products/{productId}". A 404 from the catalog is still the cart's 422 Invalid.

In the list under "A dependency that is down is a 503, never a 500", the catalog's item becomes:

• for the catalog: a 502, 503 or 504 once the client has made its last attempt, a transport error, a timeout, or a call its breaker rejected, which wraps circuit.ErrRejected;

The catalog

This section is replaced.

An openapi/v3 API, served with httpserver, with two operations:

Operation	Answers	Needs
getProduct, GET /products/{productId}	200 with the product and its unit price, 404 when there is none, or 503 Unavailable	a client certificate the internal CA issued: the cart
putProduct, PUT /products/{productId}	200 with the product, created or replaced	a bearer token with catalog:write

• It is served by httpserver, over TLS, with ClientAuth: tls.VerifyClientCertIfGiven and the internal CA as ClientCAs. Its API is wrapped in http.MaxBytesHandler at 64 KiB, as the cart's is.
• openapi.Security decides who may call what. getProduct requires auth.MutualTLS, and putProduct a bearer token with catalog:write, verified by auth/jwt against the same key set as the cart. Its document declares both, and is open at /openapi.json, as the cart's is.
• Its probes are httphealth.NewHandler, at /health/, beside its API, as the cart's are. The kubelet probes them over HTTPS with no certificate, which VerifyClientCertIfGiven serves. Nothing wraps them, so they stay open. The catalog needs no probes port.
• Its products are a table in a database of its own, as the post says.
• Its own injection point is postgresql:catalog, evaluated before each query. An abort fails the query as an unreachable database does, and getProduct answers 503 Unavailable.

Calling the catalog

This section is replaced.

The cart builds one httpclient.Client for the catalog, from CATALOG_URL and CATALOG_CA_FILE, with its client certificate for mTLS through GetClientCertificate. It keeps DefaultSettings()' retries, budget, breaker and Classify:

• a GET is idempotent, so the client retries it on a 503 or a transport error;
• a 404 is the catalog working, so the default doesn't retry it, and counts it as a success to the breaker.

It sets ForwardScenario, because the catalog is the walkthrough's own and reads the same codec.

catalogclient sends the request and turns each answer into what the cart's operations need: a price, ErrProductNotFound, or an error the operation answers as Unavailable. It decodes the catalog's JSON into types of its own, and imports nothing from catalog/, as a client of another team's service couldn't.

The settings

CATALOG_ADDR becomes CATALOG_URL. In the bullet "Certificates are PEM files", "httpserver.CheckTLS and grpcserver.CheckTLS already accept a config with GetCertificate alone" becomes "httpserver.CheckTLS already accepts a config with GetCertificate alone".

Readiness

The catalog's monitor becomes:

• the catalog, reached once: the same latch, around a GET /health/readiness sent through the catalog's httpclient Client, until one first answers 200.

"Both latch, as Connected does, for the reason Calling gRPC services gives" becomes "Both latch:", followed by the two bullets as they are.

This is added:

httpclient has no Connected, so the latch is the walkthrough's own for both dependencies. The call goes through the Client an add uses, so it proves what an add needs: the URL, a catalog certificate the CA signed, a client certificate the catalog trusts, and a catalog that has reached its own database.

Failure injection

In the table, the cart's points become: postgresql:carts, of its own; http:<catalog host>:<port>, its catalog httpclient, such as http:catalog:8443 on kind; http:<key set host>:<port>, its key set's httpclient. The catalog still trusts the cart's certificate. Its TrustScenario reads r.TLS.VerifiedChains, as Failure injection shows for httpserver.

In the bullet "A scenario aimed at the catalog's database goes through the cart", "The catalog's query fails, GetProduct ends Unavailable, and the cart's client retries it" becomes "The catalog's query fails, getProduct answers 503, and the cart's client retries it".

"Open question 3." is dropped from the first bullet.

Production-ready, as a checklist

In the row "A rollout loses no request", "httpserver.Main's and grpcserver.Main's 5s drain" becomes "httpserver.Main's 5s drain, in both services".

The tests

In the Units row, "the catalog's methods" becomes "the catalog's operations, on a fake Store; catalogclient against a handler of its own on humustest's Host, for each answer".

The Target becomes:

• the cart's base URL, and an *http.Client that trusts it;
• the catalog's base URL, and an *http.Client that trusts it, with no client certificate;
• a function that returns a token for a subject and scopes;
• CatalogPoint, the cart's point for the catalog, as the mailer's ProviderPoint names its provider's;
• the scenario's baggage key, the trust scope, and a timeout.

"It does import catalogpb, the catalog's published contract, as any client of it would" becomes: "It calls the catalog's API as any client of it would, with JSON types of its own."

These cases change:

• Prices. An item's price is the catalog's. After PUT /products/{productId} changes it, the item keeps its price and a new item gets the new one.
• Scopes. A token without carts:write gets a 403 on a change. No token gets a 401, except for /openapi.json. putProduct without catalog:write is a 403. getProduct without a client certificate is a 401, so only the cart reads the catalog.

In the failure injection cases, and in revision 1's Alone list and its case "The catalog's breaker opens", grpc:catalog.v1.Catalog/GetProduct becomes CatalogPoint. Delaying it by 2s still succeeds, inside the client's 5s AttemptTimeout. Revision 1's arithmetic stands: httpclient's default breaker also opens after 5 failures in a row, for 10s, and its MaxAttempts is also 3.

The deployment

The catalog's Deployment probes its own port over HTTPS, as the cart's does.

Alerts

CatalogErrors becomes:

Alert	When	Reads
CatalogErrors	the catalog answers 5xx to over 1% of requests for 10 minutes	http.server.request.duration's count, by status

• The services are told apart by job, which Mimir takes from each one's service.name. CartErrors, CartSlow and CatalogErrors read the same instrument, each filtered to its service's job, as the mailer's rules read job="mailer".
• Both breakers are httpclient's, so "The breakers are told apart by their instrumentation scope" becomes: the breakers are told apart by server_address, the catalog's host or the key set's, which every data point of httpclient's instruments carries. CartBreakerOpen, CartBreakerOpened and CartRetriesThrottled keep it as a label, so an alert says which dependency.

The dashboard shows the catalog's requests by status, in place of its calls by code.

The pages

Pages 5 and 6 become:

5. The catalog: its operations on openapi/v3, mTLS and openapi.Security, its injection point and its probes.
6. Calling the catalog: httpclient, its retries and breaker, ForwardScenario, and turning the catalog's answers into the cart's.

Changes to other files

• .github/workflows/build.yaml: "and generates catalogpb again, failing on a diff" is dropped from "Shoppingcart walkthrough (testcontainers)".
• example/shoppingcart/go.mod: tern, and no grpc module.

docs_test.go still needs no change.

Design decisions

The bullet "The catalog is a real service, not a stand-in" becomes:

• The catalog is a real service, over HTTP. It is small, but it is a humus service with its own data, probes, authentication and point, deployed beside the cart. WireMock stands in for an API a walkthrough invents and only calls, and has no point for a forwarded scenario to reach. gRPC is left to a gRPC walkthrough, where gRPC and protoc are already needed for the server, rather than needed here only for the catalog.

"Open question 2." and "Open question 3." are dropped from their bullets.

This is added:

• The cart's readiness calls the catalog's. httpclient has no Connected, so the latch is the walkthrough's own. The call goes through the Client an add uses, so it proves the URL, both certificates, and that the catalog has reached its database.

Testing

This is added:

The prototype's cart called a gRPC catalog. Its readiness results were Connected's, and its finding about a probe with no deadline was grpcclient's, so neither carries over. What it found of the cart's trust stands: a scenario from a token whose scopes held the trust scope was forwarded, and one from a token without it, or from no token, was stripped at the cart.

The HTTP path the scenario now takes wasn't prototyped. Each half of it is what Failure injection's own test runs (Through a humustest Host): an httpserver that trusts a caller's client certificate, and an httpclient that forwards a scenario to an httpserver on the same Host. The latch around an HTTP call wasn't prototyped either. The migrations' prototype stands.

Open questions

This section is replaced.

None. The three above are answered.

Out of scope

This is added:

• gRPC, a server or a client. A gRPC walkthrough, which a later proposal plans, shows both.

"Credentials on the cart's call to the catalog beyond its client certificate, which #870 left to service code and opts" becomes "Credentials on the cart's call to the catalog beyond its client certificate, which #870 left to service code".

References

Proposal: gRPC services and Proposal: outbound gRPC clients are dropped, here and from what the walkthrough builds on. buf is dropped from the tools.

This is added:

• Failure injection: mTLS, and Calling HTTP services: how an attempt's result counts