Proposal: gRPC walkthrough

[Zaba505]

Note

Part of the humus re-evaluation: wave 6 of Proposal: humus on the bedrock process model. Its revision 17 moved #763's handoff, a service that calls gRPC, from the API walkthrough to a gRPC walkthrough that a later proposal would plan. This is that proposal.

It keeps the points every walkthrough keeps, as the charter's revisions 13, 14 and 15 set them, so it doesn't ask them again:

• no API of its own, and standing alone;
• a stated production-ready bar;
• three levels of tests sharing one set of cases, with failure injection in them;
• an image, a kustomization with a kind overlay that up.sh brings up, and the LGTM stack;
• Grafana alert rules and a dashboard;
• CI jobs that aren't required;
• no comments, code-first pages, and a module of its own.

It also keeps what the job, RabbitMQ processor and Kafka processor walkthroughs found as they landed, which the API walkthrough's revision 1 lists.

It builds on proposals that are all done: gRPC services, outbound gRPC clients, authentication and authorization, exposing health probes, failure injection and humus's own metrics, with humus.Main and wave 1. It depends on no other walkthrough. One of its cases waits for the fix of a bug its prototype found in grpcclient, #1176.

Summary

A walkthrough is a production-ready application, kept in this repository, with pages that build it step by step. This one is gRPC's: a wallet and the ledger behind it, two gRPC services, one calling the other.

• The ledger keeps the money. ledger.v1.Ledger holds accounts, and the transfers between them, in PostgreSQL.
  • It moves money only by transfer, and each transfer is idempotent by its ID.
  • No account goes below zero.
  • Only the walkthrough's own services call it, over mTLS.
  • A watch streams an account's balance as it changes.
• The wallet is what people's apps call. wallet.v1.Wallet lets the holder of a bearer token open their account, read its balance, send money from it, and watch it. The wallet keeps no data: it asks the ledger for everything, with grpcclient.
• It is what gRPC is for:
  • a contract in protobuf, linted and checked for breaking changes with buf, and generated with buf generate, the generated code checked in;
  • unary RPCs and a server stream. A watch outlives a rollout of either service: the wallet opens the ledger's watch again on another pod, and an app watches the wallet again;
  • a deadline that crosses both services, and statuses whose details say why;
  • retries that are safe because every transfer carries its ID;
  • mTLS between the services, grpcauth on every method, gRPC health probes, and MaxConnectionAge to spread the wallet's calls across the ledger's pods.
• Failure injection crosses the two services. A trusted call to the wallet can fail the wallet's call to the ledger, or, forwarded, the ledger's own transaction. The cases show the wallet answering Unavailable with why, and recovering.
• It is tested at three levels, with the cases written once:
  • both Apps on one humustest Host inside synctest, the ledger on an in-memory store;
  • both Apps against PostgreSQL in a container;
  • the deployed image in kind.
• It runs on kind with one script, up.sh, in CI and for a reader, beside PostgreSQL and the LGTM stack. walletctl signs a token and calls the wallet, to try it by hand.

app ──gRPC, TLS + bearer──▶ wallet ──gRPC, mTLS, grpcclient──▶ ledger ──▶ PostgreSQL
                            (grpcserver)                       (grpcserver)

$ walletctl watch --subject alice
balance 2500
balance 1500    ← alice sent 1000 to bob
balance 1500    ← the ledger rolled out: the wallet opened its watch again on another pod

Background

• The API walkthrough's catalog was first a gRPC service. #948 planned the cart's catalog on grpcserver, called with grpcclient, over mTLS. Its revision 2 moved the catalog to HTTP, since gRPC and protoc would have been needed there for the catalog alone. The charter's revision 17 sent Proposal: outbound gRPC clients #763's handoff here, where the server needs them anyway. Much of what Proposal: API walkthrough #948 first designed for the catalog holds for the ledger: mTLS, grpcauth, a gRPC probes port, Connected, ForwardScenario, and buf generate with checked-in code.
• The feature pages leave the assembly to the reader.
  • gRPC services shows one server at a time: its TLS, its options, streams at shutdown, probes and grpcauth.
  • Calling gRPC services shows the client: Classify, the breaker, streams and Connected.
  • Both stand grpc-go's interop/grpc_testing service in for generated code. No page has a .proto of its own, buf generate, or two services that call each other.
• No release had a gRPC walkthrough. v0.15.0's walkthroughs were an orders REST API and a One Billion Row Challenge job. Its grpc.NewApi couldn't serve after #485.
• A retried call is safe only if a repeat does nothing. grpcclient's default retries every method on Unavailable. grpc-go also returns Unavailable when the connection breaks after the request was sent, and the server may have acted on it (Calling gRPC services). So a method that moves money must be idempotent. A transfer's ID does that, as the billing walkthrough's idempotency key does over HTTP.
• A service that relays a stream reconnects it itself. grpcserver ends every stream at shutdown with Unavailable, so its client reconnects to another pod (Streams at shutdown). grpcclient retries opening a stream, never a stream that opened (Streams). So the wallet opens the ledger's watch again itself.
• The prototype found a bug. With five or more watches open to one ledger pod, that pod's shutdown opened the wallet's breaker, because grpcclient counts each stream that grpcserver ends as a failure. Every call to the ledger was then rejected for 10s, though the other pod was healthy. MaxConnectionAgeGrace did the same. It is #1176. The walkthrough doesn't work around it: its case for a rollout of the ledger waits for the fix.

Proposed API

A walkthrough adds no API to humus. What it adds is an application, its manifests, its tests and its pages.

The application

example/wallet/
├── go.mod                 github.com/z5labs/humus/example/wallet
├── main.go                the wallet: grpcserver.Main(app.Init, WithTelemetry, WithFaults)
├── app/                   the wallet's Init, settings, key set, Faults, grpcauth rules, trust and probes
├── server/                the wallet.v1.Wallet service, on the ledger's client
├── ledger/
│   ├── main.go            the ledger: grpcserver.Main(app.Init, WithTelemetry, WithFaults)
│   ├── app/               its Init, settings, Faults, grpcauth rules, trust and probes
│   ├── server/            the ledger.v1.Ledger service, on a Store
│   └── store/             the Store on PostgreSQL, its injection point, and schema.sql
├── proto/                 wallet/v1/wallet.proto, ledger/v1/ledger.proto, buf.yaml and buf.gen.yaml
├── walletpb/              wallet.v1's generated code, checked in
├── ledgerpb/              ledger.v1's generated code, checked in
├── cmd/walletctl/         a CLI that makes the key set, signs tokens and calls the wallet, to try it by hand
├── internal/certfile/     certificates read from files, and read again when they're renewed
├── cases/                 the shared test cases, their Target, Parallel and Alone
├── functional/            the cases against kind, and the checks only kind can make (build tag functional)
├── deploy/
│   ├── base/              the Namespace, Deployments, Services, PodDisruptionBudgets and ServiceAccounts
│   ├── kind/              the kind overlay and up.sh
│   ├── alerts.yaml        Grafana's alert rules
│   └── dashboard.json     Grafana's dashboard
├── Dockerfile             one image, with the wallet and the ledger
└── Dockerfile.dockerignore

• The module replaces github.com/z5labs/humus with ../.. and github.com/z5labs/humus/grpc with ../../grpc. It isn't released.
• Beyond them, it needs pgx, grpc-go, protobuf, genproto/googleapis/rpc for the error details, and PostgreSQL's testcontainers module.
• internal/certfile is the walkthrough's own copy, as the mailer's and the order status processor's are. A walkthrough imports no other walkthrough's code.

The code

• Every Go file, tests included, starts with the license header. Each package has a one-line package comment. There are no other comments.
• Build constraints and //go:embed are directives, and stay. So does the Code generated ... DO NOT EDIT. header that buf generate writes, which tools read to skip generated code.
• The .proto files, the manifests, the script and the Dockerfile have no comments either. The pages say what each method answers.

The contract

Two packages in proto/, generated into walletpb and ledgerpb:

service Ledger {
  rpc OpenAccount(OpenAccountRequest) returns (OpenAccountResponse);
  rpc GetAccount(GetAccountRequest) returns (GetAccountResponse);
  rpc Deposit(DepositRequest) returns (DepositResponse);
  rpc Transfer(TransferRequest) returns (TransferResponse);
  rpc WatchAccount(WatchAccountRequest) returns (stream WatchAccountResponse);
}

service Wallet {
  rpc OpenAccount(OpenAccountRequest) returns (OpenAccountResponse);
  rpc GetAccount(GetAccountRequest) returns (GetAccountResponse);
  rpc Send(SendRequest) returns (SendResponse);
  rpc WatchAccount(WatchAccountRequest) returns (stream WatchAccountResponse);
}

• An Account is its ID, its balance, and when the balance last changed. A Transfer is its ID, the two accounts, the amount and when it was made. Amounts are whole cents, in one currency.
• buf lints with its STANDARD rules and checks breaking changes with FILE. buf.gen.yaml runs protoc-gen-go and protoc-gen-go-grpc as go tools, so their versions are pinned by the module's go.mod, where Renovate updates them.
• CI checks the contract in the walkthrough's container job: buf lint, buf breaking against main, and buf generate, failing on a diff.

The ledger

Method	Answers
OpenAccount	the account, opened with a zero balance, or as it is if it was already open
GetAccount	the account, or NotFound
Deposit	the deposit, as a transfer from the ledger's own external account. NotFound for no such account
Transfer	the transfer. FailedPrecondition when the source's balance is short, NotFound for either account, InvalidArgument for an amount that isn't positive or an account sent to itself, and AlreadyExists for a transfer's ID reused for another transfer
WatchAccount	the account at once, then again each time its balance changes. NotFound for no such account

• Every transfer is one transaction.

  1. It inserts the transfer, ON CONFLICT (id) DO NOTHING. A transfer already there is answered as it was made, or with AlreadyExists if it differs.
  2. It locks both accounts with SELECT … FOR UPDATE, in the order of their IDs, so two transfers between the same accounts can't deadlock.
  3. It checks the source's balance, and updates both.

  A transfer refused for its balance isn't kept, so its ID can be sent again once the money is there.

• No account but external goes below zero. The transaction checks it, and a CHECK on the table refuses it too. external's balance is the negative of every deposit, so the ledger's balances always sum to zero.

• Details say why. FailedPrecondition carries a google.rpc.PreconditionFailure naming the account, and InvalidArgument a google.rpc.BadRequest naming the field.

• A watch polls. It reads the account every WATCH_INTERVAL, 1s by default, and sends it when its balance has changed, so a watch sends the balance it settles on rather than each transfer. Open question 4.

• It is served by grpcserver, over TLS, with ClientAuth: tls.VerifyClientCertIfGiven and the internal CA as ClientCAs.

• grpcauth lets only the internal CA's certificates call it. grpcauth.Default(auth.Require(cert)), where cert is an auth.MutualTLS scheme whose principal is the certificate's subject. A call without one is Unauthenticated, which says why, rather than a failed handshake.

• Its own injection point is postgresql:ledger, evaluated before each transaction. An abort fails it as an unreachable database does, and the method answers Unavailable.

• Its probes are grpchealth.NewService, on a plaintext port of its own, since the kubelet presents no certificate and its grpc probes don't use TLS on most clusters. Its readiness is PostgreSQL, reached once: a monitor of the walkthrough's own that pings the pool until a ping first succeeds, and then stays healthy. Open question 2.

The wallet

Method	Needs	Answers
OpenAccount	wallet:write	the caller's account, opened in the ledger
GetAccount	wallet:read	the caller's account
Send	wallet:write	the transfer from the caller's account, to the account it names
WatchAccount	wallet:read	the caller's account, as the ledger's watch sends it

• A token's subject is the only account it can use. The account's ID in the ledger is the subject, and no request has a field for the source account.
• grpcauth checks the token on every method. grpcauth.Default(auth.RequireScopes(bearer, "wallet:read")), and grpcauth.Method for OpenAccount and Send, which need wallet:write. bearer is verified by auth/jwt. A handler reads the subject with bearer.From.
• A send's ID is the caller's. SendRequest requires a request_id. The transfer's ID in the ledger is the subject and that ID together. So a repeat by the app, or a retry by the wallet's client, is the same transfer, and no caller can reuse another's.
• What the ledger answers about the request is passed on, with its details: NotFound, FailedPrecondition, InvalidArgument and AlreadyExists.
• A ledger it can't reach is Unavailable, never Internal. That is the ledger's Unavailable after the client's last attempt, a call its breaker rejected, or a transport error.
  • Its google.rpc.ErrorInfo says why: LEDGER_UNAVAILABLE, or LEDGER_SHED for a call the breaker rejected without making it.
  • A shed call also carries a google.rpc.RetryInfo, from the rejection's RetryAfter, so the app knows when to try again.
• Deadlines cross both services. A caller's deadline is the wallet handler's, and grpc-go sends it on with the call to the ledger. The client's Timeout bounds it too, so a call never outlives either.
• The watch is relayed. WatchAccount opens the ledger's watch of the caller's account, through the client, and sends on what it sends.
  • When the ledger's watch ends with Unavailable after it has sent at least one message, the wallet opens it again. That is the ledger's shutdown, or a lost connection.
  • When it ends before its first message, or ends any other way, the wallet ends the caller's watch with the answer a unary method would get.
  • So a rollout of the ledger keeps every app's watch open, and a ledger that is down ends it, rather than the wallet opening it again forever.
• When the wallet shuts down, grpcserver ends the caller's watch with Unavailable, server is shutting down, and the app watches again. walletctl watch does, and so do the cases.
• It is served by grpcserver, over TLS, with no client certificates. Its probes are grpchealth.NewService on a plaintext port too.

Calling the ledger

The wallet builds one grpcclient.Client for the ledger:

• from LEDGER_ADDR and LEDGER_CA_FILE, with its client certificate through GetClientCertificate, from internal/certfile;
• with DefaultSettings()' retries, budget, breaker and Classify. Unavailable is retried for every method, which is safe because OpenAccount and Transfer are idempotent by their IDs and the rest are reads. NotFound, FailedPrecondition, InvalidArgument and AlreadyExists are the ledger working, so the breaker counts them as successes;
• with ForwardScenario set, because the ledger is the walkthrough's own and reads the same codec.

Its readiness is the Client's Connected, which latches. A release that can't reach the ledger never turns ready, and an outage after that leaves every pod ready, so apps get the wallet's own Unavailable and its reason.

Spreading the calls. Both servers pass grpc.KeepaliveParams(keepalive.ServerParameters{MaxConnectionAge: time.Minute}). Every connection is closed with a GOAWAY after about a minute, so each wallet pod's Client dials again through the ledger's Service and reaches whichever pod it picks, as Calling gRPC services describes. No MaxConnectionAgeGrace is set: an open watch keeps its old connection until it ends, so watches move only at a rollout. A grace would end them every time it ran out, which #1176 also counts against the breaker.

The settings

Every setting is sourced with config.Struct and checked, and a bad one exits 78, naming its variable. Every secret and certificate is a file, read through s.Host.FS.

The wallet:

Variable	What it is	Default
GRPC_ADDR	the address it serves on	required
GRPC_TLS_PEM_FILE	its certificate chain and private key, PEM, read again when they change	required
PROBES_ADDR	the gRPC health port, in plain text	required
LEDGER_ADDR	the ledger, host:port	required
LEDGER_CA_FILE	the CA certificates the ledger is trusted through, PEM	required
LEDGER_TLS_PEM_FILE	its client certificate and key for the ledger, read again when they change	required
LEDGER_TIMEOUT, LEDGER_MAX_ATTEMPTS	grpcclient.Settings' fields	DefaultSettings()'
TOKEN_KEYS_FILE	the JSON Web Key Set that tokens are verified with	required
TOKEN_ISSUER, TOKEN_AUDIENCE	jwt.Settings' fields	required
FAULT_TRUST_SCOPE	the scope a token needs for its scenario to be trusted	empty: none is trusted
FAULT_MAX_DELAY	fault.Settings.MaxDelay	10s
OTEL_SERVICE_NAME, OTEL_*_EXPORTER, OTEL_EXPORTER_OTLP_ENDPOINT	the telemetry Builder's	wallet, and telemetry.DefaultSettings()'

The ledger:

Variable	What it is	Default
GRPC_ADDR, GRPC_TLS_PEM_FILE, PROBES_ADDR	as the wallet's	required
GRPC_CLIENT_CA_FILE	the CA certificates its callers' certificates chain to	required
POSTGRES_URL	PostgreSQL's URL, without a password	required
POSTGRES_PASSWORD_FILE, POSTGRES_TLS_CA_FILE	its password, and the CA its certificate chains to	required
WATCH_INTERVAL	how often a watch reads its account	1s
FAULT_MAX_DELAY, the OTEL_ variables	as the wallet's	10s, and ledger

• PostgreSQL is reached as the order status processor reaches it. The URL may hold no password and no TLS parameter. Init sets sslmode=verify-full and the CA, so pgx reads nothing from the real environment or ~/.pgpass.
• The key set is a file. TOKEN_KEYS_FILE is read and parsed in Init, so a missing or bad key set exits 78. The wallet's jwt.KeySet is the walkthrough's own, and reads the file again whenever the Verifier asks: for a kid it lacks, and after MaxKeyAge. So a rotated key reaches the wallet without a restart, as a KeySet of the end user's describes. Open question 3.

Failure injection

Both services pass humus.WithFaults(Faults), with jsonscenario under the baggage member acme.fault, as every walkthrough does.

	The wallet	The ledger
Trusts a scenario from	a bearer token, verified with the wallet's own Verifier, whose scopes hold FAULT_TRUST_SCOPE	a caller whose client certificate the internal CA issued: the wallet, or the cases
Its points	grpc:ledger.v1.Ledger/<method>, one for each method its Client calls	postgresql:ledger, of its own
Sends a scenario on	to the ledger, with ForwardScenario	nowhere: it calls nothing

• The wallet's trust is a scope, not a certificate, because the wallet is the edge, and a token crosses an ingress that a client certificate doesn't. Its TrustScenario reads the token from the RPC's metadata and verifies it itself, only for a call that carries a scenario.
• The ledger's trust is a certificate, read from the peer's credentials.TLSInfo, as Failure injection shows for grpcserver.
• A scenario aimed at the ledger's database goes through the wallet. The wallet trusts it and forwards it, and the ledger trusts the wallet. The ledger's transaction fails, Transfer answers Unavailable, and the wallet's client retries it, meets the same fault, and gives up. The wallet answers Unavailable. The trace shows every attempt, and the ledger's humus.fit.evaluation events.

Telemetry

• Spans. Each RPC is a server span of otelgrpc's, in each service. Each attempt of the wallet's call to the ledger is a client span under the wallet's, and the ledger's server span is its child. So one Send is one trace across both services. A watch is one span on each side, from its opening to its end.
• Records. Beyond humus's own, each service writes one WARN record of its own: the wallet when it answers Unavailable for the ledger, with the method and the client's error; the ledger when a transaction fails, with the method and the error. No record or span holds a token.
• The walkthrough's own instruments:
  • ledger.transfers, a Counter, by ledger.transfer.outcome: posted, replayed or refused;
  • ledger.watches, an UpDownCounter of the watches open;
  • wallet.watches.reopened, a Counter of the ledger's watches that the wallet opened again.
• otelgrpc's and humus's: rpc.server.call.duration in both, rpc.client.call.duration in the wallet, and the wallet's Client's humus.circuit.state, humus.circuit.transitions, humus.circuit.calls and humus.retry.decisions.

Production-ready, as a checklist

The walkthrough's pages close with this list, and the application meets each item. The pages give each row a Where column, linking it to the page and section that shows it, as every walkthrough's does.

	How
Every setting sourced and checked; a bad one exits 78, naming its variable	config.Struct and the packages' checks, in both services
No secret in the environment, the logs or the spans	PostgreSQL's password and the private keys are files, read through s.Host.FS. The token is metadata, which otelgrpc doesn't record, and no handler logs it
Certificates renewed without a restart	internal/certfile, for both servers' certificates and the wallet's client certificate. The key set is read again for a kid it lacks
Two replicas of each service	a PodDisruptionBudget of minAvailable: 1, and a rolling update of maxUnavailable: 0
Ready once it has reached its dependency, and still ready through its outages	Connected, and the ledger's latch on PostgreSQL
A rollout loses no call	grpcserver.Main's 5s drain. A unary call is bounded by LEDGER_TIMEOUT, 10s, inside the 15s grace period. Watches end with Unavailable and are opened again. terminationGracePeriodSeconds is 30, above the drain, the grace period and the telemetry's shutdown
Calls spread across pods	MaxConnectionAge on both servers
Money is never made or lost	one transaction per transfer, locks in the order of the IDs, and a CHECK that no account but external goes below zero
A retry never moves money twice	every transfer's ID: the subject and the request's ID
The contract never breaks a caller	buf breaking against main, in CI
Least privilege	the ledger's role has SELECT, INSERT and UPDATE on its two tables. Only the internal CA's certificates call the ledger. A token can use only its subject's account. A non-root, read-only container, with no service account token
Observable	traces across both services, logs and metrics over OTLP, deploy/alerts.yaml, deploy/dashboard.json, and alerts on each Deployment from the Collector's Kubernetes cluster receiver
Failures say why	statuses with PreconditionFailure, BadRequest, ErrorInfo and RetryInfo details
Failures are rehearsed	the failure injection cases, in CI and against staging
Safe by default	FAULT_TRUST_SCOPE empty. Server reflection off. The probes ports serve gRPC health and nothing else

The tests

Level	Where it runs	What it covers
Units	go test, in the Build job, inside synctest where they take time	the wallet's methods against a ledger of the test's own on humustest's Host, for each status it answers; the relay through that ledger's shutdown, and a watch that ends before its first message; the ledger's methods on a fake Store, and the watch's polling; internal/certfile, rewriting its files under a running server; the key set; the wallet's TrustScenario; the store's injection point; walletctl's tokens, verified by auth/jwt
The store	testcontainers tag	the Store against PostgreSQL: concurrent transfers from one account never overdraw it, a repeated ID moves money once, two transfers each way between two accounts don't deadlock, and the CHECK refuses a negative balance
The shared cases, inside synctest	go test, in the Build job	both Apps on one humustest Host, with two ledger replicas sharing one in-memory Store, which keeps the point
The shared cases, in process	testcontainers tag, in "Wallet walkthrough (testcontainers)", which isn't required	both Apps on humus.OSHost(), against PostgreSQL in a container, with two ledger replicas
The shared cases, deployed	functional tag, in "Wallet walkthrough (kind)", which isn't required, and against an end user's staging	the image built from the Dockerfile, brought up by up.sh, and reached through kubectl port-forward
The telemetry as deployed	functional tag, kind only	one Send's trace in Tempo holds spans of both services. Its records are in Loki, with its trace ID. Both services' metrics are in Mimir
The alerts as deployed	functional tag, kind only	Grafana loaded every rule and the dashboard. After "The ledger's breaker opens", Mimir holds every series the rules read, and each rule has evaluated since without error
The contract	the container job	buf lint, buf breaking against main, and buf generate with no diff

Two ledger replicas in process. The in-process harnesses run two ledger Apps, at addresses of their own. The wallet's Host dials the ledger's address to one of the replicas still in its endpoints, as a ClusterIP Service does. So a rollout is one replica at a time, as on kind. The dialer is the harness's own code, and the wallet's Init doesn't change.

The shared cases are in cases. It imports neither the application nor humustest, so the compiler keeps them black-box. It does import walletpb and ledgerpb, the services' published contracts, as any client would. Each case takes a Target:

• a connection to the wallet that trusts its certificate;
• a connection to the ledger with the cases' client certificate, to open and fund accounts;
• a function that returns a token for a subject and scopes;
• RolloutWallet and RolloutLedger, which roll a Deployment, or restart each replica in turn in process, and return once it is done. On kind, RolloutWallet also forwards the wallet's port again, since kubectl port-forward follows one pod, which the rollout replaces;
• the scenario's baggage key, the trust scope, and a timeout.

They poll with plain loops, which work in every harness. Each case makes its own subjects and accounts, named with crypto/rand, so the cases run in parallel and against a shared staging environment. They mutate, so they never run against production.

Parallel and Alone. The cases are two lists, as the mailer's and the shoppingcart's are. Parallel runs at once. Alone then runs one case at a time, in every harness. A case is alone when it fails calls to the ledger, since a wallet replica's breaker and retry budget are shared by every call it serves, or when it rolls a service out.

Parallel:

• An account's life. Open an account, and open it again: one account, with a balance of 0. A deposit through the ledger is its balance.
• Sending. A send from one account to another changes both balances by its amount, and answers with the transfer.
• Sending again. The same request ID again answers with the same transfer, and the balances change once. The same ID for another amount is AlreadyExists.
• Not enough money. A send of more than the balance is FailedPrecondition, with a PreconditionFailure naming the account, and moves nothing.
• Sends at once. Ten sends at once from an account that can pay for five: five succeed, five are FailedPrecondition, and the balance ends at 0.
• Who may do what. A token without wallet:write gets PermissionDenied on Send. No token gets Unauthenticated. A call to the ledger without a client certificate is Unauthenticated.
• A watch. A watch sees the balance at once, and then each balance it settles on after two sends.
• Failure injection, each under a scenario from a token with the trust scope:
  • delaying grpc:ledger.v1.Ledger/Transfer by 2s: the send still succeeds, inside the client's 10s Timeout;
  • a scenario from a token without the trust scope is stripped, and the send succeeds.

Alone, in order:

• The ledger is down. Aborting grpc:ledger.v1.Ledger/Transfer: the send is Unavailable, LEDGER_UNAVAILABLE, and moves nothing. The same request ID without the scenario then moves the money once, polled until the breaker, if it opened, lets it through.
• The ledger's database is down. Aborting postgresql:ledger, forwarded to the ledger: the send is Unavailable, and moves nothing.
• A rollout of the wallet. With a watch open, RolloutWallet. The watch ends with Unavailable, server is shutting down. Watching again sees the balance, and a send made during the rollout either succeeds or answers Unavailable and succeeds once when repeated with its ID.
• A rollout of the ledger. With eight watches open through one wallet connection, RolloutLedger, while the case sends from each account. No watch ends. Each sees every balance its account settles on, and every send succeeds. This case waits for grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176: until it is fixed, the first replica's shutdown opens the wallet's breaker.
• The ledger's breaker opens. Under a scenario that aborts grpc:ledger.v1.Ledger/Transfer, the case sends until an answer is LEDGER_SHED, with a RetryInfo. Sends without the scenario are then polled until one succeeds. It records the breaker's transitions and the retry budget's throttled decisions, which the alerts' check needs. It runs last, because it leaves a breaker open.

The abort cases also spend the retry budget, which a success refills by a tenth of a token. So a later case may see fewer attempts than MaxAttempts, and the cases check the answer, not the number of attempts. Against a shared staging environment, the last Alone case opens the breaker of the wallet replica it reaches, and every caller's calls there are shed until it closes.

Other whole-App tests stay in each app: a bad setting exits 78, readiness waits for the dependency, and the telemetry carries the evaluations and the strips, read with humustest.Telemetry.

The deployment

deploy/base/ is a kustomization:

• a Namespace, and a ServiceAccount for each service, with automountServiceAccountToken: false;
• for each service, a Deployment with two replicas, a rolling update of maxUnavailable: 0, liveness, readiness and startup probes of grpc: on the probes port, with service: liveness and service: readiness, resources, and a non-root, read-only security context;
• a Service for each, ClusterIP, with its port named grpc;
• a PodDisruptionBudget for each, minAvailable: 1;
• the names of what each Deployment mounts, which an operator provides: each certificate as a Secret, the CA bundles, the ledger's PostgreSQL password, and the key set as a ConfigMap.

The kind overlay, deploy/kind/, adds:

• PostgreSQL, with TLS, the ledger's database, schema.sql applied by its owner, and the ledger's role with the password up.sh generates;
• the LGTM stack: an OpenTelemetry Collector that both services export to, which forwards OTLP to Loki, Tempo and Mimir, and Grafana, with all three as data sources and the alert rules and dashboard provisioned. Loki, Tempo and Mimir each run as one process, storing to the pod's disk;
• in the Collector, the k8s_attributes processor, which adds each pod's name, namespace and Deployment, and the Kubernetes cluster receiver, which reports the Deployments' replicas and the containers' restarts. A ClusterRole lets it read both. Mimir promotes the pod's attributes, and scope metadata, to labels;
• FAULT_TRUST_SCOPE set to the cases' scope.

deploy/kind/up.sh creates a kind cluster if there is none. Then it:

• generates a CA, and with it every certificate: the wallet's, its client certificate for the ledger, the ledger's, PostgreSQL's, and the cases';
• generates PostgreSQL's passwords, Grafana's admin password, and the signing key, whose key set it makes with walletctl jwks;
• loads the image into kind, applies the overlay with the Secrets and the ConfigMap it made, and waits for every rollout.

The kind job runs it, and so does a reader. No secret is kept in the repository.

The kind job builds the image, runs up.sh, and runs the shared cases and the kind-only checks through kubectl port-forward, as every walkthrough's does.

The Dockerfile builds the wallet and the ledger as static binaries, onto a distroless, non-root base. Each Deployment runs its program as the container's command.

Trying it by hand

cmd/walletctl is a small CLI:

• jwks prints the public half of a private key, as a JSON Web Key Set whose kid is the key's thumbprint. up.sh runs it to make the key set's ConfigMap.
• token signs a token with the same key and kid, for a subject, scopes, an issuer and an audience.
• open, balance, send and watch call the wallet as a subject, with a token they sign, trusting the overlay's CA. watch watches again whenever its watch ends with Unavailable.
• deposit calls the ledger with the cases' certificate, from the Secret up.sh made, so a reader can fund an account.

It runs on the reader's machine, through kubectl port-forward, so the image doesn't carry it.

Alerts

deploy/alerts.yaml is a Grafana alerting provisioning file, and deploy/dashboard.json a Grafana dashboard. The rules query Mimir, on the names the instruments have there. The kind overlay's Grafana provisions both. The pages explain each alert, and show the dashboard.

Alert	When	Reads
WalletErrors	the wallet ends over 1% of its unary calls with a server's code for 10 minutes	rpc.server.call.duration's count, by rpc.response.status_code, without WatchAccount
WalletSlow	the wallet's p99 latency of unary calls is over 1s for 10 minutes	rpc.server.call.duration, without WatchAccount
LedgerErrors	the ledger ends over 1% of its unary calls with a server's code for 10 minutes	as WalletErrors, for the ledger
WalletLedgerBreakerOpen	the wallet's breaker for the ledger is open	humus.circuit.state
WalletLedgerBreakerOpened	it opened in the last 5 minutes, however briefly	humus.circuit.transitions
WalletRetriesThrottled	the retry budget refused retries	humus.retry.decisions, throttled
WalletUnavailable, LedgerUnavailable	the Deployment has had no available replica for 5 minutes	k8s.deployment.available, from the Collector's Kubernetes cluster receiver
WalletCrashLooping, LedgerCrashLooping	a container restarted more than 3 times in 15 minutes	k8s.container.restarts, from the same receiver

• A server's codes are those grpcserver marks its spans Error for: Unknown, DeadlineExceeded, Unimplemented, Internal, Unavailable and DataLoss.
• The rules leave out WatchAccount, by rpc_method. A watch's duration is how long it was open, not how long a call took. And every rollout ends every watch with Unavailable, which an error rule would count as an outage.
• The services are told apart by job, which Mimir takes from each one's service.name.
• "In the last 5 minutes" counts a series that is new in the window, with x unless x offset 5m added to increase(x[5m]), as the mailer's rules do (#1089).
• The last four rows read what watches the services, since an App that is down reports nothing.
• The dashboard shows each service's calls by method and code, the latency of unary calls, the open watches and those opened again, the transfers by outcome, the breaker's state and transitions, the retry decisions, and each Deployment's available replicas and restarts.

The pages

They are under docs/content/walkthroughs/wallet/. Each page is a list of steps. A step is a sentence or two, then the code it arrives at, quoted from the walkthrough's files. A reason is given only where the code would be wrong without it, and in one sentence, such as why a send carries its ID, or why the locks are taken in the order of the IDs. There is no history, no alternatives and no asides.

1. Overview: what the two services do, how a call crosses them, and the files it ends with.
2. The contract: the two .proto files, buf's lint and breaking rules, and buf generate.
3. The ledger's accounts: schema.sql, a transfer's transaction, and the CHECK.
4. Serving the ledger: its methods and their statuses with details, the watch, mTLS and grpcauth, its injection point and its probes.
5. Calling the ledger: grpcclient, its retries and breaker, why every transfer is safe to retry, ForwardScenario, Connected, and turning the ledger's answers into the wallet's.
6. Serving the wallet: its methods, bearer tokens and scopes, the subject's account, and relaying the watch.
7. Init: each service's settings, the secrets, certificates and key set as files, MaxConnectionAge, readiness, BuildTelemetry, Faults and TrustScenario, and main.
8. Testing: the unit tests, the two ledger replicas, the shared cases with their Target, Parallel and Alone, and the failure injection cases.
9. Deploying: the Dockerfile, the kustomization, the gRPC probes, the disruption budgets and the drain, and the kind overlay. Running it on kind with up.sh, and trying it with walletctl.
10. Observing: one send's trace across both services in Grafana, the logs and metrics, the alerts and the dashboard.
11. Production-ready: the checklist.

The pages add the walkthrough to the Walkthroughs section's index.

Changes to other files

• .github/workflows/build.yaml:
  • a "Wallet walkthrough (testcontainers)" job, which also checks the contract with a pinned buf, and a "Wallet walkthrough (kind)" job;
  • neither is required. The vet jobs already vet every module under example/ with both tags.
• The gRPC services and Calling gRPC services pages: a link to the walkthrough from each one's introduction.
• The Walkthroughs section's index lists it.

docs_test.go needs no change. Its rule for a walkthrough's samples already accepts protobuf, JSON and the walkthrough's own tests.

Design decisions

• Two services of the walkthrough's own. Proposal: outbound gRPC clients #763's handoff is a service that calls gRPC, and only a server of the walkthrough's own shows both halves, and a scenario forwarded to a point of its own. The ledger is half of the application, deployed from deploy/base/ beside the wallet, not a stand-in for a dependency. Open question 1.
• A ledger, because its methods must not run twice. A retry that is safe only with an ID, a balance that must never go below zero, and a status that says which account is short are what a caller of an internal gRPC API has to get right.
• The wallet keeps no data. Every balance is the ledger's, so there is one source of truth, one schema and one PostgreSQL database. The wallet's job is the edge: tokens, scopes, the subject's account, and turning the ledger's failures into answers an app can act on.
• Locks, not serializable transactions. Locking both accounts in the order of their IDs serializes the transfers that touch them, with no deadlock and no transaction for a client to retry. Under SERIALIZABLE, PostgreSQL aborts transactions it can't order, and the ledger would answer Aborted for the client to retry.
• The watch is relayed, and opened again only after a message. grpcclient never retries an open stream, so the wallet opens it again itself. Requiring a message first bounds it: a ledger that keeps ending its watches at once ends the app's, rather than spinning.
• No MaxConnectionAgeGrace. A grace ends open watches every time it runs out, and the watches would move every few minutes for nothing. Watches move at a rollout, which is when a pod goes.
• gRPC health on plaintext ports. The kubelet's grpc probes use TLS only under an alpha feature gate, and present no credentials, so gRPC health on a service's own port would also need its methods opened by name in each service's grpcauth rules. The probes ports serve gRPC health and nothing else, as Health recommends for most clusters. Open question 2.
• The key set is a file. The API walkthrough fetches its key set over HTTPS from WireMock. Here a KeySet of the walkthrough's own reads a mounted file, which needs no HTTP client, and no WireMock, so the whole App runs inside synctest. Open question 3.
• Two ledger replicas in every harness. A rollout is one replica at a time, so the in-process harnesses run two, behind a dialer of their own that does what a ClusterIP Service does. That is how the prototype found grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176.
• The rules leave out the stream. A watch's duration and its end at a rollout aren't a call's latency or error.
• Operating it is out of scope. A walkthrough is a production-ready application you can run. Running one like it in production is for the end user who ships it.

Testing

The walkthrough is its own test: see The tests.

What was prototyped. A prototype ran against main at d4eadda, under go test -race, inside synctest, on one humustest Host. A grpcserver front relayed a backend's server stream, and called its unary method, through a grpcclient Client over mTLS. grpc-go's interop service stood in for walletpb and ledgerpb. It showed that:

• a forwarded scenario reaches the ledger's own point. The backend trusted the caller's verified certificate, from the peer's credentials.TLSInfo. With ForwardScenario, a scenario aborting its point aborted each of the 3 attempts DefaultSettings().Classify made, and the front answered Unavailable. Without ForwardScenario, the scenario was stripped at the client, and the call succeeded;
• a relayed watch outlives the backend's drain. After SIGTERM, the backend kept sending for its whole 5s drain. Its Shutdown then ended the stream with Unavailable, server is shutting down;
• a watch opened again moves to the other replica. With two backends behind a dialer that did what a ClusterIP Service does, and one watch open, the front opened it again on the second backend when the first shut down, and the breaker stayed closed;
• with one backend, the app watches again. The front's open failed, the front ended the app's watch with Unavailable, and the app watched again until a new backend was up and the breaker had closed;
• grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176. With six watches open, the first backend's shutdown opened the front's breaker. 4 of the 6 couldn't open again, and a unary call to the healthy backend was rejected. A Classify that ignored a stream's end with server is shutting down kept the breaker closed, and all six moved. A backend with MaxConnectionAge and MaxConnectionAgeGrace opened the breaker the same way, when its grace closed the connection.

Not prototyped: the .proto files and buf, PostgreSQL and the transfer's transaction, the watch's polling, the key set as a file, walletctl, the kind overlay and the LGTM stack.

Migration from today

Nothing migrates. The walkthrough is new, and no release had a gRPC walkthrough.

Open questions

1. The application.

   • (a) A wallet and its ledger, two gRPC services of the walkthrough's own, as proposed.
   • (b) Orders and inventory: an orders service that reserves stock in an inventory service, with a reservation's ID as its idempotency key and a watch of an order. It shows the same, but a reservation that expires and an order's states are more to explain.
   • (c) One gRPC service, with WireMock in place of its gRPC dependency. WireMock's gRPC extension answers unary methods from mappings and a descriptor set. It shows the client, but no server of the dependency's, no stream, and no point a forwarded scenario can reach.

   Recommendation: (a).

2. The probes.

   • (a) grpchealth.NewService on a plaintext port, with grpc: probes, as proposed.
   • (b) httphealth.NewService over HTTPS on a port of its own, with httpGet probes, as the processors' walkthroughs have. Every port is TLS, but a gRPC walkthrough then shows no gRPC health.
   • (c) grpchealth.NewServer on each Service's own port, with mode: TLS probes and no second port. That needs the alpha GRPCContainerProbeTLS gate, on kind and on an end user's cluster, and each service's grpcauth rules open gRPC health's methods by name.

   Recommendation: (a).

3. The wallet's key set.

   • (a) A file, read by a KeySet of the walkthrough's own, as proposed. Both Apps run inside synctest, and the kind overlay needs no WireMock.
   • (b) A RemoteKeySet that WireMock serves, as the API walkthrough's is. It is closer to an identity provider, but needs httpclient and WireMock, so the shared cases can't run inside synctest.

   Recommendation: (a).

4. How a watch learns of a change.

   • (a) It polls its account every WATCH_INTERVAL, as proposed. It costs a query a second for each watch, and a watch sends the balance it settles on, not each transfer.
   • (b) PostgreSQL's LISTEN and NOTIFY. Each transfer notifies, and each ledger replica holds a connection that listens and wakes its watches. Changes arrive at once, but the listening connection is a Service of its own, and a watch must read its account again whenever that connection is lost, since notifications aren't queued for it.

   Recommendation: (a).

Out of scope

• Fixing grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176 in this walkthrough. It is grpcclient's, and its fix is its own.
• Currencies, fees, holds, reversals, statements and interest. One currency, in whole cents.
• Deposits from a real payments provider. Deposit is the ledger's method for money that enters it, and the cases and walletctl call it.
• An ingress or gateway in front of the wallet, grpc-web, and the gRPC-JSON transcoding an app in a browser would need.
• Client-side load balancing of the ledger, which Proposal: outbound gRPC clients #763 left out of humus's scope. MaxConnectionAge spreads the calls instead.
• Server reflection. It stays off, and walletctl carries the generated code.
• Schema migrations. schema.sql is applied before the ledger runs, as the order status processor's is. The API walkthrough shows migrations.
• Authorization between internal services beyond a certificate the internal CA issued, such as which service may call Deposit.
• cert-manager in the kind overlay. up.sh issues the certificates, and the base mounts the Secrets cert-manager would write.
• Operating it in production: runbooks, and reconciling the ledger. They're for an end user who ships a service like this one.
• Publishing the image, or a release of the walkthrough.

References

• Proposal: humus on the bedrock process model, and its revisions 13, 14, 15 and 17
• Proposal: gRPC services, and the gRPC services page
• Proposal: outbound gRPC clients, and the Calling gRPC services page
• Proposal: authentication and authorization, Proposal: exposing health probes, Proposal: failure injection and Proposal: humus's own metrics
• Proposal: API walkthrough, whose catalog this walkthrough's ledger takes up, and its revisions 1 and 2
• #1176: a rollout of a dependency served by grpcserver opens the breaker of a grpcclient Client with five streams open to the pod
• buf: lint rules, breaking change detection and buf generate
• gRPC: status codes, error details, deadlines, and gRFC A9, server-side connection management
• PostgreSQL: explicit locking, and LISTEN
• WireMock's gRPC extension

[Zaba505]

Revision 1: every question answered

Questions 1, 2 and 4 are answered with (a), as proposed: a wallet and its ledger, grpchealth.NewService on a plaintext port, and a watch that polls its account every WATCH_INTERVAL. Those parts of the post don't change.

Question 3 is answered with (b): the wallet's key set is a jwt.RemoteKeySet, fetched over HTTPS with httpclient, as the API walkthrough's is.

It costs synctest nothing. The post said WireMock would keep the shared cases out of synctest, but WireMock is needed only on kind. In process, the tests serve the key set themselves: an http.Handler that writes the JSON Web Key Set, on httpserver, on the humustest Host inside synctest, and on humus.OSHost() with testcontainers. The shoppingcart's tests already do this, in app/idp_test.go. The wallet's Init doesn't change between harnesses: only the address in TOKEN_KEYS_URL and the CA in TOKEN_KEYS_CA_FILE do.

On kind, WireMock serves the key set, since no walkthrough writes a server for a dependency.

This revision adds no questions. Each section below replaces or adds to the part of the same name in the post. Everything else is unchanged.

Summary

The bullet "It runs on kind with one script" becomes:

• It runs on kind with one script, up.sh, in CI and for a reader, beside PostgreSQL, WireMock serving the key set, and the LGTM stack. walletctl signs a token and calls the wallet, to try it by hand.

The application

In the tree:

• app/ becomes "the wallet's Init, settings, Faults, grpcauth rules, trust and probes". The walkthrough has no KeySet of its own.
• deploy/kind/ becomes "the kind overlay, up.sh, and WireMock's mapping for the key set".

The wallet

The bullet "grpcauth checks the token on every method" gains:

• bearer's verify function answers Unavailable when the key set can't be fetched, which grpcauth passes on. Its google.rpc.ErrorInfo says KEYS_UNAVAILABLE. A token is never refused as Unauthenticated because the key set was down. The wallet tells the two apart by *jwt.KeySetError, as the shoppingcart's endpoint.KeysUnavailable does.

Calling the key set

This section is added, after "Calling the ledger".

The wallet builds one httpclient.Client for the key set, from TOKEN_KEYS_URL and TOKEN_KEYS_CA_FILE, with DefaultSettings()' retries, budget and breaker, and passes it to jwt.NewRemoteKeySet. The Verifier fetches the key set for a kid it lacks, and after MaxKeyAge, so a rotated key reaches the wallet without a restart.

The key set isn't part of readiness. An outage of it answers Unavailable, KEYS_UNAVAILABLE, only to a token whose key the wallet doesn't hold.

The settings

In the wallet's table, the row TOKEN_KEYS_FILE is replaced by:

Variable	What it is	Default
TOKEN_KEYS_URL	the JSON Web Key Set that tokens are verified with, an absolute https URL	required
TOKEN_KEYS_CA_FILE	the CA certificates the key set's server is trusted through, PEM	required
TOKEN_KEYS_TIMEOUT, TOKEN_KEYS_MAX_ATTEMPTS	httpclient.Settings' fields	DefaultSettings()'

The bullet "The key set is a file" is removed.

Failure injection

The wallet's points become grpc:ledger.v1.Ledger/<method>, one for each method its ledger Client calls, and http:<key set host>:<port>, its key set's Client, such as http:wiremock:443 on kind. A scenario is never forwarded to the key set.

Telemetry

The wallet's Client for the key set adds httpclient's humus.circuit.state, humus.circuit.transitions, humus.circuit.calls and humus.retry.decisions, beside its ledger Client's. Each data point carries the dependency's server.address.

Production-ready, as a checklist

The row "Certificates renewed without a restart" becomes:

	How
Certificates and signing keys renewed without a restart	internal/certfile, for both servers' certificates and the wallet's client certificate. The key set is fetched again for a kid the wallet lacks

The tests

• Units: "the key set" becomes "the wallet's answer when its key set can't be fetched: Unavailable, KEYS_UNAVAILABLE, with the tests' handler down".
• The shared cases, inside synctest: both Apps on one humustest Host, beside the tests' key set handler, with two ledger replicas sharing one in-memory Store.
• The shared cases, in process: both Apps on humus.OSHost(), against PostgreSQL in a container, with two ledger replicas, and the tests' key set handler on humus.OSHost(). The job runs no WireMock.

The Target's function that returns a token signs it with the tests' key in process, and with the key from the Secret up.sh made on kind.

The deployment

In deploy/base/, the wallet mounts the key set's CA bundle instead of the key set.

The kind overlay adds WireMock, serving the key set over HTTPS from the mapping deploy/kind/wiremock/jwks.json and the key set up.sh makes, as the API walkthrough plans for its own.

up.sh also issues WireMock's certificate, and makes the key set WireMock serves with walletctl jwks.

Trying it by hand

jwks's last sentence becomes: "up.sh runs it to make the key set WireMock serves."

Alerts

WalletLedgerBreakerOpen and WalletLedgerBreakerOpened become WalletBreakerOpen and WalletBreakerOpened, a breaker of the wallet's, the ledger's or the key set's. They and WalletRetriesThrottled keep server_address as a label, so an alert names the dependency, as the API walkthrough plans for its own.

The dashboard's breaker and retry panels are split by server_address.

Design decisions

The bullet "The key set is a file" becomes:

• The key set is fetched, as an identity provider's is. The wallet uses a jwt.RemoteKeySet over httpclient. In process, the tests serve the key set from a handler of their own, so both Apps still run inside synctest. On kind, WireMock serves it from a mapping the walkthrough owns.

Out of scope

Added:

• An identity provider in the overlay. WireMock serves a static key set, and the cases sign their own tokens. A real provider is the platform's.

Open questions

1. (a)
2. (a)
3. (b), with the tests serving the key set from their own handler in process, so synctest is kept.
4. (a)

[Zaba505]

Closing: the gRPC walkthrough has landed

The revision's stories are done. example/wallet is a wallet service calling a ledger service over gRPC, a module of its own:

• story(example): define the wallet's and the ledger's contracts in protobuf, checked by buf #1189 defined the wallet's and the ledger's contracts in protobuf, checked by buf (story(example): define the wallet's and the ledger's contracts in protobuf, checked by buf #1212);
• story(example): keep the ledger's accounts and transfers in PostgreSQL, where money is never made or lost #1190 kept the ledger's accounts and transfers in PostgreSQL, where money is never made or lost (story(example): keep the ledger's accounts and transfers in PostgreSQL, where money is never made or lost #1216), and story(example): serve the ledger's methods and its watch, with statuses whose details say why #1191 served the ledger's methods and its watch, with statuses whose details say why (story(example): serve the ledger's methods and its watch, with statuses whose details say why #1220);
• story(example): read the wallet's and the ledger's certificates from files, again when they're renewed #1192 read both services' certificates from files, again when they're renewed (story(example): read the wallet's and the ledger's certificates from files, again when they're renewed #1222), and story(example): wire the ledger's App: settings, mTLS, grpcauth, probes and main #1193 wired the ledger's App: settings, mTLS, grpcauth, probes and main (story(example): wire the ledger's App: settings, mTLS, grpcauth, probes and main #1224);
• story(example): serve the wallet's methods on the ledger, for the token's subject only #1194 served the wallet's methods on the ledger, for the token's subject only (story(example): serve the wallet's methods on the ledger, for the token's subject only #1226), and story(example): wire the wallet's App: settings, the ledger's client, tokens, probes and main #1195 wired the wallet's App: settings, the ledger's client, tokens, probes and main (story(example): wire the wallet's App: settings, the ledger's client, tokens, probes and main #1228);
• story(example): make the key set, sign tokens and call the wallet by hand with walletctl #1196 made the key set, signed tokens and called the wallet by hand with walletctl (story(example): make the key set, sign tokens and call the wallet by hand with walletctl #1230);
• story(example): run the wallet's shared cases in process, inside synctest and against PostgreSQL #1197 ran the shared cases in process, inside synctest and against PostgreSQL (story(example): run the wallet's shared cases in process, inside synctest and against PostgreSQL #1233);
• story(example): rehearse failures across the wallet and the ledger #1198 rehearsed failures across the wallet and the ledger (story(example): rehearse failures across the wallet and the ledger #1238), and story(example): rehearse failures in the wallet's shared cases #1199 added them to the shared cases (story(example): rehearse failures in the wallet's shared cases #1241);
• story(example): keep every watch open through a rollout of the ledger #1200 kept every watch open through a rollout of the ledger (story(example): keep every watch open through a rollout of the ledger #1242), once story(grpcclient): a dependency's rollout or connection age leaves the Client's breaker closed #1180 fixed grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176;
• story(example): trace, log and measure the wallet and the ledger, without a token #1201 traced, logged and measured both services, without a token (story(example): trace, log and measure the wallet and the ledger, without a token #1243);
• story(example): build the wallet's image and its manifests #1202 built the image and its manifests (story(example): build the wallet's image and its manifests #1245), story(example): bring the wallet and the ledger up on kind with one script #1203 brought both services up on kind with up.sh (story(example): bring the wallet and the ledger up on kind with one script #1246), and story(example): run the wallet's shared cases against kind #1204 ran the shared cases against kind (story(example): run the wallet's shared cases against kind #1247);
• story(example): send the wallet's and the ledger's telemetry to the LGTM stack in kind #1205 sent their telemetry to the LGTM stack in kind (story(example): send the wallet's and the ledger's telemetry to the LGTM stack in kind #1248), and story(example): alert on the wallet and the ledger in Grafana, and give them a dashboard #1206 gave them Grafana alerts and a dashboard (story(example): alert on the wallet and the ledger in Grafana, and give them a dashboard #1253);
• story(docs): walk through building the wallet and the ledger #1207, story(docs): walk through testing and deploying the wallet and the ledger #1208 and story(docs): walk through observing the wallet and the ledger, and their production-ready checklist #1209 walked through building, testing, deploying and observing them, and their production-ready checklist (story(docs): walk through building the wallet and the ledger #1281, story(docs): walk through testing and deploying the wallet and the ledger #1283, story(docs): walk through observing the wallet and the ledger, and their production-ready checklist #1285).

The revision left no open questions.

The rest stays out of scope, as "Out of scope" lists: currencies, fees, holds, reversals, statements and interest; deposits from a real payments provider; an ingress or gateway, grpc-web and gRPC-JSON transcoding; client-side load balancing; server reflection; schema migrations; authorization between internal services beyond the internal CA's certificate; cert-manager in the kind overlay; operating it in production; and publishing the image or releasing the walkthrough.