v4.0.0 — Canary: API v3

The canary release: API v3, rolled out next to v2 (3.1.0) and promoted to 100% by the pipeline.

API v3

• /v3/items: every item carries stockValue, what its stock is worth (quantity × unitPrice), computed on every response and never stored. v1 and v2 never show it, and no request body may set it.
• /api/... without a version serves v3; every resource is served under /v3.
• Expand/Contract: v3 changes the API, not the database. It reads the columns v2 reads, so 3.1.0 keeps running on the same schema, untouched, and a rollback loses nothing.

Breaking, hence a major

v3 no longer has the bin location that v3.2.0 added. It lived in a column of its own; a computed field leaves the schema exactly as v2 built it. Removing a field from a published API is incompatible. The pipeline fixes of v3.2.1 and v3.2.2 are all here.

Verified on AWS

Run	Canary share measured	Canary requests	Outcome
deploy	0%, reached with x-canary: true		4.0.0 canary with stockValue, 3.1.0 stable without it
promote, fault=error	8.9% at stage 1	396, all 5xx	gate → ABORT → 0%
promote, no fault	10.0% → 48.6% → 97.4%	448 → 2,174 → 4,342	PROMOTE at every stage; stable rebuilt with 4.0.0

A price written through v3 while the canary held 0% was read by 3.1.0 at once, and survived the abort and the promotion.

v3.2.2 — Canary: API v3

The canary release: API v3, rolled out next to v2 and promoted to 100% by the pipeline. It carries everything in v3.2.0 and v3.2.1.

API v3

• /v3/items: every item carries its bin location, readable in GET and writable in POST and PATCH. v1 and v2 never see it.
• /api/... without a version serves v3; every resource is served under /v3.
• Expand/Contract: the canary adds the nullable column before it starts; 3.1.0 keeps working on the same database, and a rollback loses nothing.

Fixed in the pipeline while rehearsing it

• v3.2.1: each stage waits for alarms left by an earlier rollout to clear before moving traffic.
• v3.2.2: both alarms count a minute without canary traffic as healthy (FILL(..., 0)). With no new data, a CloudWatch alarm keeps looking back at the minutes that broke it, so after a rollback it never cleared.

Verified on AWS

Run	Canary share measured	Canary requests	Outcome
promote, fault=error	9.0% at stage 1	399, all 5xx	alarm canary-errors → ABORT → 0%
promote, fault=latency	8.9% at stage 1	397, p95 1,548 ms	alarm canary-latency → ABORT → 0%
promote, no fault	9.5% → 47.9% → 97.3%	426 → 2,140 → 4,354	PROMOTE at every stage; stable rebuilt with 3.2.2

A bin location written through v3 while the canary held 0% survived both aborts and the promotion.

v3.2.1 — Stages wait for stale alarms

Superseded by v3.2.2.

Fixed

• A promotion started right after an abort failed at once: the alarm the abort had fired was still in ALARM, and the first stage read it as evidence against the new rollout. Each stage now waits for alarms left by an earlier rollout to clear before it moves any traffic.

v3.2.0 — API v3

API v3, the version the canary rolls out. Superseded by v3.2.2, which fixes two pipeline issues found while rehearsing it.

API v3

• /v3/items: every item carries its bin location ("A-03-2"), readable in GET and writable in POST and PATCH.
• v1 and v2 never see the column: it is not selected unless asked for, so their answers, cached copies included, keep their exact shape, and v2 rejects it in a body.
• Groups, movements, exports and interop are served under /v3 too, so /api/... without a version, which now means v3, never loses a route.

Expand / Contract

• Before the canary starts, its seed step expands the table with ALTER TABLE items ADD COLUMN IF NOT EXISTS location varchar(40): additive, idempotent, nullable, so 3.1.0 keeps working on the same database and a rollback loses nothing.

Docs

• README: the canary architecture, the routing, the stages and their observation times, the thresholds, fault injection, the dashboard and the Expand/Contract strategy.

v3.1.0 — Canary-ready baseline

The stable baseline of the canary delivery, and everything a canary rollout needs.

Release identity

• Every response, errors and docs included, carries X-App-Version and X-App-Track (stable or canary).
• /api/... without a version serves the newest version the build has: v2 in this release.

Traffic splitting

• An Application Load Balancer in front of two EC2 machines, one per track, each with its own image.
• Weighted forward between the two target groups, set by canary_weight; a rule sends x-canary: true to the canary at any weight.
• Image tags in ECR are immutable: a commit SHA always names the same image.

Safe schema for two versions at once

• The schema is built from the entities only on an empty database; production never synchronizes, so an older build can never drop what a newer one added.
• Both tracks seed under a PostgreSQL advisory lock.

Observability and automatic abort

• Fault injection on the canary only: x-fault: error (500) or x-fault: latency (+1.5 s).
• The API publishes memory and movements recorded (the business metric) to CloudWatch, per track.
• A CloudWatch dashboard compares stable and canary side by side; two alarms watch the canary's errors and latency.
• canary/slo.json: versioned promote / pause / abort thresholds, read by both the alarms and the gate.
• Canary workflow: deploy (stage 0, header only), promote (10% → 50% → 100% under load, judged by canary/gate.ts, automatic rollback on abort), rollback.

Verified on AWS: a 50% weight split 87 / 113 out of 200 requests; a promotion with injected errors aborted at 10% (423 canary requests, all 5xx, alarm fired) and the canary went back to 0%.

v3.0.0 — Terraform

The API now deploys to AWS from code: Terraform creates four services, a GitHub Actions pipeline builds the API and installs it on them, and the same pipeline destroys everything.

File	Service	What it is for
ec2.tf	EC2	One machine that runs the API with Docker, with its firewall and role
rds.tf	RDS	The PostgreSQL database, reachable only from the machine
ecr.tf	ECR	The registry the pipeline pushes the API image to
s3.tf	S3	The bucket for the CSV exports

Plus providers.tf, variables.tf, outputs.tf and the machine's boot script, user-data.sh.

Pipeline: Actions → Infrastructure → Run workflow.

• apply creates the registry, pushes the image tagged with the commit, creates the rest and waits until the API answers. About 9 minutes.
• destroy removes all 12 resources, data included. About 3 minutes. Only the bucket that holds the Terraform state is left.

About 0.03 USD per hour while it exists. See terraform/README.md.

Full Changelog: v2.1.0...v3.0.0

v2.4.0 — four services

Four AWS services, one file each: the simplest infrastructure that runs the API, still created, deployed and destroyed by Terraform from the pipeline.

File	Service	What it is for
ec2.tf	EC2	One machine that runs the API with Docker, with its firewall and role
rds.tf	RDS	The PostgreSQL database, reachable only from the machine
ecr.tf	ECR	The registry the pipeline pushes the API image to
s3.tf	S3	The bucket for the CSV exports

• Parameter Store and CloudWatch Logs are gone: the machine gets the database URL in its boot script.
• Each service keeps its own firewall and permissions in its own file.
• 12 resources, about 0.03 USD per hour while it exists.
• Measured from the pipeline: about 9 minutes to deploy and install the API, about 3 to destroy.

See terraform/README.md to run it.

v2.3.0 — one machine

The simplest way to run the API on AWS: one virtual machine with Docker, still created, deployed and destroyed by Terraform from the pipeline.

Terraform

• The API runs on a single EC2 t3.micro with Amazon Linux 2023. On first boot it installs Docker, pulls the image from ECR, seeds the database and starts the API on port 80.
• ECS, the load balancer and their roles are gone: 15 resources instead of 21, and about 0.03 USD per hour instead of 0.07.
• The machine reads the database URL from SSM Parameter Store at boot, so the password never appears in its user data.
• One role for the machine, scoped to its image, its parameter, its bucket and its log group, plus Session Manager for a shell without SSH keys.
• A new image means a new machine: Terraform replaces it and leaves the database untouched.

Pipeline

• The seeding step is gone: the machine does it on boot.
• The smoke test now waits for the machine to boot and the API to answer /health/ready.
• Measured: about 8 minutes to deploy, about 3 to destroy.

See terraform/README.md to run it.

v2.2.0 — infrastructure as code

The infrastructure-as-code release: everything the API needs on AWS is created, deployed and destroyed by Terraform, from a pipeline.

Terraform

• terraform/ holds one file per service: network rules, ECR, RDS PostgreSQL, SSM Parameter Store, S3, IAM, CloudWatch Logs, an Application Load Balancer and ECS Fargate.
• The database password is generated by Terraform and reaches the container from an encrypted parameter, never through the task definition.
• The state lives in S3 with native locking, so a deploy from the pipeline can be destroyed from anywhere.
• terraform destroy leaves nothing behind: the registry, the bucket and the database are allowed to go with their contents.

Pipeline

• Infrastructure workflow, run on demand with apply or destroy.
• apply builds the API image tagged with the commit, deploys it, waits until it is healthy, seeds the database with a one-off task and smoke-tests /health/ready.
• Measured: about 9 minutes to deploy, about 6 to destroy.

See terraform/README.md to run it.

v2.1.0 — multicloud

The multicloud release: the inventory API runs on AWS, talks to its partner API on Google Cloud through an orchestrator, and is observable end to end.

Cross-cloud integration

• GET /v2/items/:id adds a partner block with a record fetched live from the other cloud (ok, unavailable or disabled; never an error).
• GET /v2/interop/random hands one item to the other cloud in a domain-agnostic contract.
• POST /v2/interop/messages is this API's step in the flow: appends an item to the message and stores the accumulated JSON in object storage. ?itemId= picks the item, so a live change shows up in the next message.
• Partner records are cached for 30 s, failures never are, and DELETE /v2/interop/cache forces a refresh.
• Every route under /v2 also answers under /api/v2.
• X-Correlation-Id is honoured, generated when missing, returned and logged.

Kubernetes

• Liveness and readiness probes, requests and limits, ConfigMap and Secret.
• Horizontal autoscaling from 2 to 6 replicas on CPU.
• A PodDisruptionBudget and do-not-disrupt, so node consolidation no longer takes both replicas down at once.

Observability

• Prometheus metrics: request rate, errors and latency by route pattern, cache hits and misses, cross-cloud lookups.
• OpenTelemetry, injected by the cluster rather than bundled, sent through a collector to both AWS X-Ray and Grafana Cloud Tempo.
• Every request log line carries the trace id; every span carries the correlation id.
• A Grafana Cloud dashboard and alert, templated in observability/.

Deployment

• deploy/ manifests as templates; filled-in copies stay out of the repository.