Skip to content

Fix the red API deploy, halve the image, and smoke-test before traffic - #10821

Merged
MarkusNeusinger merged 1 commit into
mainfrom
slim-api-image-and-smoke
Sep 1, 2026
Merged

Fix the red API deploy, halve the image, and smoke-test before traffic#10821
MarkusNeusinger merged 1 commit into
mainfrom
slim-api-image-and-smoke

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Summary

Three things in the API's build and deploy path. The first one is urgent.

1. The API deploy has been failing since 2026-08-30

#10813 added a build-time assertion on features.check('raqm'), on the understanding that the locked Pillow 12.3.0 manylinux wheel bundles libraqm. It does not — the wheel dlopen()s libraqm at runtime. In a bare python:3.13-slim:

pillow 12.3.0
raqm      : False
freetype2 : True
harfbuzz    version: None
fribidi     version: None

So the assertion has failed every build of this image since it landed. The deploy-api trigger build 1c2c5b92 of 2026-08-30 21:31 died on exactly that line, and anyplot-api:latest is still the image built on 2026-08-28 — nothing merged since has shipped.

Installing the Debian libraqm0 package (32 KB plus its HarfBuzz/FriBiDi dependencies) turns the check True, verified in the built image. That does not just unblock the build: it makes #10813's actual regression go away, rather than merely guarding against it.

The assertion also moves to the runtime stage, where it checks the image that actually serves. In the builder stage it would pass on a venv whose runtime never received the library — precisely the false green the guard exists to prevent.

2. The image is built in two stages

The single-stage version produced a 1.62 GB image (502 MB compressed). Two layers were ballast: build-essential, which never compiled anything because all 108 packages this image installs ship wheels (the two sdist-only entries in uv.lock, esprima and matplotlib-venn, belong to the plotting extra, which this image does not install), and a chown -R appuser:appuser /app after the venv was in place, which rewrites every touched file into a fresh layer and so duplicated the whole environment.

uncompressed
old (single-stage, currently serving) 1.62 GB
new (multi-stage) 693 MB

3. The deploy smoke-tests a candidate revision before it takes traffic

Until now this pipeline deployed straight onto live traffic, so a bad image served users until someone noticed. It now deploys with --no-traffic --tag=candidate and a deterministic --revision-suffix, probes that revision on its tag URL, and only then shifts traffic to exactly the revision it smoked — never --to-latest, which could promote a concurrent build's unsmoked revision.

Probes: /health, /libraries, /languages, /plots/filter (the one that takes require_db, so it fails when Cloud SQL is unreachable — /libraries and /languages fall back to static metadata via optional_db and would not notice), and /debug/status, which must answer 401 from the fail-closed admin gate.

Adopted from the sibling repo kurrentschrift, which has had this net from the start. Its step chain and this one are now identical apart from kurrentschrift's migrate job.

Test plan

  • Built in Cloud Build (e60104fe) — succeeds without build-essential.
  • features.check('raqm') → True in the built image.
  • Root cause isolated in a standalone probe (47f8bf05): the same wheel gives raqm: False in bare python:3.13-slim and raqm: True with libraqm0 installed.
  • Regression pinned to the old Dockerfile, not to this change: trigger build 1c2c5b92 (30.08., single-stage) fails on the identical line.
  • In the built image: heavy wheels import, api.main imports, uvicorn runs, the user is appuser, plots/ and core/ are present.
  • api/cloudbuild.yaml parses; step chain build → push → deploy → smoke → promote → get-url.
  • The smoke step itself first runs on merge — it cannot be exercised from a branch, since it probes a deployed candidate revision.

Notes

The size change affects Artifact Registry growth per deploy and rollout time, not user-facing latency — min-instances 1 already covers the cold start.

CI note: this diff touches no Python, so CI: Lint and CI: Tests skip themselves by design. Cloud Build is the only real gate, which is part of why the smoke step is in here.

@MarkusNeusinger @claude
Fix the red API deploy, halve the image, and smoke-test before traffic
95a50c2 
Three things that all live in the API's build and deploy path.

1. The deploy has been failing since 2026-08-30. PR #10813 added a
   build-time assertion on features.check('raqm'), on the understanding
   that the locked Pillow 12.3.0 manylinux wheel bundles libraqm. It does
   not — the wheel dlopen()s libraqm at runtime, so in a bare
   python:3.13-slim the check is False and HarfBuzz and FriBiDi report no
   version at all. The assertion therefore failed every build of this
   image: the deploy-api trigger build of 2026-08-30 21:31 died on
   exactly that line, and the serving revision is still the one built on
   2026-08-28, so nothing merged since has shipped. Installing the Debian
   libraqm0 package (32 KB plus its HarfBuzz/FriBiDi dependencies) turns
   the check True — verified in the built image. That also makes #10813's
   actual regression go away rather than merely guarding it.

   The assertion moves to the runtime stage, where it checks the image
   that serves. In the builder stage it would pass on a venv whose
   runtime never received the library — precisely the false green the
   guard exists to prevent.

2. The image is built in two stages. The single-stage version produced a
   1.62 GB image, of which two layers were ballast: build-essential,
   which never compiled anything because all 108 packages this image
   installs ship wheels, and a `chown -R appuser:appuser /app` after the
   venv was in place, which rewrites every touched file into a fresh
   layer and so duplicated the whole environment. Installing in a builder
   stage and copying the finished venv with `COPY --chown` gives 693 MB,
   measured in Cloud Build against the serving image.

3. The deploy smoke-tests a candidate revision before it takes traffic.
   Until now this pipeline deployed straight onto live traffic, so a bad
   image served users until someone noticed. It now deploys with
   --no-traffic --tag=candidate and a deterministic --revision-suffix,
   probes that revision on its tag URL, and only then shifts traffic to
   exactly the revision it smoked — never --to-latest, which could
   promote a concurrent build's unsmoked revision. Adopted from the
   sibling repo kurrentschrift, which has had this net from the start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSuSdGbMwpFkewFTEoDfq8
Copilot AI lite review requested due to automatic review settings September 1, 2026 18:28
Copilot AI reviewed Sep 1, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes production deployment behavior (Cloud Build/Cloud Run smoke+promote) and introduces a new pre-traffic gate whose reliability depends on builder-image tooling assumptions.

Pull request overview

This PR fixes the API’s broken deploy pipeline by ensuring Pillow’s text shaping (libraqm) is actually available in the serving image, reduces the production API image size via a multi-stage Docker build, and adds a pre-traffic Cloud Run smoke-test/promote sequence so failed revisions don’t immediately serve users.

Changes:

  • Install libraqm0 in the runtime image and run the Pillow RAQM guard in the runtime stage.
  • Convert api/Dockerfile to a multi-stage build and avoid a venv-duplicating chown -R layer.
  • Update api/cloudbuild.yaml to deploy with --no-traffic + candidate tag, smoke-test the tagged revision, then promote that exact revision.
File summaries
File Description
CHANGELOG.md Adds Unreleased entries documenting the deploy fix, image size reduction, and new smoke-test promotion flow.
api/Dockerfile Switches to multi-stage build; installs libraqm0 in runtime; copies venv with COPY --chown; moves RAQM assert to runtime stage.
api/cloudbuild.yaml Deploys no-traffic candidate revision, smokes key endpoints (including DB path + admin gate), then promotes the smoked revision.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread api/cloudbuild.yaml
@MarkusNeusinger
MarkusNeusinger merged commit df2dbe1 into main Sep 1, 2026
10 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the slim-api-image-and-smoke branch September 1, 2026 18:49
This was referenced Sep 2, 2026
Merged
Merged
MarkusNeusinger added a commit that referenced this pull request Sep 2, 2026
@MarkusNeusinger @claude
ci: build the API image and smoke the container before merge (#11205)
e842261 
## Why

The first build attempt of a changed `api/Dockerfile` happened in
**Cloud Build — after the merge**. That is not a hypothetical: the
`deploy-api` trigger sat red from 2026-08-30 until #10821, and every PR
check was green the whole time, because nothing in CI ever built or ran
the image. The sibling repo kurrentschrift added the same job for the
same class of miss (its `pyproject.toml` fell out of the runtime stage,
so the API would have reported version `0.0.0` in production and no
check could have seen it). Transferred here per the sibling rule; keep
the two jobs in the same shape.

## What the new `ci-image.yml` does

A single `image` job, in the shape `ci-lint.yml` / `ci-tests.yml`
already use:

1. **Change detection.** Builds only when `api/**`, `core/**`,
`pyproject.toml`, `uv.lock`, `README.md`, a root `.dockerignore`,
`app/Dockerfile` or the workflow itself changed. `plots/**` is
deliberately excluded — the automated plot pipeline opens hundreds of
PRs that touch nothing the image serves, and each would otherwise pay
for a container build. Verified against a sample file list:
`plots/…/plot.py`, `app/src/App.tsx`, `app/README.md` and `docs/*.md` do
not match; `api/main.py`, `core/images.py`, `pyproject.toml`, `uv.lock`,
`README.md`, `.dockerignore`, `app/Dockerfile` and
`.github/workflows/ci-image.yml` do. One `case` branch per event, and
each **fails closed**: a PR or merge-group diff that cannot be computed
stops the job rather than reading as "nothing changed", and a push diffs
its **whole range** (`github.event.before..after`, so an earlier commit
touching `api/` under an unrelated tip commit still counts) with a new
branch or an unreachable force-pushed tip building rather than skipping.
All four branches simulated against a throwaway repo — see Evidence.
2. **Build** with `docker/build-push-action`, `push: false` / `load:
true` (nothing reaches a registry — Cloud Build still owns the published
image; `load` puts the result in the local daemon so it can actually be
run) and `cache-from`/`cache-to: type=gha`.
3. **DB-free container smoke.** `docker run` with no database and no
secrets — `api/main.py` guards its DB init with `is_db_configured()`, so
a bare container boots. Then:
- `/health` answers `"healthy"` within a 90 s readiness window (this
image imports matplotlib, scikit-learn, statsmodels and the MCP server
first).
- **The version assert.** `/health`'s `version` must equal
`pyproject.toml`'s. `api/version.py` reads the *installed
distribution's* metadata and falls back to `0.0.0+unknown` when it is
absent — silently, in a field `/health`, `/openapi.json` and the MCP
server all report. The builder stage installs the project from a context
that has `pyproject.toml` and `uv.lock` but no source yet, so that
dist-info is a genuinely fragile artefact of the stage ordering and
nothing else in CI looks at it.
- **The COPY list.** `test -f /app/api/static/og-image.png` — the one
payload the runtime stage must ship that no import would catch.
`og_images.py` reads it off disk as the last resort when dynamic OG
rendering fails, i.e. exactly in a fresh container that cannot reach the
font bucket.
- **Non-root.** `id -u` must be 1000. `USER appuser` is one line above
the `CMD`; nothing else notices if a rebuild drops it, and Cloud Run
runs whatever the image says.
   - Container logs are dumped on failure.
4. **Two hadolint steps**, threshold `warning` rather than a
non-blocking run, so a warning of any other code blocks — which is the
point of having the linter. The three exceptions are named at their
line. Note honestly what `ignore` is: hadolint applies it file-wide, so
a *second* DL3013/DL3008/DL3025 elsewhere in `api/Dockerfile` is
suppressed too. Line-scoping needs `# hadolint ignore=<code>` comments
in the Dockerfile itself — the better home, and the named follow-up
below, but a Dockerfile edit is not this change's to make.

## Evidence

Both runs on this PR, `Build API image and smoke the container`:

| | build step | whole job |
|---|---|---|
| cold (no GHA layer cache) — [run
33683067699](https://github.com/MarkusNeusinger/anyplot/actions/runs/33683067699)
| 1 m 59 s | 2 m 25 s |
| warm (cache from the run above) — [run
33683725363](https://github.com/MarkusNeusinger/anyplot/actions/runs/33683725363)
| 32 s | 1 m 01 s |

The job runs in parallel with the other CI workflows, so the pipeline's
wall clock is unchanged; it only adds runner minutes, and only on PRs
that touch the image.

Smoke output, verbatim:

```
health OK
version OK: 3.2.0
COPY list OK
non-root OK: uid 1000
```

(Container answered `/health` about 6 s after `docker run`; `3.2.0` is
`pyproject.toml`'s `project.version`.)

Hadolint was reproduced locally against **2.15.1**, the version
`hadolint-action@v3.5.0` pins:

| Dockerfile | threshold `warning`, no ignores | with the ignores in
this PR |
|---|---|---|
| `api/Dockerfile` | `DL3013` (line 25), `DL3008` (47), `DL3025` (90) →
exit 1; `DL3066` (80) at info | exit 0 |
| `app/Dockerfile` | exit 0 | — (no ignores needed; clean even at
`--failure-threshold info`) |

The three ignores and why each is a deliberate choice, not a
suppression:

- `DL3013` — `pip install uv` unpinned. uv is the *installer*; the
versions that matter are pinned in `uv.lock`, which the very next line
honours with `uv sync --frozen`.
- `DL3008` — unpinned apt `curl` / `libraqm0`. Pinning a Debian point
release breaks the build on every security update of the base image.
- `DL3025` — shell-form `HEALTHCHECK CMD … || exit 1`. The fallback
needs a shell; JSON form cannot express it.

`DL3066` (non-numeric `USER`) also fires but only at info level, so it
stays visible in the log without blocking — `useradd -u 1000` already
gives the user a fixed uid.

**Change detection simulated** against a throwaway repo, all four
branches:

| case | result |
|---|---|
| push, 2 commits, only the *first* touches `api/`, tip is unrelated |
`should_build=true` (the old `HEAD~1..HEAD` form saw only
`unrelated.txt`) |
| push, new branch (`before` = zeroes) | build, with a warning |
| push, force-pushed / unreachable `before` | build, with a warning |
| `pull_request` with an unreachable base sha | **exit 1**,
`::error::…refusing to decide…` |
| push, nothing image-relevant anywhere in the range |
`should_build=false` |

`actionlint` 1.7.7 on the new file: clean (exit 0). `bash -n` on the
extracted change-detection script: clean.

## Decisions taken here (routine, flagged for override)

- **A third `ci-*.yml` rather than a job inside `ci-tests.yml`.** The
repo already separates CI concerns file by file (`ci-lint`, `ci-tests`,
`bot-serving-check`, `notify-deployment`, `sync-postgres`), and a docker
build has a different runtime profile and a different skip condition
than the Python test job. This extends the existing layout rather than
changing it.
- **Event data reaches the change-detection shell through `env:`, not
`${{ }}` inside the script.** Same logic as the sibling files, one notch
safer; a shell that never sees interpolated event data cannot be made to
execute it.
- **The non-root assert** is not in kurrentschrift's version. It is
three lines, guards a property nothing else covers, and belongs to the
same question the COPY assert asks ("is the image still what the
Dockerfile says").
- **New `uses:` are pinned by commit SHA** of the current major tag, as
everything else in `.github/workflows/` is; all three SHAs were verified
against the upstream tag lists (`docker/setup-buildx-action` v4.3.0 =
v4, `docker/build-push-action` v7.3.0 = v7, `hadolint/hadolint-action`
v3.5.0). Dependabot's `github-actions` ecosystem keeps them current.

## Findings for the author — not changed here, `api/` is out of this
PR's scope

1. **`api/.dockerignore` is dead.** The build context is the repo root
(`-f api/Dockerfile .`), so Docker looks for `/.dockerignore`, which
does not exist. Proof: `api/.dockerignore` excludes `*.md`, yet the
builder's `COPY pyproject.toml uv.lock README.md ./` succeeds — it could
not if that file were in effect. Consequence: every build ships the full
repo (~210 MB incl. `.git` and `plots/`) as context.
2. **The three hadolint exceptions belong in `api/Dockerfile` as `#
hadolint ignore=<code>` comments**, one line above `RUN pip install uv`,
the `apt-get install`, and the `HEALTHCHECK`. That makes them
line-scoped, so a *new* occurrence of the same code elsewhere blocks —
which the workflow-level `ignore` cannot do. Three comment lines; left
out only because `api/` is not this PR's to touch.
3. **`COPY plots/ ./plots/` in the runtime stage looks like dead
weight** (~98 MB). Nothing under `api/` or `core/` reads the directory
at runtime — implementations are served from Postgres
(`sync-postgres.yml`). If that holds, dropping it would take about a
fifth off the image. I did not touch it: it is a behaviour change in a
file this PR only lints.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01PBQdMbboxo59sSThGSbfke

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
MarkusNeusinger added a commit that referenced this pull request Sep 2, 2026
@MarkusNeusinger @claude
infra(app): deploy the frontend through a smoked candidate revision (#…
511a52d 
…11207)

## Why

`api/cloudbuild.yaml` already deploys through a candidate revision
(`--no-traffic` → smoke → `update-traffic`, #10821).
`app/cloudbuild.yaml` did not — it deployed straight onto live traffic.

That is backwards, because the app service is the one carrying the whole
crawler path in `app/nginx.conf`: the `$is_bot` map, the `location =`
bypasses for robots/llms/sitemap/og, the `@seo_proxy` upstream. That is
the very file whose breakage served **every bot an HTTP 502 for four
weeks** (2026-06-12 → 2026-07-09, a `proxy_ssl_verify_depth` default
against a 4-deep Let's Encrypt chain) while humans, Plausible and CI all
saw a healthy site. Until now a typo in it went live unchecked and the
daily `bot-serving-check.yml` was the only net — a night later.

The sibling repo kurrentschrift carries the same chain in *its*
`app/cloudbuild.yaml`; transferred here per the sibling rule.

## Every changed line

**Header comment (new).** Records the shared candidate-rollout pattern
(and the two steps where the app and API chains differ) and why this
service earns the same care as the API.

**`build-image` step** — unchanged args, gains `id: 'build-image'`. Both
tags are still built; only the *push* of `:latest` moves.

**Push step** — was `push --all-tags …/anyplot-app`, now pushes only
`…:$BUILD_ID`, with `id: 'push-image'` / `waitFor: ['build-image']`. A
`--all-tags` push before the rollout means `:latest` names an image that
may never have served a request. `:latest` gets its own step at the
bottom, behind `promote`.

**`deploy` step** — every existing flag (`--memory 512Mi`, `--cpu 1`,
`--timeout 60`, `--min-instances 0`, `--max-instances 3`, `--port 8080`,
gen2, `--cpu-throttling`, `--concurrency 15`, `--allow-unauthenticated`)
is **unchanged**, including the scale-to-zero decision from #10812.
Three flags are added:
- `--no-traffic` — the revision goes up serving nobody.
- `--tag=candidate` — gives it a stable tag URL to probe.
- `--revision-suffix=b$BUILD_ID` — a deterministic revision name so
`promote` can target *exactly* the revision this build smoked, never a
concurrent build's newer one. The `b` prefix is required: Cloud Run
wants the suffix to start with a lowercase letter and `$BUILD_ID` is a
UUID that usually starts with a digit.
Plus `id: 'deploy'` / `waitFor: ['push-image']`.

**`smoke` step (new).** Resolves the `candidate` tag's URL *and* asserts
the tag points at this build's revision — **before and again after** the
probes. `candidate` is a shared tag, so a concurrent build could move it
mid-smoke and this build would then have smoked someone else's revision
while promoting its own; a competing build only ever tags its OWN
revision and never ours back, so seeing our revision at both ends means
every probe in between hit it. Between them, six probes, every one
retried (`--retry 5 --retry-delay 5 --retry-all-errors`) because at
`min-instances 0` the candidate is *always* cold and the crawler probes
additionally wait on the first upstream call to `api.anyplot.ai`:

| UA | path | assertion | what it proves |
|---|---|---|---|
| browser | `/` | `<div id="root">` | humans still get the SPA shell |
| Googlebot | `/` | `<link rel="canonical" href="https://anyplot.ai/"
/>` | the `$is_bot` → `@seo_proxy` hop ran |
| Googlebot | `/scatter-basic` |
`…href="https://anyplot.ai/scatter-basic" />` | the request URI reached
the upstream |
| Googlebot | `/robots.txt` | `User-agent: Bytespider` | served by the
`location =` bypass, not the proxy |
| Googlebot | `/llms.txt` | `# anyplot` | same |
| Googlebot | `/llms.txt` | content-type carries `charset=utf-8` | its
em dashes don't decode as Latin-1 mojibake |

The **canonical link is the prerender marker**. anyplot has no committed
prerender files and no marker comment, but the SPA shell
(`app/index.html`) carries no `<link rel="canonical">` at all, while
every `api/routers/seo.py` page emits `href="https://anyplot.ai{route}"`
— so one `grep -F` proves both "this is the prerendered page" and "it is
the right route". Verified live on all twelve current bot routes (see
below).

**`promote` step (new).** `gcloud run services update-traffic
--to-revisions=anyplot-app-b$BUILD_ID=100`, never `--to-latest` (which
could promote a concurrent build's unsmoked revision).

**`push-latest` step (new).** `:latest` moves only after this build
promoted, so the tag can no longer name an image that was never rolled
out (which the old `--all-tags` push before the deploy did). It is not a
cross-build guarantee — two overlapping deploys still race for the tag —
which is fine, because neither pipeline *reads* `:latest`: both deploy
`:$BUILD_ID`.

**`timeout: '1200s'` (new).** deploy → cold-start smoke → promote does
not fit the default 10 min with a retrying probe. Matches
`api/cloudbuild.yaml`.

`images:` and `options:` are unchanged.

## Evidence

**The smoke script was expanded through the Cloud Build substitution
rules** (`${_VAR}`/`$BUILTIN` replaced, `$$` → literal `$` applied last
so an escape can never be re-read as a substitution), then
syntax-checked and executed:

- `bash -n` on the expanded script: clean.
- A scan of the whole file for substitutions Cloud Build cannot resolve
(after removing `$$` escapes): **none**. This matters — `$request_uri`
appears in a comment *inside* the script and had to be written
`$$request_uri`; an unescaped one fails the build before a single step
runs, and YAML comments (which is where `$is_bot` sits) are stripped
before substitution and are safe.
- The expanded assertions, run against the **live Cloud Run origin** of
`anyplot-app` (reads only, no deploy, no writes):

```
smoke against candidate https://anyplot-app-r3tvmejsmq-ez.a.run.app (live-origin)
OK: /
OK: /
OK: /scatter-basic
OK: /robots.txt
OK: /llms.txt
smoke OK
```

- **Negative control** — the same script with the deep-route probe sent
as a *browser* UA (i.e. served the SPA shell, which is exactly the
regression this smoke exists to catch):

```
a crawler did not get the prerendered page for this route (/scatter-basic is missing: <link rel="canonical" href="https://anyplot.ai/scatter-basic" />)
EXIT=1
```

- The canonical marker was confirmed against every static bot route at
the origin — `/`, `/plots`, `/specs`, `/libraries`, `/legal`, `/mcp`,
`/about`, `/palette`, `/map`, `/stats`, `/scatter-basic`,
`/scatter-basic/python/matplotlib` — all `200`, all with
`href="https://anyplot.ai{route}"`, none with `<div id="root">`.

### What the tag check does *not* prove

`status.traffic` is the **control-plane tag assignment**, not proof of
which revision answered a given request. `candidate` is reused across
builds, so a probe issued while a re-assignment is still propagating can
reach the previous revision. Closing that needs either a build-unique
tag (tag URLs then accumulate on the service without bound) or a build
id the app serves in its own response (an `app/Dockerfile` + Vite
change). Neither is taken here; both the smoke step's comment and
`agentic/docs/project-guide.md` now say so. The residual is narrow and
its worst case is mild — it needs the *previous* candidate to pass every
probe too, so the failure would be "promoted a revision we believed we
smoked", not "shipped a page we know is broken", and it is still
strictly better than the no-smoke chain this replaces.

## Decisions taken here (routine, flagged for override)

- **Each probe fetches to a file instead of piping into `grep -q`.**
`grep -q` exits at the first match and SIGPIPEs curl, so the pipe form
(which `api/cloudbuild.yaml` and the kurrentschrift original both use)
passes *only* because `bash -ceu` carries no `pipefail`. Adding
`pipefail` — a normal hardening instinct — would red every deploy. I hit
exactly that while testing. The file form is correct under any shell
options, checks curl's own exit status, and lets each failure name which
probe failed and what it expected. This is the one place the file
deliberately does not mirror `api/cloudbuild.yaml` byte for byte; happy
to align if you'd rather have strict parity.
- **The canonical link rather than a title as the crawler marker.** A
title is copy and goes stale (anyplot's `bot-serving-check.yml` has a
comment recording ten red nights from exactly that); the canonical is
generated from the route and cannot drift.
- **A deep route in addition to the home page.** Both go through the
same `location /` mechanism, so the deep route is not about coverage of
the map — it is the only probe that would notice `$request_uri` failing
to reach the upstream.
- **`api/cloudbuild.yaml` untouched.** Its `push-latest` still waits on
`build-image` rather than `promote`, so the API's `:latest` can still
name an image that never served. Left for the author / the parallel
`api/` work rather than taken here — see below.

## Findings for the author

1. **`api/cloudbuild.yaml`'s `push-latest` sits behind `build-image`,
not `promote`.** Same one-line fix as here (`waitFor: ['promote']`); I
stayed out of that file to avoid colliding with the parallel work in
`api/`.
1. **`api/cloudbuild.yaml` has the same shared-tag race this PR just
closed** — it resolves `candidate` once before its probes and never
re-checks, so two overlapping API builds can promote a revision that was
not the one smoked. The fix is the four lines added here; same reason
for not applying it in that file. `agentic/docs/project-guide.md` now
records the asymmetry explicitly rather than claiming the guarantee for
both pipelines.
2. This PR does not run the chain — a Cloud Build run is a production
deploy. The assertions are proven against the live origin, but the first
*real* exercise of `--no-traffic`/`--tag=candidate`/`update-traffic` on
the `anyplot-app` service happens on the first deploy after merge. Worth
watching that one build.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01PBQdMbboxo59sSThGSbfke

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MarkusNeusinger