Skip to content

Pin the third-party control-plane images by digest - #439

Merged
onel merged 2 commits into
devfrom
fix/432-pin-control-plane-images
Sep 4, 2026
Merged

Pin the third-party control-plane images by digest#439
onel merged 2 commits into
devfrom
fix/432-pin-control-plane-images

Conversation

@onel

@onel onel commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Closes #432.

The two third-party control-plane images (caddy:2-alpine, tecnativa/docker-socket-proxy:v0.4.2) and the hosted Caddy build's two upstream bases were all named by mutable tag, so two builds of the same malmo commit weeks apart could hold different bytes with nothing recording which. Not a live-box risk — a box docker loads the baked tarballs and never pulls — but a build-reproducibility and supply-chain gap, and the opposite of what the catalog already does for every app image (APP_LIFECYCLE.md # Locked: image digest pinning).

Step 1 — verifying the issue's claims

All of them hold. Two corrections worth recording:

What changed

  • dev/control-plane/images.lock (new) — four NAME=name:tag@sha256:... pins. The Makefile includes it and is the only reader — everything else that builds one of these images goes through a make target, stage-control-plane.sh included. Lines are plain NAME=value with no spaces, which is what include accepts and also keeps the file source-able if a script ever needs a pin without going through make. Multi-arch index digests, so a pin does not assume an architecture. Recording is the file itself: git show v0.4.0:dev/control-plane/images.lock answers "which Caddy was in v0.4.0?" from a version number alone. The digests stay out of the release manifest, which is about the two images an update can move.
  • Makefilecontrol-plane-images pulls by digest then re-tags to the plain tag before docker save. That is load-bearing, not cosmetic: a box loads the tarball offline and dev/control-plane/compose.yml names the image by tag, so a tarball saved under a digest reference would leave the compose naming an image that is not there. New caddy-acmedns-image target.
  • dev/control-plane/caddy-acmedns/Dockerfile — both bases are build args with no default. A default would be a second copy of the pin, free to drift, and a bare docker build would then quietly bake unpinned bytes. # check=skip=InvalidDefaultArgInFrom silences the BuildKit hint about exactly that intended state.
  • dev/cloud/stage-control-plane.sh — calls make caddy-acmedns-image instead of running docker build itself. It already shells out to make control-plane-images a few lines above, so this is the same seam.
  • internal/hostagent/controlplane/imagepins_test.go (new) — fails if a pin loses its digest, or if a pinned tag stops matching the four places that name the image by tag (compose.yml, the two systemd drop-ins, host-agent's MALMO_PROXY_IMAGE default). Drift there is the failure that breaks a boot.

Specs

BUILD.md gains # 5c Third-party image pins — what is pinned, why a build-time pull still needs a digest, the save-by-tag step, where the digests are recorded, and a two-line bump recipe with upstream Caddy security releases named as the case that matters. Locked-decisions bullet added. Progress entry: docs/progress/third-party-image-pins.md.

Tested

  • make caddy-acmedns-image builds against the pinned bases; caddy list-modules in the result still lists dns.providers.acmedns, so the build-arg rewrite did not lose the module.
  • Pull → re-tag twice with docker rmi + docker image prune -f in between: identical image IDs both times (caddy:2-alpineaf555904…, socket proxy → 16bbd120…). The saved caddy.tar carries RepoTags: ["caddy:2-alpine"], which is what a box looks up. This is the issue's "Done when".
  • The guard test fails as intended when a pin is edited down to a bare tag, and when a pinned tag stops matching compose.yml.
  • Go suite green (make test-nopam, make fmt-check, make openapi-check). Full make check does not complete on my machine for two reasons that predate this branch and reproduce on a clean dev: go vet on the PAM binding (C.RTLD_NEXT) and gitignored dev/cloud/mkosi.tools/ build residue that ./... walks into. CI runs the full gate.
  • No VM boot run. Nothing a box does changed — the bundle holds the same four images under the same four tags — so no image canary is bumped.

New dependencies

None.

Known gaps

  • cmd/brain/Dockerfile and web-ui/Dockerfile still ride mutable bases. Out of scope here, and pinning them alone would not buy a reproducible image while both run apt-get / npm ci against live indexes. Left for its own issue.
  • dev/docker-compose.yml (the inner-loop dev Caddy) stays on the bare tag — a developer-machine container that ships to nobody.
  • The xcaddy build is still not reproducible: --with github.com/caddy-dns/acmedns resolves the module at build time with no version constraint. The pin fixes the bases, which is what the issue asked for.

The two third-party control-plane images and the hosted Caddy build's two
upstream bases were all named by mutable tag. `caddy:2-alpine` is rebuilt
upstream whenever its base is patched, so two builds of the same malmo
commit weeks apart could hold different Caddy bytes with nothing recording
which. This is not a live-box risk (a box docker-loads the baked tarballs
and never pulls) but it is a build-reproducibility and supply-chain gap,
and the opposite of what the catalog already does for every app image.

All four are now pinned in dev/control-plane/images.lock, one checked-in
file the Makefile includes and the cloud staging script sources. The file
being in git is the record: `git show v0.4.0:dev/control-plane/images.lock`
answers "which Caddy was in v0.4.0?" from a version number alone.

control-plane-images pulls by digest, then re-tags to the plain tag before
docker save. That step is load-bearing: a box loads the tarball offline and
the control-plane compose looks the image up by tag, so a tarball saved
under a digest reference would leave the compose naming an image that is
not there.

The acmedns Dockerfile takes both bases as build args with no default. A
default would be a second copy of the pin, free to drift, and a bare
`docker build` would then quietly bake unpinned bytes; with no default it
fails instead. `make caddy-acmedns-image` feeds them from the pin file and
stage-control-plane.sh calls that target.

The guard test fails if a pin loses its digest, or if a pinned tag stops
matching the four places that name the image by tag - drift there is what
breaks a boot, since the tarball would load under one tag and the control
plane ask for another.
@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown

Confidence Score: 5/5

The PR appears safe to merge; no actionable new issue remains, and the previously reported documentation error is resolved.

The current build flow pulls or builds from digest-qualified image references, re-tags only for the required offline lookup contract, and tests alignment with runtime consumers. The documentation now consistently explains that cloud staging reaches the pins through the Makefile rather than sourcing the lock file.

Important Files Changed

Filename Overview
Makefile Loads the shared image pins, preserves tag-based offline lookup, and provides a pinned hosted-Caddy build target.
dev/cloud/stage-control-plane.sh Routes hosted Caddy construction through the Makefile target that supplies digest-pinned bases.
dev/control-plane/caddy-acmedns/Dockerfile Replaces mutable base references with required build arguments supplied by the pinned build target.
dev/control-plane/images.lock Records four third-party control-plane image references using multi-architecture SHA-256 index digests.
internal/hostagent/controlplane/imagepins_test.go Guards digest syntax and verifies that saved image tags remain aligned with all offline runtime consumers.
docs/progress/third-party-image-pins.md Accurately documents the pinning flow and now identifies the Makefile as the lock file’s only reader.

Reviews (2): Last reviewed commit: "Say who actually reads the pin file" | Re-trigger Greptile

Comment thread docs/progress/third-party-image-pins.md
Greptile on #439: the progress entry and the index row said
dev/cloud/stage-control-plane.sh sources images.lock. It does not — it
calls a make target, and the Makefile is the only reader. Corrected there,
in the file's own header, in BUILD.md # 5c and in the guard test's comment.
The no-spaces rule stays, but its reason is make's include; being
shell-sourceable is a spare property, not a live consumer.
@onel
onel merged commit c79e058 into dev Sep 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Third-party control-plane images are pulled by mutable tag and never recorded

1 participant