Pin the third-party control-plane images by digest - #439
Merged
Conversation
The two third-party control-plane images and the hosted Caddy build's two upstream bases were all named by mutable tag. `caddy:2-alpine` is rebuilt upstream whenever its base is patched, so two builds of the same malmo commit weeks apart could hold different Caddy bytes with nothing recording which. This is not a live-box risk (a box docker-loads the baked tarballs and never pulls) but it is a build-reproducibility and supply-chain gap, and the opposite of what the catalog already does for every app image. All four are now pinned in dev/control-plane/images.lock, one checked-in file the Makefile includes and the cloud staging script sources. The file being in git is the record: `git show v0.4.0:dev/control-plane/images.lock` answers "which Caddy was in v0.4.0?" from a version number alone. control-plane-images pulls by digest, then re-tags to the plain tag before docker save. That step is load-bearing: a box loads the tarball offline and the control-plane compose looks the image up by tag, so a tarball saved under a digest reference would leave the compose naming an image that is not there. The acmedns Dockerfile takes both bases as build args with no default. A default would be a second copy of the pin, free to drift, and a bare `docker build` would then quietly bake unpinned bytes; with no default it fails instead. `make caddy-acmedns-image` feeds them from the pin file and stage-control-plane.sh calls that target. The guard test fails if a pin loses its digest, or if a pinned tag stops matching the four places that name the image by tag - drift there is what breaks a boot, since the tarball would load under one tag and the control plane ask for another.
Confidence Score: 5/5The PR appears safe to merge; no actionable new issue remains, and the previously reported documentation error is resolved. The current build flow pulls or builds from digest-qualified image references, re-tags only for the required offline lookup contract, and tests alignment with runtime consumers. The documentation now consistently explains that cloud staging reaches the pins through the Makefile rather than sourcing the lock file.
|
| Filename | Overview |
|---|---|
| Makefile | Loads the shared image pins, preserves tag-based offline lookup, and provides a pinned hosted-Caddy build target. |
| dev/cloud/stage-control-plane.sh | Routes hosted Caddy construction through the Makefile target that supplies digest-pinned bases. |
| dev/control-plane/caddy-acmedns/Dockerfile | Replaces mutable base references with required build arguments supplied by the pinned build target. |
| dev/control-plane/images.lock | Records four third-party control-plane image references using multi-architecture SHA-256 index digests. |
| internal/hostagent/controlplane/imagepins_test.go | Guards digest syntax and verifies that saved image tags remain aligned with all offline runtime consumers. |
| docs/progress/third-party-image-pins.md | Accurately documents the pinning flow and now identifies the Makefile as the lock file’s only reader. |
Reviews (2): Last reviewed commit: "Say who actually reads the pin file" | Re-trigger Greptile
Greptile on #439: the progress entry and the index row said dev/cloud/stage-control-plane.sh sources images.lock. It does not — it calls a make target, and the Makefile is the only reader. Corrected there, in the file's own header, in BUILD.md # 5c and in the guard test's comment. The no-spaces rule stays, but its reason is make's include; being shell-sourceable is a spare property, not a live consumer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #432.
The two third-party control-plane images (
caddy:2-alpine,tecnativa/docker-socket-proxy:v0.4.2) and the hosted Caddy build's two upstream bases were all named by mutable tag, so two builds of the same malmo commit weeks apart could hold different bytes with nothing recording which. Not a live-box risk — a boxdocker loads the baked tarballs and never pulls — but a build-reproducibility and supply-chain gap, and the opposite of what the catalog already does for every app image (APP_LIFECYCLE.md# Locked: image digest pinning).Step 1 — verifying the issue's claims
All of them hold. Two corrections worth recording:
dev/cloud/stage-control-plane.sh, notdev/cloud/test/bootstrap.shas the issue said — the build moved when staging was shared between the two cloud lanes in Production hosted cloud image (make build-cloud-image) has no first-boot wiring — boots network-less, no seed/host-agent/control-plane (surfaced at cloud#6 CL6 live run) #242.internal/hostagent/brainlaunch/brainlaunch.go: there is no pull path for either image, only adocker loadwhen the image is absent. So "not a live-box risk" is confirmed in code, not assumed.What changed
dev/control-plane/images.lock(new) — fourNAME=name:tag@sha256:...pins. TheMakefileincludes it and is the only reader — everything else that builds one of these images goes through a make target,stage-control-plane.shincluded. Lines are plainNAME=valuewith no spaces, which is whatincludeaccepts and also keeps the filesource-able if a script ever needs a pin without going through make. Multi-arch index digests, so a pin does not assume an architecture. Recording is the file itself:git show v0.4.0:dev/control-plane/images.lockanswers "which Caddy was in v0.4.0?" from a version number alone. The digests stay out of the release manifest, which is about the two images an update can move.Makefile—control-plane-imagespulls by digest then re-tags to the plain tag beforedocker save. That is load-bearing, not cosmetic: a box loads the tarball offline anddev/control-plane/compose.ymlnames the image by tag, so a tarball saved under a digest reference would leave the compose naming an image that is not there. Newcaddy-acmedns-imagetarget.dev/control-plane/caddy-acmedns/Dockerfile— both bases are build args with no default. A default would be a second copy of the pin, free to drift, and a baredocker buildwould then quietly bake unpinned bytes.# check=skip=InvalidDefaultArgInFromsilences the BuildKit hint about exactly that intended state.dev/cloud/stage-control-plane.sh— callsmake caddy-acmedns-imageinstead of runningdocker builditself. It already shells out tomake control-plane-imagesa few lines above, so this is the same seam.internal/hostagent/controlplane/imagepins_test.go(new) — fails if a pin loses its digest, or if a pinned tag stops matching the four places that name the image by tag (compose.yml, the two systemd drop-ins, host-agent'sMALMO_PROXY_IMAGEdefault). Drift there is the failure that breaks a boot.Specs
BUILD.mdgains # 5c Third-party image pins — what is pinned, why a build-time pull still needs a digest, the save-by-tag step, where the digests are recorded, and a two-line bump recipe with upstream Caddy security releases named as the case that matters. Locked-decisions bullet added. Progress entry:docs/progress/third-party-image-pins.md.Tested
make caddy-acmedns-imagebuilds against the pinned bases;caddy list-modulesin the result still listsdns.providers.acmedns, so the build-arg rewrite did not lose the module.docker rmi+docker image prune -fin between: identical image IDs both times (caddy:2-alpine→af555904…, socket proxy →16bbd120…). The savedcaddy.tarcarriesRepoTags: ["caddy:2-alpine"], which is what a box looks up. This is the issue's "Done when".compose.yml.make test-nopam,make fmt-check,make openapi-check). Fullmake checkdoes not complete on my machine for two reasons that predate this branch and reproduce on a cleandev:go veton the PAM binding (C.RTLD_NEXT) and gitignoreddev/cloud/mkosi.tools/build residue that./...walks into. CI runs the full gate.New dependencies
None.
Known gaps
cmd/brain/Dockerfileandweb-ui/Dockerfilestill ride mutable bases. Out of scope here, and pinning them alone would not buy a reproducible image while both runapt-get/npm ciagainst live indexes. Left for its own issue.dev/docker-compose.yml(the inner-loop dev Caddy) stays on the bare tag — a developer-machine container that ships to nobody.--with github.com/caddy-dns/acmednsresolves the module at build time with no version constraint. The pin fixes the bases, which is what the issue asked for.