build: consolidate reproducible builder scripts#583
Merged
Conversation
Move pin-packages.sh, config-qemu.sh, and common build functions into a single build/shared/ directory, eliminating divergent copies across gateway, kms, and verifier builders. Key changes: - Create build/shared/build-lib.sh with shared functions (ensure_buildkit, docker_build, extract_packages, sync_shared_scripts, check_clean_tree) - Fix pin-packages.sh: use printf instead of echo for correct newline handling, proper quoting, use HTTP for snapshot.debian.org (APT does its own GPG verification) - Unify snapshot date to 20260317T000000Z across all services - Rewrite all build-image.sh scripts to source shared build-lib.sh - Add DSTACK_SRC_URL ARG to gateway Dockerfile (was hardcoded) - Fix kms GIT_REV handling to use Dockerfile ARG instead of host file - Rename kms-pinned-packages.txt to builder-pinned-packages.txt for consistency across services - Regenerate all pinned-packages.txt files with unified snapshot date - Fix verifier's corrupt builder-pinned-packages.txt (was 435 lines of '=')
The CI workflows use docker/build-push-action directly (not build-image.sh), so shared scripts must be copied into each service's shared/ directory before building. Also update REUSE.toml for the renamed kms pinned-packages file, and remove the obsolete .GIT_REV file creation from kms workflows (now handled via Dockerfile ARG).
Instead of gitignoring the per-service copies and copying them at build time, track them in git so `docker build` works standalone without build-image.sh. - Remove .gitignore files, track pin-packages.sh and config-qemu.sh in each service's shared/ directory - build-image.sh now verifies copies match build/shared/ (not copies) - CI: add check-shared-scripts job that fails if copies are out of sync - CI: remove manual copy steps (no longer needed) - CI: remove obsolete .GIT_REV file creation from kms workflows - CI: add DSTACK_SRC_URL build-arg to gateway workflows
After each Docker image build, extract installed packages with dpkg -l and diff against the committed pinned-packages.txt files. Fails CI if Dockerfile changes cause package drift without regenerating the pinned lists. This catches the scenario where someone modifies a Dockerfile (e.g., adds a new apt package or updates the base image) but forgets to run build-image.sh to regenerate pinned-packages.txt.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Instead of copying pin-packages.sh and config-qemu.sh into each service's shared/ directory, use BuildKit's named build contexts (--build-context build-shared=build/shared) so Dockerfiles can directly COPY --from=build-shared. This means: - Scripts exist only in build/shared/ (single source of truth) - No copies to track, no sync checks needed - docker build works standalone with: --build-context build-shared=build/shared - build-image.sh passes it automatically via build-lib.sh - Each service's shared/ dir contains only pinned-packages data files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pin-packages.sh,config-qemu.sh, and common build functions intobuild/shared/, eliminating 5 divergent copies across gateway, kms, and verifier builderspin-packages.sh: useprintfinstead ofechofor correct newline handling (gateway/kms had a bug), use HTTP for snapshot.debian.org (APT does GPG verification independently)20260317T000000Zand regenerate all pinned-packages.txt filesbuild/shared/build-lib.shwith shared functions (ensure_buildkit,docker_build,extract_packages, etc.)kms-pinned-packages.txt→builder-pinned-packages.txtfor consistencybuilder-pinned-packages.txt(was 435 lines of bare=)DSTACK_SRC_URLARG to gateway Dockerfile (was hardcoded)Test plan
build-image.shruns successfully, pinned packages generated (477 builder + 127 runtime)build-image.shruns successfully, pinned packages generated (477 builder + 236 qemu)build-image.shruns successfully, pinned packages generated (435 builder + 236 qemu + 108 runtime)