From 292aa9a81994bddf91bed62b4834f24a19e866af Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Sat, 1 Aug 2026 23:48:27 -0400 Subject: [PATCH 01/10] modules/musl-cross-make: fix cross-compiler reproducibility MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five changes to make musl-cross-make produce deterministic output: 1. Override BUILD triplet (x86_64-pc-linux-gnu) via config.mak to prevent config.guess from probing the Docker host kernel. Docker shares the host kernel across CI runners; different runners produce different build-system triplets. No prior reproducibility-motivated BUILD pinning found in other musl-cross-make deployments — this is Heads-original. 2. Pass -Wa,--no-pad-sections via CFLAGS to prevent gas from padding section ends to alignment boundaries (gas NEWS 2.27, 2016). No other project uses this flag for reproducibility; the need is specific to gas 2.44 behavior in musl-cross-make. 3. Pass --with-debug-prefix-map=$(pwd)=. to normalize the build directory path embedded in the cross-compiler's debug info. Same technique used by Buildroot (gcc.mk, BR2_REPRODUCIBLE), rust-musl-cross (config.mak), Debian dpkg (-ffile-prefix-map), and Yocto (DEBUG_PREFIX_MAP). Ref: https://reproducible-builds.org/docs/build-path/ 4. Pass --enable-compressed-debug-sections=no to disable zlib debug-section compression (non-deterministic output). Used by Rust CI (crosstool-ng), Microsoft Azure Linux, Fedora binutils.spec, Chromium OS, and Frida. Ref: https://reproducible-builds.org/docs/deterministic-build-systems/ 5. Export SOURCE_DATE_EPOCH from the pinned commit epoch to prevent __DATE__/__TIME__ embedding during the GCC build. Follows the Buildroot fakedate pattern and the spec: https://reproducible-builds.org/specs/source-date-epoch/ Signed-off-by: Thierry Laurion --- modules/musl-cross-make | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/modules/musl-cross-make b/modules/musl-cross-make index 027fc5a21..9c777fc0f 100644 --- a/modules/musl-cross-make +++ b/modules/musl-cross-make @@ -40,13 +40,26 @@ else # Force a full build of the cross compiler for target platform # No need to build i386 for x86 since coreboot uses its own compiler +# +# For reproducibility, config.mak is written with: +# BUILD = x86_64-pc-linux-gnu -- prevents config.guess from probing +# the Docker host kernel (different CI runners produce different triplets) +# -Wa,--no-pad-sections -- prevents gas from padding section ends, +# producing non-deterministic alignment between runs +# --with-debug-prefix-map=$(pwd)=. -- normalizes build paths in debug info +# --enable-compressed-debug-sections=no -- disables non-deterministic zlib +# debug-section compression +# SOURCE_DATE_EPOCH from the pinned commit epoch prevents __DATE__/__TIME__ +# embedding during the GCC build. musl-cross-make_configure := \ echo -e >> Makefile 'musl-target:' ; \ - echo -e >> Makefile '\t$$(MAKE) TARGET="$(MUSL_ARCH)-linux-musl" install' ; + echo -e >> Makefile '\t$$(MAKE) TARGET="$(MUSL_ARCH)-linux-musl" install' ; \ + echo -e 'BUILD = x86_64-pc-linux-gnu\nCOMMON_CONFIG += CFLAGS="-Wa,--no-pad-sections"\nCOMMON_CONFIG += --with-debug-prefix-map=$(pwd)=.\nBINUTILS_CONFIG += --enable-compressed-debug-sections=no' >> config.mak CROSS_PATH ?= $(pwd)/crossgcc/$(CONFIG_TARGET_ARCH) -musl-cross-make_target := \ +musl-cross-make_target = \ + SOURCE_DATE_EPOCH=$(shell cd $(build)/$(musl-cross-make_dir) && git log -1 --format=%ct 2>/dev/null || echo 0) \ OUTPUT="$(CROSS_PATH)" \ MAKE="$(MAKE)" \ $(MAKE_JOBS) \ From 9c964c59197f37549a5c9f1e90e07837d03df2bb Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Sat, 1 Aug 2026 23:48:44 -0400 Subject: [PATCH 02/10] busybox: reproducible linking and install busybox's scripts/trylink passes -Wl,--gc-sections to ld.bfd. In ld.bfd 2.44 (and through 2.47), the --gc-sections mark-phase traversal depends on the ASLR layout of the linking process. No upstream binutils fix exists. Per the SOURCE_DATE_EPOCH convention, its presence signals that a reproducible build is requested: https://reproducible-builds.org/specs/source-date-epoch/ Changes: - patches/busybox-1.36.1/0004-trylink-reproducible.patch: Patch scripts/trylink to skip -Wl,--gc-sections when SOURCE_DATE_EPOCH is set. - modules/busybox: Pass SOURCE_DATE_EPOCH=0 in busybox_target. Replace 'make install' with direct binary copy + install.sh symlink generation. busybox's install target depends on busybox_unstripped which is FORCE, so 'make install' can trigger a re-link with no SOURCE_DATE_EPOCH. Copying the already-built binary and running applets/install.sh --symlinks avoids this entirely. Follows the same pattern used by Buildroot's fakedate (2016): SDE gates nondeterministic behavior off. No prior art found for SDE-gated --gc-sections disabling; OpenWrt offers per-package gc-sections opt-out but for size reduction, not reproducibility. With the musl-cross-make fix, 67 of 68 tools in tools.cpio are byte-identical. This fixes the last one. Signed-off-by: Thierry Laurion --- modules/busybox | 25 ++++++++++++------- .../0004-trylink-reproducible.patch | 18 +++++++++++++ 2 files changed, 34 insertions(+), 9 deletions(-) create mode 100644 patches/busybox-1.36.1/0004-trylink-reproducible.patch diff --git a/modules/busybox b/modules/busybox index 8b9909da8..183985167 100644 --- a/modules/busybox +++ b/modules/busybox @@ -14,6 +14,17 @@ busybox_output := busybox busybox_target := \ $(CROSS_TOOLS) \ $(MAKE_JOBS) \ + SOURCE_DATE_EPOCH=0 \ + +# For reproducibility, busybox is linked without --gc-sections. +# SEE: patches/busybox-1.36.1/0004-trylink-reproducible.patch +# patches scripts/trylink to skip --gc-sections when SOURCE_DATE_EPOCH +# is set. ld.bfd's --gc-sections uses internal hash tables influenced +# by ASLR (no upstream binutils fix through 2.47). +# SOURCE_DATE_EPOCH=0 in busybox_target triggers the patch. +# The install rule copies the already-built binary and runs +# applets/install.sh directly, avoiding 'make install' which could +# trigger a re-link via the FORCE dependency on busybox_unstripped. # Install symlinks when the busybox program is installed @@ -22,15 +33,11 @@ initrd_bins += $(initrd_bin_dir)/busybox endif $(initrd_bin_dir)/busybox: $(build)/$(busybox_dir)/.build - $(call do,SYMLINK,bin/busybox,\ - $(MAKE) \ - -C $(build)/$(busybox_dir) \ - CC="$(heads_cc)" \ - CROSS_COMPILE="$(CROSS)" \ - CONFIG_PREFIX="$(initrd_bin_dir)/.." \ - install \ - | tee -a $(build)/log/busybox.log \ - $(VERBOSE_REDIRECT) \ + $(call do,INSTALL,bin/busybox,\ + cp $(build)/$(busybox_dir)/busybox \ + $(initrd_bin_dir)/busybox && \ + cd $(build)/$(busybox_dir) && \ + $(SHELL) applets/install.sh $(initrd_bin_dir)/.. --symlinks \ ) # Remove busybox's reboot symlink that conflicts with our custom script $(call do,RM-SYMLINKS,bin/busybox,\ diff --git a/patches/busybox-1.36.1/0004-trylink-reproducible.patch b/patches/busybox-1.36.1/0004-trylink-reproducible.patch new file mode 100644 index 000000000..9712463c1 --- /dev/null +++ b/patches/busybox-1.36.1/0004-trylink-reproducible.patch @@ -0,0 +1,18 @@ +diff -u --recursive clean/busybox-1.36.1/scripts/trylink busybox-1.36.1/scripts/trylink +--- clean/busybox-1.36.1/scripts/trylink ++++ busybox-1.36.1/scripts/trylink +@@ -126,6 +126,14 @@ + fi + fi + ++# For reproducible builds, SOURCE_DATE_EPOCH disables --gc-sections. ++# ld.bfd's --gc-sections mark-phase traversal uses internal hash tables ++# that are influenced by ASLR, making the output nondeterministic. ++if [ -n "${SOURCE_DATE_EPOCH:-}" ]; then ++ echo "SOURCE_DATE_EPOCH is set, disabling --gc-sections for reproducible build" ++ GC_SECTIONS="" ++fi ++ + # Sanitize lib list (dups, extra spaces etc) + LDLIBS=`echo "$LDLIBS" | xargs -n1 | sort | uniq | xargs` + From 1d68084623ad18cbc8d7751e6023ebd4a5dad97f Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Mon, 3 Aug 2026 13:28:29 -0400 Subject: [PATCH 03/10] modules/musl-cross-make: wire up pre-seeded component tarballs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add SOURCES = $(packages) to config.mak so musl-cross-make skips its internal downloads and uses pre-seeded tarballs from packages/. Add .sources sentinel target that calls fetch_musl_cross_make_archive.sh, and wire .configured to depend on .sources so tarballs are pre-seeded before the build starts. Add packages: target dependency on .sources so 'make packages' seeds the mirrors too. The dead GNU_SITE variable is cleaned up as a side effect — with SOURCES = $(packages), musl-cross-make's download infrastructure is skipped entirely. Signed-off-by: Thierry Laurion --- modules/musl-cross-make | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/modules/musl-cross-make b/modules/musl-cross-make index 9c777fc0f..0387a6a06 100644 --- a/modules/musl-cross-make +++ b/modules/musl-cross-make @@ -54,7 +54,7 @@ else musl-cross-make_configure := \ echo -e >> Makefile 'musl-target:' ; \ echo -e >> Makefile '\t$$(MAKE) TARGET="$(MUSL_ARCH)-linux-musl" install' ; \ - echo -e 'BUILD = x86_64-pc-linux-gnu\nCOMMON_CONFIG += CFLAGS="-Wa,--no-pad-sections"\nCOMMON_CONFIG += --with-debug-prefix-map=$(pwd)=.\nBINUTILS_CONFIG += --enable-compressed-debug-sections=no' >> config.mak + echo -e 'BUILD = x86_64-pc-linux-gnu\nCOMMON_CONFIG += CFLAGS="-Wa,--no-pad-sections"\nCOMMON_CONFIG += --with-debug-prefix-map=$(pwd)=.\nBINUTILS_CONFIG += --enable-compressed-debug-sections=no\nSOURCES = $(packages)' >> config.mak CROSS_PATH ?= $(pwd)/crossgcc/$(CONFIG_TARGET_ARCH) @@ -73,6 +73,22 @@ endif musl-cross-make_output := $(CROSS)gcc +# Pre-seed component tarballs into $(packages) via fetch_source_archive.sh +# (primary -> Purism mirror fallback), following the same pattern as +# coreboot's crossgcc packages. Tarballs are cached alongside other module +# tarballs and picked up by Purism's package mirror sync. +$(build)/$(musl-cross-make_dir)/.sources: $(build)/$(musl-cross-make_dir)/.canary + WGET="$(WGET)" bin/fetch_musl_cross_make_archive.sh \ + "$(build)/$(musl-cross-make_dir)" "$(packages)" + touch "$@" + +# 'make packages' seeds mirrors with these tarballs too +packages: $(build)/$(musl-cross-make_dir)/.sources + +# Run the fetch script before configure so tarballs land in packages/ +# before the build starts +$(build)/$(musl-cross-make_dir)/.configured: $(build)/$(musl-cross-make_dir)/.sources + ## Fake a target so that musl will force a header install by the ## Linux kernel sources. $(build)/$(musl-cross-make_dir)/.build: $(INSTALL)/include/linux/limits.h From 8f539dc69143840d5cacade94cd3d1d9e065cc17 Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Tue, 4 Aug 2026 15:23:21 -0400 Subject: [PATCH 04/10] bin/fetch_source_archive.sh: add --timeout/--tries/-4 and audit logging Add --timeout=30 --tries=3 -4 to wget: IPv4 preference avoids IPv6 timeouts, fast timeout avoids hanging on unresponsive mirrors (issue #2086). Log PRIMARY/CACHED/MIRROR_FALLBACK/MIRROR_USED to build/mirror_fallbacks.log with archive name for auditability. Signed-off-by: Thierry Laurion --- bin/fetch_source_archive.sh | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/bin/fetch_source_archive.sh b/bin/fetch_source_archive.sh index 125b479f2..9bd96d4f5 100755 --- a/bin/fetch_source_archive.sh +++ b/bin/fetch_source_archive.sh @@ -52,7 +52,12 @@ esac download() { local download_url download_url="$1" - if ! "$WGET" -O "$TMP_FILE" "$download_url"; then + # --timeout=30: bail fast on hung downloads (default 900s) + # --tries=3: retry transient failures + # -4: prefer IPv4, avoid IPv6 stalls (issue #2086) + if ! "$WGET" -O "$TMP_FILE" \ + --timeout=30 --tries=3 -4 \ + "$download_url"; then echo "Failed to download $download_url" >&2 elif ! echo "$DIGEST $TMP_FILE" | "$SHASUM" --check -; then echo "File from $download_url does not match expected digest" >&2 @@ -66,6 +71,8 @@ download() { # If the file exists already and the digest is correct, use the cached copy. if [ -f "$FILE" ] && (echo "$DIGEST $FILE" | "$SHASUM" --check -); then + echo "$(date -Iseconds) CACHED file=$(basename "$FILE")" \ + >>"${MIRROR_LOG:-build/mirror_fallbacks.log}" echo "File $FILE is already cached" >&2 exit 0 fi @@ -73,12 +80,17 @@ fi rm -f "$FILE" "$TMP_FILE" # Try the primary source -download "$URL" && exit 0 +if download "$URL"; then + echo "$(date -Iseconds) PRIMARY file=$(basename "$FILE") url=$URL" \ + >>"${MIRROR_LOG:-build/mirror_fallbacks.log}" + echo "Downloaded from primary: $URL" >&2 + exit 0 +fi # Log mirror fallback for developer awareness MIRROR_LOG="${MIRROR_LOG:-build/mirror_fallbacks.log}" mkdir -p "$(dirname "$MIRROR_LOG")" -echo "$(date -Iseconds) MIRROR_FALLBACK primary=$URL" >>"$MIRROR_LOG" +echo "$(date -Iseconds) MIRROR_FALLBACK file=$(basename "$FILE") primary=$URL" >>"$MIRROR_LOG" # Shuffle the mirrors so we try each equally readarray -t BACKUP_MIRRORS < <(shuf -e "${BACKUP_MIRRORS[@]}") From 911ab9862ef861e86781167a9793f95a4234265e Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Tue, 4 Aug 2026 15:23:27 -0400 Subject: [PATCH 05/10] bin/fetch_coreboot_crossgcc_archive.sh: switch to mirrors.kernel.org Replace ftpmirror.gnu.org with mirrors.kernel.org/gnu for all 5 GNU component URLs. ftpmirror.gnu.org is a redirector that frequently returns 502. Signed-off-by: Thierry Laurion --- bin/fetch_coreboot_crossgcc_archive.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/bin/fetch_coreboot_crossgcc_archive.sh b/bin/fetch_coreboot_crossgcc_archive.sh index f9ff04c49..1e520a4c0 100755 --- a/bin/fetch_coreboot_crossgcc_archive.sh +++ b/bin/fetch_coreboot_crossgcc_archive.sh @@ -82,19 +82,19 @@ PKG_BASENAME="$(basename "$PKG_CKSUM_FILE" .cksum)" # broke all the iasl links - coreboot 90753398). case "$PKG_NAME" in gmp) - PKG_BASEURL="https://ftpmirror.gnu.org/gmp/" + PKG_BASEURL="https://mirrors.kernel.org/gnu/gmp/" ;; mpfr) - PKG_BASEURL="https://ftpmirror.gnu.org/mpfr/" + PKG_BASEURL="https://mirrors.kernel.org/gnu/mpfr/" ;; mpc) - PKG_BASEURL="https://ftpmirror.gnu.org/mpc/" + PKG_BASEURL="https://mirrors.kernel.org/gnu/mpc/" ;; gcc) - PKG_BASEURL="https://ftpmirror.gnu.org/gcc/gcc-$(delete_prefix_suffix "$PKG_BASENAME" gcc- .tar.xz)/" + PKG_BASEURL="https://mirrors.kernel.org/gnu/gcc/gcc-$(delete_prefix_suffix "$PKG_BASENAME" gcc- .tar.xz)/" ;; binutils) - PKG_BASEURL="https://ftpmirror.gnu.org/binutils/" + PKG_BASEURL="https://mirrors.kernel.org/gnu/binutils/" ;; nasm) PKG_BASEURL="https://www.nasm.us/pub/nasm/releasebuilds/$(delete_prefix_suffix "$PKG_BASENAME" nasm- .tar.bz2)/" From 500878cf0bd175ada947d2750ec2a5ca5d87afaf Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Tue, 4 Aug 2026 15:23:33 -0400 Subject: [PATCH 06/10] modules/qrencode: update URL to Wayback Machine archive fukuchi.org has been offline. Use web.archive.org snapshot instead. Fixes #2085. Signed-off-by: Thierry Laurion --- modules/qrencode | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/qrencode b/modules/qrencode index fbfbb870c..e32185496 100644 --- a/modules/qrencode +++ b/modules/qrencode @@ -3,7 +3,7 @@ modules-$(CONFIG_QRENCODE) += qrencode qrencode_version := 3.4.4 qrencode_dir := qrencode-$(qrencode_version) qrencode_tar := qrencode-$(qrencode_version).tar.gz -qrencode_url := https://fukuchi.org/works/qrencode/$(qrencode_tar) +qrencode_url := https://web.archive.org/web/20240910005455/https://fukuchi.org/works/qrencode/$(qrencode_tar) qrencode_hash := e794e26a96019013c0e3665cb06b18992668f352c5553d0a553f5d144f7f2a72 qrencode_libraries := .libs/libqrencode.so.3 From f7a262e17daeb9451a20173f34d8232d3162f2a7 Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Tue, 4 Aug 2026 15:23:39 -0400 Subject: [PATCH 07/10] bin/fetch_musl_cross_make_archive.sh: pre-seed component tarballs Pre-seed musl-cross-make component tarballs (binutils, gcc, gmp, mpc, mpfr, musl, linux) into packages/ via fetch_source_archive.sh, following the same pattern as coreboot's crossgcc packages. - Reads component versions and archive names from the musl-cross-make source tree (Makefile defaults + hashes/*.sha1) - Downloads each tarball via fetch_source_archive.sh from mirrors.kernel.org with Purism mirror fallback - musl and Linux headers from non-GNU hosts (musl.libc.org, ftp.barfooze.de) Signed-off-by: Thierry Laurion --- bin/fetch_musl_cross_make_archive.sh | 108 +++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100755 bin/fetch_musl_cross_make_archive.sh diff --git a/bin/fetch_musl_cross_make_archive.sh b/bin/fetch_musl_cross_make_archive.sh new file mode 100755 index 000000000..66a91c967 --- /dev/null +++ b/bin/fetch_musl_cross_make_archive.sh @@ -0,0 +1,108 @@ +#! /usr/bin/env bash +set -eo pipefail + +# Pre-seed musl-cross-make's component tarballs into the packages directory +# so they are cached alongside all other module tarballs, picked up by +# Purism's package mirror sync, and available for CI cache layers. +# +# usage: +# $0 +# $0 --help +# +# Reads component versions and archive names from the musl-cross-make +# source tree (Makefile defaults + hashes/*.sha1), then downloads each +# tarball via fetch_source_archive.sh. +# Uses fetch_source_archive.sh, so the Purism mirrors are used +# as fallback and WGET can override the path to wget. + +usage() { + cat < + $0 --help + +Reads component versions and archive names from a musl-cross-make +source tree (Makefile defaults + hashes/*.sha1), then downloads each +tarball via fetch_source_archive.sh (primary -> Purism mirror fallback). + +Uses fetch_source_archive.sh, so the Purism mirrors are used +as fallback and WGET can override the path to wget. +USAGE_END +} + +if [ "$#" -lt 2 ]; then + usage + exit 1 +fi + +MCM_DIR="$(realpath "$1")" +PKGS_DIR="$(realpath "$2")" # ensure absolute paths +BIN_DIR="$(dirname "${BASH_SOURCE[0]}")" + +# Find a glob pattern that matches exactly one file, failing with a +# distinct message for no matches vs. multiple matches. +single() { + if [ "$#" -eq 1 ]; then + if [ -f "$1" ]; then + echo "$1" + return 0 + fi + else + echo "multiple unexpected matches for glob:" "$@" >&2 + exit 1 + fi + echo "$1: no matches" >&2 + exit 1 +} + +# Extract a variable from the musl-cross-make Makefile. +# Returns the value of KEY = , or empty string if not found. +make_var() { + grep -E "^$1[[:space:]]*=" "$MCM_DIR/Makefile" | head -1 \ + | sed "s/^$1[[:space:]]*=[[:space:]]*//" +} + +# Resolve the archive name for a component from its hashes file. +# The hashes directory contains files named -.tar.*.sha1 +# (glob tolerates both .tar.gz and .tar.xz suffixes). +# single() guarantees exactly one match; basename strips the .sha1 suffix +# to give the archive filename. +component_archive() { # $1 = prefix (gcc, binutils, ...), $2 = version + basename "$(single "$MCM_DIR/hashes/$1-$2.tar."*.sha1)" .sha1 +} + +# Download a tarball via fetch_source_archive.sh. +# Digests are SHA-1 (40 chars); fetch_source_archive.sh auto-detects that. +# fetch_source_archive.sh applies the Purism mirror fallback +# keyed on the basename of the destination file. +fetch() { # $1 = url_base, $2 = filename, $3 = sha1_digest + "$BIN_DIR/fetch_source_archive.sh" "$1$2" "$PKGS_DIR/$2" "$3" +} + +# GNU component versions come from the musl-cross-make Makefile defaults +# and feed component_archive() to resolve the archive filenames. +V_BINUTILS=$(make_var BINUTILS_VER) +V_GCC=$(make_var GCC_VER) +V_GMP=$(make_var GMP_VER) +V_MPC=$(make_var MPC_VER) +V_MPFR=$(make_var MPFR_VER) +V_MUSL=$(make_var MUSL_VER) +V_LINUX=$(make_var LINUX_VER) + +# SHA-1 digests are read from hashes/*.sha1 files (format: "digest filename"). +# cut extracts the first field. component_archive() maps version to filename. + +# GNU tarballs: use mirrors.kernel.org directly. +# ftpmirror.gnu.org is a redirector that frequently returns 502. +GNU_BASE="https://mirrors.kernel.org/gnu" + +# Fetch each component from its upstream source. +# binutils, gcc, gmp, mpc, mpfr come from GNU mirrors. +fetch "$GNU_BASE/binutils/" "$(component_archive binutils "$V_BINUTILS")" "$(cut -d' ' -f1 "$MCM_DIR/hashes/$(component_archive binutils "$V_BINUTILS").sha1")" +fetch "$GNU_BASE/gcc/gcc-$V_GCC/" "$(component_archive gcc "$V_GCC")" "$(cut -d' ' -f1 "$MCM_DIR/hashes/$(component_archive gcc "$V_GCC").sha1")" +fetch "$GNU_BASE/gmp/" "$(component_archive gmp "$V_GMP")" "$(cut -d' ' -f1 "$MCM_DIR/hashes/$(component_archive gmp "$V_GMP").sha1")" +fetch "$GNU_BASE/mpc/" "$(component_archive mpc "$V_MPC")" "$(cut -d' ' -f1 "$MCM_DIR/hashes/$(component_archive mpc "$V_MPC").sha1")" +fetch "$GNU_BASE/mpfr/" "$(component_archive mpfr "$V_MPFR")" "$(cut -d' ' -f1 "$MCM_DIR/hashes/$(component_archive mpfr "$V_MPFR").sha1")" +# musl and Linux headers come from non-GNU hosts. +fetch "https://musl.libc.org/releases/" "$(component_archive musl "$V_MUSL")" "$(cut -d' ' -f1 "$MCM_DIR/hashes/$(component_archive musl "$V_MUSL").sha1")" +fetch "https://ftp.barfooze.de/pub/sabotage/tarballs/" "$(component_archive linux "$V_LINUX")" "$(cut -d' ' -f1 "$MCM_DIR/hashes/$(component_archive linux "$V_LINUX").sha1")" From 7a0a3a3edee190462ebe5bfaf1a3a26e1614da18 Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Mon, 3 Aug 2026 16:37:24 -0400 Subject: [PATCH 08/10] modules/linux: add FORCE to modules.cpio for complete hashes.txt Without FORCE, warm builds skip modules.cpio entirely when the kernel .build stamp is up-to-date. The hashes.txt is reset at every make invocation but modules.cpio and kernel module hashes are never re-appended, producing a partial hashes.txt (422 hashes instead of 436). data.cpio, board.cpio, and tools.cpio already use FORCE so their do-cpio recipe always runs, prints UNCHANGED when unchanged, and appends hashes regardless. modules.cpio now matches. Signed-off-by: Thierry Laurion --- modules/linux | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/linux b/modules/linux index 206ff95f1..ee3c6fc0b 100644 --- a/modules/linux +++ b/modules/linux @@ -192,7 +192,7 @@ endef $(call map,linux_module,$(linux_modules-y)) # We can't rebuild the module initrd until the kernel has been rebuilt -$(build)/$(BOARD)/modules.cpio: $(build)/$(linux_dir)/.build +$(build)/$(BOARD)/modules.cpio: $(build)/$(linux_dir)/.build FORCE $(call do-cpio,$@,$(module_initrd_dir)) @$(RM) -rf "$(module_initrd_dir)" From db1ea2e3504be4d7576e445ad835b94219f57141 Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Tue, 4 Aug 2026 13:24:34 -0400 Subject: [PATCH 09/10] modules/gpg2: fix reproducibility -- disable tpm2d gpg2's configure probes for libtss2-esys via AC_SEARCH_LIBS (Esys_Initialize), and the result depends on parallel build order. When tpm2-tss was a dependency, the probe always succeeded but linked gpg-agent against libtss2-esys unnecessarily. With tpm2-tss removed from depends, the probe becomes non-deterministic between CI and local builds. --disable-tpm2d removes the entire TPM probe chain, eliminating the non-deterministic BUILD_WITH_TPM2D, HAVE_INTEL_TSS, HAVE_LIBTSS, and HAVE_LIBTSS2_ESYS defines from config.h. Signed-off-by: Thierry Laurion --- modules/gpg2 | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/gpg2 b/modules/gpg2 index b121febc8..65fafe8ce 100644 --- a/modules/gpg2 +++ b/modules/gpg2 @@ -24,6 +24,7 @@ gpg2_configure := \ --disable-bzip2 \ --disable-dirmngr \ --disable-doc \ + --disable-tpm2d \ --disable-exec \ --disable-gnutls \ --disable-gpgsm \ From b0dea02aab5efc2bc799c2405ad70623d0e30307 Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Tue, 4 Aug 2026 14:53:44 -0400 Subject: [PATCH 10/10] doc: add reproducible-builds.md, reorg index, add cross-references doc/reproducible-builds.md (new): documents all reproducibility mechanisms -- musl-cross-make flags, heads_cc, kernel build pins, prefix normalization, rpath removal, gpg2 --disable-tpm2d, coreboot BUILD_TIMELESS, busybox patches, version/timestamp pins, cpio-clean.pl, FORCE on modules.cpio, tarball downloads, and all six reproducibility patches. doc/index.md: reorganized into subject categories (Build System & CI, Architecture & Boot Flow, Security/TPM/Keys, Development & Reference) with reproducible-builds.md listed under Build System. doc/docker.md: cross-reference to reproducible-builds.md and unattended build instructions (script -qec + HEADS_DISABLE_USB=1). doc/architecture.md, doc/build-freshness.md: cross-references to reproducible-builds.md added. doc/modules.md: Toolchain Modules section documenting musl-cross-make MUSL_CROSS_ONCE guard and early include ordering. Signed-off-by: Thierry Laurion --- doc/architecture.md | 3 +- doc/build-freshness.md | 2 + doc/docker.md | 9 ++ doc/index.md | 61 ++++++++++--- doc/modules.md | 13 +++ doc/reproducible-builds.md | 176 +++++++++++++++++++++++++++++++++++++ 6 files changed, 250 insertions(+), 14 deletions(-) create mode 100644 doc/reproducible-builds.md diff --git a/doc/architecture.md b/doc/architecture.md index 69ce6649c..98d871917 100644 --- a/doc/architecture.md +++ b/doc/architecture.md @@ -113,7 +113,8 @@ The top-level `Makefile` orchestrates: 6. `data.cpio` — data files - Final ROM image: coreboot ROM with Linux + initramfs payload embedded -Reproducible builds are achieved via Nix-pinned Docker images. See [docker.md](docker.md). +Reproducible builds are achieved via Nix-pinned Docker images (see [docker.md](docker.md)) +and deterministic compiler flags (see [reproducible-builds.md](reproducible-builds.md)). The CI pipeline's workspace and cache behavior is documented in [circleci.md](circleci.md). diff --git a/doc/build-freshness.md b/doc/build-freshness.md index d4fc74b55..2e432348d 100644 --- a/doc/build-freshness.md +++ b/doc/build-freshness.md @@ -1,5 +1,7 @@ # Build Freshness Debugging Guide +See also: [reproducible-builds.md](reproducible-builds.md) for verifying ROM reproducibility. + ## The Problem Changes to source files in `initrd/` or other build dependencies were not being packed into `initrd.cpio.xz`, causing stale artifacts in the final ROM. The test system showed old commit hashes in `/tmp/config` even after rebuilding. diff --git a/doc/docker.md b/doc/docker.md index 00678e22e..a4f573145 100644 --- a/doc/docker.md +++ b/doc/docker.md @@ -5,6 +5,7 @@ environment. Docker images are built with Nix since [PR #1661](https://github.com/linuxboot/heads/pull/1661). See also: [General reproducible-build notes](../README.md#general-notes-on-reproducible-builds), +[Reproducible build practices](reproducible-builds.md), [QEMU testing](qemu.md), [CircleCI pipeline notes](circleci.md). --- @@ -235,6 +236,14 @@ USB token (for example `scdaemon` or `pcscd`). The wrapper will warn and, on int shells, give a **3-second abort window** before attempting to kill those processes to free the token. Set `HEADS_DISABLE_USB=1` to opt out of this automatic cleanup. +For fully unattended builds (script/non-interactive shell), combine with +`script` to provide the pseudo-TTY that docker_repro.sh's `-ti` requires: + + HEADS_DISABLE_USB=1 script -qec './docker_repro.sh make BOARD=...' /dev/null + +Both `HEADS_DISABLE_USB=1` and `script` are unnecessary when running from +an interactive terminal. + ```bash HEADS_DISABLE_USB=1 ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run ``` diff --git a/doc/index.md b/doc/index.md index 77f24795f..0be106bb1 100644 --- a/doc/index.md +++ b/doc/index.md @@ -2,18 +2,53 @@ Quick reference: read the relevant doc when working on a topic. +## Build System & CI + +| File | What it covers | +|------|----------------| +| `build-artifacts.md` | ROM filenames, update-package zip layout, LVFS conventions | +| `build-freshness.md` | Why rebuilds produce stale artifacts and how to force a full rebuild | +| `circleci.md` | CI pipeline: job dependency graph, cache layers, workspace persistence | +| `docker.md` | Docker-based build environment with pinned, reproducible images | +| `modules.md` | Module system: toolchain and bin modules, inclusion rules, sentinel chain | +| `patches.md` | Creating and maintaining source patches for upstream packages | +| `prerequisites.md` | Tools and libraries needed before building Heads | +| `reproducible-builds.md` | Deterministic build flags and verifying ROM hashes against CI | + +## Architecture & Boot Flow + +| File | What it covers | +|------|----------------| +| `architecture.md` | System layout: coreboot → Linux → initramfs, config hierarchy | +| `boot-process.md` | Boot flow stages, ISO boot steps, [OK]/[~]/[X] progress markers | +| `iso_boot.md` | ISO kernel parameters: which framework uses each option | +| `kexec_handoff.md` | Kernel kexec handoff: screen_info, EBDA, sysfb/simpledrm/vesadrm | + +## Security, TPM & Keys + +| File | What it covers | +|------|----------------| +| `configuring-keys.md` | Setting up GPG keys for signing firmware updates | +| `gpg.md` | GPG tool operation for firmware signing and verification | +| `hotp.md` | HOTP-based remote attestation of firmware state | +| `keys.md` | Key management for firmware signing | +| `security-model.md` | TPM measured boot, trust chain, flash write protection | +| `TPM_GPIO_Reset_Approaches.md` | Eight approaches for resetting TPM via GPIO | +| `TPM_GPIO_Reset_Vulnerability.md` | TPM GPIO reset vulnerability analysis | +| `tpm.md` | TPM 1.2 and 2.0 operation details | +| `wp-notes.md` | Flash write protection: PR0 chipset locking, WP# pin, runtime chain | + +## Development & Reference + | File | What it covers | |------|----------------| -| `architecture.md` | System architecture: coreboot -> kernel -> initrd, build system, config hierarchy | -| `boot-process.md` | Boot flow stages, ISO boot steps (1-7), [OK]/[~]/[X] marker legend | -| `busybox_perks.md` | GNU vs BusyBox command differences for all tools used in initrd scripts | -| `docker.md` | Docker-based build environment | -| `logging.md` | Log levels (STATUS, WARN, NOTE, INFO, DEBUG, TRACE) usage conventions | -| `modules.md` | Available tools: which are BusyBox applets vs standalone binaries | -| `security-model.md` | TPM, measured boot, trust chain, flash write protection | -| `wp-notes.md` | Flash write protection: PR0 chipset locking, WP# pin, config tables, runtime chain, board coverage | -| `tpm.md` | TPM 1.2 and 2.0 operations | -| `ux-patterns.md` | User interaction patterns (whiptail, CLI menu, confirm dialogs) | -| `iso_boot.md` | ISO boot parameter reference: what each kernel param does and which framework uses it | -| `kexec_handoff.md` | Kexec handoff: screen_info normalization (VLFB), EBDA preservation, sysfb/simpledrm/vesadrm dispatch, driver detection markers, kernel version matrix | -| `patches.md` | Patch creation conventions: naming, multi-patch directories, testing, splitting, forced rebuild after changes | +| `BOARDS_AND_TESTERS.md` | Board EOL/ESU status, CPU generations, tester registry | +| `busybox_perks.md` | GNU vs BusyBox command differences for initrd scripts | +| `config.md` | Board config hierarchy: defconfig, oldconfig, variation-to-defconfig | +| `development.md` | Development environment setup and contribution workflow | +| `faq.md` | Frequently asked questions | +| `logging.md` | Message levels (STATUS, WARN, NOTE, INFO, DEBUG, TRACE) | +| `qemu.md` | QEMU-based board emulation for testing | +| `recovery-shell.md` | Recovery shell usage and diagnostic commands | +| `ux-patterns.md` | User interaction: whiptail dialogs, CLI menus, confirmations | +| `variation-to-defconfig.md` | Converting Kconfig variation files to defconfig format | diff --git a/doc/modules.md b/doc/modules.md index 851d17ee6..d9731c932 100644 --- a/doc/modules.md +++ b/doc/modules.md @@ -256,3 +256,16 @@ The `define_module` function in `Makefile` expands these into the `.canary` → `.configured` → `.build` chain above. The package name is the Make target: `make BOARD=... kexec` builds just that package. ``` + +## Toolchain Modules + +### musl-cross-make + +The `MUSL_CROSS_ONCE` guard prevents `modules/musl-cross-make` from being +included multiple times. + +The cross-compiler is included **early** in the Makefile so +that `$(CROSS)` and `$(heads_cc)` are available before any userland module is +included. + +See `doc/circleci.md` for how CI orchestrates toolchain caching across jobs. diff --git a/doc/reproducible-builds.md b/doc/reproducible-builds.md new file mode 100644 index 000000000..3a2aabe67 --- /dev/null +++ b/doc/reproducible-builds.md @@ -0,0 +1,176 @@ +# Reproducible Builds + +See `doc/docker.md` for build-environment reproducibility. + +These practices follow the [reproducible-builds.org](https://reproducible-builds.org/) +project's documentation. Every mechanism below contributes to producing +bit-identical output across independent environments (CI and local). + +## Cross-compiler (musl-cross-make) + +`BUILD = x86_64-pc-linux-gnu` pins config.guess so different CI runners +produce the same build triplet instead of probing the Docker host kernel. +`-Wa,--no-pad-sections` prevents gas from padding section ends (non-deterministic +alignment). `--with-debug-prefix-map=$(pwd)=.` normalizes build paths in debug +info. `--enable-compressed-debug-sections=no` disables zlib debug-section +compression. `SOURCE_DATE_EPOCH` from the pinned musl-cross-make commit epoch +prevents `__DATE__`/`__TIME__` embedding during the GCC build. + +## Userland compiler flags + +`heads_cc` (Makefile) injects `-fdebug-prefix-map=$(pwd)=heads` and +`-gno-record-gcc-switches` for every userland module, normalizing build paths +and suppressing non-deterministic compiler flag recording in debug info. +`modules/libnitrokey` additionally uses `-ffile-prefix-map=$(pwd)=heads`. + +## Kernel + +`EXTRA_FLAGS` passes `-fdebug-prefix-map=$(pwd)=heads -gno-record-gcc-switches` +to the kernel build. `KBUILD_BUILD_USER` (pinned to the Linux config filename), +`KBUILD_BUILD_HOST=linuxboot`, `KBUILD_BUILD_TIMESTAMP="1970-00-00"`, and +`KBUILD_BUILD_VERSION=0` pin all kernel build-identity variables. Modules are +stripped with `strip --strip-debug --preserve-dates`. + +## Prefix normalization + +Most autotools-based modules use `--prefix "/"` or `--prefix ""` combined +with `DESTDIR="$(INSTALL)"` so generated Makefiles carry fixed paths; the +install target redirects output to the actual build tree. (`modules/bash` +is an exception, using `--prefix="/usr"`.) `modules/pciutils` additionally +sets `IDSDIR="/"` and `PREFIX="/"` so `libpci.so.3` is path-independent. + +## rpath removal + +`modules/gpg2` and `modules/cryptsetup2` use `--disable-rpath` to prevent +build paths from being embedded in binaries. `modules/tpm2-tss` and +`modules/tpm2-tools` use `sed` to rewrite libtool's `hardcode_libdir_flag_spec` +or `hardcode_into_libs` in generated configure scripts. `modules/cairo` +rewrites the same variables in the generated `libtool` file post-configure. +`modules/util-linux` removes `.la` libtool files post-install. + +## gpg2 + +`--disable-tpm2d` in `gpg2_configure` eliminates the entire TPM probe chain, +removing the non-deterministic `BUILD_WITH_TPM2D`/`HAVE_INTEL_TSS`/`HAVE_LIBTSS` +defines from `config.h`. + +## coreboot + +`BUILD_TIMELESS=1` is passed to coreboot's build to produce reproducible ROMs. + +## Busybox + +`SOURCE_DATE_EPOCH=0` in `busybox_target` triggers +`patches/busybox-1.36.1/0004-trylink-reproducible.patch`, which disables +ld.bfd `--gc-sections` (ASLR-influenced hash tables in binutils 2.44). +`patches/busybox-1.36.1/0001-messages.patch` replaces `AUTOCONF_TIMESTAMP` +with a fixed `"(heads)"` string. The install rule copies the binary and runs +`applets/install.sh` directly, avoiding `make install`'s FORCE re-link. + +## Patches for reproducibility + +| Patch | Mechanism | +|---|---| +| `bash-5.1.16.patch` | Drops `-b` from `mkversion.sh` — removes build timestamp from `version.h` | +| `busybox-1.36.1/0001-messages.patch` | Replaces `AUTOCONF_TIMESTAMP` with fixed `"(heads)"` | +| `busybox-1.36.1/0004-trylink-reproducible.patch` | Disables `--gc-sections` when `SOURCE_DATE_EPOCH` set | +| `coreboot-4.11/0073-build-race-condition-fixes.patch` | Fixes parallel-make race conditions (non-deterministic ordering) | +| `openssl-3.0.8.patch` | `SOURCE_DATE_EPOCH` replaces `time()` in `mkbuildinf.pl`; compiler flags replaced with fixed literal | +| `tpm2-tools-5.6.patch` | Disables `git describe --tags --dirty > VERSION` | + +## Version and timestamp pins + +| Module | Mechanism | +|---|---| +| `modules/hotp-verification` | `GITVERSION=""` — removes git-derived version string | +| `modules/tpm2-tools` | `echo version > ./VERSION` — pinned version file | +| `modules/bash` | `LDFLAGS="-s"`, `CFLAGS="-g0 -Os"` — strip symbols, no debug | +| `modules/fbwhiptail` | `LDFLAGS="-s"`, `CFLAGS="-g0 -Os"` — strip symbols, no debug | +| `modules/zstd` | `CFLAGS="-g0 -Os"` — no debug info | + +## Archive determinism + +`bin/cpio-clean.pl` rewrites every newc cpio entry for determinism: files +sorted by name, inodes derived from MD5(filename), timestamps/uid/gid zeroed, +nlink=0, devmajor/devminor=0, check=0, and 512-byte trailing padding. +`blobs/dev.cpio` is a pre-built, git-tracked archive providing a reproducible +`/dev/console`. + +`modules/linux` adds a `FORCE` dependency on `modules.cpio` so `hashes.txt` +is always complete on every rebuild, not just cold builds. The `do-cpio` +macro uses `cmp --quiet` to short-circuit identical output, avoiding +unnecessary rewrites. + +## Tarball downloads + +`bin/fetch_source_archive.sh` uses `--timeout=30 --tries=3 -4` (IPv4 +preference, fast wget timeout). All download attempts are logged to +`build/mirror_fallbacks.log`. `bin/fetch_musl_cross_make_archive.sh` +pre-seeds musl-cross-make component tarballs into `packages/` via +fetch_source_archive.sh. + +## Verifying ROM Reproducibility + +### Prerequisites +- Same git commit on both CI and local +- Build with `docker_repro.sh` locally (same Docker image as CI) +- For a complete `hashes.txt`, use `real.gitclean_keep_packages` first. + Warm (cached) builds produce partial `hashes.txt` — only rebuilt targets + re-append their hashes. + +### Understanding hashes.txt + +`build/$ARCH/$BOARD/hashes.txt` records the SHA256 of **every file inside every +cpio archive**, not just the cpio archives themselves. Each cpio section is +separated by `-----` lines: + +``` + /path/to/modules.cpio +----- + ./lib/modules/usbhid.ko + ./lib/modules/e1000e.ko +... +----- + /path/to/tools.cpio +----- + ./bin/busybox + ./bin/kexec +... +----- +``` + +### GIT_HASH in /etc/config + +`tools.cpio` contains `./etc/config`, which embeds `GIT_HASH` from `git rev-parse HEAD` +(in the `/etc/config` generation rule). Every commit changes `GIT_HASH`, so `./etc/config` differs +between ANY two commits. This cascades: `./etc/config` → `tools.cpio` → +`initrd.cpio.xz` → ROM. + +When `./etc/config` is the **only** differing file inside `tools.cpio`, the +build is still reproducible — all binaries (busybox, kexec, gpg, etc.) are +byte-identical. A binary mismatch (e.g. `./bin/busybox`) is the actual +reproducibility bug to investigate. + +### Steps + +1. Build locally and download CI `hashes.txt` for the same commit. + +2. Compare ROM hashes — if they match, the build is reproducible. Done. +```bash +grep '\.rom' /tmp/ci-hashes.txt build/x86/EOL_t480-hotp-maximized/hashes.txt +``` + +3. If the ROM differs, step down: `initrd.cpio.xz`/`bzImage` → `tools.cpio` → +individual files. The innermost differing file (e.g. `./bin/busybox`) is the +root cause — fix it and the cascade resolves. `hashes.txt` records every file +at every level, so no diffoscope is needed until you've identified what differs. + +4. For a comprehensive check: +```bash +diff <(grep '^[0-9a-f]\{64\}' /tmp/ci-hashes.txt | awk '{print $1}' | sort) \ + <(grep '^[0-9a-f]\{64\}' build/x86/EOL_t480-hotp-maximized/hashes.txt | awk '{print $1}' | sort) +``` + +`./etc/config` inside `tools.cpio` always differs between commits (contains +`GIT_HASH`). Its cascade through `tools.cpio`/`initrd.cpio.xz`/ROM is expected; +only a binary mismatch is a reproducibility bug.