Skip to content

build: complete warm-build hashes.txt coverage (bzImage, tools.cpio, initrd.cpio.xz FORCE; busybox symlinks fix; doc cleanup) - #2179

Merged
tlaurion merged 4 commits into
linuxboot:masterfrom
tlaurion:fix-doc-verification
Aug 5, 2026
Merged

build: complete warm-build hashes.txt coverage (bzImage, tools.cpio, initrd.cpio.xz FORCE; busybox symlinks fix; doc cleanup)#2179
tlaurion merged 4 commits into
linuxboot:masterfrom
tlaurion:fix-doc-verification

Conversation

@tlaurion

@tlaurion tlaurion commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Supersedes #2178. Completes the warm-build hashes.txt completeness work started in #2174. Fixes emergency bug introduced by #2174.

Commit 1: build: improve clean target @echo messages and fix reproducibility doc

Makefile clean targets

  • Rewrite real.gitclean / gitclean_keep_packages / gitclean_keep_packages_and_build @echo messages to name the git clean -fxd command and describe exact preserve rules
  • Tighten real.remove_canary_files-extract_patch_rebuild_what_changed @echo messages: make clear this target deletes .canary stamps and the re-extract/rebuild cascade happens on the next make invocation

doc/reproducible-builds.md verification section

  • Split verification into "Same commit: everything should match" vs "Different commits: ROMs always differ" (GIT_HASH in /etc/config causes expected cascade)
  • Remove stale real.gitclean_keep_packages prerequisite -- after Fix build reproducibility regression from PR #2136 (musl-cross-make binutils bump) #2174 FORCE fixes, warm builds produce complete hashes.txt
  • Remove incorrect warm-build partial-hashes.txt claim
  • Add CI hashes.txt download reference with CircleCI artifact URL example
  • Add clean working tree prerequisite (dirty tree sets GIT_STATUS=dirty, changing ROM output)
  • Restore bzImage in same-commit step-down path
  • Upgrade comprehensive diff from hash-only to hash+path comparison

Commit 2: modules/linux: add FORCE and UNCHANGED guard to bzImage rule; fix busybox symlinks

bzImage FORCE + UNCHANGED

  • Add FORCE to bzImage copy/hash rule prerequisite -- matching the modules.cpio fix in Fix build reproducibility regression from PR #2136 (musl-cross-make binutils bump) #2174 -- so warm builds always re-run the recipe that records the kernel hash
  • Replace direct do-copy-to-target with cmp-based UNCHANGED pattern: copy to .tmp, compare, mv only when changed, print UNCHANGED when identical, always append hash/size to hashes.txt
  • When unchanged, preserve mtime to prevent cascading .bundled rebuild

Emergency fix: busybox symlinks broken by #2174

  • Fix build reproducibility regression from PR #2136 (musl-cross-make binutils bump) #2174 replaced make install with applets/install.sh --symlinks to avoid re-linking without SOURCE_DATE_EPOCH. However, install.sh reads symlink names from busybox.links, which was not regenerated and is missing after cache restore or make clean. Result: only the bare busybox binary was installed; all 137 applet symlinks ([, ash, cat, cp, ls, ...) were absent from tools.cpio, breaking the initrd.
  • Fix: regenerate busybox.links via applets/busybox.mkll before calling install.sh, using HOSTCC=gcc (needed outside busybox's make context) and include/autoconf.h (busybox 1.36.1 uses this, not config.h), with a test -s guard so empty generation fails the build loudly.

Verification (commit c4c59764b64)

Built locally with ./docker_repro.sh make BOARD=EOL_t480-hotp-maximized (warm build, clean GIT_STATUS).

137 busybox symlinks now present in tools.cpio:

$ cpio -itv < tools.cpio | grep -c -- "-> busybox"
137
$ cpio -itv < tools.cpio | grep -- "-> busybox" | head -5
lrwx------   0 root    0 root    7 Dec 31  1969 bin/[ -> busybox
lrwx------   0 root    0 root    7 Dec 31  1969 bin/[[ -> busybox
lrwx------   0 root    0 root    7 Dec 31  1969 bin/ash -> busybox
lrwx------   0 root    0 root    7 Dec 31  1969 bin/cat -> busybox
lrwx------   0 root    0 root    7 Dec 31  1969 bin/cp -> busybox

Local ROM hash: [pending]
CI cross-check: [pending CircleCI pipeline for c4c59764b64]

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the ROM reproducibility verification documentation to reflect recent Makefile target renames and to clarify what can (and cannot) be expected to match when comparing builds from the same vs different commits.

Changes:

  • Removes stale prerequisite guidance (including the renamed real.gitclean_keep_packages target reference and the outdated warm-build hashes.txt claim).
  • Splits verification guidance into Same commit (ROM should match) vs Different commits (ROM expected to differ due to embedded GIT_HASH).
  • Rephrases the troubleshooting flow to emphasize drilling down from ROM → initrd/bzImage → tools.cpio → individual files.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread doc/reproducible-builds.md Outdated
Comment thread doc/reproducible-builds.md Outdated
@tlaurion
tlaurion force-pushed the fix-doc-verification branch 2 times, most recently from f4cc4d9 to ee03b11 Compare August 5, 2026 15:04
@tlaurion tlaurion changed the title doc/reproducible-builds.md: fix verification section build: improve clean target @echo messages and fix reproducibility doc Aug 5, 2026
@tlaurion
tlaurion requested a lite review from Copilot August 5, 2026 15:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (1)

doc/reproducible-builds.md:120

  • Wording: “Forged example URL” can read as recommending tampering; this looks like a placeholder/example format. Consider renaming to a neutral description so readers don’t misinterpret it.
  Forged example URL:

@tlaurion
tlaurion force-pushed the fix-doc-verification branch 4 times, most recently from dc5c17d to fa50b2e Compare August 5, 2026 16:59
- Rewrite real.gitclean / gitclean_keep_packages / gitclean_keep_packages_and_build
  @echo messages to name the git clean command and describe exact preserve rules
- Tighten real.remove_canary_files-extract_patch_rebuild_what_changed @echo messages:
  make clear that this target deletes .canary stamps and the re-extract/rebuild
  happens on the next make invocation
- Split reproducibility doc verification into Same/Different commit sections
- Remove stale warm-build partial-hashes.txt claim (FORCE fixes make it obsolete)
- Add CI hashes.txt download reference with example URL
- Add clean working tree prerequisite
- Simplify comprehensive diff command (hash+path comparison, drop awk)
- Restore bzImage in step-down path

Signed-off-by: Thierry Laurion <insurgo@riseup.net>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

modules/linux:209

  • cmp --quiet "$@.tmp" "$@" will print an error on the first build because $@ doesn’t exist yet. Even though the ! cmp condition will take the "changed" branch, the stderr noise is avoidable by explicitly handling the missing-target case.
	@if ! cmp --quiet "$@.tmp" "$@" ; then \
		mv "$@.tmp" "$@" ; \
		touch "$@" ; \
	else \
		echo "$(DATE) UNCHANGED $(@:$(pwd)/%=%)" ; \

modules/linux:205

  • Using do-copy with $@.tmp is problematic here: do-copy runs cmp "$1" "$2" before copying and prints sha256/stat for the destination. Since $@.tmp usually doesn’t exist yet, this will emit cmp: ... No such file noise and log hash/size lines for the temporary file rather than the real target.

This issue also appears on line 205 of the same file.

$(build)/$(BOARD)/$(LINUX_IMAGE_FILE): $(build)/$(linux_dir)/.build FORCE
	$(call do-copy,$(dir $<)/$(linux_output),$@.tmp)
	@if ! cmp --quiet "$@.tmp" "$@" ; then \

@tlaurion
tlaurion force-pushed the fix-doc-verification branch from 45cd771 to c4c5976 Compare August 5, 2026 18:15
…lete hashes.txt

- Add FORCE to bzImage copy/hash rule prerequisite -- matching the
  modules.cpio fix in linuxboot#2174 -- so warm builds always re-run the recipe
  that records the kernel hash
- Replace direct do-copy-to-target with cmp-based UNCHANGED pattern
  matching the modules.cpio precedent: copy to .tmp, compare, mv only
  when changed, print UNCHANGED when identical, always append hash/size
  to hashes.txt
- When unchanged, preserve mtime to prevent cascading .bundled rebuild

This completes the work started in PR linuxboot#2174 (modules.cpio FORCE fix).

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
…ll change

PR linuxboot#2174 replaced 'make install' with 'applets/install.sh --symlinks'
to avoid re-linking without SOURCE_DATE_EPOCH.  However, install.sh
reads symlink names from busybox.links, which was not regenerated and
can be missing after cache restore or make clean.  Result: only the
bare busybox binary was installed; all 137 applet symlinks were absent
from tools.cpio, breaking the initrd.

Fixes:
- Regenerate busybox.links via applets/busybox.mkll before install.sh
- Use include/autoconf.h (busybox 1.36.1 uses this, not config.h)
- Add busybox_hostcc ?= gcc and pass HOSTCC to mkll (needed outside
  busybox's make context where HOSTCC is undefined)
- Add test -s guard so empty link-list generation fails the build
  loudly instead of silently producing no symlinks

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
@tlaurion
tlaurion force-pushed the fix-doc-verification branch from c4c5976 to 0bd856e Compare August 5, 2026 18:17
…hes.txt

tools.cpio and initrd.cpio.xz were the last two hashes.txt writers
without FORCE.  On warm builds, their recipes were skipped, leaving
their entries (and all per-file hashes inside them) missing from
hashes.txt.  CI build comparison would show spurious diffs where
the local side simply lacked entries, not where content diverged.

- tools.cpio: add FORCE to prerequisites; recipe uses do-cpio (does
  not reference $^), so FORCE is safe
- initrd.cpio.xz: add FORCE to prerequisites; recipe uses $^ in
  cpio-clean.pl input, so filter to $(filter-out FORCE,$^) to
  avoid passing FORCE as a filename argument

With this, all cpio archives (modules, tools, heads, data, board),
bzImage, and initrd.cpio.xz now have FORCE, completing the warm-build
hashes.txt completeness work across every rule that writes to it.

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
@tlaurion
tlaurion requested a lite review from Copilot August 5, 2026 18:38
@tlaurion tlaurion changed the title build: improve clean target @echo messages and fix reproducibility doc build: complete warm-build hashes.txt coverage (bzImage, tools.cpio, initrd.cpio.xz FORCE; busybox symlinks fix; doc cleanup) Aug 5, 2026
@tlaurion tlaurion added this to the reproduciblebuilds milestone Aug 5, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

modules/linux:205

  • The bzImage rule copies to $@.tmp via do-copy, but do-copy always prints sha256/stat output for its destination. That makes the build log show hash/size for the temporary file (and it may be deleted on the UNCHANGED path), which is misleading noise in addition to the later tee -a $(HASHES)/$(SIZES) lines for $@. Prefer a plain cp -a (or a helper that doesn’t emit hashes) for the temporary copy.
$(build)/$(BOARD)/$(LINUX_IMAGE_FILE): $(build)/$(linux_dir)/.build FORCE
	$(call do-copy,$(dir $<)/$(linux_output),$@.tmp)
	@if ! cmp --quiet "$@.tmp" "$@" ; then \

@tlaurion
tlaurion marked this pull request as ready for review August 5, 2026 18:53
@tlaurion

tlaurion commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

pushing as bugfix otherwise busybox missing symlinks causes brick

@tlaurion
tlaurion merged commit 47c1952 into linuxboot:master Aug 5, 2026
5 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants