build: complete warm-build hashes.txt coverage (bzImage, tools.cpio, initrd.cpio.xz FORCE; busybox symlinks fix; doc cleanup) - #2179
Conversation
There was a problem hiding this comment.
Pull request overview
Updates the ROM reproducibility verification documentation to reflect recent Makefile target renames and to clarify what can (and cannot) be expected to match when comparing builds from the same vs different commits.
Changes:
- Removes stale prerequisite guidance (including the renamed
real.gitclean_keep_packagestarget reference and the outdated warm-buildhashes.txtclaim). - Splits verification guidance into Same commit (ROM should match) vs Different commits (ROM expected to differ due to embedded
GIT_HASH). - Rephrases the troubleshooting flow to emphasize drilling down from ROM → initrd/bzImage → tools.cpio → individual files.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
f4cc4d9 to
ee03b11
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (1)
doc/reproducible-builds.md:120
- Wording: “Forged example URL” can read as recommending tampering; this looks like a placeholder/example format. Consider renaming to a neutral description so readers don’t misinterpret it.
Forged example URL:
dc5c17d to
fa50b2e
Compare
- Rewrite real.gitclean / gitclean_keep_packages / gitclean_keep_packages_and_build @echo messages to name the git clean command and describe exact preserve rules - Tighten real.remove_canary_files-extract_patch_rebuild_what_changed @echo messages: make clear that this target deletes .canary stamps and the re-extract/rebuild happens on the next make invocation - Split reproducibility doc verification into Same/Different commit sections - Remove stale warm-build partial-hashes.txt claim (FORCE fixes make it obsolete) - Add CI hashes.txt download reference with example URL - Add clean working tree prerequisite - Simplify comprehensive diff command (hash+path comparison, drop awk) - Restore bzImage in step-down path Signed-off-by: Thierry Laurion <insurgo@riseup.net>
fa50b2e to
45cd771
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (2)
modules/linux:209
cmp --quiet "$@.tmp" "$@"will print an error on the first build because$@doesn’t exist yet. Even though the! cmpcondition will take the "changed" branch, the stderr noise is avoidable by explicitly handling the missing-target case.
@if ! cmp --quiet "$@.tmp" "$@" ; then \
mv "$@.tmp" "$@" ; \
touch "$@" ; \
else \
echo "$(DATE) UNCHANGED $(@:$(pwd)/%=%)" ; \
modules/linux:205
- Using
do-copywith$@.tmpis problematic here:do-copyrunscmp "$1" "$2"before copying and prints sha256/stat for the destination. Since$@.tmpusually doesn’t exist yet, this will emitcmp: ... No such filenoise and log hash/size lines for the temporary file rather than the real target.
This issue also appears on line 205 of the same file.
$(build)/$(BOARD)/$(LINUX_IMAGE_FILE): $(build)/$(linux_dir)/.build FORCE
$(call do-copy,$(dir $<)/$(linux_output),$@.tmp)
@if ! cmp --quiet "$@.tmp" "$@" ; then \
45cd771 to
c4c5976
Compare
…lete hashes.txt - Add FORCE to bzImage copy/hash rule prerequisite -- matching the modules.cpio fix in linuxboot#2174 -- so warm builds always re-run the recipe that records the kernel hash - Replace direct do-copy-to-target with cmp-based UNCHANGED pattern matching the modules.cpio precedent: copy to .tmp, compare, mv only when changed, print UNCHANGED when identical, always append hash/size to hashes.txt - When unchanged, preserve mtime to prevent cascading .bundled rebuild This completes the work started in PR linuxboot#2174 (modules.cpio FORCE fix). Signed-off-by: Thierry Laurion <insurgo@riseup.net>
…ll change PR linuxboot#2174 replaced 'make install' with 'applets/install.sh --symlinks' to avoid re-linking without SOURCE_DATE_EPOCH. However, install.sh reads symlink names from busybox.links, which was not regenerated and can be missing after cache restore or make clean. Result: only the bare busybox binary was installed; all 137 applet symlinks were absent from tools.cpio, breaking the initrd. Fixes: - Regenerate busybox.links via applets/busybox.mkll before install.sh - Use include/autoconf.h (busybox 1.36.1 uses this, not config.h) - Add busybox_hostcc ?= gcc and pass HOSTCC to mkll (needed outside busybox's make context where HOSTCC is undefined) - Add test -s guard so empty link-list generation fails the build loudly instead of silently producing no symlinks Signed-off-by: Thierry Laurion <insurgo@riseup.net>
c4c5976 to
0bd856e
Compare
…hes.txt tools.cpio and initrd.cpio.xz were the last two hashes.txt writers without FORCE. On warm builds, their recipes were skipped, leaving their entries (and all per-file hashes inside them) missing from hashes.txt. CI build comparison would show spurious diffs where the local side simply lacked entries, not where content diverged. - tools.cpio: add FORCE to prerequisites; recipe uses do-cpio (does not reference $^), so FORCE is safe - initrd.cpio.xz: add FORCE to prerequisites; recipe uses $^ in cpio-clean.pl input, so filter to $(filter-out FORCE,$^) to avoid passing FORCE as a filename argument With this, all cpio archives (modules, tools, heads, data, board), bzImage, and initrd.cpio.xz now have FORCE, completing the warm-build hashes.txt completeness work across every rule that writes to it. Signed-off-by: Thierry Laurion <insurgo@riseup.net>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
modules/linux:205
- The bzImage rule copies to
$@.tmpviado-copy, butdo-copyalways prints sha256/stat output for its destination. That makes the build log show hash/size for the temporary file (and it may be deleted on the UNCHANGED path), which is misleading noise in addition to the latertee -a $(HASHES)/$(SIZES)lines for$@. Prefer a plaincp -a(or a helper that doesn’t emit hashes) for the temporary copy.
$(build)/$(BOARD)/$(LINUX_IMAGE_FILE): $(build)/$(linux_dir)/.build FORCE
$(call do-copy,$(dir $<)/$(linux_output),$@.tmp)
@if ! cmp --quiet "$@.tmp" "$@" ; then \
|
pushing as bugfix otherwise busybox missing symlinks causes brick |
Supersedes #2178. Completes the warm-build
hashes.txtcompleteness work started in #2174. Fixes emergency bug introduced by #2174.Commit 1:
build: improve clean target @echo messages and fix reproducibility docMakefile clean targets
real.gitclean/gitclean_keep_packages/gitclean_keep_packages_and_build@echo messages to name thegit clean -fxdcommand and describe exact preserve rulesreal.remove_canary_files-extract_patch_rebuild_what_changed@echo messages: make clear this target deletes.canarystamps and the re-extract/rebuild cascade happens on the nextmakeinvocationdoc/reproducible-builds.md verification section
real.gitclean_keep_packagesprerequisite -- after Fix build reproducibility regression from PR #2136 (musl-cross-make binutils bump) #2174 FORCE fixes, warm builds produce completehashes.txthashes.txtdownload reference with CircleCI artifact URL exampleGIT_STATUS=dirty, changing ROM output)bzImagein same-commit step-down pathCommit 2:
modules/linux: add FORCE and UNCHANGED guard to bzImage rule; fix busybox symlinksbzImage FORCE + UNCHANGED
FORCEto bzImage copy/hash rule prerequisite -- matching themodules.cpiofix in Fix build reproducibility regression from PR #2136 (musl-cross-make binutils bump) #2174 -- so warm builds always re-run the recipe that records the kernel hashdo-copy-to-target with cmp-based UNCHANGED pattern: copy to.tmp, compare,mvonly when changed, printUNCHANGEDwhen identical, always append hash/size tohashes.txt.bundledrebuildEmergency fix: busybox symlinks broken by #2174
make installwithapplets/install.sh --symlinksto avoid re-linking withoutSOURCE_DATE_EPOCH. However,install.shreads symlink names frombusybox.links, which was not regenerated and is missing after cache restore ormake clean. Result: only the barebusyboxbinary was installed; all 137 applet symlinks ([,ash,cat,cp,ls, ...) were absent fromtools.cpio, breaking the initrd.busybox.linksviaapplets/busybox.mkllbefore callinginstall.sh, usingHOSTCC=gcc(needed outside busybox's make context) andinclude/autoconf.h(busybox 1.36.1 uses this, notconfig.h), with atest -sguard so empty generation fails the build loudly.Verification (commit
c4c59764b64)Built locally with
./docker_repro.sh make BOARD=EOL_t480-hotp-maximized(warm build,cleanGIT_STATUS).137 busybox symlinks now present in
tools.cpio:Local ROM hash: [pending]
CI cross-check: [pending CircleCI pipeline for
c4c59764b64]