Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Make ISO build (almost) reproducible, given the same set of input packages #26

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

marmarek
Copy link
Member

@marmarek marmarek commented Oct 6, 2018

This PR makes the ISO build almost reproducible, assuming individual input
packages are the same. The only missing part is reproducing efiboot.img build
(FAT filesystem image for EFI boot).

After this being merged (and updated installer-qubes-os packages released), the
way to reproduce ISO should be, in qubes-builder configured for appropriate release:

make INSTALLER_KICKSTART=/home/user/qubes-src/installer-qubes-os/conf/travis-iso-full.ks COMPONENTS= DISTS_VM= QUBES_RELEASE="(version to reproduce)" iso.

The INSTALLER_KICKSTART is needed to download all the packages from yum.qubes-os.org, instead of local build - to ensure stable input. This requirement should be gone once all qubes packages will build reproducibly.
For devel builds, use ISO_VERSION instead of QUBES_RELEASE.

Until this is merged, to test this one need build installer (make installer-qubes-os) and manually copy qubes-src/installer-qubes-os/pkgs from one build environment to another, then set COMPONENTS=installer-qubes-os (instead of empty).

As for efiboot.img, list of problems:

  • mkfs.vfat is not reproducible - embed some timestamp/random identifier; mformat doesn't have this problem, but is a PITA to use (the need to provide full "disk" geometry in sectors, heads, cylinders etc)
  • kernel driver doesn't deterministically create files; even after manually clamping timestamps in file metadata, there are some differences in directory entries (seen as "inode" numbers when mounted on Linux); there are also differences at the very end of fs image, not sure about the origin
  • mtools's mcopy is slightly better - it's possible to keep file metadata the same with mcopy -i "$img" -mp source-file ::destination-file iterated over all the files in stable order; but it still leaves differences at the end of fs image, similar to kernel driver

Most of those patches (or equivalent) are already submitted upstream:

QubesOS/qubes-issues#816

For devel builds we use current date as ISO version. Allow to provide
specific value, without marking the ISO as final.
This isn't exactly "last modification time of source", as the source for
the final ISO include also all rpm packages from other components (and
from Fedora), which may be newer than installer-qubes-os itself. But in
practice it shouldn't be a problem as most of them will be only included
as files without relying on any specific timestamps inside.
Package root filesystem directly as squashfs image. The most important
gain is getting rid of greatly unreproducible ext4 image creation. But
also it makes the output image slightly smaller, and the build faster.
Also, dmsquash-live dracut module is not needed anymore, which reduce
initramfs size.
Since moving to squashfs + overlay fs, it isn't needed anymore. And size
of initrd for efi is critical.
…image

Remove various caches not really needed.
Sort group files and packages list.
Strip timestamps from gconf file.
Clamp mtime in fontconfig cache to SOURCE_DATE_EPOCH.
@marmarek marmarek changed the title Make ISO build (almost) reproducible, given the same set of input packages WIP: Make ISO build (almost) reproducible, given the same set of input packages Oct 7, 2018
@marmarek
Copy link
Member Author

marmarek commented Oct 7, 2018

This should not be merged as is. Instead, patched packages should be extracted to separate repositories. This include at least:

  • squashfs-tools
  • mtools (TBD)
  • createrepo (instead of monkey patching it)

@Conan-Kudo
Copy link

@marmarek You should consider changing from createrepo to createrepo_c, as that's actively maintained and supports more features. It is also replacing createrepo in Fedora and RHEL/CentOS.

@marmarek
Copy link
Member Author

marmarek commented Oct 9, 2018

The pungi version we currently use rely on createrepo. But yes, for the future version we'll definitely use something newer. Porting the patch shouldn't be that hard, it's just different language, right? ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants