New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Make ISO build (almost) reproducible, given the same set of input packages #26

Open
wants to merge 9 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@marmarek
Member

marmarek commented Oct 6, 2018

This PR makes the ISO build almost reproducible, assuming individual input
packages are the same. The only missing part is reproducing efiboot.img build
(FAT filesystem image for EFI boot).

After this being merged (and updated installer-qubes-os packages released), the
way to reproduce ISO should be, in qubes-builder configured for appropriate release:

make INSTALLER_KICKSTART=/home/user/qubes-src/installer-qubes-os/conf/travis-iso-full.ks COMPONENTS= DISTS_VM= QUBES_RELEASE="(version to reproduce)" iso.

The INSTALLER_KICKSTART is needed to download all the packages from yum.qubes-os.org, instead of local build - to ensure stable input. This requirement should be gone once all qubes packages will build reproducibly.
For devel builds, use ISO_VERSION instead of QUBES_RELEASE.

Until this is merged, to test this one need build installer (make installer-qubes-os) and manually copy qubes-src/installer-qubes-os/pkgs from one build environment to another, then set COMPONENTS=installer-qubes-os (instead of empty).

As for efiboot.img, list of problems:

  • mkfs.vfat is not reproducible - embed some timestamp/random identifier; mformat doesn't have this problem, but is a PITA to use (the need to provide full "disk" geometry in sectors, heads, cylinders etc)
  • kernel driver doesn't deterministically create files; even after manually clamping timestamps in file metadata, there are some differences in directory entries (seen as "inode" numbers when mounted on Linux); there are also differences at the very end of fs image, not sure about the origin
  • mtools's mcopy is slightly better - it's possible to keep file metadata the same with mcopy -i "$img" -mp source-file ::destination-file iterated over all the files in stable order; but it still leaves differences at the end of fs image, similar to kernel driver

Most of those patches (or equivalent) are already submitted upstream:

QubesOS/qubes-issues#816

marmarek added some commits Oct 4, 2018

Allow to override ISO_VERSION for devel builds from builder.conf
For devel builds we use current date as ISO version. Allow to provide
specific value, without marking the ISO as final.
Set SOURCE_DATE_EPOCH for the build as last commit time
This isn't exactly "last modification time of source", as the source for
the final ISO include also all rpm packages from other components (and
from Fedora), which may be newer than installer-qubes-os itself. But in
practice it shouldn't be a problem as most of them will be only included
as files without relying on any specific timestamps inside.
anaconda, lorax: drop rootfs.img inside squashfs
Package root filesystem directly as squashfs image. The most important
gain is getting rid of greatly unreproducible ext4 image creation. But
also it makes the output image slightly smaller, and the build faster.
Also, dmsquash-live dracut module is not needed anymore, which reduce
initramfs size.
lorax-templates-qubes: drop dmsquash-live module in efi initrd image
Since moving to squashfs + overlay fs, it isn't needed anymore. And size
of initrd for efi is critical.
lorax-templates-qubes: strip various non-determinisms from installer …
…image

Remove various caches not really needed.
Sort group files and packages list.
Strip timestamps from gconf file.
Clamp mtime in fontconfig cache to SOURCE_DATE_EPOCH.

@marmarek marmarek changed the title from Make ISO build (almost) reproducible, given the same set of input packages to WIP: Make ISO build (almost) reproducible, given the same set of input packages Oct 7, 2018

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Oct 7, 2018

Member

This should not be merged as is. Instead, patched packages should be extracted to separate repositories. This include at least:

  • squashfs-tools
  • mtools (TBD)
  • createrepo (instead of monkey patching it)
Member

marmarek commented Oct 7, 2018

This should not be merged as is. Instead, patched packages should be extracted to separate repositories. This include at least:

  • squashfs-tools
  • mtools (TBD)
  • createrepo (instead of monkey patching it)
@Conan-Kudo

This comment has been minimized.

Show comment
Hide comment
@Conan-Kudo

Conan-Kudo Oct 9, 2018

@marmarek You should consider changing from createrepo to createrepo_c, as that's actively maintained and supports more features. It is also replacing createrepo in Fedora and RHEL/CentOS.

Conan-Kudo commented Oct 9, 2018

@marmarek You should consider changing from createrepo to createrepo_c, as that's actively maintained and supports more features. It is also replacing createrepo in Fedora and RHEL/CentOS.

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Oct 9, 2018

Member

The pungi version we currently use rely on createrepo. But yes, for the future version we'll definitely use something newer. Porting the patch shouldn't be that hard, it's just different language, right? ;)

Member

marmarek commented Oct 9, 2018

The pungi version we currently use rely on createrepo. But yes, for the future version we'll definitely use something newer. Porting the patch shouldn't be that hard, it's just different language, right? ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment