Make debian package build reproducible #26
Conversation
marmarek
force-pushed
the
reprobuild
branch
10 times, most recently
from
9221007
to
a2ae71b
Compare
|
After a little more work, it works in Travis too. And packages build there have the same hash as my local build! |
|
Is "qubesiation" of packages i.e. creating |
|
To build with qubes-builder, yes, you still need |
Makefile.debian
Outdated
| fi; \ | ||
| fi | ||
|
|
||
| git -C $(CHROOT_DIR)/$(DIST_SRC)/$(DEBIAN_BUILD_DIRS)/.. clean -f -d -X |
There was a problem hiding this comment.
This unfortunately is too extensive. For example in vmm-xen, it removes extras/mini-os, tools/hotplug/Linux/init.d and more (all part of the upstream tarball). While some of it may be a bug in gitignore file, others are not (external repositories bundled into upstream tarball, but excluded in gitignore to not commit accidentally into xen.git - like mini-os). full log
My current plan is to revert to removing just deb, rpm and pkgs. @HW42 any better idea?
There was a problem hiding this comment.
Indeed for upstream tarballs .gitignore will not work as intended (also I wonder a bit why they are included in the tarballs). I don't really have a better idea. I guess at some point we will need a way to extend that list (maybe a SOURCE_TAR_EXCLUDE variable defined by the components is enough), but I think this can wait until a concrete need is there.
There was a problem hiding this comment.
To be honest, similar problem is with vmm-xen-stubdom-linux, where we have both upstream tarballs (excluded with .gitignore) and our tarball. It's Fedora package only there, but the problem is similar - git clean -X removes just downloaded tarballs. Since we do control .gitignore there, I was thinking about removing tarballs in top dir from .gitignore (keep only dl/ dir) and add separate file .tarignore to be used with --exclude-ignore=.tarignore in create-archive script.
It's only similar problem, because there we do control .gitignore, so we can keep git clean. Also, when create-archive script is used (generating orig.tar.gz from git checkout) removing object files is a good idea, especially when we mangle timestamps of source files. Otherwise sooner or later make will get confused and will use some out of date output file from some development build.
|
I've build all R4.0 Debian packages twice, on the same machine:
This isn't full test as environment was very similar, but it was something :) |
In preparation for pbuilder usage, move old code to the separate file. This will allow choosing old or new one with an option.
|
Building R3.2 packages fails for many reasons (mostly related to source package hacks). Since R3.2 will be EOL soon, instead of fixing all the package, I've separated old code to separate file and make it configurable whether legacy method or pbuilder should be used. Similar as we've done for Fedora (mock vs non-mock). This is now controlled with a single option: |
This replace custom code based on debootstrap and apt-get. The major gain here is much smaller build environment (only declared dependencies are installed), and also sharing tools with the upstream. Also it simplifies the builder a lot. The downside is slower operation (pbuilder re-install all the build dependencies each time). This work was done collectively with @HW42, during Reproducible Builds Summit. QubesOS/qubes-issues#816
Remove base.tgz if pbuilder create/update fails - it may leave broken image. Setup Build-Origin: Qubes Minor fixes.
Make it easier to build on non-Debian system, do not require debian-archive-keyring package (or equivalent) installed.
Apparently --basetgz is ignored in some cases and BASETGZ from pbuilderrc is used instead. To avoid this confusion, drop --basetgz and use the right value in pbuilderrc.
Make it possible to use both pbuilder and legacy builder in the same qubes-builder instance. Creating base directory is a bit tricky, as CHROOT_DIR may already exists and be root-owned (lagacy build first). In that case, create base dir as root and chown it later. Otherwise create base dir as normal user, so CHROOT_DIR is also owned by normal user and other subdirs (like home/user/qubes-src) can be created there normally.
Use pbuilder for build environment preparation
This replace custom code based on debootstrap and apt-get. The major
gain here is much smaller build environment (only declared dependencies
are installed), and also sharing tools with the upstream. Also it
simplifies the builder a lot.
The downside is slower operation (pbuilder re-install all the build dependencies each time).
This work was done collectively with @HW42, during Reproducible Builds Summit.
QubesOS/qubes-issues#816
Tests to do before merging: