Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upCreate deterministic Debian source packages #1244
Comments
marmarek
added
enhancement
P: major
C: Debian
labels
Sep 26, 2015
marmarek
added this to the Release 3.1 milestone
Sep 26, 2015
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
adrelanos
Sep 30, 2015
Member
Afaik upstream tarball creation is usually is implemented by make dist. For a Debian policy conform and deterministic shell function on how to create a upstream tarball, have a look at this genmkfile function:
https://github.com/Whonix/genmkfile/blob/472850c285714a080316778296dda4aefd2b06c6/usr/share/genmkfile/make-helper.bsh#L491-L515
(needs a patch using touch to the same date, that I have ready and need to clean up)
Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available.
For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=650077
So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic.
|
Afaik upstream tarball creation is usually is implemented by Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available. For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug: So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Sep 30, 2015
Member
On Wed, Sep 30, 2015 at 10:00:19AM -0700, Patrick Schleizer wrote:
Afaik upstream tarball creation is usually is implemented by
make dist. For a Debian policy conform and deterministic shell function on how to create a upstream tarball, have a look at this genmkfile function:
https://github.com/Whonix/genmkfile/blob/472850c285714a080316778296dda4aefd2b06c6/usr/share/genmkfile/make-helper.bsh#L491-L515
(needs a patch usingtouchto the same date, that I have ready and need to clean up)
I don't think that's enough. For example tar file header contains device
major:minor of the filesystem from which it was created. git archive
handles all of such quirks.
Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available.
git archive doesn't include .git in the output tarball. Here I meant
to use git archive to prepare "upstream tarball" before debian
package build. Currently it is done using plain tar call (which, as you
know, is far from being deterministic).
For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=650077So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic.
Yes, getting deterministic build process it totally different task. This
one is prerequisite for it.
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
|
On Wed, Sep 30, 2015 at 10:00:19AM -0700, Patrick Schleizer wrote:
I don't think that's enough. For example tar file header contains device
Yes, getting deterministic build process it totally different task. This Best Regards, |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
nrgaway
Jan 28, 2016
I have been working on the Debian packaging over the last few weeks.
I have created a solution that removes the creation of the tars and patching from Makefile.builder and moves it into debian/rules. I moved the code into debian/rules instead of Makefile.debian to allow building package independent of Makefile.builder using targets recommended by Debian standards (get-orig-source and patch).
Currently I created a sub-directory called snippets within builder-debian and install those small Makefile includes the the chroots /usr/share/qubesbuilder directory, but plan on either creating a package that can be included as a build depend for packages that require it, or maybe adding it as a gitrepo (like gittree) and include the requires Makefile snippet in debian/rules, if needed.
Other related items I have completed are:
- Removing the need for
debianto be placed indebian-vmfolder on packages likevmm-xen. - Installation of choot environment using
pbuilderorsbuild. I am still working onsbuildover the next few days since I initially needed to build the packages in a Debian AppVM as itsbuildwas not available for Fedora. I could packagesbuildfor Fedora if we find it useful. - Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory
- Ability to apply patches based of DIST or VENDOR (wily, debian)
- Added
buildinfoto each build. - Experimented with using CDBS, Common Debian Build System
which I used forcore-agent-linuxandvmm-xen` but decided it is not as flexible and transparent as I would like. - Qubuntu refresh almost complete
I will have four qubes repos ready for inspection by end of weekend (builder-debian, vmm-xen, core-agent-linux and giu-agent-linux) for more feed back (currently in qubuntu feature branches). Currently those feature branches have some CDBS and pbuilder code in them I need to rip back out.
Sample of get-orig-source snippet:
DEB_ORIG_TGZ = $(tarball_dir)/$(DEB_SOURCE)_$(DEB_VERSION_UPSTREAM).orig.tar.gz
### Create Debian <package>_<version>.orig.tar.gz file
$(DEB_ORIG_TGZ):
@$(shell rm -f $@)
@$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)
# @$(shell git archive HEAD --output=$@)
get-orig-source: $(DEB_ORIG_TGZ)
@true
Sample of patch snippet:
# Top source directory to apply patches in
QUILT_TOPDIR ?= $(DEB_SRCDIR)
QUILT_DIR ?= $(debdir)/patches
QUILT_SERIES ?= series
QUILT_SERIES_PATH = $(QUILT_DIR)/$(QUILT_SERIES)
# Master series patch file (if it exists) will be used to copy patches contained
# in its series file to $QUILT_DIR and will append its series entry with the
# new patches.
#
# This allows patches to be shared amongst other packaging or allow different
# sets of patches for both Debian, Ubuntu, etc while making sure the patches are
# in place to allow creation of a proper orig.tar.gz.
MASTER_QUILT_SERIES ?= \
$(wildcard \
series-$(DEB_DISTRIBUTION)-$(packageset).conf \
series-$(call LOWERCASE, $(DEB_VENDOR))-$(packageset).conf \
)
# Path to stamp file to use
DEB_STAMP ?= debian/stamp-patched
### Move any patches to debian/patches
$(QUILT_SERIES_PATH):
@# Clear patch series.conf file to prepare for update
@$(shell rm -f $(QUILT_SERIES_PATH))
@$(shell touch $(QUILT_SERIES_PATH))
@# Move patches to debian/patches directory
@while read patch_file; do \
echo "I: $(red)patch_file: $(blue)$(packagedir)/$${patch_file}$(normal)"; \
if [ -e "$(packagedir)/$${patch_file}" ]; then \
echo "$${patch_file##*/}" >> "$(QUILT_SERIES_PATH)"; \
cp "$(packagedir)/$${patch_file}" "$(QUILT_DIR)"; \
fi \
done < $(firstword $(MASTER_QUILT_SERIES))
$(MASTER_QUILT_SERIES): $(QUILT_SERIES_PATH)
@true
prepare-patches: $(MASTER_QUILT_SERIES)
@true
patch: prepare-patches $(DEB_STAMP)
$(DEB_STAMP):
@cd $(QUILT_TOPDIR) && \
QUILT_SERIES=$(QUILT_SERIES) QUILT_PATCHES=$(QUILT_DIR) \
quilt --quiltrc /dev/null push -a || test $$? = 2
@touch $(DEB_STAMP)
unpatch:
@cd $(QUILT_TOPDIR) && \
QUILT_SERIES=$(QUILT_SERIES) QUILT_PATCHES=$(QUILT_DIR) \
quilt --quiltrc /dev/null pop -a -R || test $$? = 2
@rm -rf .pc $(DEB_STAMP)
.PHONY: patch unpatch
nrgaway
commented
Jan 28, 2016
|
I have been working on the Debian packaging over the last few weeks. I have created a solution that removes the creation of the tars and patching from Currently I created a sub-directory called Other related items I have completed are:
I will have four qubes repos ready for inspection by end of weekend ( Sample of
Sample of
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Jan 28, 2016
Member
- Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory
Try to make it as easy as possible to build in non-qubes-builder. Ideally if the source package could be build any standard Debian upstream build tool without any Qubes-specific tools for that. For example:
apt-get source qubes-core-agent
apt-get build-dep qubes-core-agent
cd qubes-core-agent*/
dpkg-buildpackage
If that means changing something in our packaging files, that's fine, lets do it (but keep builder also compatible with old packages - so it will be possible to build for example R3.0 packages).
If that means some additional build dependency with those snippets, it may be an option, but preferably package should be buildable using standard environment. Having some custom things done by qubes-builder (like currently orig.tar.gz...) is no-go.
Be as close to the upstream solution (package format and build process) as possible.
### Create Debian <package>_<version>.orig.tar.gz file $(DEB_ORIG_TGZ): @$(shell rm -f $@) @$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)
This is not deterministic, take a look here: https://reproducible-builds.org/docs/archives/
Also note that z option (gzip) makes it non-deterministic because of timestamp included. AFAIR there is some option to skip it - search linked page.
or ...
# @$(shell git archive HEAD --output=$@)
... use this one.
Try to make it as easy as possible to build in non-qubes-builder. Ideally if the source package could be build any standard Debian upstream build tool without any Qubes-specific tools for that. For example:
If that means changing something in our packaging files, that's fine, lets do it (but keep builder also compatible with old packages - so it will be possible to build for example R3.0 packages). If that means some additional build dependency with those snippets, it may be an option, but preferably package should be buildable using standard environment. Having some custom things done by qubes-builder (like currently orig.tar.gz...) is no-go. Be as close to the upstream solution (package format and build process) as possible.
This is not deterministic, take a look here: https://reproducible-builds.org/docs/archives/ Also note that or ...
... use this one. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
nrgaway
Jan 29, 2016
On 28 January 2016 at 17:50, Marek Marczykowski-Górecki <
notifications@github.com> wrote:
- Ability to have different binary depends, recommends, etc per DIST
or VENDOR (wily, debian) for any package using a packagename.depends
configuration file within the debian directoryTry to make it as easy as possible to build in non-qubes-builder. Ideally
if the source package could be build any standard Debian upstream build
tool without any Qubes-specific tools for that. For example:apt-get source qubes-core-agent
apt-get build-dep qubes-core-agent
cd qubes-core-agent*/
dpkg-buildpackageIf that means changing something in our packaging files, that's fine, lets
do it (but keep builder also compatible with old packages - so it will be
possible to build for example R3.0 packages).If that means some additional build dependency with those snippets, it may
be an option, but preferably package should be buildable using standard
environment. Having some custom things done by qubes-builder (like
currently orig.tar.gz...) is no-go.I agree. You will see the four packages I refactored will be able to built
in that manner since I am actually working on getting them to build with
buildd as well. At the same time I am also being careful that they will
still build for 3.0 and making sure stuff builds in a Debian environment (I
can build all the components, just can't create the template rpm yet).
I think if you don't want to have to depend on a yet to be created
qubes-builder package that would install the snippets onto the chroot
that I should consider using git-subrepo (
https://github.com/ingydotnet/git-subrepo) which would allow updating the
snippets without using submodules. I want to stay away from submodules
since there are issues using them with Debian packaging using some of the
git package tools (I got an error saying git submodules are not yet
supported).
Be as close to the upstream solution (package format and build process) as
possible.Create Debian _.orig.tar.gz file
$(DEB_ORIG_TGZ):
@$(shell rm -f $@)
@$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)This is not deterministic, take a look here:
https://reproducible-builds.org/docs/archives/Also note that z option (gzip) makes it non-deterministic because of
timestamp included. AFAIR there is some option to skip it - search linked
page.
I have code to create a tar that is deterministic, just have not
implemented it yet (https://wiki.debian.org/onlyjob/get-orig-source). I am
also going to change get-orig-source to do just that and create the .orig
using the dist target.
I am starting to also use dh_buildinfo:
Register the versions of build-dependencies used to build a package.
dh_buildinfo is a debhelper program that registers in a file the list of
packages declared as build-time dependencies, as well as build-essential
packages, together with their versions, as installed in the build
machine.
As well as --list-missing which looks for files that are not in package but
built, similar to what happens at the end of building an rpm.
Finally, I also noticed none of the source packages are being signed. I
will fix that as well look into using dput to move packages to repo.
Oh, one last thing in regards to using native Debian packages like some of
the qubes packages do:
5.3. Native Debian package
https://www.debian.org/doc/manuals/debmake-doc/ch05.en.html
The non-native Debian package in the “3.0 (quilt)” format is the most
normal Debian binary package format. The above workflow and the following
packaging examples always use this format.
The native Debian package is the rare Debian binary package format. It may
be used only when the package is useful and valuable only for Debian. Thus,
its use is generally deprecated.
or ...
@$(shell git archive HEAD --output=$@)
... use this one.
—
Reply to this email directly or view it on GitHub
#1244 (comment)
.
nrgaway
commented
Jan 29, 2016
|
On 28 January 2016 at 17:50, Marek Marczykowski-Górecki <
I think if you don't want to have to depend on a yet to be created
I have code to create a tar that is deterministic, just have not I am starting to also use As well as --list-missing which looks for files that are not in package but Finally, I also noticed none of the source packages are being signed. I Oh, one last thing in regards to using native Debian packages like some of The non-native Debian package in the “3.0 (quilt)” format is the most The native Debian package is the rare Debian binary package format. It may
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Jan 29, 2016
Member
I think if you don't want to have to depend on a yet to be created
qubes-builderpackage that would install thesnippetsonto thechrootthat I should consider using git-subrepo ( https://github.com/ingydotnet/git-subrepo) which would allow updating the snippets without using submodules.
If some additional snippets are really required, build-depends on them is ok. But make sure that builder package itself can be built without them (to not introduce dependency cycle)... And generally if some package can be built without such additions, do not depend on that package.
If some additional snippets are really required, build-depends on them is ok. But make sure that builder package itself can be built without them (to not introduce dependency cycle)... And generally if some package can be built without such additions, do not depend on that package. |
marmarek commentedSep 26, 2015
Currently orig.tar.gz is created at each build, using plain tar. This means that each time it is different, even if no change was made (because of mtimes, files order and such issues). Later, when adding such package to repository (linux-deb),
repreprocomplains about adding already existing package but with different size/checksum (this happens because orig.tar.gz is added as part of every target release -jessie,wheezy,stretch).This is the reason why no source packages are uploaded to
http://deb.qubes-os.org/.Additionally the command creating orig.tar.gz should be standarized and placed in
builder-debian- currently it is duplicated in every component, for example:https://github.com/QubesOS/qubes-core-agent-linux/blob/master/Makefile.builder#L14-L18
https://github.com/QubesOS/qubes-core-qubesdb/blob/master/Makefile.builder#L25-L29
Most of packages (in quilt format) have exacty the same lines. Two of them are slightly different:
vmm-xenandapp-thunderbird. This is probably easy to handle.For creating stable tar.gz,
git archive --format=tar.gz HEADshould be used - it takes care of mtimes and such problems. The archive will contain only committed content and dpkg-buildpackage will complain about local changes (if any), but that's not a problem - in case of local changesbuilder-debianmay simply skip source package creation as it would be useless anyway (dpkg-buildpackage -b). Or even skip source package creation for every devel build (INCREMENT_DEVEL_VERSION=1).Packages in native format I believe are properly handled by
dpkg-sourcealready (care needs to be taken to not include binary packages in the source archive, but currently is seems to be ok).