New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create deterministic Debian source packages #1244

Closed
marmarek opened this Issue Sep 26, 2015 · 6 comments

Comments

Projects
None yet
5 participants
@marmarek
Member

marmarek commented Sep 26, 2015

Currently orig.tar.gz is created at each build, using plain tar. This means that each time it is different, even if no change was made (because of mtimes, files order and such issues). Later, when adding such package to repository (linux-deb), reprepro complains about adding already existing package but with different size/checksum (this happens because orig.tar.gz is added as part of every target release - jessie, wheezy, stretch).
This is the reason why no source packages are uploaded to http://deb.qubes-os.org/.

Additionally the command creating orig.tar.gz should be standarized and placed in builder-debian - currently it is duplicated in every component, for example:
https://github.com/QubesOS/qubes-core-agent-linux/blob/master/Makefile.builder#L14-L18
https://github.com/QubesOS/qubes-core-qubesdb/blob/master/Makefile.builder#L25-L29

Most of packages (in quilt format) have exacty the same lines. Two of them are slightly different: vmm-xen and app-thunderbird. This is probably easy to handle.

For creating stable tar.gz, git archive --format=tar.gz HEAD should be used - it takes care of mtimes and such problems. The archive will contain only committed content and dpkg-buildpackage will complain about local changes (if any), but that's not a problem - in case of local changes builder-debian may simply skip source package creation as it would be useless anyway (dpkg-buildpackage -b). Or even skip source package creation for every devel build (INCREMENT_DEVEL_VERSION=1).

Packages in native format I believe are properly handled by dpkg-source already (care needs to be taken to not include binary packages in the source archive, but currently is seems to be ok).

@adrelanos

This comment has been minimized.

Show comment
Hide comment
@adrelanos

adrelanos Sep 30, 2015

Member

Afaik upstream tarball creation is usually is implemented by make dist. For a Debian policy conform and deterministic shell function on how to create a upstream tarball, have a look at this genmkfile function:
https://github.com/Whonix/genmkfile/blob/472850c285714a080316778296dda4aefd2b06c6/usr/share/genmkfile/make-helper.bsh#L491-L515
(needs a patch using touch to the same date, that I have ready and need to clean up)

Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available.

For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=650077

So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic.

Member

adrelanos commented Sep 30, 2015

Afaik upstream tarball creation is usually is implemented by make dist. For a Debian policy conform and deterministic shell function on how to create a upstream tarball, have a look at this genmkfile function:
https://github.com/Whonix/genmkfile/blob/472850c285714a080316778296dda4aefd2b06c6/usr/share/genmkfile/make-helper.bsh#L491-L515
(needs a patch using touch to the same date, that I have ready and need to clean up)

Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available.

For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=650077

So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic.

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Sep 30, 2015

Member

On Wed, Sep 30, 2015 at 10:00:19AM -0700, Patrick Schleizer wrote:

Afaik upstream tarball creation is usually is implemented by make dist. For a Debian policy conform and deterministic shell function on how to create a upstream tarball, have a look at this genmkfile function:
https://github.com/Whonix/genmkfile/blob/472850c285714a080316778296dda4aefd2b06c6/usr/share/genmkfile/make-helper.bsh#L491-L515
(needs a patch using touch to the same date, that I have ready and need to clean up)

I don't think that's enough. For example tar file header contains device
major:minor of the filesystem from which it was created. git archive
handles all of such quirks.

Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available.

git archive doesn't include .git in the output tarball. Here I meant
to use git archive to prepare "upstream tarball" before debian
package build. Currently it is done using plain tar call (which, as you
know, is far from being deterministic).

For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=650077

So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic.

Yes, getting deterministic build process it totally different task. This
one is prerequisite for it.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Member

marmarek commented Sep 30, 2015

On Wed, Sep 30, 2015 at 10:00:19AM -0700, Patrick Schleizer wrote:

Afaik upstream tarball creation is usually is implemented by make dist. For a Debian policy conform and deterministic shell function on how to create a upstream tarball, have a look at this genmkfile function:
https://github.com/Whonix/genmkfile/blob/472850c285714a080316778296dda4aefd2b06c6/usr/share/genmkfile/make-helper.bsh#L491-L515
(needs a patch using touch to the same date, that I have ready and need to clean up)

I don't think that's enough. For example tar file header contains device
major:minor of the filesystem from which it was created. git archive
handles all of such quirks.

Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available.

git archive doesn't include .git in the output tarball. Here I meant
to use git archive to prepare "upstream tarball" before debian
package build. Currently it is done using plain tar call (which, as you
know, is far from being deterministic).

For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=650077

So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic.

Yes, getting deterministic build process it totally different task. This
one is prerequisite for it.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@nrgaway

This comment has been minimized.

Show comment
Hide comment
@nrgaway

nrgaway Jan 28, 2016

I have been working on the Debian packaging over the last few weeks.

I have created a solution that removes the creation of the tars and patching from Makefile.builder and moves it into debian/rules. I moved the code into debian/rules instead of Makefile.debian to allow building package independent of Makefile.builder using targets recommended by Debian standards (get-orig-source and patch).

Currently I created a sub-directory called snippets within builder-debian and install those small Makefile includes the the chroots /usr/share/qubesbuilder directory, but plan on either creating a package that can be included as a build depend for packages that require it, or maybe adding it as a gitrepo (like gittree) and include the requires Makefile snippet in debian/rules, if needed.

Other related items I have completed are:

  • Removing the need for debian to be placed in debian-vm folder on packages like vmm-xen.
  • Installation of choot environment using pbuilder or sbuild. I am still working on sbuild over the next few days since I initially needed to build the packages in a Debian AppVM as it sbuild was not available for Fedora. I could package sbuild for Fedora if we find it useful.
  • Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory
  • Ability to apply patches based of DIST or VENDOR (wily, debian)
  • Added buildinfo to each build.
  • Experimented with using CDBS, Common Debian Build Systemwhich I used forcore-agent-linuxandvmm-xen` but decided it is not as flexible and transparent as I would like.
  • Qubuntu refresh almost complete

I will have four qubes repos ready for inspection by end of weekend (builder-debian, vmm-xen, core-agent-linux and giu-agent-linux) for more feed back (currently in qubuntu feature branches). Currently those feature branches have some CDBS and pbuilder code in them I need to rip back out.

Sample of get-orig-source snippet:

DEB_ORIG_TGZ = $(tarball_dir)/$(DEB_SOURCE)_$(DEB_VERSION_UPSTREAM).orig.tar.gz  

### Create Debian <package>_<version>.orig.tar.gz file                           
$(DEB_ORIG_TGZ):                                                                 
        @$(shell rm -f $@)                                                       
        @$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)     
#        @$(shell git archive HEAD --output=$@)                                   

get-orig-source: $(DEB_ORIG_TGZ)                                                 
        @true

Sample of patch snippet:

# Top source directory to apply patches in                                       
QUILT_TOPDIR ?= $(DEB_SRCDIR)                                                    

QUILT_DIR ?= $(debdir)/patches                                                   
QUILT_SERIES ?= series                                                           
QUILT_SERIES_PATH = $(QUILT_DIR)/$(QUILT_SERIES)                                 

# Master series patch file (if it exists) will be used to copy patches contained 
# in its series file to $QUILT_DIR and will append its series entry with the     
# new patches.                                                                   
#                                                                                
# This allows patches to be shared amongst other packaging or allow different    
# sets of patches for both Debian, Ubuntu, etc while making sure the patches are 
# in place to allow creation of a proper orig.tar.gz.                            
MASTER_QUILT_SERIES ?= \                                                         
        $(wildcard \                                                             
            series-$(DEB_DISTRIBUTION)-$(packageset).conf \                      
            series-$(call LOWERCASE, $(DEB_VENDOR))-$(packageset).conf \         
)                                                                                

# Path to stamp file to use                                                      
DEB_STAMP ?= debian/stamp-patched                                                

### Move any patches to debian/patches
$(QUILT_SERIES_PATH):                                                            
        @# Clear patch series.conf file to prepare for update                    
        @$(shell rm -f $(QUILT_SERIES_PATH))                                     
        @$(shell touch $(QUILT_SERIES_PATH))                                     

        @# Move patches to debian/patches directory                              
        @while read patch_file; do \                                             
            echo "I: $(red)patch_file: $(blue)$(packagedir)/$${patch_file}$(normal)"; \
            if [ -e "$(packagedir)/$${patch_file}" ]; then \                     
                echo "$${patch_file##*/}" >> "$(QUILT_SERIES_PATH)"; \           
                cp "$(packagedir)/$${patch_file}" "$(QUILT_DIR)"; \              
            fi \                                                                 
        done < $(firstword $(MASTER_QUILT_SERIES))                               

$(MASTER_QUILT_SERIES): $(QUILT_SERIES_PATH)                                     
         @true

prepare-patches: $(MASTER_QUILT_SERIES)                                          
        @true

patch: prepare-patches $(DEB_STAMP)                                              
$(DEB_STAMP):                                                                    
        @cd $(QUILT_TOPDIR) && \                                                 
        QUILT_SERIES=$(QUILT_SERIES) QUILT_PATCHES=$(QUILT_DIR) \                
                quilt --quiltrc /dev/null push -a || test $$? = 2                
        @touch $(DEB_STAMP)                                                      

unpatch:                                                                         
        @cd $(QUILT_TOPDIR) && \                                                 
        QUILT_SERIES=$(QUILT_SERIES) QUILT_PATCHES=$(QUILT_DIR) \                
                quilt --quiltrc /dev/null pop -a -R || test $$? = 2              
        @rm -rf .pc $(DEB_STAMP)                                                 

.PHONY: patch unpatch

nrgaway commented Jan 28, 2016

I have been working on the Debian packaging over the last few weeks.

I have created a solution that removes the creation of the tars and patching from Makefile.builder and moves it into debian/rules. I moved the code into debian/rules instead of Makefile.debian to allow building package independent of Makefile.builder using targets recommended by Debian standards (get-orig-source and patch).

Currently I created a sub-directory called snippets within builder-debian and install those small Makefile includes the the chroots /usr/share/qubesbuilder directory, but plan on either creating a package that can be included as a build depend for packages that require it, or maybe adding it as a gitrepo (like gittree) and include the requires Makefile snippet in debian/rules, if needed.

Other related items I have completed are:

  • Removing the need for debian to be placed in debian-vm folder on packages like vmm-xen.
  • Installation of choot environment using pbuilder or sbuild. I am still working on sbuild over the next few days since I initially needed to build the packages in a Debian AppVM as it sbuild was not available for Fedora. I could package sbuild for Fedora if we find it useful.
  • Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory
  • Ability to apply patches based of DIST or VENDOR (wily, debian)
  • Added buildinfo to each build.
  • Experimented with using CDBS, Common Debian Build Systemwhich I used forcore-agent-linuxandvmm-xen` but decided it is not as flexible and transparent as I would like.
  • Qubuntu refresh almost complete

I will have four qubes repos ready for inspection by end of weekend (builder-debian, vmm-xen, core-agent-linux and giu-agent-linux) for more feed back (currently in qubuntu feature branches). Currently those feature branches have some CDBS and pbuilder code in them I need to rip back out.

Sample of get-orig-source snippet:

DEB_ORIG_TGZ = $(tarball_dir)/$(DEB_SOURCE)_$(DEB_VERSION_UPSTREAM).orig.tar.gz  

### Create Debian <package>_<version>.orig.tar.gz file                           
$(DEB_ORIG_TGZ):                                                                 
        @$(shell rm -f $@)                                                       
        @$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)     
#        @$(shell git archive HEAD --output=$@)                                   

get-orig-source: $(DEB_ORIG_TGZ)                                                 
        @true

Sample of patch snippet:

# Top source directory to apply patches in                                       
QUILT_TOPDIR ?= $(DEB_SRCDIR)                                                    

QUILT_DIR ?= $(debdir)/patches                                                   
QUILT_SERIES ?= series                                                           
QUILT_SERIES_PATH = $(QUILT_DIR)/$(QUILT_SERIES)                                 

# Master series patch file (if it exists) will be used to copy patches contained 
# in its series file to $QUILT_DIR and will append its series entry with the     
# new patches.                                                                   
#                                                                                
# This allows patches to be shared amongst other packaging or allow different    
# sets of patches for both Debian, Ubuntu, etc while making sure the patches are 
# in place to allow creation of a proper orig.tar.gz.                            
MASTER_QUILT_SERIES ?= \                                                         
        $(wildcard \                                                             
            series-$(DEB_DISTRIBUTION)-$(packageset).conf \                      
            series-$(call LOWERCASE, $(DEB_VENDOR))-$(packageset).conf \         
)                                                                                

# Path to stamp file to use                                                      
DEB_STAMP ?= debian/stamp-patched                                                

### Move any patches to debian/patches
$(QUILT_SERIES_PATH):                                                            
        @# Clear patch series.conf file to prepare for update                    
        @$(shell rm -f $(QUILT_SERIES_PATH))                                     
        @$(shell touch $(QUILT_SERIES_PATH))                                     

        @# Move patches to debian/patches directory                              
        @while read patch_file; do \                                             
            echo "I: $(red)patch_file: $(blue)$(packagedir)/$${patch_file}$(normal)"; \
            if [ -e "$(packagedir)/$${patch_file}" ]; then \                     
                echo "$${patch_file##*/}" >> "$(QUILT_SERIES_PATH)"; \           
                cp "$(packagedir)/$${patch_file}" "$(QUILT_DIR)"; \              
            fi \                                                                 
        done < $(firstword $(MASTER_QUILT_SERIES))                               

$(MASTER_QUILT_SERIES): $(QUILT_SERIES_PATH)                                     
         @true

prepare-patches: $(MASTER_QUILT_SERIES)                                          
        @true

patch: prepare-patches $(DEB_STAMP)                                              
$(DEB_STAMP):                                                                    
        @cd $(QUILT_TOPDIR) && \                                                 
        QUILT_SERIES=$(QUILT_SERIES) QUILT_PATCHES=$(QUILT_DIR) \                
                quilt --quiltrc /dev/null push -a || test $$? = 2                
        @touch $(DEB_STAMP)                                                      

unpatch:                                                                         
        @cd $(QUILT_TOPDIR) && \                                                 
        QUILT_SERIES=$(QUILT_SERIES) QUILT_PATCHES=$(QUILT_DIR) \                
                quilt --quiltrc /dev/null pop -a -R || test $$? = 2              
        @rm -rf .pc $(DEB_STAMP)                                                 

.PHONY: patch unpatch
@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Jan 28, 2016

Member
  • Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory

Try to make it as easy as possible to build in non-qubes-builder. Ideally if the source package could be build any standard Debian upstream build tool without any Qubes-specific tools for that. For example:

apt-get source qubes-core-agent
apt-get build-dep qubes-core-agent
cd qubes-core-agent*/
dpkg-buildpackage

If that means changing something in our packaging files, that's fine, lets do it (but keep builder also compatible with old packages - so it will be possible to build for example R3.0 packages).

If that means some additional build dependency with those snippets, it may be an option, but preferably package should be buildable using standard environment. Having some custom things done by qubes-builder (like currently orig.tar.gz...) is no-go.

Be as close to the upstream solution (package format and build process) as possible.

### Create Debian <package>_<version>.orig.tar.gz file                           
$(DEB_ORIG_TGZ):                                                                 
        @$(shell rm -f $@)                                                       
        @$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)     

This is not deterministic, take a look here: https://reproducible-builds.org/docs/archives/

Also note that z option (gzip) makes it non-deterministic because of timestamp included. AFAIR there is some option to skip it - search linked page.

or ...

# @$(shell git archive HEAD --output=$@)

... use this one.

Member

marmarek commented Jan 28, 2016

  • Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory

Try to make it as easy as possible to build in non-qubes-builder. Ideally if the source package could be build any standard Debian upstream build tool without any Qubes-specific tools for that. For example:

apt-get source qubes-core-agent
apt-get build-dep qubes-core-agent
cd qubes-core-agent*/
dpkg-buildpackage

If that means changing something in our packaging files, that's fine, lets do it (but keep builder also compatible with old packages - so it will be possible to build for example R3.0 packages).

If that means some additional build dependency with those snippets, it may be an option, but preferably package should be buildable using standard environment. Having some custom things done by qubes-builder (like currently orig.tar.gz...) is no-go.

Be as close to the upstream solution (package format and build process) as possible.

### Create Debian <package>_<version>.orig.tar.gz file                           
$(DEB_ORIG_TGZ):                                                                 
        @$(shell rm -f $@)                                                       
        @$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)     

This is not deterministic, take a look here: https://reproducible-builds.org/docs/archives/

Also note that z option (gzip) makes it non-deterministic because of timestamp included. AFAIR there is some option to skip it - search linked page.

or ...

# @$(shell git archive HEAD --output=$@)

... use this one.

@nrgaway

This comment has been minimized.

Show comment
Hide comment
@nrgaway

nrgaway Jan 29, 2016

On 28 January 2016 at 17:50, Marek Marczykowski-Górecki <
notifications@github.com> wrote:

  • Ability to have different binary depends, recommends, etc per DIST
    or VENDOR (wily, debian) for any package using a packagename.depends
    configuration file within the debian directory

Try to make it as easy as possible to build in non-qubes-builder. Ideally
if the source package could be build any standard Debian upstream build
tool without any Qubes-specific tools for that. For example:

apt-get source qubes-core-agent
apt-get build-dep qubes-core-agent
cd qubes-core-agent*/
dpkg-buildpackage

If that means changing something in our packaging files, that's fine, lets
do it (but keep builder also compatible with old packages - so it will be
possible to build for example R3.0 packages).

If that means some additional build dependency with those snippets, it may
be an option, but preferably package should be buildable using standard
environment. Having some custom things done by qubes-builder (like
currently orig.tar.gz...) is no-go.

I agree. You will see the four packages I refactored will be able to built
in that manner since I am actually working on getting them to build with
buildd as well. At the same time I am also being careful that they will
still build for 3.0 and making sure stuff builds in a Debian environment (I
can build all the components, just can't create the template rpm yet).

I think if you don't want to have to depend on a yet to be created
qubes-builder package that would install the snippets onto the chroot
that I should consider using git-subrepo (
https://github.com/ingydotnet/git-subrepo) which would allow updating the
snippets without using submodules. I want to stay away from submodules
since there are issues using them with Debian packaging using some of the
git package tools (I got an error saying git submodules are not yet
supported).

Be as close to the upstream solution (package format and build process) as
possible.

Create Debian _.orig.tar.gz file

$(DEB_ORIG_TGZ):
@$(shell rm -f $@)
@$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)

This is not deterministic, take a look here:
https://reproducible-builds.org/docs/archives/

Also note that z option (gzip) makes it non-deterministic because of
timestamp included. AFAIR there is some option to skip it - search linked
page.

I have code to create a tar that is deterministic, just have not
implemented it yet (https://wiki.debian.org/onlyjob/get-orig-source). I am
also going to change get-orig-source to do just that and create the .orig
using the dist target.

I am starting to also use dh_buildinfo:
Register the versions of build-dependencies used to build a package.
dh_buildinfo is a debhelper program that registers in a file the list of
packages declared as build-time dependencies, as well as build-essential
packages, together with their versions, as installed in the build
machine.

As well as --list-missing which looks for files that are not in package but
built, similar to what happens at the end of building an rpm.

Finally, I also noticed none of the source packages are being signed. I
will fix that as well look into using dput to move packages to repo.

Oh, one last thing in regards to using native Debian packages like some of
the qubes packages do:
5.3. Native Debian package
https://www.debian.org/doc/manuals/debmake-doc/ch05.en.html

The non-native Debian package in the “3.0 (quilt)” format is the most
normal Debian binary package format. The above workflow and the following
packaging examples always use this format.

The native Debian package is the rare Debian binary package format. It may
be used only when the package is useful and valuable only for Debian. Thus,
its use is generally deprecated.

or ...

@$(shell git archive HEAD --output=$@)

... use this one.


Reply to this email directly or view it on GitHub
#1244 (comment)
.

nrgaway commented Jan 29, 2016

On 28 January 2016 at 17:50, Marek Marczykowski-Górecki <
notifications@github.com> wrote:

  • Ability to have different binary depends, recommends, etc per DIST
    or VENDOR (wily, debian) for any package using a packagename.depends
    configuration file within the debian directory

Try to make it as easy as possible to build in non-qubes-builder. Ideally
if the source package could be build any standard Debian upstream build
tool without any Qubes-specific tools for that. For example:

apt-get source qubes-core-agent
apt-get build-dep qubes-core-agent
cd qubes-core-agent*/
dpkg-buildpackage

If that means changing something in our packaging files, that's fine, lets
do it (but keep builder also compatible with old packages - so it will be
possible to build for example R3.0 packages).

If that means some additional build dependency with those snippets, it may
be an option, but preferably package should be buildable using standard
environment. Having some custom things done by qubes-builder (like
currently orig.tar.gz...) is no-go.

I agree. You will see the four packages I refactored will be able to built
in that manner since I am actually working on getting them to build with
buildd as well. At the same time I am also being careful that they will
still build for 3.0 and making sure stuff builds in a Debian environment (I
can build all the components, just can't create the template rpm yet).

I think if you don't want to have to depend on a yet to be created
qubes-builder package that would install the snippets onto the chroot
that I should consider using git-subrepo (
https://github.com/ingydotnet/git-subrepo) which would allow updating the
snippets without using submodules. I want to stay away from submodules
since there are issues using them with Debian packaging using some of the
git package tools (I got an error saying git submodules are not yet
supported).

Be as close to the upstream solution (package format and build process) as
possible.

Create Debian _.orig.tar.gz file

$(DEB_ORIG_TGZ):
@$(shell rm -f $@)
@$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)

This is not deterministic, take a look here:
https://reproducible-builds.org/docs/archives/

Also note that z option (gzip) makes it non-deterministic because of
timestamp included. AFAIR there is some option to skip it - search linked
page.

I have code to create a tar that is deterministic, just have not
implemented it yet (https://wiki.debian.org/onlyjob/get-orig-source). I am
also going to change get-orig-source to do just that and create the .orig
using the dist target.

I am starting to also use dh_buildinfo:
Register the versions of build-dependencies used to build a package.
dh_buildinfo is a debhelper program that registers in a file the list of
packages declared as build-time dependencies, as well as build-essential
packages, together with their versions, as installed in the build
machine.

As well as --list-missing which looks for files that are not in package but
built, similar to what happens at the end of building an rpm.

Finally, I also noticed none of the source packages are being signed. I
will fix that as well look into using dput to move packages to repo.

Oh, one last thing in regards to using native Debian packages like some of
the qubes packages do:
5.3. Native Debian package
https://www.debian.org/doc/manuals/debmake-doc/ch05.en.html

The non-native Debian package in the “3.0 (quilt)” format is the most
normal Debian binary package format. The above workflow and the following
packaging examples always use this format.

The native Debian package is the rare Debian binary package format. It may
be used only when the package is useful and valuable only for Debian. Thus,
its use is generally deprecated.

or ...

@$(shell git archive HEAD --output=$@)

... use this one.


Reply to this email directly or view it on GitHub
#1244 (comment)
.

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Jan 29, 2016

Member

I think if you don't want to have to depend on a yet to be created qubes-builder package that would install the snippets onto the chroot that I should consider using git-subrepo ( https://github.com/ingydotnet/git-subrepo) which would allow updating the snippets without using submodules.

If some additional snippets are really required, build-depends on them is ok. But make sure that builder package itself can be built without them (to not introduce dependency cycle)... And generally if some package can be built without such additions, do not depend on that package.

Member

marmarek commented Jan 29, 2016

I think if you don't want to have to depend on a yet to be created qubes-builder package that would install the snippets onto the chroot that I should consider using git-subrepo ( https://github.com/ingydotnet/git-subrepo) which would allow updating the snippets without using submodules.

If some additional snippets are really required, build-depends on them is ok. But make sure that builder package itself can be built without them (to not introduce dependency cycle)... And generally if some package can be built without such additions, do not depend on that package.

@marmarek marmarek modified the milestones: Release 4.0, Release 3.1 Feb 8, 2016

andrewdavidwong added a commit that referenced this issue May 31, 2016

andrewdavidwong added a commit that referenced this issue Jun 12, 2016

@rootkovska rootkovska removed the C: label Jun 30, 2016

marmarek added a commit to marmarek/qubes-builder-debian that referenced this issue Apr 25, 2017

Don't try to sign source package inside chroot, do it outside instead
There is no access to signing keys from inside of chroot.

Related QubesOS/qubes-issues#1244

marmarek added a commit to marmarek/qubes-builder-debian that referenced this issue Apr 25, 2017

Reuse timestamp of previous changelog entry for "+debXu1" one
Don't embed build time into debian/changelog, it would break all
$SOURCE_DATE_EPOCH handling.

QubesOS/qubes-issues#1244

marmarek added a commit to marmarek/qubes-builder-debian that referenced this issue Apr 25, 2017

andrewdavidwong added a commit that referenced this issue Apr 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment