Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the whole qubes-builder deterministic #816

Open
marmarek opened this issue Mar 8, 2015 · 9 comments
Open

Make the whole qubes-builder deterministic #816

marmarek opened this issue Mar 8, 2015 · 9 comments

Comments

@marmarek
Copy link
Member

@marmarek marmarek commented Mar 8, 2015

Reported by joanna on 11 Apr 2014 09:09 UTC
... to allow easy, independent comparison of rpms/ISOs build from the same sources on different machines and by different people.

In the future: allow rpm/iso signatures by ""M out of N" signers.

Migrated-From: https://wiki.qubes-os.org/ticket/816

@marmarek
Copy link
Member Author

@marmarek marmarek commented Mar 8, 2015

Comment by joanna on 11 Apr 2014 09:11 UTC
We need deterministic gcc, rpmbuld, and probably many other tools.

A great task for somebody from the community! :)

@marmarek marmarek added this to the Release 3 milestone Mar 8, 2015
@marmarek
Copy link
Member Author

@marmarek marmarek commented Mar 8, 2015

Comment by marmarek on 17 Apr 2014 00:05 UTC
Some thoughts on similar idea from Debian project:
https://wiki.debian.org/ReproducibleBuilds

@marmarek
Copy link
Member Author

@marmarek marmarek commented Mar 8, 2015

Modified by joanna on 20 Apr 2014 17:12 UTC

@marmarek marmarek added C: builder and removed C: other labels Mar 8, 2015
@marmarek
Copy link
Member Author

@marmarek marmarek commented Mar 8, 2015

Comment by axon on 11 Aug 2014 09:05 UTC
Historical reference:
https://groups.google.com/d/topic/qubes-devel/D2Ca4Ef-dh4/discussion

@marmarek marmarek modified the milestones: Release 3.1, Release 3.0 May 13, 2015
@marmarek
Copy link
Member Author

@marmarek marmarek commented Sep 26, 2015

@andrewdavidwong
Copy link
Member

@andrewdavidwong andrewdavidwong commented Dec 24, 2016

In the future: allow rpm/iso signatures by ""M out of N" signers.

Since this is actually a separate issue, I've created #2535.

marmarek added a commit to marmarek/old-qubes-builder-debian that referenced this issue Nov 3, 2017
This replace custom code based on debootstrap and apt-get. The major
gain here is much smaller build environment (only declared dependencies
are installed), and also sharing tools with the upstream. Also it
simplifies the builder a lot.

The downside is slower operation (pbuilder re-install all the build
dependencies each time).

This work was done collectively with @HW42, during Reproducible Builds
Summit.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Dec 13, 2018
marmarek added a commit to marmarek/qubes-builder that referenced this issue Dec 13, 2018
marmarek added a commit to marmarek/qubes-builder that referenced this issue Dec 13, 2018
marmarek added a commit to marmarek/qubes-builder that referenced this issue Dec 13, 2018
Make source tarball clean, without build outputs.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-debian that referenced this issue Dec 13, 2018
This replace custom code based on debootstrap and apt-get. The major
gain here is much smaller build environment (only declared dependencies
are installed), and also sharing tools with the upstream. Also it
simplifies the builder a lot.

The downside is slower operation (pbuilder re-install all the build
dependencies each time).

This work was done collectively with @HW42, during Reproducible Builds
Summit.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-debian that referenced this issue Dec 14, 2018
It's required for key handling if pbuilder is run on Ubuntu (including
Travis CI).

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder that referenced this issue Dec 15, 2018
We install newer pbuilder, which is in new package format (the one
having contri.tar.xz, instead of control.tar.gz).

QubesOS/qubes-issues#816
marmarek added a commit to QubesOS/qubes-builder-debian that referenced this issue Feb 14, 2019
This replace custom code based on debootstrap and apt-get. The major
gain here is much smaller build environment (only declared dependencies
are installed), and also sharing tools with the upstream. Also it
simplifies the builder a lot.

The downside is slower operation (pbuilder re-install all the build
dependencies each time).

This work was done collectively with @HW42, during Reproducible Builds
Summit.

QubesOS/qubes-issues#816
marmarek added a commit to QubesOS/qubes-builder-debian that referenced this issue Feb 14, 2019
This replace custom code based on debootstrap and apt-get. The major
gain here is much smaller build environment (only declared dependencies
are installed), and also sharing tools with the upstream. Also it
simplifies the builder a lot.

The downside is slower operation (pbuilder re-install all the build
dependencies each time).

This work was done collectively with @HW42, during Reproducible Builds
Summit.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-debian that referenced this issue Feb 15, 2019
This replace custom code based on debootstrap and apt-get. The major
gain here is much smaller build environment (only declared dependencies
are installed), and also sharing tools with the upstream. Also it
simplifies the builder a lot.

The downside is slower operation (pbuilder re-install all the build
dependencies each time).

This work was done collectively with @HW42, during Reproducible Builds
Summit.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-linux-deb that referenced this issue Feb 16, 2019
A preliminary method for distributing buildinfo files.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-debian that referenced this issue Feb 18, 2019
If repository configuration is created by builder-debian plugin (not the
case for official repositories), include buildinfo files too. Useful for
reproducing packages.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Mar 2, 2019
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Mar 2, 2019
Call rpmbuildinfo script in the build environment to collect relevant
information.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Mar 2, 2019
Since RPM packages are signed, include also signed checksums. This
should ease matching buildinfo files to specific binary packages to
verify. Note it isn't possible to reproduce the signature without access
to private key, so for actual verification one needs to strip the
signature anyway.

Once buildinfo contains final hashes, sign the buildinfo file itself.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Mar 2, 2019
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Mar 2, 2019
Call rpmbuildinfo script in the build environment to collect relevant
information.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Mar 2, 2019
Since RPM packages are signed, include also signed checksums. This
should ease matching buildinfo files to specific binary packages to
verify. Note it isn't possible to reproduce the signature without access
to private key, so for actual verification one needs to strip the
signature anyway.

Once buildinfo contains final hashes, sign the buildinfo file itself.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Mar 8, 2019
Call rpmbuildinfo script in the build environment to collect relevant
information.

QubesOS/qubes-issues#816
marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Mar 8, 2019
Since RPM packages are signed, include also signed checksums. This
should ease matching buildinfo files to specific binary packages to
verify. Note it isn't possible to reproduce the signature without access
to private key, so for actual verification one needs to strip the
signature anyway.

Once buildinfo contains final hashes, sign the buildinfo file itself.

QubesOS/qubes-issues#816
unman added a commit to unman/live that referenced this issue Jul 16, 2019
This replace custom code based on debootstrap and apt-get. The major
gain here is much smaller build environment (only declared dependencies
are installed), and also sharing tools with the upstream. Also it
simplifies the builder a lot.

The downside is slower operation (pbuilder re-install all the build
dependencies each time).

This work was done collectively with @HW42, during Reproducible Builds
Summit.

QubesOS/qubes-issues#816
@marmarek
Copy link
Member Author

@marmarek marmarek commented Aug 9, 2019

Status:

  • Debian packages for R4.0 build twice in the same environment are reproducible (at least we don't have timestamp issues)
  • Fedora packages for R4.0 dom0 are not (too old rpm), R4.1 should be fine
  • Fedora packages for fc30 VM mostly are reproducible (tested local build vs travis) - with exception to very few packages like core-agent-linux (see #4625)

Besides packages itself, some tooling is missing to reproduce environment (frozen set of dependencies). We do have buildinfo files, but not an automated way to prepare build environment based on it.

cc @iprid23

@garlicgambit
Copy link

@garlicgambit garlicgambit commented Jul 26, 2020

@marmarek what is the current status of reproducible builds? Is the Qubes install iso deterministically reproducible?

@marmarek
Copy link
Member Author

@marmarek marmarek commented Jul 26, 2020

No, it isn't. Not much changed from the comment above, but we do have scheduled some work on this later this year.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.