Publish `.buildinfo` files somewhere #6356

legoktm · 2022-03-23T02:43:45Z

Building Debian packages generates a .buildinfo file (see spec: https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles). These files document what the build environment was at the time of building, so that if someone wanted to reproduce the build, they'd know exactly what packages they needed. Tools like debrebuild use buildinfo files to verify that packages are actually reproducible.

Debian currently has a proof-of-concept buildinfo server (https://buildinfo.debian.net/) but no official API, so we can really do whatever we want here.

My proposal is to create a new public Git repository, freedomofpress/buildinfo (or if that's too generic, freedomofpress/debian-buildinfo) that just contains these as an archive for now. Similar to the build-logs repository, maintainers will copy and commit any *.buildinfo files to this repo. CI that builds nightlies will also copy these over. We can split them into directories by month (2022-03/foo.buildinfo) to avoid one directory from being super giant. The idea would be that we always just keep adding to this, old buildinfo metadata would never be purged from it.

Anyone who wants to try rebuilding our packages can either clone the Git repo or download individual files via the GitHub web interface. And if one day a standardized API exists for serving buildinfo files, we can set that up on top of the Git repo, or import everything in the Git repo into that and archive the repo.

The text was updated successfully, but these errors were encountered:

legoktm · 2023-02-22T20:58:58Z

Kev suggested just putting these in the existing build-logs repo, I think that's probably fine.

Create a new "Build metadata" page that consolidates documentation from <https://github.com/freedomofpress/securedrop/wiki/Build-logs> and parts of the release management guides. We are also going to start requiring the publishing of `.buildinfo` files (see freedomofpress/securedrop#6754), so document how they work and where they should be published. Refs <freedomofpress/securedrop#6356>.

To theoretically try independently reproducing these builds, just like we want to for release builds. Refs <freedomofpress/securedrop#6356>.

To theoretically try independently reproducing these builds, just like we want to for release builds. This also provides the checksum of packages in a machine-readable format. We push to build-logs before apt-test to allow eventual CI in apt-test to verify that packages in that repository match the published buildinfo. Refs <freedomofpress/securedrop#6356>.

buildinfo files contain package checksums in a machine-readable format, so script checking newly added packages against those. This will be added to CI for securedrop-apt-test and securedrop-apt-prod. The main iffy part of this is how it compares against "origin/main", but I think for PRs it'll mostly do the right thing. We only check new packages because old ones don't have buildinfo published. Maybe once we no longer have any legacy cases left, we just check everything in the repository. Likely there are more checks that could be added, but this is a start. Refs <freedomofpress/securedrop#6356>.

CI verifies that newly added packages had a buildinfo file pushed to the build-logs repository and the package checksum matches that. The check-buildinfo script is added in <freedomofpress/securedrop-builder#423>. Refs <freedomofpress/securedrop#6356>.

buildinfo files contain package checksums in a machine-readable format, so script checking newly added packages against those. This will be added to CI for securedrop-apt-test and securedrop-apt-prod. The main iffy part of this is how it compares against "origin/main", but I think for PRs it'll mostly do the right thing. We only check new packages because old ones don't have buildinfo published. Maybe once we no longer have any legacy cases left, we just check everything in the repository. Likely there are more checks that could be added, but this is a start. Refs <freedomofpress/securedrop#6356>.

CI verifies that newly added packages had a buildinfo file pushed to the build-logs repository and the package checksum matches that. The check-buildinfo script is added in <freedomofpress/securedrop-builder#423>. Refs <freedomofpress/securedrop#6356>.

buildinfo files contain package checksums in a machine-readable format, so script checking newly added packages against those. This will be added to CI for securedrop-apt-test and securedrop-apt-prod. The main iffy part of this is how it compares against "origin/main", but I think for PRs it'll mostly do the right thing. We only check new packages because old ones don't have buildinfo published. Maybe once we no longer have any legacy cases left, we just check everything in the repository. Likely there are more checks that could be added, but this is a start. Refs <freedomofpress/securedrop#6356>.

CI verifies that newly added packages had a buildinfo file pushed to the build-logs repository and the package checksum matches that. The check-buildinfo script is added in <freedomofpress/securedrop-builder#423>. Refs <freedomofpress/securedrop#6356>.

legoktm · 2023-03-11T00:53:20Z

Overview of the ongoing buildinfo work:

Fix versioning to match what unattended-upgrades wants; force removal of old kernels kernel-builder#33 output buildinfo files; were committed for the first time in freedomofpress/build-logs@b75928b
Remove molecule from the package building process #6754 will start outputting buildinfo files for server packages
Publish buildinfo files to build-logs repository securedrop-builder#422 will start auto-publishing buildinfo for SDW nightlies
Add script to check newly added packages against buildinfo files securedrop-builder#423 script to verify new packages match their buildinfo, added to CI in Have CI verify new packages match buildinfo files securedrop-apt-test#188
Document .buildinfo files, remove build container documentation securedrop-dev-docs#40 documents buildinfo files

legoktm · 2023-03-15T18:35:51Z

One thing that was raised during team discussion is whether we could embed other metadata in buildinfo files, like a git version or something. At the time I had suggested that we could stick it in the environment and then it'll get picked up and exported into the Environment section. After reading the dpkg code, it has a hardcoded set of environment variables to check.

I think we can still get away with a simple cat " SD_GIT_VERSION=\"foo\"" >> foo.buildinfo though.

CI verifies that newly added packages had a buildinfo file pushed to the build-logs repository and the package checksum matches that. The check-buildinfo script was added in <freedomofpress/securedrop-builder#423>. Refs <freedomofpress/securedrop#6356>.

legoktm mentioned this issue Mar 23, 2022

Improve and streamline Debian package building process #6310

Open

10 tasks

legoktm changed the title ~~Publish .buildinfo files (see spec: https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles)~~ Publish .buildinfo files somewhere Mar 23, 2022

legoktm mentioned this issue Dec 9, 2022

Stop pinning the "builder" docker image #6706

Closed

legoktm mentioned this issue Feb 22, 2023

Remove molecule from the package building process #6754

Merged

13 tasks

legoktm mentioned this issue Mar 10, 2023

Document .buildinfo files, remove build container documentation freedomofpress/securedrop-dev-docs#40

Merged

3 tasks

legoktm mentioned this issue Mar 10, 2023

Publish buildinfo files to build-logs repository freedomofpress/securedrop-builder#422

Merged

legoktm self-assigned this Mar 10, 2023

legoktm mentioned this issue Mar 10, 2023

Add script to check newly added packages against buildinfo files freedomofpress/securedrop-builder#423

Merged

legoktm mentioned this issue Mar 10, 2023

Have CI verify new packages match buildinfo files freedomofpress/securedrop-apt-test#188

Merged

legoktm mentioned this issue Mar 24, 2023

WIP: Script to rebuild a package from buildinfo freedomofpress/securedrop-builder#433

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish `.buildinfo` files somewhere #6356

Publish `.buildinfo` files somewhere #6356

legoktm commented Mar 23, 2022 •

edited

Loading

legoktm commented Feb 22, 2023

legoktm commented Mar 11, 2023

legoktm commented Mar 15, 2023

Publish .buildinfo files somewhere #6356

Publish .buildinfo files somewhere #6356

Comments

legoktm commented Mar 23, 2022 • edited Loading

legoktm commented Feb 22, 2023

legoktm commented Mar 11, 2023

legoktm commented Mar 15, 2023

Publish `.buildinfo` files somewhere #6356

Publish `.buildinfo` files somewhere #6356

legoktm commented Mar 23, 2022 •

edited

Loading