Reproducible Builds Collector

This is the very early development of a collector for rbvf. The rbvf files contains rebuild results to show what distributed binaries of open source projects are reproducible. Results are stored in a database to generate stats, warnings and if provided by the rebuilder, artifacts like diffoscope outputs or actual binaries.

The collector essentially uses origins and rebuilders, where origins are open source projects like Archlinux, Debian or OpenWrt and rebuilders are whoever wants to verify that files provided on download servers are what they state. Those two information sources are stored and kept track on.

Currently a OpenWrt snapshot verifier is implemented, however more projects are planned.

Design

The collector tries to be modular to support a variation of origins and rebuilders. Find current implementations in the folders sources/ and results/. All fetched data is stored in either a Sqlite or PostgreSQL database, later render via Jinja2 to a static website.

Currently supported source fetching from origins:

• Archlinux
• OpenWrt

Currently supported result fetching from rebuilders:

• GitLab CI artifacts
• GitHub CI artifacts
• Rebuilderd

Templates for the website are found in the templates/ folder.

Development setup

Ideally setup a Python virtual environment and install all requirements and the collector itself via the following command:

pip install -e .

Copy config.yml.example over to config.yml and insert secret tokens in case you want to use the artifacts from GitLab CI or GitHub CI.

Using test data for the database

Instead of parsing remote sources it is also possible to use test data. To load the data from ./tests/testdata run the following command:

rbcollector util load --dir ./tests/testdata

Rendering the site

To render all content of the database run the following command:

rbcollector render-site

The default destination is ./public.

Setup PostgreSQL database

Run the following command to setup a PostgreSQL database. Once done the database in config.yml can be modified to use PostgreSQL instead of Sqlite.

psql --username postgres -c "CREATE DATABASE collector"
psql --username postgres -c "CREATE USER collector WITH PASSWORD 'secret'"
psql --username postgres -c "GRANT ALL ON DATABASE collector TO collector"