Permalink
Switch branches/tags
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
137 lines (105 sloc) 5.58 KB

Overview

The piwheels project is designed to automate building of wheels from packages on PyPI for a set of pre-configured ABIs. As the name suggests, it was originally built for Raspberry Pis but there's nothing particular in the codebase that should limit it to that platform. The system relies on the following components:

Component Description
:doc:`master` Coordinates the various build slaves, using the database to store all relevant information, and keeps the web site up to date.
:doc:`slaves` Builds package on behalf of the piwheels master. Is intended to run on separate machines to the master, partly for performance and partly for security.
:doc:`monitor` Provides a friendly curses-based UI for interacting with the piwheels master.
:doc:`sense` Provides a friendly Sense HAT-based UI for interacting with the piwheels master.
:doc:`initdb` A simple maintenance script for initializing or upgrading the database to the current version.
:doc:`importer` A tool for importing wheels manually into the piwheels database and file-system.
:doc:`remove` A tool for manually removing builds from the database and file-system.
database server Currently only PostgreSQL is supported (and frankly that's all we're ever likely to support). This provides the master's data store.
web server Anything that can serve from a static directory is fine here. We use Apache in production.

Note

At present the master is a monolithic application, but the internal architecture is such that it could, in future, be split into three parts: one that deals exclusively with the database server, one that deals exclusively with the file-system served by the web server, and one that talks to the piwheels slave and monitor processes.

Deployment

A typical deployment of the master service on a Raspbian server goes something like this (all chunks assume you start as root):

  1. Install the pre-requisite software:

    # apt install postgresql-9.6 apache2 python3-psycopg2 python3-geoip
    # apt install python3-sqlalchemy python3-urwid python3-zmq
    # pip install piwheels[monitor,master,log]
  2. Set up the (unprivileged) piwheels user and the output directory:

    # groupadd piwheels
    # useradd -g piwheels -m piwheels
    # mkdir /var/www/piwheels
    # chown piwheels:piwheels /var/www/piwheels
  3. Set up the database:

    # su - postgres
    $ createuser piwheels
    $ createdb -O postgres piwheels
    $ piw-initdb
  4. Set up the web server:

    • Point the document root to the output path (:file:`/var/www/piwheels` above, but it can be anywhere your piwheels user has write access to; naturally you want to make sure your web-server's user only has read access to the location).
    • Set up SSL for the web server (e.g. with Let's Encrypt; the dehydrated utility is handy for getting and maintaining the SSL certificates).
  5. Start the master running (it'll take quite a while to populate the list of packages and versions from PyPI on the initial run so get this going before you start bringing up build slaves):

    # su - piwheels
    $ piw-master -v
  6. Deploy some build slaves on separate machines:

    # wget https://raw.githubusercontent.com/bennuttall/piwheels/master/deploy_slave.sh
    # chmod +x deploy_slave.sh
    # ./deploy_slave.sh
  7. Start the build slave running (assuming your master's IP address is 10.0.0.1):

    # su - piwheels
    $ piw-slave -v -m 10.0.0.1

Upgrades

The master will check that build slaves have the same version number and will reject them if they do not. Furthermore, it will check the version number in the database's configuration table matches its own and fail if it does not. Re-run the :program:`piw-initdb` script as the postgres super-user to upgrade the database between versions (downgrades are not supported, so take a backup first!).