Adds bootsrapped wheels for the build tool

``` python3 -m venv .venv source .venv/bin/activate python3 -m pip install pip-tools pip-compile --allow-unsafe --generate-hashes --output-file=requirements.txt requirements.in python3 -m pip install -r requirements.txt ./scripts/build-sync-wheels --cache ./bootstrap -p $PWD BOOTSTRAP=true ./scripts/sync-sha256sums gpg --armor --output bootstrap-sha256sums.txt.asc --detach-sig bootstrap-sha256sums.txt BOOTSTRAP=true ./scripts/verify-sha256sum-signature PKG_DIR=$PWD BOOTSTRAP=true ./scripts/update-requirements ``` This PR also updates the CI steps for the reproducible wheels test.
freedomofpress · Mar 31, 2021 · 4e7da57 · 4e7da57
1 parent cf80374
commit 4e7da57
Show file tree

Hide file tree

Showing 43 changed files with 583 additions and 48 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -255,6 +255,7 @@ jobs:
             virtualenv -p python3 .venv
             source .venv/bin/activate
             pip install -r test-requirements.txt
+            pip install --require-hashes --no-index --no-deps --no-cache-dir -r build-requirements.txt --find-links ./bootstrap/
             pytest -vvs tests/test_reproducible_wheels.py
 
   reprotest-debs:

diff --git a/README.md b/README.md
@@ -28,6 +28,37 @@ sure that you install them into the template for your debian packaging AppVM.
 The install target will configure [git-lfs](https://git-lfs.github.com/), used for storing
 binary wheel files.
 
+## Updating our bootstrapped build tools
+
+We use [build](https://pypa-build.readthedocs.io/en/latest/) toolchain to build our reproducible wheels.
+
+Remember to put your OpenPGP public key in the `pubkeys/` directory as `maintainer_name.pub`.
+Right now we have the release key, and kushal's key there.
+
+If we have to update the tool, use the following steps
+
+```
+# First create a new fresh virtualenv and install pip-tools
+python3 -m venv .venv
+source .venv/bin/activate
+python3 -m pip install pip-tools
+# Then update the requirements.in file as required
+pip-compile --allow-unsafe --generate-hashes --output-file=requirements.txt requirements.in
+python3 -m pip install -r requirements.txt
+# now we are ready for bootstrapping
+./scripts/build-sync-wheels --cache ./bootstrap -p $PWD
+# Here we have the new wheels ready
+# Now let us recreate our new sha256sums for bootstrapping
+BOOTSTRAP=true ./scripts/sync-sha256sums
+# now let us sign the list of sha256sums
+gpg --armor --output bootstrap-sha256sums.txt.asc --detach-sig  bootstrap-sha256sums.txt
+# We can even verify if we want
+BOOTSTRAP=true ./scripts/verify-sha256sum-signature
+# Update the build-requirements.txt file
+PKG_DIR=$PWD BOOTSTRAP=true ./scripts/update-requirements
+```
+
+
 ## Updating Python wheels
 
 Maintainers of `securedrop-client` and `securedrop-proxy` must ensure that
@@ -38,15 +69,27 @@ If new dependencies were added in the `build-requirements.txt` of that
 repo that are not in the FPF PyPI mirror (`./localwheels/` in this repository), then the maintainer needs
 to do the following (we are taking `securedrop-client` project as example):
 
-### 0. Create updated build-requirements.txt for the project
+### 0. Enable the virtualenv
+
+You can create a fresh virtualenv and install the build tools from our bootstrapped wheels.
+
+```
+python3 -m venv .venv
+source .venv/bin/activate
+python3 -m pip install --require-hashes --no-index --no-deps --no-cache-dir -r build-requirements.txt --find-links ./bootstrap/
+```
+
+Remember that the following steps needs to be done from the same virtual environment.
+
+### 1. Create updated build-requirements.txt for the project
 
 From the `securedrop-debian-packaging` directory,
 
 ```
 PKG_DIR=/home/user/code/securedrop-client make requirements
 ```
 
-This will create the proper `requirements.txt` file in the project directory along with the binary wheel
+This will create the proper `build-requirements.txt` file in the project directory along with the binary wheel
 hashes from our own Python package index server.
 
 If we are missing any wheels from our cache/build/server, it will let you know with a following message.
@@ -66,7 +109,7 @@ The next step is to build the wheels. To do this step, you will need an owner
 of the SecureDrop release key to build the wheel and sign the updated sha256sums file
 with the release key. If you're not sure who to ask, ping @redshiftzero for a pointer.
 
-### 1. Build wheels
+### 2. Build wheels
 
 This must be done in an environment for building production artifacts:
 
@@ -84,7 +127,7 @@ Then navigate back to the project's code directory and run the following command
 python3 setup.py sdist
 ```
 
-### 2. Commit changes to the localwheels directory (if only any update of wheels)
+### 3. Commit changes to the localwheels directory (if only any update of wheels)
 
 Now add these built artifacts to version control:
 
@@ -93,32 +136,11 @@ git add localwheels/
 git commit
 ```
 
-### 3. Update the index files for the bucket (required for Debian builds)
-
-If there is any completely new Python package (source/wheel), then only we will have to update our index.
-
-```
-./scripts/createdirs.py ~/code/securedrop-client/requirements.txt
-```
-Then update the corresponding packages's `index.html`.
-
-If there is a new package, then update the main index.
-
-```
-./scripts/updateindex.py
-```
-
 Finally, submit a PR containing the new wheels and updated files.
 If you wish to test the new wheels in a local build before submitting a PR,
 or as part of PR review, you can do so by:
 
-```
-python3 -m http.server # serve local wheels via HTTP
-vim $PKG_NAME/debian/rules # edit index URL to http://localhost:8000/simple
-```
-
-Then run e.g. `PKG_VERSION=0.0.11 make securedrop-client`, and you'll see the GET
-requests in the console running the HTTP server.
+Then run e.g. `PKG_VERSION=0.4.1 make securedrop-client` to verify that the new wheels are working.
 
 ## Make a release
 

diff --git a/bootstrap-sha256sums.txt b/bootstrap-sha256sums.txt
@@ -0,0 +1,32 @@
+3fe8fac398ae76f534dee92c0db01c2960e271094f8cd52aa9d24a633c244e59  build-0.3.0-py2.py3-none-any.whl
+0eb95b2c8d770d7c4c9b92c68c227c350bbf65f3ec83551ace9097c18cc15fdd  build-0.3.0.tar.gz
+8c9eec7e9de2a30861ca347d0a149cc1482de12fc765fa06c414930e8ce20d0a  Cython-0.29.22-cp37-cp37m-linux_x86_64.whl
+df6b83c7a6d1d967ea89a2903e4a931377634a297459652e4551734c48195406  Cython-0.29.22.tar.gz
+a14d6cca50a51eff2c418fc3f8e887cd31a3233a6b465451fd57074e232d7c4c  flit_core-2.3.0-py2.py3-none-any.whl
+a50bcd8bf5785e3a7d95434244f30ba693e794c5204ac1ee908fc07c4acdbf80  flit_core-2.3.0.tar.gz
+35fc3ab05e060b85739da17db4ce33e8532ce1010d892d178f85a211e080e1ff  importlib_metadata-3.7.0-py3-none-any.whl
+24499ffde1b80be08284100393955842be4a59c7c16bbf2738aad0e464a8e0aa  importlib_metadata-3.7.0.tar.gz
+44698376bc57a48290dc82bfdc20d339684856b24d6b8c3760b284de4eebc498  packaging-20.9-py2.py3-none-any.whl
+5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5  packaging-20.9.tar.gz
+bfe04a1859fd3cbd3bc2a1ffd2a6dbf30e912bf7fc36ed55d4c838d642c55934  pep517-0.9.1-py2.py3-none-any.whl
+aeb78601f2d1aa461960b43add204cc7955667687fbcf9cdb5170f00556f117f  pep517-0.9.1.tar.gz
+f266816461e421e2bbdb61e6e122d01fe638b5710698879ab283cc456aabd1d1  pip-21.0.1-py3-none-any.whl
+99bbde183ec5ec037318e774b0d8ae0a64352fe53b2c7fd630be1d07e94f41e5  pip-21.0.1.tar.gz
+4d00a1a6d2bf41a459fa6f47d41a1c5d157715283c2dc4a4b291972037bc9620  pyparsing-2.4.7-py2.py3-none-any.whl
+c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1  pyparsing-2.4.7.tar.gz
+3084ada00452395171c8f10199d2b5d3b74c6948518c2b6bd75e948c77ca2280  pytest_runner-5.3.0-py3-none-any.whl
+ca3f58ff4957e8be6c54c55d575b235725cbbcf4dc0d5091c29c6444cfc8a5fe  pytest-runner-5.3.0.tar.gz
+97e4f6bd5d1d2a32f82d5c6ec9bb90a1b60db3af81b6427ee9a5949021d56d7b  pytoml-0.1.21-py2.py3-none-any.whl
+8eecf7c8d0adcff3b375b09fe403407aa9b645c499e5ab8cac670ac4a35f61e7  pytoml-0.1.21.tar.gz
+d55810b08ce4cef4d4e354e4fc1013c435abf3a9712daa4f48dcfc8694f4722f  setuptools-54.0.0-py3-none-any.whl
+34efee89c4c879204f5739ec6d9d3635195b0b7d2b51e25c9261a327367ec5ff  setuptools-54.0.0.tar.gz
+96d120093688708a75155802077cc3fefd4352b2b5e30386821e86bbc4c8c402  setuptools_scm-5.0.2-py2.py3-none-any.whl
+83a0cedd3449e3946307811a4c7b9d89c4b5fd464a2fb5eeccd0a5bb158ae5c8  setuptools_scm-5.0.2.tar.gz
+931471fdf4a532e4095922de7eb81ef13da584d91af4f93e2e7284b29f56a65b  toml-0.10.2-py2.py3-none-any.whl
+b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f  toml-0.10.2.tar.gz
+98c8fbd44f53e83e4114c5669b7ecdb6da6e8f7373777234bfb44ce29401d165  typing_extensions-3.7.4.3-py3-none-any.whl
+99d4073b617d30288f569d3f13d2bd7548c3a7e4c8de87db09a9d29bb3a4a60c  typing_extensions-3.7.4.3.tar.gz
+870fcd67bd55da8e73be17468e970d365ad926ac9c5fee0e00ad778019c579d9  wheel-0.36.2-py2.py3-none-any.whl
+e11eefd162658ea59a60a0f6c7d493a7190ea4b9a85e335b33489d9f17e0245e  wheel-0.36.2.tar.gz
+8306623852c6278fc72c19b9b7297179beb75cef462bf25f150d85db858c806f  zipp-3.4.1-py3-none-any.whl
+3607921face881ba3e026887d8150cca609d517579abe052ac81fc5aeffdbd76  zipp-3.4.1.tar.gz
diff --git a/bootstrap-sha256sums.txt.asc b/bootstrap-sha256sums.txt.asc
@@ -0,0 +1,16 @@
+-----BEGIN PGP SIGNATURE-----
+
+iQIzBAABCgAdFiEEC8E1El6y/5oPiO4cxl/wB8dXZu0FAmBkfbYACgkQxl/wB8dX
+Zu1thQ/+N/xT77cYfr6QgFQPTnzqZau5mefevPunJ4WQWlwl5Fzh+hxWfkWp7vqZ
+JRonnO25bRbWC8wKZUH0WOr0XRhaopsgokKHpx/qMcmWlbiLB4LwBjI4g26fKJSS
+a8lbiKKRaBSgf0TmsM9hdn7Fu9UUwl0+iGua3RLR8hh9erdNYDovGcLZ99rYFDk8
+zfFlvFICdpN7Rj812AsKp2lnxr+AndFfvBByoUus9fZUmRkBwREVgpQxtXvj0WB1
+W8cwDh4FvnMyMg4MB3abOhP6yXFnNKIYOmEIbSRRBDikrP3IBiHnZ+Vtd4WbDG7q
+Fe6zsLFBB7l0hOF2URpFVTo/ntT040neL6B8HGD1ygfoMfGxgixN3tqdJ9aW5Q7B
+hW9sDvamlSLaDINvudaKlNcUwZ7/SBR1B9pIZ0QQyrofbRbviKObnaUUMvSDcSbX
+FsjKgOaXMiABztvitF+Ch6mUB8maTS7tJ2YB0B8YUhy0sy4rMJAV+pqmLW/diWcU
+j6KiNhWPB3oTUi+KuTcM4oLDmToVU5Iq4MvjbZpb22scYa1wAA4sjk8sS2rD3Npk
+Pa15XBafTTZSmLfwbSKFAt+IzhBrMgyM+31XPm6iYnmqzTDDkE2uZ3cutVCW/L65
+M97/1slMFYEj5tadcCkO6/UqGjsGaPg2vpFNIh72ncvBcPYyNNw=
+=z5W/
+-----END PGP SIGNATURE-----
diff --git a/bootstrap/Cython-0.29.22-cp37-cp37m-linux_x86_64.whl b/bootstrap/Cython-0.29.22-cp37-cp37m-linux_x86_64.whl
diff --git a/bootstrap/Cython-0.29.22.tar.gz b/bootstrap/Cython-0.29.22.tar.gz
diff --git a/bootstrap/build-0.3.0-py2.py3-none-any.whl b/bootstrap/build-0.3.0-py2.py3-none-any.whl
diff --git a/bootstrap/build-0.3.0.tar.gz b/bootstrap/build-0.3.0.tar.gz
diff --git a/bootstrap/flit_core-2.3.0-py2.py3-none-any.whl b/bootstrap/flit_core-2.3.0-py2.py3-none-any.whl
diff --git a/bootstrap/flit_core-2.3.0.tar.gz b/bootstrap/flit_core-2.3.0.tar.gz
diff --git a/bootstrap/importlib_metadata-3.7.0-py3-none-any.whl b/bootstrap/importlib_metadata-3.7.0-py3-none-any.whl
diff --git a/bootstrap/importlib_metadata-3.7.0.tar.gz b/bootstrap/importlib_metadata-3.7.0.tar.gz
diff --git a/bootstrap/packaging-20.9-py2.py3-none-any.whl b/bootstrap/packaging-20.9-py2.py3-none-any.whl
diff --git a/bootstrap/packaging-20.9.tar.gz b/bootstrap/packaging-20.9.tar.gz
diff --git a/bootstrap/pep517-0.9.1-py2.py3-none-any.whl b/bootstrap/pep517-0.9.1-py2.py3-none-any.whl
diff --git a/bootstrap/pep517-0.9.1.tar.gz b/bootstrap/pep517-0.9.1.tar.gz
diff --git a/bootstrap/pip-21.0.1-py3-none-any.whl b/bootstrap/pip-21.0.1-py3-none-any.whl
diff --git a/bootstrap/pip-21.0.1.tar.gz b/bootstrap/pip-21.0.1.tar.gz
diff --git a/bootstrap/pyparsing-2.4.7-py2.py3-none-any.whl b/bootstrap/pyparsing-2.4.7-py2.py3-none-any.whl
diff --git a/bootstrap/pyparsing-2.4.7.tar.gz b/bootstrap/pyparsing-2.4.7.tar.gz
diff --git a/bootstrap/pytest-runner-5.3.0.tar.gz b/bootstrap/pytest-runner-5.3.0.tar.gz
diff --git a/bootstrap/pytest_runner-5.3.0-py3-none-any.whl b/bootstrap/pytest_runner-5.3.0-py3-none-any.whl
diff --git a/bootstrap/pytoml-0.1.21-py2.py3-none-any.whl b/bootstrap/pytoml-0.1.21-py2.py3-none-any.whl
diff --git a/bootstrap/pytoml-0.1.21.tar.gz b/bootstrap/pytoml-0.1.21.tar.gz
diff --git a/bootstrap/setuptools-54.0.0-py3-none-any.whl b/bootstrap/setuptools-54.0.0-py3-none-any.whl
diff --git a/bootstrap/setuptools-54.0.0.tar.gz b/bootstrap/setuptools-54.0.0.tar.gz
diff --git a/bootstrap/setuptools_scm-5.0.2-py2.py3-none-any.whl b/bootstrap/setuptools_scm-5.0.2-py2.py3-none-any.whl
diff --git a/bootstrap/setuptools_scm-5.0.2.tar.gz b/bootstrap/setuptools_scm-5.0.2.tar.gz
diff --git a/bootstrap/toml-0.10.2-py2.py3-none-any.whl b/bootstrap/toml-0.10.2-py2.py3-none-any.whl
diff --git a/bootstrap/toml-0.10.2.tar.gz b/bootstrap/toml-0.10.2.tar.gz
diff --git a/bootstrap/typing_extensions-3.7.4.3-py3-none-any.whl b/bootstrap/typing_extensions-3.7.4.3-py3-none-any.whl
diff --git a/bootstrap/typing_extensions-3.7.4.3.tar.gz b/bootstrap/typing_extensions-3.7.4.3.tar.gz
diff --git a/bootstrap/wheel-0.36.2-py2.py3-none-any.whl b/bootstrap/wheel-0.36.2-py2.py3-none-any.whl
diff --git a/bootstrap/wheel-0.36.2.tar.gz b/bootstrap/wheel-0.36.2.tar.gz
diff --git a/bootstrap/zipp-3.4.1-py3-none-any.whl b/bootstrap/zipp-3.4.1-py3-none-any.whl
diff --git a/bootstrap/zipp-3.4.1.tar.gz b/bootstrap/zipp-3.4.1.tar.gz
diff --git a/build-requirements.txt b/build-requirements.txt
@@ -0,0 +1,16 @@
+build==0.3.0 --hash=sha256:3fe8fac398ae76f534dee92c0db01c2960e271094f8cd52aa9d24a633c244e59
+cython==0.29.22 --hash=sha256:8c9eec7e9de2a30861ca347d0a149cc1482de12fc765fa06c414930e8ce20d0a
+flit-core==2.3.0 --hash=sha256:a14d6cca50a51eff2c418fc3f8e887cd31a3233a6b465451fd57074e232d7c4c
+importlib-metadata==3.7.0 --hash=sha256:35fc3ab05e060b85739da17db4ce33e8532ce1010d892d178f85a211e080e1ff
+packaging==20.9 --hash=sha256:44698376bc57a48290dc82bfdc20d339684856b24d6b8c3760b284de4eebc498
+pep517==0.9.1 --hash=sha256:bfe04a1859fd3cbd3bc2a1ffd2a6dbf30e912bf7fc36ed55d4c838d642c55934
+pyparsing==2.4.7 --hash=sha256:4d00a1a6d2bf41a459fa6f47d41a1c5d157715283c2dc4a4b291972037bc9620
+pytest-runner==5.3.0 --hash=sha256:3084ada00452395171c8f10199d2b5d3b74c6948518c2b6bd75e948c77ca2280
+pytoml==0.1.21 --hash=sha256:97e4f6bd5d1d2a32f82d5c6ec9bb90a1b60db3af81b6427ee9a5949021d56d7b
+setuptools-scm==5.0.2 --hash=sha256:96d120093688708a75155802077cc3fefd4352b2b5e30386821e86bbc4c8c402
+toml==0.10.2 --hash=sha256:931471fdf4a532e4095922de7eb81ef13da584d91af4f93e2e7284b29f56a65b
+typing-extensions==3.7.4.3 --hash=sha256:98c8fbd44f53e83e4114c5669b7ecdb6da6e8f7373777234bfb44ce29401d165
+wheel==0.36.2 --hash=sha256:870fcd67bd55da8e73be17468e970d365ad926ac9c5fee0e00ad778019c579d9
+zipp==3.4.1 --hash=sha256:8306623852c6278fc72c19b9b7297179beb75cef462bf25f150d85db858c806f
+pip==21.0.1 --hash=sha256:f266816461e421e2bbdb61e6e122d01fe638b5710698879ab283cc456aabd1d1
+setuptools==54.0.0 --hash=sha256:d55810b08ce4cef4d4e354e4fc1013c435abf3a9712daa4f48dcfc8694f4722f