Tools for deterministic builds of standalone binaries and distribution archives #121

achow101 · 2019-02-10T06:31:34Z

This PR adds several scripts and tools for making standalone binaries of the hwi.py script and for creating distribution archives that will go on pypi.org.

To achieve deterministic builds, the dependencies used must be locked to specific versions and hashes. To do this, I have added configuration files for using the Poetry dependency manager. Because Poetry uses a pyproject.toml file instead of setup.py, I have created a helper script which will automatically generate the proper setup.py file from pyproject.toml. The reason I chose Poetry instead of Pipenv for this task is because it has the ability to do deterministic builds of the distribution archives (python wheel and source tar for pypi.org) which Pipenv does not have.

Additionally scripts have been added to the newly created contrib/ folder which will perform the deterministic builds of the binaries and distribution archives. The builds of the binaries are done using pyinstaller. In order to build for different platforms, the contrib/build_bin.sh script needs to be run on each of the platforms we wish to release for. It can also be run in wine to do windows builds (see contrib/build_wine.sh). The configuration file that pyinstaller needs has also been added.

Lastly the pyenv version for this project has been bumped to 3.6.8 since using python 3.5.x produced standalone binaries that did not work.

This PR is built on #120 as reducing the number of dependencies fixed several issues with the standalone binary builds.

Sjors · 2019-02-10T15:32:06Z

Concept ACK. Will review after 120 is merged.

I'm thinking of adding a (mandatory?) -signer_hash argument for bitcoind in bitcoin/bitcoin#14912, so Bitcoin Core can first run sha256 against the driver before calling it.

instagibbs

I'm not really well-equipped to review this.

What's the suggested review process?

instagibbs · 2019-02-14T20:01:47Z

docs/release-process.md

@@ -0,0 +1,8 @@
+# Release Process
+
+1. Bump version number in `pyproject.toml`, generate the setup.py file, and tag release


git tag? what tag?

Git tag.

Clarified.

achow101 · 2019-02-21T21:39:24Z

The best way to review this is probably to just have everyone do the build and check that their hashes match. All of the builds can be done with contrib/build_bin.sh && contrib/build_dist.sh && contrib/build_wine.sh. Note that contrib/build_bin.sh builds the binary for your type of system, so it will differ between e.g. linux and MacOS.

These are what I got on commit 2b5f16f:

$ sha256sum dist/*
e5009335aa1a73edaf209182af271c812c686aee6da6831d4ccb442bdade42a4  dist/hwi
ceae5bfc684f3f17cf25e901055893734b69a3c93ea131e1292c7eabdf6c14b4  dist/hwi-0.0.5-py3-none-any.whl
b250bff5c054ffcc08163ac42d0879b9e098b57bf287c4287ffe72118c96d88c  dist/hwi-0.0.5.tar.gz
167b96ed351bddefa00440f8f9be0453ea46c91aa7dee1e2feffbddb571ac688  dist/hwi.exe

Once I get access to a Mac, I'll do the MacOS build too.

MacOS build:

a6e64a4120859cce21ddca7b72f741f789e6dfc30698e72d61f76dce5d899905  dist/hwi

Sjors · 2019-02-26T10:58:05Z

I had to jump through some hoops on macOS, including PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.6.8 (see pyenv/pyenv#1095). Hopefully that doesn't impact using the binaries.

For macOS maybe point to brew install libfaketime wine p7zip

The wine part doesn't work for me, see logs.

I'm get the same checksum for dist/hwi-0.0.5-py3-none-any.whl, but the others are different.

ef74d3df7a63876fa6cb5db8026294eacdb60a3909dc9b65f7c1274269ac8406  dist/hwi
ceae5bfc684f3f17cf25e901055893734b69a3c93ea131e1292c7eabdf6c14b4  dist/hwi-0.0.5-py3-none-any.whl
bdc1a36230a99a2e5c222ef16831ddef3fb9301ffc7a450a8c69eed161c3fcc6  dist/hwi-0.0.5.tar.gz

achow101 · 2019-02-26T16:42:51Z

For macOS, I had issues getting faketime to work. I have a fix for poetry that removes the need to use faketime but is waiting to be merged. The difference for dist/hwi-0.0.5.tar.gz is expected because of this.

I think the reason the resulting binaries are different are because the installed python binaries are different since we are using pyenv which compiles locally. I'll have to investigate this.

fanquake · 2019-03-01T03:05:54Z

docs/release-process.md

+
+Now install [Poetry](https://github.com/sdispater/poetry) with `pip install --user poetry`
+
+You will also need to install `faketime`. On Ubuntu/Debian: `sudo apt install faketime`. On Mac: `brew install faketime`


brew install libfaketime

fanquake · 2019-03-01T03:22:30Z

Nice. Getting setup was easy (on macOS 10.14.3).

Building with 88ce6ab, I'm getting these hashes:

976354df3090a8b910fea64889481528be5db5ae75ec6b7771521aa90d7a2c8b  HWI/dist/hwi
ceae5bfc684f3f17cf25e901055893734b69a3c93ea131e1292c7eabdf6c14b4  HWI/dist/hwi-0.0.5-py3-none-any.whl
a4179a11c38cb7e6f84a1b2e2d93c912a3eb035b661a7a3172cb48316649cdbd  HWI/dist/hwi-0.0.5.tar.gz

When I try and do the windows build contrib/build_wine.sh, I'm seeing a page fault.
Backtrace and log prior:

backtrace.txt

You should consider upgrading via the 'python -m pip install --upgrade pip' command.
003b:fixme:ntdll:NtQueryInformationJobObject stub: 0x48 9 0x22f960 144 0x22f8d0
003d:fixme:msvcrt:_configure_wide_argv (1) stub
003d:fixme:msvcrt:_initialize_wide_environment stub
003d:fixme:winsock:set_dont_fragment IP_DONTFRAGMENT for IPv6 not supported in this platform
003d:fixme:ntdll:server_get_file_info Unsupported info class e
003d:fixme:ntdll:server_get_file_info Unsupported info class e
0040:fixme:ntdll:server_get_file_info Unsupported info class e
0042:fixme:ntdll:NtQueryInformationJobObject stub: 0x20 9 0x23e420 144 0x23e390
0044:fixme:msvcrt:_configure_wide_argv (1) stub
0044:fixme:msvcrt:_initialize_wide_environment stub
0044:fixme:ntdll:server_get_file_info Unsupported info class e
0044:fixme:ntdll:server_get_file_info Unsupported info class e
0044:fixme:ntdll:server_get_file_info Unsupported info class e
wine: Unhandled page fault on read access to 0xffffffffffffffff at address 0x14003301d (thread 0042), starting debugger...

pyproject.toml contains everything that was in the setup.py. The setup.py file is replaced with the one that poetry automatically generates.

achow101 · 2019-03-02T05:33:10Z

I've investigated the non-determinism and I think I've finally fixed it. Since for linux and macOS we are using pyenv which self compiles python, the python installation that each person was using was non-deterministic and thus introducing non-determinism. There are now instructions in docs/release-process.md which explain how to create a deterministic python install with pyenv. To test this, you'll need to uninstall Python 3.6.8 from pyenv first, and then you can install the deterministic one.

I've also added more to the build_bin.sh script which makes the python library files included in the build to be deterministic as building python deterministically is not enough.

Lastly I've added another travis job that builds the binary and checks that it works.

Below are the hashes of the result of the complete build as of 464c51d .

14126ca063b72e357b36b52a3e44b413c48ed707c3ffb92af93da209feac1898  dist/hwi
7050ab9b4c50c3735d64ed02834b4528b4bf44b6c90cf5f7316f8bce045b03f2  dist/hwi-0.0.5-py3-none-any.whl
e74a8a333deb1aabcee23f17bda063edeb68de93f0bdd9ffd8c92f66817229d8  dist/hwi-0.0.5.tar.gz
2bea2fd3e3ea55ec75a28300eefafa8fd58e31b175226c330ab1fe650693b9c4  dist/hwi.exe

I haven't done the macOS build yet. I will do so once I get to a mac. I will also investigate the wine issue on macOS.

fanquake · 2019-03-04T02:48:48Z

Tested building at 464c51d:

zipped /dist - dist.zip.

c59c643acd2c3ed2a75ba6229674e9e24743ab8b76689baef2f283dff3d4b286 dist/hwi
7050ab9b4c50c3735d64ed02834b4528b4bf44b6c90cf5f7316f8bce045b03f2 dist/hwi-0.0.5-py3-none-any.whl
77bad3308ed5bb9a3c85cc24076abd37759b57e68a00eba50a0baa52b6646639 dist/hwi-0.0.5.tar.gz

contrib/build_wine.sh fails with:

Extracting archive: libusb.7z
--
Path = libusb.7z
Type = 7z
Physical Size = 980895
Headers Size = 682
Method = LZMA2:23 BCJ
Solid = +
Blocks = 2

Everything is Ok

Folders: 18
Files: 32
Size:       6294517
Compressed: 980895
0032:fixme:msvcrt:_configure_wide_argv (1) stub
0032:fixme:msvcrt:_initialize_wide_environment stub
Traceback (most recent call last):
  File "c:\python3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "c:\python3\lib\site-packages\pip\__main__.py", line 16, in <module>
    from pip._internal import main as _main  # isort:skip # noqa
  File "c:\python3\lib\site-packages\pip\_internal\__init__.py", line 4, in <module>
    import locale
  File "c:\python3\lib\locale.py", line 16, in <module>
    import re
  File "c:\python3\lib\re.py", line 283, in <module>
    _pattern_type = type(sre_compile.compile("", 0))
  File "c:\python3\lib\sre_compile.py", line 579, in compile
    groupindex, indexgroup
TypeError: compile() argument 6 must be tuple, not list

achow101 · 2019-03-04T05:12:40Z

It looks like faketime isn't working on mac which is why dist/hwi-0.0.5.tar.gz have different hashes.

achow101 · 2019-03-07T05:14:17Z

I've updated the dockerfile to use debian stretch-slim as the base image. There's also a travis job that runs the determinstic build, tests the linux binary, and outputs the hashes of the result. Note that it doesn't do the Windows build because that (for some reason) has a tendency to hang. Another travis job does the macOS build but does not run any tests.

The hashes I got were:

503f375e671264dd6ab83f33d06568209e0a6a50773c12694f9d5b9df552383b  dist/hwi
7050ab9b4c50c3735d64ed02834b4528b4bf44b6c90cf5f7316f8bce045b03f2  dist/hwi-0.0.5-py3-none-any.whl
f450ecd14dcbb508321bc3d7151a516f94b6ba3876767a8615314d2ff6410279  dist/hwi-0.0.5.tar.gz
4e7130b4eef7a76d262a4a0a4139baf9304b10b42390b6ad89f50e9ee3cf0cbb  dist/hwi.exe

fanquake · 2019-03-07T06:44:58Z

I'm seeing the same hashes as you:

root@e10a4cdc76f4:/opt/hwi# sha256sum dist/*
503f375e671264dd6ab83f33d06568209e0a6a50773c12694f9d5b9df552383b  dist/hwi
7050ab9b4c50c3735d64ed02834b4528b4bf44b6c90cf5f7316f8bce045b03f2  dist/hwi-0.0.5-py3-none-any.whl
f450ecd14dcbb508321bc3d7151a516f94b6ba3876767a8615314d2ff6410279  dist/hwi-0.0.5.tar.gz
4e7130b4eef7a76d262a4a0a4139baf9304b10b42390b6ad89f50e9ee3cf0cbb  dist/hwi.exe

As mentioned, sometimes the Windows build hangs (using contrib/build_wine.sh), particularly when installing dependencies via poetry. However running the commands manually inside the container works fine..

Adds builds scripts that are used to build releases deterministically. Also adds documentation that explains the release process and what the build scripts do.

achow101 · 2019-03-08T06:06:06Z

I've made a few changes to .travis.yml to build the macOS binary with an older version for better compatibility. That travis job will also upload the resulting binary to an AWS S3 bucket. That upload will be what I will use for releases.

Can someone with a mac test that this binary works and actually communicates with devices? It can be downloaded from https://s3.amazonaws.com/hwi-travis-builds/achow101/hardware-wallet-interface/114/114.4/hwi. This was built in a travis job on my personal fork of this repo: https://travis-ci.com/achow101/hardware-wallet-interface/jobs/183298042.

Sjors · 2019-03-08T09:13:48Z

wget https://s3.amazonaws.com/hwi-travis-builds/achow101/hardware-wallet-interface/114/114.4/hwi
sha256sum hwi
d9acc15b507ce2079134b372e68278f71780ff0c762249c906778aefdf47f5f4  hwi
chmod +x hoi

enumerate and getkeypool work

achow101 · 2019-03-08T20:03:18Z

Merging as there is agreement on the produced binary hashes and that they all work.

… distribution archives 8931ae0 Run tests using the binary distribution (Andrew Chow) 39a6fc9 Fixes for Windows (Andrew Chow) a229de0 Update .travis.yml to use poetry for build (Andrew Chow) 77257a1 Add build scripts and documentation for building releases (Andrew Chow) 9e04d1a Add a hwi.spec file for pyinstaller to build standalone binaries (Andrew Chow) d6b24b8 Add pyproject.toml and poetry.lock for poetry dependency manager (Andrew Chow) Pull request description: This PR adds several scripts and tools for making standalone binaries of the `hwi.py` script and for creating distribution archives that will go on pypi.org. To achieve deterministic builds, the dependencies used must be locked to specific versions and hashes. To do this, I have added configuration files for using the [Poetry dependency manager](https://github.com/sdispater/poetry). Because Poetry uses a `pyproject.toml` file instead of `setup.py`, I have created a helper script which will automatically generate the proper `setup.py` file from `pyproject.toml`. The reason I chose Poetry instead of Pipenv for this task is because it has the ability to do deterministic builds of the distribution archives (python wheel and source tar for pypi.org) which Pipenv does not have. Additionally scripts have been added to the newly created `contrib/` folder which will perform the deterministic builds of the binaries and distribution archives. The builds of the binaries are done using [pyinstaller](http://www.pyinstaller.org/). In order to build for different platforms, the `contrib/build_bin.sh` script needs to be run on each of the platforms we wish to release for. It can also be run in wine to do windows builds (see `contrib/build_wine.sh`). The configuration file that pyinstaller needs has also been added. Lastly the pyenv version for this project has been bumped to 3.6.8 since using python 3.5.x produced standalone binaries that did not work. This PR is built on #120 as reducing the number of dependencies fixed several issues with the standalone binary builds. Tree-SHA512: abc1a6ac06d663b1316cde254980b0b1e8c392a6ffe478710df7c8e48a344cd57105e83555ec8fdcdc30e2b7d6d9cd6464367afda80653cb3cbc3acaf6119f48

achow101 force-pushed the det-build-poetry branch 2 times, most recently from 3521cf1 to fd79eb0 Compare February 14, 2019 06:04

instagibbs mentioned this pull request Feb 14, 2019

TODO for 1.0 release #91

Closed

12 tasks

instagibbs reviewed Feb 14, 2019

View reviewed changes

achow101 force-pushed the det-build-poetry branch from fd79eb0 to ade217c Compare February 14, 2019 22:10

achow101 added this to the 1.0 milestone Feb 14, 2019

achow101 force-pushed the det-build-poetry branch from ade217c to 2b5f16f Compare February 21, 2019 21:32

achow101 force-pushed the det-build-poetry branch 2 times, most recently from d54402d to 88ce6ab Compare March 1, 2019 02:28

fanquake reviewed Mar 1, 2019

View reviewed changes

Add pyproject.toml and poetry.lock for poetry dependency manager

d6b24b8

pyproject.toml contains everything that was in the setup.py. The setup.py file is replaced with the one that poetry automatically generates.

achow101 force-pushed the det-build-poetry branch 10 times, most recently from 00f818f to c47eb22 Compare March 2, 2019 05:30

achow101 force-pushed the det-build-poetry branch from c47eb22 to 464c51d Compare March 2, 2019 06:08

achow101 force-pushed the det-build-poetry branch 8 times, most recently from 2bf186c to 44269e3 Compare March 7, 2019 05:07

achow101 force-pushed the det-build-poetry branch 7 times, most recently from 56674ab to 346f624 Compare March 8, 2019 04:58

achow101 added 4 commits March 8, 2019 00:17

Add a hwi.spec file for pyinstaller to build standalone binaries

9e04d1a

Add build scripts and documentation for building releases

77257a1

Adds builds scripts that are used to build releases deterministically. Also adds documentation that explains the release process and what the build scripts do.

Update .travis.yml to use poetry for build

a229de0

Fixes for Windows

39a6fc9

achow101 force-pushed the det-build-poetry branch from 346f624 to b3f750d Compare March 8, 2019 05:17

Run tests using the binary distribution

8931ae0

achow101 force-pushed the det-build-poetry branch from b3f750d to 8931ae0 Compare March 8, 2019 05:42

achow101 merged commit 8931ae0 into bitcoin-core:master Mar 8, 2019

SomberNight mentioned this pull request May 1, 2021

Reproducible build for macOS/OSX (.dmg) spesmilo/electrum#7266

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tools for deterministic builds of standalone binaries and distribution archives #121

Tools for deterministic builds of standalone binaries and distribution archives #121

achow101 commented Feb 10, 2019

Sjors commented Feb 10, 2019 •

edited

instagibbs left a comment

instagibbs Feb 14, 2019

achow101 Feb 14, 2019

achow101 commented Feb 21, 2019 •

edited

Sjors commented Feb 26, 2019

achow101 commented Feb 26, 2019 •

edited

fanquake Mar 1, 2019

fanquake commented Mar 1, 2019 •

edited

achow101 commented Mar 2, 2019 •

edited

fanquake commented Mar 4, 2019

achow101 commented Mar 4, 2019

achow101 commented Mar 7, 2019 •

edited

fanquake commented Mar 7, 2019

achow101 commented Mar 8, 2019

Sjors commented Mar 8, 2019 •

edited

achow101 commented Mar 8, 2019

		@@ -0,0 +1,8 @@
		# Release Process

		1. Bump version number in `pyproject.toml`, generate the setup.py file, and tag release


		Now install [Poetry](https://github.com/sdispater/poetry) with `pip install --user poetry`

		You will also need to install `faketime`. On Ubuntu/Debian: `sudo apt install faketime`. On Mac: `brew install faketime`

Tools for deterministic builds of standalone binaries and distribution archives #121

Tools for deterministic builds of standalone binaries and distribution archives #121

Conversation

achow101 commented Feb 10, 2019

Sjors commented Feb 10, 2019 • edited

instagibbs left a comment

Choose a reason for hiding this comment

instagibbs Feb 14, 2019

Choose a reason for hiding this comment

achow101 Feb 14, 2019

Choose a reason for hiding this comment

achow101 commented Feb 21, 2019 • edited

Sjors commented Feb 26, 2019

achow101 commented Feb 26, 2019 • edited

fanquake Mar 1, 2019

Choose a reason for hiding this comment

fanquake commented Mar 1, 2019 • edited

achow101 commented Mar 2, 2019 • edited

fanquake commented Mar 4, 2019

achow101 commented Mar 4, 2019

achow101 commented Mar 7, 2019 • edited

fanquake commented Mar 7, 2019

achow101 commented Mar 8, 2019

Sjors commented Mar 8, 2019 • edited

achow101 commented Mar 8, 2019

Sjors commented Feb 10, 2019 •

edited

achow101 commented Feb 21, 2019 •

edited

achow101 commented Feb 26, 2019 •

edited

fanquake commented Mar 1, 2019 •

edited

achow101 commented Mar 2, 2019 •

edited

achow101 commented Mar 7, 2019 •

edited

Sjors commented Mar 8, 2019 •

edited