Reproducible builds

- Build releases in a consistent Docker container environment.

- Drop PE timestamps in Windows exe builds.

- Pack tar.gz/zip archives with static/removed metadata attrs and stable
  file ordering.

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Reproducible builds using Docker
+
 - Electrum plugin: Fix hot wallet test (#47)
 
 - Electrum: Fix docker image libssl dependency with the `http` feature (#48)

README.md

@@ -39,6 +39,7 @@ Support development: [⛓️ on-chain or ⚡ lightning via BTCPay](https://btcpa
   - [Miscellaneous](#miscellaneous)
 - [Web Hooks](#web-hooks)
 - [Developing](#developing) 👩‍💻
+- [Reproducible builds](#reproducible-builds)
 - [Thanks](#thanks)
 
 ## Intro
@@ -90,6 +91,8 @@ $ ./bwt-0.1.4-x86_64-linux/bwt --xpub <xpub> ...
 
 The signature verification should show `Good signature from "Nadav Ivgi <nadav@shesek.info>" ... Primary key fingerprint: FCF1 9B67 ...` and `bwt-0.1.4-x86_64-linux.tar.gz: OK`.
 
+The builds are reproducible and can be verified against Travis CI. See [more details here](#reproducible-builds).
+
 #### From source
 
 [Install Rust](https://rustup.rs/) and:
@@ -115,7 +118,7 @@ $ bwt --xpub <xpub>
 Assuming your bitcoin datadir is at `~/.bitcoin`,
 
 ```bash
-$ docker run -it --net host -v ~/.bitcoin:/bitcoin shesek/bwt --xpub <xpub> ...
+$ docker run -it --net host -v ~/.bitcoin:/bitcoin:ro shesek/bwt --xpub <xpub> ...
 ```
 
 (Mounting the bitcoin datadir is not necessary if you're not using the cookie file.)
@@ -1190,6 +1193,16 @@ The only guideline is to use `cargo fmt`.
 You can check out [the list of enhancement issues](https://github.com/shesek/bwt/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement)
 for some ideas to work on (output script descriptors <3).
 
+## Reproducible builds
+
+The builds can be reproduced in a Docker container environment as follows:
+
+```
+$ git clone https://github.com/shesek/bwt && cd bwt
+$ docker build -t bwt-builder -f scripts/builder.Dockerfile .
+$ docker run -it --rm -v `pwd`:/usr/src/bwt bwt-builder
+$ sha256sum dist/*
+```
 
 ## Thanks
 

scripts/build.sh

@@ -0,0 +1,66 @@
+#!/bin/bash
+set -xeo pipefail
+
+build() {
+  name=$1; platform=$2; features=$3
+  ext=$([[ $platform != *"-windows-"* ]] || echo .exe)
+  dest=dist/$name
+  mkdir -p $dest
+
+  echo Building $name for $platform with features $features
+
+  # drop PE timestamps in windows builds for reproducibility (https://wiki.debian.org/ReproducibleBuilds/TimestampsInPEBinaries#building_with_mingw-w64)
+  RUSTFLAGS="$RUSTFLAGS $([[ $platform != *"-windows-"* ]] || echo '-Clink-arg=-Wl,--no-insert-timestamp')" \
+  cargo build --release --target $platform --no-default-features --features "$features"
+
+  mv target/$platform/release/bwt$ext $dest
+  strip $dest/bwt$ext
+
+  cp README.md LICENSE $dest/
+
+  pack $name
+}
+
+# pack an tar.gz/zip archive file, with fixed/removed metadata attrs and deterministic file order for reproducibility
+pack() {
+  name=$1; dir=${2:-$1}
+  pushd dist
+  touch -t 1711081658 $name $name/*
+  if [[ $name == *"-linux" ]]; then
+    TZ=UTC tar --mtime='2017-11-08 16:58:00' --owner=0 --sort=name -I 'gzip --no-name' -chf $name.tar.gz $dir
+  else
+    find -H $dir | sort | xargs zip -X -q $name.zip
+  fi
+  popd
+}
+
+version=`cat Cargo.toml | egrep '^version =' | cut -d'"' -f2`
+
+rm -rf dist/*
+
+for target in linux,x86_64-unknown-linux-gnu \
+              win,x86_64-pc-windows-gnu; do
+  IFS=',' read pnick platform <<< $target
+
+  build bwt-$version-x86_64-$pnick $platform http,electrum,webhooks,track-spends
+  build bwt-$version-electrum_only-x86_64-$pnick $platform electrum
+done
+
+echo Building electrum plugin
+for pnick in linux win; do
+  name=bwt-$version-electrum_plugin-x86_64-$pnick
+  dest=dist/$name
+  mkdir $dest
+  cp contrib/electrum-plugin/*.py $dest
+  cp dist/bwt-$version-electrum_only-x86_64-$pnick/* $dest
+  # needs to be inside a directory with a name that matches the plugin module name for electrum to load it,
+  # create a temporary link to get tar/zip to pack it properly. (can also be done for tar.gz with --transform)
+  ln -s $name dist/bwt
+  pack $name bwt
+  rm dist/bwt
+done
+
+# remove subdirectories, keep release tarballs
+rm -r dist/*/
+
+[ -n "$OWNER" ] && chown -R $OWNER dist || true

scripts/builder.Dockerfile

@@ -0,0 +1,11 @@
+FROM rust:1.46-slim as builder
+RUN apt-get update && apt-get install -y pkg-config libssl-dev gcc-mingw-w64-x86-64 zip && \
+    rustup target add x86_64-pc-windows-gnu
+
+WORKDIR /usr/src/bwt
+COPY Cargo.toml Cargo.lock ./
+RUN mkdir src .cargo && touch src/lib.rs src/main.rs && \
+    cargo vendor > .cargo/config
+
+VOLUME /usr/src/bwt
+ENTRYPOINT [ "/usr/src/bwt/scripts/build.sh" ]

scripts/release-footer.md

@@ -4,7 +4,7 @@
 
 Installation instructions are [available on the README](https://github.com/shesek/bwt#installation).
 
-### Verifying signatures
+#### Verifying signatures
 
 The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on [keybase](https://keybase.io/nadav), [github](https://api.github.com/users/shesek/gpg_keys), [twitter](https://twitter.com/shesek) and [HN](https://news.ycombinator.com/user?id=nadaviv). The signature can be verified as follows (replace `x86_64-linux` with your download):
 
@@ -20,8 +20,12 @@ $ wget -qO - https://github.com/shesek/bwt/releases/download/vVERSION/SHA256SUMS
 
 You should see `Good signature from "Nadav Ivgi <nadav@shesek.info>" ... Primary key fingerprint: FCF1 9B67 ...` and `bwt-VERSION-x86_64-linux.tar.gz: OK`.
 
-### Electrum plugin
+#### Reproducible builds
+
+The builds are fully reproducible. See [more details here](https://github.com/shesek/bwt#reproducible-builds).
+
+#### Electrum plugin
 
 The [Electrum plugin](https://github.com/shesek/bwt#electrum-plugin) is available for download for Linux and Windows, as the `electrum_plugin` package.
 
-> ⚠️ **NOTE:** The plugin supports watch-only wallets only and **cannot be used with hot wallets**. This is done as a security measure, which is expected to eventually be relaxed. You can use the plugin with hardware wallets or with an offline Electrum setup. For hot wallets, you will need to setup a standalone server instead of using the plugin, ideally away from your keys.
+> ⚠️ **NOTE:** The plugin supports watch-only wallets only and **cannot be used with hot wallets**. This is done as a security measure, which is expected to eventually be lifted. You can use the plugin with hardware wallets or with an offline Electrum setup. For hot wallets, you will need to setup a standalone server instead of using the plugin, ideally far away from your keys.

scripts/release.sh

@@ -34,47 +34,14 @@ cargo fmt -- --check
 ./scripts/check.sh
 
 if [ -z "$SKIP_BUILD" ]; then
-  echo Building executables...
-  rm -rf dist/*
-
-  pack() {
-    pname=$1; name=$2; dir=${3:-$2}
-    pushd dist
-    if [ $pname == 'linux' ]; then tar -czhf $name.tar.gz $dir
-    else zip -rq $name.zip $dir; fi
-    popd
-  }
-  build() {
-    pname=$1; name=$2; toolchain=$3; platform=$4; features=$5; ext=$6
-    echo "- Building $name using $toolchain for $platform with features: $features"
-    cargo +$toolchain build --release --target $platform --no-default-features --features "$features"
-    mkdir -p dist/$name
-    mv target/$platform/release/bwt$ext dist/$name
-    strip dist/$name/bwt$ext
-    cp README.md LICENSE dist/$name
-    pack $pname $name
-  }
-  for cfg in linux,stable,x86_64-unknown-linux-gnu, \
-             win,nightly,x86_64-pc-windows-gnu,.exe; do
-    IFS=',' read pname toolchain platform ext <<< $cfg
-    build $pname bwt-$version-x86_64-$pname $toolchain $platform 'http electrum webhooks track-spends' $ext
-    build $pname bwt-$version-electrum_only-x86_64-$pname $toolchain $platform electrum $ext
-  done
-
-  echo - Building electrum plugin
-  for pname in linux win; do
-    name=bwt-$version-electrum_plugin-x86_64-$pname
-    mkdir dist/$name
-    cp contrib/electrum-plugin/*.py dist/$name
-    cp dist/bwt-$version-electrum_only-x86_64-$pname/bwt* README.md LICENSE dist/$name
-    # needs to be inside a directory with a name that matches the plugin module name for electrum to load it,
-    # create a temporary link to get tar/zip to pack it properly. (can also be done for tar.gz with --transform)
-    ln -s $name dist/bwt
-    pack $pname $name bwt
-    rm dist/bwt
-  done
+  echo Building releases...
 
-  rm -r dist/*/
+  if [ -z "$NO_DOCKER_BUILD" ]; then
+    docker build -t bwt-builder -f scripts/builder.Dockerfile .
+    docker run -it --rm -e OWNER=`id -u` -v `pwd`:/usr/src/bwt bwt-builder
+  else
+    ./scripts/build.sh
+  fi
 
   echo Making SHA256SUMS...
   (cd dist && sha256sum *) | gpg --clearsign --digest-algo sha256 > SHA256SUMS.asc
@@ -97,7 +64,7 @@ if [[ -z "$SKIP_UPLOAD" && -n "$GH_TOKEN" ]]; then
   echo Uploading to github...
   gh_auth="Authorization: token $GH_TOKEN"
   gh_base=https://api.github.com/repos/$gh_repo
-  release_text="## Changelog"$'\n'$'\n'$changelog$'\n'$'\n'`cat scripts/release-footer.md | sed "s/VERSION/$version/g"`
+  release_text="### Changelog"$'\n'$'\n'$changelog$'\n'$'\n'`cat scripts/release-footer.md | sed "s/VERSION/$version/g"`
   release_opt=`jq -n --arg version v$version --arg text "$release_text" \
     '{ tag_name: $version, name: $version, body: $text, draft:true }'`
   gh_release=`curl -sf -H "$gh_auth" $gh_base/releases/tags/v$version \