[work in progress] Reproducible builds #521

fwalch · Jan 17, 2015

I noticed you had SHA1 sums on the build release page. With #496, you cannot really verify this hash locally, only get it from Travis:

diff --git a/tools/make-chromium.sh b/tools/make-chromium.sh
index cd5598e..f410d05 100755
--- a/tools/make-chromium.sh
+++ b/tools/make-chromium.sh
@@ -26,6 +26,7 @@ if [ "$1" = all ]; then
     echo "*** uBlock.chromium: Creating package..."
     pushd $(dirname $DES/)
     zip uBlock.chromium.zip -qr $(basename $DES/)/*
+    sha1sum uBlock.chromium.zip
     popd
 fi

I tried to create reproducible builds, i.e. the ZIPs generated by Travis are exactly the same as the ones generated locally. With this blog post, I got pretty far, but something's still off:

$ ./tools/make-firefox.sh all
$ wget https://github.com/fwalch/uBlock/releases/download/0.8.5.4-TEST/uBlock.firefox.xpi
$ cmp uBlock.firefox.xpi dist/build/uBlock.firefox.xpi
uBlock.firefox.xpi dist/build/uBlock.firefox.xpi differ: char 11704927, line 399465
$ xxd uBlock.firefox.xpi > ff.1
$ xxd dist/build/uBlock.firefox.xpi > ff.2
$ diff -u ff.1 ff.2

(partial diff)

 0b2d190: 4688 3f7e af51 0400 0051 0400 000e 0000  F.?~.Q...Q......
-0b2d1a0: 0000 0000 0000 0000 00b4 81b9 95b2 0077  ...............w
+0b2d1a0: 0000 0000 0000 0000 00a4 81b9 95b2 0077  ...............w
 0b2d1b0: 6869 7465 6c69 7374 2e68 746d 6c50 4b05  hitelist.htmlPK.

So while all the headers and contents of the ZIPs are equal, the end of central directory records don't match. I'll see if that can be fixed as well.

julianxhokaxhiu · Jan 28, 2015

Maybe http://pivotallabs.com/barriers-deterministic-reproducible-zip-files/ will help you.

Also this could be helpful: http://stackoverflow.com/questions/19523063/zip-utility-giving-me-different-md5sum-every-time-in-linux (practically you should use the -X argument to discard special meta data information)

fwalch · Feb 5, 2015

Thanks, but I think I'm already doing all of that. Debian has some strip-nondeterminism scripts, maybe that would help (didn't check them out yet).

gorhill referenced this pull request Feb 9, 2015
Closed
Please could you PGP sign the binary releases? #747

fwalch added some commits Jan 17, 2015

fwalch Test with fork. 8ab627d

fwalch Perform reproducible builds. d853199

chrisaljoudi closed this Apr 7, 2015

@@ @@ -8,10 +8,10 @@ script: ./tools/make-${BROWSER}.sh all @@
      deploy:
        provider: releases
        api_key:
-    -    secure: BCCzIkUFHKotFPABHqPHHyPP8O6ZTPmtt6Ja/JltfO9XBwf6pk9jRiNuRmx5AeETGE2BpojvfqW622q9lCOpgtBit6xkHQC/r2NKt35qMBIPgGCb7GUr5KK7lHNTVOviK9gsOuxcVTeYw3w+zmGLUTpjieELgn4tyz9E8DcOeHE=
+    +    secure: FknPhrs+Y3fXAhXiK9feaSvfHrfEoa4Gxv5iOu13JHaZN0RVmnbTAkx376LU786lEtuG1RJPumkMJ0Bc0U/sgy9dz3iWkwXtXTSa4BAvuJOQA4oeeDLXoHiy0b41VFMbMKUGlUaskW5X3Q7cxpHKEZrm5HRU+dO1jemTTSlI0P0=
        file: dist/build/uBlock.${BROWSER}.${EXT}
        skip_cleanup: true
        on:
-    -    repo: gorhill/uBlock
+    +    repo: fwalch/uBlock
          tags: true
-    -    all_branches: true
+    +    all_branches: true

@@ -1,9 +1,9 @@
-    -#!/bin/bash
+    +#!/bin/bash -e
      #
      # This script assumes a linux environment
-    -echo "*** uBlock.chromium: Creating web store package"
-    -echo "*** uBlock.chromium: Copying files"
+    +echo "*** uBlock.chromium: Creating web store package."
+    +echo "*** uBlock.chromium: Copying files."
      DES=dist/build/uBlock.chromium
      rm -rf $DES
@@ @@ -25,9 +25,18 @@ cp LICENSE.txt $DES/ @@
      if [ "$1" = all ]; then
          echo "*** uBlock.chromium: Creating package..."
-    -    pushd $(dirname $DES/)
-    -    zip uBlock.chromium.zip -qr $(basename $DES/)/*
-    -    popd
+    +    # Get timestamp of latest commit and change
+    +    # files in $DES to have this timestamp.
+    +    timestamp="$(git log -1 --pretty=format:"%cD")"
+    +    find $DES/ -exec touch -d "$timestamp" {} +
+    +
+    +    cd "$(dirname $DES/)"
+    +    rm -f uBlock.chromium.zip
+    +    find "$(basename $DES/)" -type f -print |
+    +      sort -d -f |
+    +      zip -qX0@ uBlock.chromium.zip
+    +    echo "*** uBlock.chromium: Created $(dirname "$DES")/uBlock.chromium.zip."
+    +    echo "*** uBlock.chromium: SHA $(sha256sum uBlock.chromium.zip | cut -f1 -d' ')."
      fi
      echo "*** uBlock.chromium: Package done."

@@ -1,8 +1,8 @@
-    -#!/bin/bash
+    +#!/bin/bash -e
      #
      # This script assumes a linux environment
-    -echo "*** uBlock.firefox: Copying files"
+    +echo "*** uBlock.firefox: Copying files."
      DES=dist/build/uBlock.firefox
      rm -rf $DES
@@ @@ -29,9 +29,18 @@ python tools/make-firefox-meta.py $DES/ @@
      if [ "$1" = all ]; then
          echo "*** uBlock.firefox: Creating package..."
-    -    pushd $DES/
-    -    zip ../uBlock.firefox.xpi -qr *
-    -    popd
+    +    # Get timestamp of latest commit and change
+    +    # files in $DES to have this timestamp.
+    +    timestamp="$(git log -1 --pretty=format:"%cD")"
+    +    find $DES/ -exec touch -d "$timestamp" {} +
+    +
+    +    cd "$DES/"
+    +    rm -f ../uBlock.firefox.xpi
+    +    find . -type f -print |
+    +      sort -d -f |
+    +      zip -qX0@ ../uBlock.firefox.xpi
+    +    echo "*** uBlock.firefox: Created $(dirname "$DES")/uBlock.firefox.xpi."
+    +    echo "*** uBlock.firefox: SHA $(sha256sum ../uBlock.firefox.xpi | cut -f1 -d' ')."
      fi
      echo "*** uBlock.firefox: Package done."

chrisaljoudi/uBlock

[work in progress] Reproducible builds #521

Labels

Milestone

Assignee

3 participants

Please could you PGP sign the binary releases? #747