reproducible builds

[obfusk]

Reproducible Builds

Current Status

• still requires the umask & build path to be identical
• armeabi-v7a seems to consistently build reproducibly
  • ✔️ identical between different builds on the same VM (clean build)
  • ✔️ identical between different builds on GitHub Actions
  • ✔️ identical between builds on buster VM and builds on GitHub Actions w/ buster container (& usrmerge)
  • ✔️ identical between builds on focal VM and builds on GitHub Actions w/ focal container (& umask 022)
  • ✔️ identical between builds on focal VM and builds on GitHub Actions... except for .pyc files ❌
• arm64-v8a seems to consistently build reproducibly... but only with ndk r22 (clang 11.0.5)
  • ✔️ identical between different builds on the same VM (clean build)
  • ✔️ identical between different builds on GitHub Actions
  • ✔️ identical between builds on buster VM and builds on GitHub Actions w/ buster container (& usrmerge)
  • ✔️ identical between builds on focal VM and builds on GitHub Actions w/ focal container (& umask 022)
  • ✔️ identical between builds on focal VM and builds on GitHub Actions... except for .pyc files ❌
• arm64-v8a builds almost reproducibly with other ndk versions
  • consistently produces one of two almost identical versions (see diff below)
    • between different builds on the same VM (clean build)
    • between different builds on GitHub Actions
  • probably a clang/llvm bug:
    • ndk 20b (clang 8.0.7) has a single nondeterministic output (register size, see diff below)
    • ndk 21d (clang 9.0.8) has a few more nondeterministic outputs (some asm reordering as well)

---

TODO

NB: can be done in subsequent PRs.

• use chmod to fix file modes so umask is irrelevant
• use -ffile-prefix-map=OLD=NEW (with gcc & clang >= 10) to fix build path (though it may still be used elsewhere)
• test with more recipes, bootstraps & apps

---

Makefile Example

NB: you need to export something like this in your build environment to get reproducible builds.

export LC_ALL             := C
export TZ                 := UTC
export SOURCE_DATE_EPOCH  := $(shell git log -1 --pretty=%ct)
export PYTHONHASHSEED     := $(SOURCE_DATE_EPOCH)
export BUILD_DATE         := $(shell LC_ALL=C TZ=UTC date +'%b %e %Y' -d @$(SOURCE_DATE_EPOCH))
export BUILD_TIME         := $(shell LC_ALL=C TZ=UTC date +'%H:%M:%S' -d @$(SOURCE_DATE_EPOCH))

---

Diff

--- 1.apk
+++ 2.apk
├── zipinfo {}
│ @@ -1,17 +1,17 @@
│ -Zip file size: 86458780 bytes, number of entries: 15
│ +Zip file size: 86458783 bytes, number of entries: 15
│  -rw----     0.0 fat     3848 b- defN 80-000-00 00:00 AndroidManifest.xml
│  -rw----     2.4 fat       87 b- defN 80-000-00 00:00 META-INF/MANIFEST.MF
│  -rw----     2.4 fat 83167080 bx stor 80-000-00 00:00 assets/private.mp3
│  -rw----     2.4 fat    46652 b- defN 80-000-00 00:00 classes.dex
│  -rw----     2.4 fat  3266024 b- defN 80-000-00 00:00 lib/arm64-v8a/libcrypto1.1.so
│  -rw----     2.4 fat    55416 b- defN 80-000-00 00:00 lib/arm64-v8a/libffi.so
│  -rw----     2.4 fat    18440 b- defN 80-000-00 00:00 lib/arm64-v8a/libmain.so
│  -rw----     2.4 fat   739504 b- defN 80-000-00 00:00 lib/arm64-v8a/libpcre.so
│  -rw----     2.4 fat  3239808 b- defN 80-000-00 00:00 lib/arm64-v8a/libpython3.9.so
│  -rw----     2.4 fat  1215048 b- defN 80-000-00 00:00 lib/arm64-v8a/libsqlite3.so
│  -rw----     2.4 fat   749216 b- defN 80-000-00 00:00 lib/arm64-v8a/libssl1.1.so
│ --rw----     2.4 fat    11251 b- stor 80-000-00 00:00 res/drawable/icon.png
│ +-rw----     2.4 fat    11251 bx stor 80-000-00 00:00 res/drawable/icon.png
│  -rw----     2.4 fat    11584 bx stor 80-000-00 00:00 res/drawable/presplash.jpg
│  -rw----     0.0 fat      544 b- defN 80-000-00 00:00 res/layout/main.xml
│  -rw----     0.0 fat     1092 bx stor 80-000-00 00:00 resources.arsc
│ -15 files, 92525594 bytes uncompressed, 86456914 bytes compressed:  6.6%
│ +15 files, 92525594 bytes uncompressed, 86456903 bytes compressed:  6.6%
├── lib/arm64-v8a/libpython3.9.so
│ ├── objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text {}
│ │ @@ -190721,18 +190721,18 @@
│ │     sub     w13, w10, #0x1
│ │     cmp     w14, #0xff
│ │     b.ne    1b8ddc <PyUnicode_BuildEncodingMap@@Base+0x668>  // b.any
│ │     add     w16, w9, #0x1
│ │     strb    w9, [x20, x15]
│ │     mov     w14, w9
│ │     mov     w9, w16
│ │ -   and     w12, w12, #0x7f
│ │ -   and     w14, w14, #0xff
│ │ -   bfi     w12, w14, #7, #8
│ │ -   strb    w13, [x21, w12, uxtw]
│ │ +   and     x12, x12, #0x7f
│ │ +   and     x14, x14, #0xff
│ │ +   bfi     x12, x14, #7, #8
│ │ +   strb    w13, [x21, x12]
│ │     sxtw    x12, w10
│ │     cmp     x25, x12
│ │     add     w10, w10, #0x1
│ │     b.gt    1b8da0 <PyUnicode_BuildEncodingMap@@Base+0x62c>
│ │     b       1b87c4 <PyUnicode_BuildEncodingMap@@Base+0x50>
│ │  
│ │  00000000001b8e00 <_PyUnicode_EncodeCharmap@@Base>:

[obfusk]

It "works for me" 😅 and I don't see how these modifications would impact anyone who doesn't want reproducible builds, so I think this can be merged.

[AndreMiras]

Very nice idea and approach.
Thanks for taking time investigating this and documenting code and outcome.
Would you mind rebasing/squashing?
I've made some tiny comments, but nothing blocking

[obfusk]

Rebased & squashed.

[obfusk]

@AndreMiras where would be a good place to document this?

[AndreMiras]

Nice thanks!

[SomberNight]

I would be very interested in building my apk reproducibly; any chance you have a list of things you needed to do?

[obfusk]

I would be very interested in building my apk reproducibly; any chance you have a list of things you needed to do?

The code for my app (including build scripts & patches) is here.

IIRC I needed:

• this PR (which has been merged into p4a develop \o/)
• this patch for buildozer (not yet merged)
• this patch for p4a (not yet merged)
• android.ndk = 22 in buildozer.spec (for a sufficiently recent clang, requires those patches to work)

Makefile:

# for reproducible builds
export LC_ALL             := C
export TZ                 := UTC
export SOURCE_DATE_EPOCH  := $(shell git log -1 --pretty=%ct)
export PYTHONHASHSEED     := $(SOURCE_DATE_EPOCH)
export BUILD_DATE         := $(shell LC_ALL=C TZ=UTC date +'%b %e %Y' -d @$(SOURCE_DATE_EPOCH))
export BUILD_TIME         := $(shell LC_ALL=C TZ=UTC date +'%H:%M:%S' -d @$(SOURCE_DATE_EPOCH))

And of course sufficiently identical build environments (OS, packages, user, umask, build path).

[obfusk]

@SomberNight mind letting me know if you get it to work?

Also: I don't think I've tested reproducible builds with python < 3.9. Not sure if that matters, but it might.
[edit: seems to work fine w/ 3.8]

[obfusk]

I just found out I get different .pyc files depending on whether liblzma-dev is installed or not.

Presumably this is what caused the differences between GitHub Actions and my focal VM.

[obfusk]

@AndreMiras I'd like to document this. What would be a good place/way to do that?

[AndreMiras]

Yeah good initiative, thank you! You could document it somewhere within https://github.com/kivy/python-for-android/tree/b809239/doc/source
Don't overthink it too much, we could always move things around later

[obfusk]

I've been a bit busy working on Reproducible Builds for F-Droid. But I intend to document this sometime soonish :)

[obfusk]

Another issue I've encountered is that .pyc files ended up different b/c PYTHONPATH was set.

[SomberNight]

@obfusk Thank you for your work, and for the instructions.
With spesmilo/electrum#7263, the next release of Electrum for Android should be reproducible.