Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducible Android builds #8203

Closed
PombeirP opened this issue May 14, 2019 · 9 comments

Comments

@PombeirP
Copy link
Member

commented May 14, 2019

Problem

In order to provide verifiability in the builds we provide to users, we want to make our builds reproducible (Android first). This will also allow us to publish the Android app on F-Droid.

Implementation

The recent effort of migrating to Nix (including pinning our tools, SDKs and ensuring a pure build environment) should have done most of the work required for reproducible builds (Nix takes care for instance of setting build artifacts' timestamps to a known value).

Comparing 2 builds from our CI build servers yielded a single type of differences in index.android.bundle, related to how our logging library and Leiningen interact. Leiningen normally creates a temp file path that ends up in the js code, and this file path is used by Timbre to calculate a callsite ID (although fixing the path is not enough because Timbre also adds a random number before hashing everything). We'll need to address this by either replacing Leiningen or modifying/replacing the Timbre logging library.

image

Acceptance Criteria

  • Distinct builds from CI servers produce the exact same output binary (bitwise equality).
  • Distinct builds from 2 different developer machines (same host and target platform) produce the exact same output binary (bitwise equality).

Notes

Issue for similar effort @ Briar: https://code.briarproject.org/briar/briar/issues/1273
Reproducible builds post @ F-Droid: https://f-droid.org/en/docs/Reproducible_Builds/
SOURCE_DATE_EPOCH: https://reproducible-builds.org/docs/source-date-epoch/

Future Steps

  • Modify our canary build to compare hashes with the previous build and fail if it ever changes.
  • Address other platforms.

@PombeirP PombeirP self-assigned this May 14, 2019

@PombeirP

This comment has been minimized.

Copy link
Member Author

commented May 14, 2019

After a quick test of disabling our usage of Timbre, this is the report on the differences between 2 builds of the same commit:

image

The main issue seems to be the lib folder (which contains artifacts built by RN Android using the NDK). The differences in META_INF can be assumed to be implicit.

Comparing the .so files with objdump -xs shows that the differences in the .note.gnu.build-id section:

image

@PombeirP

This comment has been minimized.

Copy link
Member Author

commented May 14, 2019

Since we cannot trust the build ID (.note.gnu.build-id) to be constant across build machines (e.g. source path differences count towards the build ID), we need to disable it with -Wl,--build-id=none in two places:

  • in React Native where it is explicitly enabled and used for stack trace purposes (hopefully not impactful for us);
  • on our status-go build, where a random number is used for the GOPATH variable when compiling Go programs. The strategy is to unpack the output .aar file, replace the occurences with fixed values and pack it again.
@PombeirP

This comment has been minimized.

Copy link
Member Author

commented May 15, 2019

I was able to get a bitwise identical Android build from 2 separate CI builds (https://ci.status.im/job/status-react/job/combined/job/mobile-android/10934/ and https://ci.status.im/job/status-react/job/combined/job/mobile-android/10935/):

$ md5sum ~/Downloads/StatusIm-190514-220205-4b220e-manual.apk 
b0ea652c97176d267710383475fcfd87  /home/pedro/Downloads/StatusIm-190514-220205-4b220e-manual.apk
$ md5sum ~/Downloads/StatusIm-190514-220233-4b220e-manual.apk 
b0ea652c97176d267710383475fcfd87  /home/pedro/Downloads/StatusIm-190514-220233-4b220e-manual.apk

for this I had to remove both .note.gnu.build-id and .gnu.version_d sections. The only part changing in .gnu.version_d is the vd_hash field (see http://refspecs.linuxbase.org/LSB_1.3.0/gLSB/gLSB/symverdefs.html). This is probably all due to the different build paths, so after some investigation I found a more promising and less intrusive way of having a stable build ID: -fdebug-prefix-map=old=new (see https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html). I'll be working to try to get React Native and gomobile to build with that option and validating the reproducibility of the output.

@PombeirP

This comment has been minimized.

Copy link
Member Author

commented May 15, 2019

Another issue that I've come across is that React Native's dependencies (e.g. boost, folly, glog) make heavy use of __FILE__, causing the host build path to end up in the .rodata section of libreactnativejni.so. This is an obstacle for cross-machine reproducible builds. One way I've seen suggested to get around this is to pass relative paths to gcc. I've tried modifying RN's ReactAndroid/build.gradle to use relative paths but couldn't get it to find the files, so not sure if that'd work.

If I take a build from the CI server and search for jenkins, I get the following hits:

lib/armeabi-v7a/libfb.so:9
lib/armeabi-v7a/libjscexecutor.so:6
lib/armeabi-v7a/libreactnativejni.so:8
lib/armeabi-v7a/libglog.so:3
lib/armeabi-v7a/libfolly_json.so:4
lib/x86/libreactnativejni.so:8
lib/x86/libfb.so:9
lib/x86/libjscexecutor.so:6
lib/x86/libglog.so:3
lib/armeabi-v7a/librealmreact.so:37
lib/x86/libfolly_json.so:4
lib/x86/librealmreact.so:35
assets/index.android.bundle:3

so basically it breaks down into RNAndroid compilation and transpiling of the ClojureScript code.

@PombeirP

This comment has been minimized.

Copy link
Member Author

commented May 16, 2019

I’ve set up a public Git repo where I’m working on a Hello Word Nix-based React Native app here: https://github.com/PombeirP/nix-react-native-test. I was able to successfully build it and deploy to Android, so clearly something else is the problem in status-react, most likely one (or several) of the RN modules that we import.

Update: When talking with Roman, I was remembered that we build RN from sources as it is sometimes necessary to patch it (although not currently).

@StatusWrike

This comment has been minimized.

Copy link

commented May 24, 2019

➤ Igor Mandrigin commented:

Nabil NaghdyJarrad Hope I feel like having reproducible bulds for Android is important for 1.0 release, so we can side-load app and share it to F-Droid, etc. Marked it as a part of critical path, ping me if you disagree.

@StatusWrike

This comment has been minimized.

Copy link

commented May 24, 2019

➤ Jarrad Hope commented:

Igor Mandrigin I agree, its really great work

@StatusWrike

This comment has been minimized.

Copy link

commented Jun 17, 2019

➤ Nabil Naghdy commented:

Pedro Pombeiro any update on this?

@StatusWrike

This comment has been minimized.

Copy link

commented Jun 17, 2019

➤ Pedro Pombeiro commented:

Nabil Naghdy I've just updated Build <code>lein prod-build-android</code> and <code>gradle assembleRelease</code> in Nix build to be in progress, which is what I'm working on. It's taking some time, since it's a big undertaking, but I expect to have it finished by the end of the week, and reviewed the week after.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.