App from GooglePlay is not verifiable #758
Comments
|
Issue-Label Bot is automatically applying the label Links: app homepage, dashboard and code for this bot. |
|
This setup is tested on macOS and Linux, if you are using other setup, please help us figure out what is missing on it. Open an Android Emulator or Connect an Android device. The run: similar issue being described here #759 |
|
Verifiability is important, we’ll definitely spend some time to make builds
verifiable. I’ll start with updating stand-alone build instructions, as atm
all builds are setup and automated on CI
…On Sat, 14 Dec 2019 at 10:54, Nuno Coelho ***@***.***> wrote:
This setup is tested on macOS and Linux, if you are using other setup,
please help us figure out what is missing on it.
Open an Android Emulator or Connect an Android device.
The run:
$ npm i && npm start
$ react-native run-android
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#758?email_source=notifications&email_token=AAOTD6NSMJFJEPTW2V7SF4LQYS3NVA5CNFSM4J2YAVNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG376RY#issuecomment-565706567>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOTD6NHTP4QMXJURTZNNSDQYS3NVANCNFSM4J2YAVNA>
.
|
|
If you build on CI, you should be there already. The more it can be automated, the better. Just strip the signing part or provide a fallback dummy key. Signature is obviously ignored for the verification. I have a few other open source wallets that reacted to my shout-out and I'll review all asap again. Yes, verification is important. Any non-verified release might be the end of the project. As a release manager myself I certainly want every tempted criminal to understand that putting me under distress won't get them all our customers' coins. |
|
#801 using |
|
@Giszmo you are right, this is not for building an APK, it's for local development only. |
|
Can you try for me it builds this file : |
|
I use assembleRelease instead of bundleRelease |
|
I tried again to verify the build but
Please provide tested and working build instructions on your main Readme.md. Relevant part of review: but this also didn't succeed: Different but same for |
|
well fuck me. I'll have to get myself a clean system and setup builds from scratch writing down every step I make. stay tuned... |
|
staying tuned ... |
|
thanks for your patience! |
|
ok, I adapted our travisCI scenario to build apk. works on clean ubuntu 18, tested it on cheap digitalocean vps. don't run as root. here's the script: sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install unzip git npm -y
sudo npm i n -g
sudo n stable
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
export COMPILE_API=29
export ANDROID_BUILD_TOOLS=29.0.2
export ABI=x86_64
export ADB_INSTALL_TIMEOUT=8
export ANDROID_HOME=${HOME}/android-sdk
export ANDROID_TOOLS_URL="https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip"
export EMU_FLAVOR=default # use google_apis flavor if no default flavor emulator
export GRAVIS="https://raw.githubusercontent.com/DanySK/Gravis-CI/master/"
export JDK="1.8"
export TOOLS=${ANDROID_HOME}/tools
export PATH=${ANDROID_HOME}:${ANDROID_HOME}/emulator:${TOOLS}:${TOOLS}/bin:${ANDROID_HOME}/platform-tools:${PATH}
export API=28
export TRAVIS_OS_NAME="linux"
# Set up JDK 8 for Android SDK
curl "${GRAVIS}.install-jdk-travis.sh" --output ~/.install-jdk-travis.sh
export TARGET_JDK="${JDK}"
source ~/.install-jdk-travis.sh
# Set up Android SDK
wget -q "${ANDROID_TOOLS_URL}" -O android-sdk-tools.zip
unzip -q android-sdk-tools.zip -d ${ANDROID_HOME}
rm android-sdk-tools.zip
mkdir ~/.android # avoid harmless sdkmanager warning
echo 'count=0' > ~/.android/repositories.cfg # avoid harmless sdkmanager warning
yes | sdkmanager --licenses >/dev/null # accept all sdkmanager warnings
echo y | sdkmanager --no_https "platform-tools" >/dev/null
echo y | sdkmanager --no_https "tools" >/dev/null # A second time per Travis docs, gets latest versions
echo y | sdkmanager --no_https "build-tools;${ANDROID_BUILD_TOOLS}" >/dev/null # Implicit gradle dependency - gradle drives changes
echo y | sdkmanager --no_https "platforms;android-${COMPILE_API}" >/dev/null # We need the API of the current compileSdkVersion from gradle.properties
git clone https://github.com/BlueWallet/BlueWallet
cd BlueWallet
npm i
cd android
./gradlew assembleRelease
keytool -genkeypair -v -keystore detox.keystore -alias detox -keyalg RSA -keysize 2048 -validity 10000 -storepass 123456 -keypass 123456 -dname 'cn=Unknown, ou=Unknown, o=Unknown, c=Unknown'
jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore detox.keystore ./app/build/outputs/apk/release/app-release-unsigned.apk detox -storepass 123456it should build APK and sign it with random key. let me know how is it going |
|
if script works for you I will include it in repo under |
|
https://github.com/junderw/BlueWallet/tree/dockerBuildAndroid/scripts/docker This works. To change the version you need to do the following before running the docker run again:
If anyone wants to clean up that docker so it will work more deterministically (hard because that Gravis-CI stuff can be changed on a whim) cool. I tested the apk. It works fine. |
|
I gave it another try and while sort of following @Overtorment's code yielded a compile error, maybe because I deviated from the script, @junderw's docker did compile fine but the result differs a lot from the apk I got from Google Play. My attempt at running the commands mostly in some (kind of random) docker container. Should have used Running the docker: The Dockerfile works to compile the apk but it should require a revision/tag as command line parameter and not require the user to edit a script for that. Edit: As the line with diffoscope might not be clear if you don't know the tool: apk tools necessarily are not bit by bit reproducible, as the verifier cannot reproduce the release signature, which is part of the apk. Therefore build verification requires to unpack the apk. This can be done with unzip or with apktool but the following result means the binary on google play is not reproducible: Edit 2: The file compiled with docker does run on my AVD and looks complete but the diff in the files is also very significant, looking at the file size: |
|
we can work on that later, when we move dockerfile to repo. ok my guess is that by default can you try rerunning but with |
|
might also want to do npm ci instead, since any patches for react dependencies would get pulled in if we had any ^ dependencies |
|
we do not have any ^ dependencies for production. some devDependencies do have ^ |
|
|
also, i noticed that our nodejs ver on CI is set to 10.x |
|
got some progress. bad news, is that the size difference is there. when you build from docker its ~29 Mb, and builds that we get from our CI/CD (appcenter.ms) is ~27 Mb. a bit in-depth... so, you can unzip apk and see its contents. i started examining apk from docker and apk from appcenter. the main executable is quick googling showed that this is javascriptcore binaries that are used to actually bootstrap javascript https://github.com/react-native-community/jsc-android-buildscripts |
|
some progress, found out specific versions of build container on appcenter: https://github.com/actions/virtual-environments/blob/master/images/macos/macos-10.15-Readme.md @junderw how hard it is you think to bake those specific versions in our docker? Im looking into Azure containers atm |
|
Very hard... Not to mention it looks like appcenter is macOS, and iirc there are no macOS docker containers.....??? |
To my understanding of how Docker works, you can't have a macOS container as Docker is using the host's kernel which would not be the same on macOS. So if there was a macOS container, it would only run on macOS. On the other hand I'm very confused right now. Docker is still magic to me and due to it not being as flexible as virtual machines and not designed to provide security, I am considering to switch away from docker to VMWare or whatever. Please let me know if that would be necessary or helpful in your case. |
|
@Giszmo would that be an option for you to use appcenter to setup your own CI for BW to make at least 1 build to compare and count that as verifiable? otherwise, I have some thoughts on renting https://www.macincloud.com to install specified software versions and try to do a verifiable build there. |
|
It's probably much work that would be obsolete with the next release. If you build this on a Mac and it can only be reproduced on a Mac, I would prefer to get neutral, trusted reproducers to join the project and rebuild it on their infrastructure but ideally somebody would figure out how to do this with a VM. I never ran a MacOS VM but I assume that's possible. No idea if that would require a license or if there is some more accessible way of doing this. |
|
I occasionally rent a Mac on macincloud to do some dev work, works pretty
good
…On Fri, 19 Jun 2020 at 02:37, Leo Wandersleb ***@***.***> wrote:
It's probably much work that would be obsolete with the next release.
If you build this on a Mac and it can only be reproduced on a Mac, I would
prefer to get neutral, trusted reproducers to join the project and rebuild
it on their infrastructure but ideally somebody would figure out how to do
this with a VM. I never ran a MacOS VM but I assume that's possible. No
idea if that would require a license or if there is some more accessible
way of doing this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#758 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOTD6I7DBGOSKHVBG6V6A3RXK6NTANCNFSM4J2YAVNA>
.
|
|
Building latest version, tag |
|
Both 2 issues in previous comment are fixed, I Tested
Regarding the diffs in .so files, running diffoscope is needed to look at the diff & also compiling from MACOSX which I might look into it in the future when most of the above issues are fixed... Containerfile used to build the apk is: build: FROM frolvlad/alpine-glibc
RUN set -ex; \
apk update; \
apk add --no-cache \
openjdk8 \
git \
npm; \
adduser -D appuser;
USER appuser
ENV ANDROID_SDK_ROOT="/home/appuser/app/sdk" \
ANDROID_HOME="/home/appuser/app/sdk" \
NODE_ENV="production"
RUN set -ex; \
mkdir -p "/home/appuser/app/sdk/licenses" "/home/appuser/app/bluewallet/"; \
printf "\n24333f8a63b6825ea9c5514f83c2829b004d1fee" > "/home/appuser/app/sdk/licenses/android-sdk-license"; \
cd /home/appuser/app/bluewallet/; \
git clone https://github.com/BlueWallet/BlueWallet; \
cd BlueWallet; \
git checkout v6.1.4;
WORKDIR /home/appuser/app/bluewallet/BlueWallet/
RUN set -ex; \
npm ci --no-optional --no-audit --no-fund --ignore-scripts; \
npm run postinstall; \
cd android; \
./gradlew assembleRelease |
I tried these steps on a Digital Ocean Ubuntu 18.04 (LTS) x64 to reproduce the build for v6.2.7. The setup and build works fine, I end up with the following diff: As noted in #3239, the .json files differ on user name. |
|
A few quirks later:
I end up with this diff Further inspection with diff/diffoscope Lines appear in different order. Lines appear in different order and there's a comment at the end of line. |
|
Here's the script I'm using: With this I am down to the following diff: If there's a way to resolve the small diff in |
|
Tagging @bmwiedemann maybe you know how to resolve & why happening the diff in ordering & additonal comment inside |
|
My usual approach in such cases is to find out how the files are produced. Chances are, that |
|
Latest version 6.3.2 on Github is kinda reproducible with Containerfile below (Requires to find workaround for ordering issue in Binary AndroidManifest.xml, opened issue also in reproducible-apk-tools repo as they might add generic solution to order this file obfusk/reproducible-apk-tools#15 also related to additional tag in Google Play version at #3219 ) Compile with: podman build --pull --ulimit nofile=8192:8192 --rm -t bluewallet_build_apk_632 -f Containerfile632APK generated in:
Containerfile632 content: FROM docker.io/node:16.18.1-bullseye
RUN set -ex; \
apt-get update; \
DEBIAN_FRONTEND=noninteractive apt-get install --yes -o APT::Install-Suggests=false --no-install-recommends openjdk-11-jre-headless; \
rm -rf /var/lib/apt/lists/*; \
useradd -ms /bin/bash appuser; \
mkdir -p /Users/runner/work/1/; \
chown -R appuser:appuser /Users/;
USER appuser
ENV ANDROID_SDK_ROOT="/home/appuser/sdk" \
ANDROID_HOME="/home/appuser/sdk" \
NODE_ENV="production"
RUN set -ex; \
mkdir -p "/home/appuser/sdk/licenses"; \
printf "\n24333f8a63b6825ea9c5514f83c2829b004d1fee" > "/home/appuser/sdk/licenses/android-sdk-license"; \
cd /Users/runner/work/1/; \
git clone --branch v6.3.2 https://github.com/BlueWallet/BlueWallet /Users/runner/work/1/s/;
WORKDIR /Users/runner/work/1/s/
RUN set -ex; \
npm install --production --no-optional --omit=optional --no-audit --no-fund --ignore-scripts; \
npm run postinstall; \
sed -i "s/versionCode 1/versionCode 1668358190/g" android/app/build.gradle; \
sed -i '/^\s*<\/application>\s*/i <meta-data android:name="com.bugsnag.android.BUILD_UUID" android:value="8699182a-9b70-4ef4-ab1c-d7a0d2020499"\/>' android/app/src/main/AndroidManifest.xml; \
echo '"master"' > current-branch.json;
RUN set -ex; \
cd /Users/runner/work/1/s/android; \
./gradlew assembleReleaseThe diff between it and version on github in https://github.com/BlueWallet/BlueWallet/releases/download/v6.3.2/BlueWallet-6.3.2.apk is: sha256sum of apk from github: 939275561273a0901ce80bb34c37457b8a1a6a1bb11f8267837e7b8db1d0fbf5 In the decoded AndroidManifest.xml the diff is: 96d95
< <meta-data android:name="com.bugsnag.android.BUILD_UUID" android:value="8699182a-9b70-4ef4-ab1c-d7a0d2020499"/>
141a141
> <meta-data android:name="com.bugsnag.android.BUILD_UUID" android:value="8699182a-9b70-4ef4-ab1c-d7a0d2020499"/>The sed command was used to add this entry during compilation (as without it the value generated will be different then the one in github APK), but it ends up in different location in the decoded AndroidManifest (not before the Opened issue #5474 to suggest using a "static" value that changes in tags/branch for the What left is to check version downloaded from Google Play directly which may require more changes to build script (as just comparing the github APK to APK downloaded from https://apk.support and https://apkcombo.com/ websites will show diffs in some files) |
|
Thanks @emanuelb! This is a really good improvement. I think as long as the diff is easy to verify and it is in the manifest then it's acceptable. Look forward to also being able to verify the Google Play APK and future improvements on this. |
|
@emanuelb First, thank you! Second, I successfully built 6.3.2 following your instructions above but was not able to build 6.4.4. I basically got stuck at "info Done copying assets". Do we need to modify the content of Containerfilexxx to make it work with future versions? Here are the logs below: w: file:///Users/runner/work/1/s/node_modules/rn-ldk/android/src/main/java/com/rnldk/RnLdkModule.kt:741:15 'reject(String!): Unit' is deprecated. Deprecated in Java
w: file:///Users/runner/work/1/s/node_modules/rn-ldk/android/src/main/java/com/rnldk/RnLdkModule.kt:817:46 'reject(String!): Unit' is deprecated. Deprecated in Java
w: file:///Users/runner/work/1/s/node_modules/rn-ldk/android/src/main/java/com/rnldk/RnLdkModule.kt:818:48 'reject(String!): Unit' is deprecated. Deprecated in Java
w: file:///Users/runner/work/1/s/node_modules/rn-ldk/android/src/main/java/com/rnldk/RnLdkModule.kt:819:46 'reject(String!): Unit' is deprecated. Deprecated in Java
w: file:///Users/runner/work/1/s/node_modules/rn-ldk/android/src/main/java/com/rnldk/RnLdkModule.kt:923:20 'toUpperCase(): String' is deprecated. Use uppercase() instead.
w: file:///Users/runner/work/1/s/node_modules/rn-ldk/android/src/main/java/com/rnldk/RnLdkModule.kt:942:85 'toLowerCase(): String' is deprecated. Use lowercase() instead.
> Task :app:checkReleaseAarMetadata
> Task :rn-ldk:extractReleaseAnnotations
> Task :rn-ldk:compileReleaseJavaWithJavac
> Task :rn-ldk:mergeReleaseGeneratedProguardFiles
> Task :rn-ldk:mergeReleaseConsumerProguardFiles
> Task :rn-ldk:mergeReleaseJavaResource
> Task :app:generateReleaseResValues
> Task :rn-ldk:syncReleaseLibJars
> Task :rn-ldk:bundleReleaseLocalLintAar
> Task :app:processReleaseGoogleServices
> Task :app:mapReleaseSourceSetPaths
> Task :app:createBundleReleaseJsAndAssets
warning: the transform cache was reset.
Welcome to Metro v0.73.9
Fast - Scalable - Integrated
warn Package react-native-fingerprint-scanner contains invalid configuration: "dependency.assets" is not allowed,"dependency.hooks" is not allowed. Please verify it's properly linked using "react-native config" command and contact the package maintainers about this.
info Writing bundle output to:, /Users/runner/work/1/s/android/app/build/generated/assets/createBundleReleaseJsAndAssets/index.android.bundle
info Writing sourcemap output to:, /Users/runner/work/1/s/android/app/build/intermediates/sourcemaps/react/release/index.android.bundle.packager.map
info Done writing bundle output
info Done writing sourcemap output
info Copying 64 asset files
info Done copying assets |
|
@ nobody77777 Yes, some modifications are needed for every version change (like using different node version) , for version 6.4.4 I posted a Containerfile at WalletScrutiny project in comment for MR for BlueWallet at: also the stuck is not unusual and I experienced it myself in latest versions, had to run the script / last operation ( |
|
@emanuelb Ok I understand it better now. Thank you! |
|
I tried to reproduce 6.4.13 from Play Store and got a big diff. |
|
Again, I tried to reproduce v6.5.0 and the diff was huge. I used this Dockerfile to build: and these build commands: |
|
Diff is huge for 6.6.1, too. With diffoscope I see no pattern to take a guess on what might be the issue. I used basically the same script as above. |
Please check my findings on the wallet's verifiability and let me know when I should give it another try. We are already in touch but I'm pinging all the wallets we failed to verify, so here's an issue for this.
The text was updated successfully, but these errors were encountered: