F-Droid reproducible build failed #414

linsui · 2021-05-31T11:28:42Z

Describe the bug

DOES NOT VERIFY
ERROR: APK Signature Scheme v2 signer #1: APK integrity check failed. CHUNKED_SHA256 digest mismatch. Expected: <a4192e5d7a62b86c4c02a19860b9984102da180d9a6e0ad9707029beb10e91c0>, actual: <11e234eb288257ac5c47e1090a760391291155ea0d4b59e8e82398a8908e0c47>

Colud you please take a look? Thanks!

The text was updated successfully, but these errors were encountered:

nielsvanvelzen · 2021-05-31T11:30:01Z

Can you give us some more information, like what version this happened with?

nielsvanvelzen · 2021-05-31T11:34:41Z

Actually, this might be because #330 removed the unsigned release builds from repo.jellyfin.org. I'll look into it before releasing 2,3.0 :)

linsui · 2021-05-31T11:48:38Z

The log is here. https://monitor.f-droid.org/builds/log/org.jellyfin.mobile/2030003 But there is not much information.

nielsvanvelzen · 2021-06-05T07:48:21Z

I compared 2.3.0-beta.1 (last successful release on F-Droid, second beta was skipped for some reason) with 2.3.0-beta.3 (the failed build) and I don't see any significant changes that would cause the verification to fail. This makes me believe the fault is at F-Droid. The lack of useful logging in their build also doesn't help me to figure out what the problem might be.

Here's hoping our next beta release just works because I'm not interested in spending hours to figuring out the problem.

Comparison: v2.3.0-beta.1...v2.3.0-beta.3

dkanada · 2021-06-05T08:00:40Z

If it's not on their end, my bet is the new signing config section.

Maxr1998 · 2021-06-05T08:04:47Z

It shouldn't be, without the environment vars, it's a no-op on their side.

However, it's possible that the default signing configuration within Gradle uses a different scheme than apksigner.

Here's the relevant line in fdroidserver.

khimaros · 2021-06-12T14:36:21Z

@IzzySoft may be able to help.

IzzySoft · 2021-06-12T15:00:08Z

Not with reproducible builds, sorry – I'm no Android dev.

nielsvanvelzen · 2021-06-18T08:16:57Z

I've published a new beta release (v2.3.0-beta.4). Let's see if F-Droid can build this one or fails again.

CarlosOlivo · 2021-06-19T07:10:22Z

However, it's possible that the default signing configuration within Gradle uses a different scheme than apksigner.

Bingo!

Checking the F-Droid server source code I found the culprit
https://gitlab.com/fdroid/fdroidserver/-/blob/63444f86544fffd51a704338e410d43cbcf39c0e/fdroidserver/common.py#L3311

Doing a quick rebuild of v2.3.0-beta.3 throws the same error as the build log 2030003

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ apksigner verify --print-certs --verbose jellyfin-android-v2.3.0-beta.3-libre-release.apk
Verifies
Verified using v1 scheme (JAR signing): true
Verified using v2 scheme (APK Signature Scheme v2): true
Verified using v3 scheme (APK Signature Scheme v3): false
Number of signers: 1
Signer #1 certificate DN: CN=Joshua Boniface, OU=Jellyfin Team, O=Jellyfin, L=Burlington, ST=Ontario, C=CA
Signer #1 certificate SHA-256 digest: d881796ed2a67ff6ef9f676828723c6b1fa18e09388962cba4abc4a594a69131
Signer #1 certificate SHA-1 digest: 44f9fbf25aaac9ecc4e57f32bd0ad9795d25bdbe
Signer #1 certificate MD5 digest: 9f696f9d71014e880173ce4f67a89088
Signer #1 key algorithm: RSA
Signer #1 key size (bits): 2048
Signer #1 public key SHA-256 digest: bcececa370a6bcf29866c62acd1a05ab380cb6ad43fe2a1483af189730a781f6
Signer #1 public key SHA-1 digest: e7695a9d794136c924cf5279a5a5b78303299c66
Signer #1 public key MD5 digest: 784a247d522b5f13019b1d76d9d16c53
...

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ apksigcopier copy --v1-only auto jellyfin-android-v2.3.0-beta.3-libre-release.apk jellyfin-android-v2.3.0-beta.3-libre-release-unsigned.apk jellyfin-android-v2.3.0-beta.3-libre-release-apksigcopier.apk

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ apksigner verify --print-certs --verbose jellyfin-android-v2.3.0-beta.3-libre-release-apksigcopier.apk
DOES NOT VERIFY
ERROR: APK Signature Scheme v2 signer #1: APK integrity check failed. CHUNKED_SHA256 digest mismatch. Expected: <a4192e5d7a62b86c4c02a19860b9984102da180d9a6e0ad9707029beb10e91c0>, actual: <11e234eb288257ac5c47e1090a760391291155ea0d4b59e8e82398a8908e0c47>

Reviewing apksigcopier I find 2 important details:

The version stored in F-Droid server is out of date v0.4.0 vs v1.0.0
There is a note in the code that says:

FIXME: makes certain assumptions and doesn't handle all valid ZIP files!

Comparison of v2.3.0-beta.1

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ diffoscope jellyfin-android-v2.3.0-beta.1-libre-release-unsigned.apk jellyfin-android-v2.3.0-beta.1-libre-release.apk
--- jellyfin-android-v2.3.0-beta.1-libre-release-unsigned.apk
+++ jellyfin-android-v2.3.0-beta.1-libre-release.apk
├── zipinfo /dev/stdin
│ @@ -1,8 +1,8 @@
│ -Zip file size: 4531178 bytes, number of entries: 1540
│ +Zip file size: 4649360 bytes, number of entries: 1543
│  -rw-rw-rw-  0.0 unx       55 b- defN 81-Jan-01 01:01 META-INF/com/android/build/gradle/app-metadata.properties
│  -rw-rw-rw-  0.0 unx  5602260 b- defN 81-Jan-01 01:01 classes.dex
│  -rw-rw-rw-  0.0 unx    17545 b- defN 81-Jan-01 01:01 assets/native/EventEmitter.js
│  -rw-rw-rw-  0.0 unx     3578 b- defN 81-Jan-01 01:01 assets/native/ExoPlayerPlugin.js
│  -rw-rw-rw-  0.0 unx     3787 b- defN 81-Jan-01 01:01 assets/native/ExternalPlayerPlugin.js
│  -rw-rw-rw-  0.0 unx      384 b- defN 81-Jan-01 01:01 assets/native/NavigationPlugin.js
│  -rw-rw-rw-  0.0 unx    62704 b- defN 81-Jan-01 01:01 assets/native/chrome.cast.js
│ @@ -1535,8 +1535,11 @@
│  -rw----     2.0 fat      200 b- defN 81-Jan-01 01:01 kotlin/coroutines/coroutines.kotlin_builtins
│  -rw----     2.0 fat      758 b- defN 81-Jan-01 01:01 kotlin/internal/internal.kotlin_builtins
│  -rw----     2.0 fat    14151 b- defN 81-Jan-01 01:01 kotlin/kotlin.kotlin_builtins
│  -rw----     2.0 fat     2301 b- defN 81-Jan-01 01:01 kotlin/ranges/ranges.kotlin_builtins
│  -rw----     2.0 fat     2338 b- defN 81-Jan-01 01:01 kotlin/reflect/reflect.kotlin_builtins
│  -rw----     2.0 fat      218 b- defN 81-Jan-01 01:01 okhttp3/internal/publicsuffix/NOTICE
│  -rw----     2.0 fat    37730 b- defN 81-Jan-01 01:01 okhttp3/internal/publicsuffix/publicsuffixes.gz
│ -1540 files, 7987719 bytes uncompressed, 4365816 bytes compressed:  45.3%
│ +-rw----     2.0 fat   134594 b- defN 81-Jan-01 01:01 META-INF/JELLYFIN.SF
│ +-rw----     2.0 fat     1410 b- defN 81-Jan-01 01:01 META-INF/JELLYFIN.RSA
│ +-rw----     2.0 fat   134467 b- defN 81-Jan-01 01:01 META-INF/MANIFEST.MF
│ +1543 files, 8258190 bytes uncompressed, 4477682 bytes compressed:  45.8%

META-INF/JELLYFIN.SF - apksigner

Signature-Version: 1.0
Created-By: 1.0 (Android)
SHA-256-Digest-Manifest: Kcn7it1jv52R2CU5tLEUdWpghMjN+gvXLXG9vnArLk8=
X-Android-APK-Signed: 2, 3

Comparison of v2.3.0-beta.3

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ diffoscope jellyfin-android-v2.3.0-beta.3-libre-r
elease-unsigned.apk jellyfin-android-v2.3.0-beta.3-libre-release.apk
--- jellyfin-android-v2.3.0-beta.3-libre-release-unsigned.apk
+++ jellyfin-android-v2.3.0-beta.3-libre-release.apk
├── zipinfo /dev/stdin
│ @@ -1,8 +1,8 @@
│ -Zip file size: 4729127 bytes, number of entries: 1470
│ +Zip file size: 4853160 bytes, number of entries: 1473
│  -rw-rw-rw-  0.0 unx       55 b- defN 81-Jan-01 01:01 META-INF/com/android/build/gradle/app-metadata.properties
│  -rw-rw-rw-  0.0 unx  5726292 b- defN 81-Jan-01 01:01 classes.dex
│  -rw-rw-rw-  0.0 unx   460220 b- defN 81-Jan-01 01:01 classes2.dex
│  -rw-rw-rw-  0.0 unx    17545 b- defN 81-Jan-01 01:01 assets/native/EventEmitter.js
│  -rw-rw-rw-  0.0 unx     3578 b- defN 81-Jan-01 01:01 assets/native/ExoPlayerPlugin.js
│  -rw-rw-rw-  0.0 unx     3895 b- defN 81-Jan-01 01:01 assets/native/ExternalPlayerPlugin.js
│  -rw-rw-rw-  0.0 unx      384 b- defN 81-Jan-01 01:01 assets/native/NavigationPlugin.js
│ @@ -1465,8 +1465,11 @@
│  -rw----     2.0 fat      200 b- defN 81-Jan-01 01:01 kotlin/coroutines/coroutines.kotlin_builtins
│  -rw----     2.0 fat      758 b- defN 81-Jan-01 01:01 kotlin/internal/internal.kotlin_builtins
│  -rw----     2.0 fat    14151 b- defN 81-Jan-01 01:01 kotlin/kotlin.kotlin_builtins
│  -rw----     2.0 fat     2301 b- defN 81-Jan-01 01:01 kotlin/ranges/ranges.kotlin_builtins
│  -rw----     2.0 fat     2338 b- defN 81-Jan-01 01:01 kotlin/reflect/reflect.kotlin_builtins
│  -rw----     2.0 fat      218 b- defN 81-Jan-01 01:01 okhttp3/internal/publicsuffix/NOTICE
│  -rw----     2.0 fat    37730 b- defN 81-Jan-01 01:01 okhttp3/internal/publicsuffix/publicsuffixes.gz
│ -1470 files, 8552242 bytes uncompressed, 4570611 bytes compressed:  46.6%
│ +-rw-rw-rw-  0.0 unx   128856 b- defN 81-Jan-01 01:01 META-INF/CERT.SF
│ +-rw-rw-rw-  0.0 unx     1410 b- defN 81-Jan-01 01:01 META-INF/CERT.RSA
│ +-rw-rw-rw-  0.0 unx   128782 b- defN 81-Jan-01 01:01 META-INF/MANIFEST.MF
│ +1473 files, 8811290 bytes uncompressed, 4681890 bytes compressed:  46.9%

META-INF/CERT.SF - Gradle

Signature-Version: 1.0
Created-By: Android Gradle 4.2.1
SHA-256-Digest-Manifest: xlx7Ms9JC4UuipLv6ggXfKgcvqC6U57AXypSzFftvfA=
X-Android-APK-Signed: 2

Missing APK Signature Scheme v3 in v2.3.0-beta.3 and changes in the metadata of the .zip file

So in summary, if you want to continue publishing in F-Droid, it is mandatory to sign with apksigner, at least until apksigcopier become compatible with gradle signed files

A test run with the debug certificate using apksigner

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ apksigner sign --ks ~/.android/debug.keystore --ks-pass pass:android --out jellyfin-android-v2.3.0-beta.3-libre-release-signed.apk --verbose jellyfin-android-v2.3.0-beta.3-libre-release-unsigned.apk
Signed

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ apksigner verify --print-certs --verbose jellyfin-android-v2.3.0-beta.3-libre-release-signed.apk
Verifies
Verified using v1 scheme (JAR signing): true
Verified using v2 scheme (APK Signature Scheme v2): true
Verified using v3 scheme (APK Signature Scheme v3): true
Number of signers: 1
Signer #1 certificate DN: C=US, O=Android, CN=Android Debug
Signer #1 certificate SHA-256 digest: a405812c6c2db59925de281505537940644ac7ece0cd628cd36b68745924c470
Signer #1 certificate SHA-1 digest: 1c0b318994c2fd75d53761d96a7fef3cd4e6a9dd
Signer #1 certificate MD5 digest: 03e57959e52033f3d5ec2c538e47df35
Signer #1 key algorithm: RSA
Signer #1 key size (bits): 2048
Signer #1 public key SHA-256 digest: c3041a3881f318e249e3c013983ea0d09535f8d7ead5eee9c856c8f64c793018
Signer #1 public key SHA-1 digest: 3cb6f8928affd7ca7e1b67865262b73b2abc3168
Signer #1 public key MD5 digest: 3e093e24dc86e4045826151d92ca0bd9
...

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ apksigcopier copy --v1-only auto jellyfin-android-v2.3.0-beta.3-libre-release-signed.apk jellyfin-android-v2.3.0-beta.3-libre-release-unsigned.apk jellyfin-android-v2.3.0-beta.3-libre-release-apksigcopier.apk

carlos@DELL-PC:~/jellyfin-android/app/build/outputs/apk/libre/release$ apksigner verify --print-certs --verbose jellyfin-android-v2.3.0-beta.3-libre-release-apksigcopier.apk
Verifies
Verified using v1 scheme (JAR signing): true
Verified using v2 scheme (APK Signature Scheme v2): true
Verified using v3 scheme (APK Signature Scheme v3): true
Number of signers: 1
Signer #1 certificate DN: C=US, O=Android, CN=Android Debug
Signer #1 certificate SHA-256 digest: a405812c6c2db59925de281505537940644ac7ece0cd628cd36b68745924c470
Signer #1 certificate SHA-1 digest: 1c0b318994c2fd75d53761d96a7fef3cd4e6a9dd
Signer #1 certificate MD5 digest: 03e57959e52033f3d5ec2c538e47df35
Signer #1 key algorithm: RSA
Signer #1 key size (bits): 2048
Signer #1 public key SHA-256 digest: c3041a3881f318e249e3c013983ea0d09535f8d7ead5eee9c856c8f64c793018
Signer #1 public key SHA-1 digest: 3cb6f8928affd7ca7e1b67865262b73b2abc3168
Signer #1 public key MD5 digest: 3e093e24dc86e4045826151d92ca0bd9
...

nielsvanvelzen · 2021-06-19T08:03:24Z

Thanks @CarlosOlivo for the investigation. I didn't expect that much of a difference between Gradle signing and apksigner (I actually expected that apksigner would be used in the background). What you're saying is that the v3 signing is missing right now. I think there is an option to enable it with the Gradle signer. But that won't solve the complete issue right?

CarlosOlivo · 2021-06-19T08:15:45Z

That's right, although the option enableV3Signing can be added, the .apk header & .zip metadata are different and support by apksigcopier is required first.

obfusk/apksigcopier#29

nielsvanvelzen · 2021-06-19T08:17:15Z

Ok thanks, will look into going back to apksigner. Maybe this is something @h1dden-da3m0n wants to work on.

Maxr1998 · 2021-06-19T08:32:46Z

Fwiw, it shouldn't be necessary to change the build.gradle.kts, we should he able to fully handle this in the CI - just don't provide the secrets to the Gradle task to disable Gradle signing.

Thanks a lot for the investigation @CarlosOlivo!

nielsvanvelzen · 2021-06-20T20:36:38Z

A new beta was released today with the mentioned changes, this should fix the build in F-Droid. I'll keep this issue open until the update is available and F-Droid.

nielsvanvelzen · 2021-06-21T09:12:13Z

Seems like it failed again because the hash for Gradle 7.1 is not known in the F-Droid build server:

==== detail begin ====
Found 7.1 via distributionUrl
No hash for gradle version 7.1! Exiting...
==== detail end ====

https://monitor.f-droid.org/builds/log/org.jellyfin.mobile/2030006

nielsvanvelzen · 2021-06-21T09:18:13Z

The hash is available in the fdroidserver source (https://gitlab.com/fdroid/fdroidserver/-/blob/master/gradlew-fdroid#L169) so this probably means the F-Droid team needs to update their build server. I'm not sure if the app will be rebuild after this is done.

linsui · 2021-06-21T09:46:53Z

Yes, the buildserver needs to be updated manully.

CarlosOlivo · 2021-06-24T22:35:30Z

Build successful!
PS: Clear cache if you see the previous build log

2021-06-24 16:01:36,971 DEBUG: unsigned/binaries/org.jellyfin.mobile_2030006.binary.apk: Verifies
2021-06-24 16:01:37,766 DEBUG: /tmp/tmpdnma5t6g/sigcp_org.jellyfin.mobile_2030006.apk: Verifies
2021-06-24 16:01:37,766 INFO: ...successfully verified
2021-06-24 16:01:37,767 INFO: compared built binary to supplied reference binary successfully
2021-06-24 16:01:37,768 INFO: success: org.jellyfin.mobile
2021-06-24 16:01:37,769 INFO: Finished
2021-06-24 16:01:37,769 INFO: 1 build succeeded

Now we just have to wait for the publishing that will be happen in the next few hours/days

nielsvanvelzen · 2021-06-30T07:03:48Z

Beta 6 is now available on f-droid! Closing the issue.

licaon-kter · 2023-05-17T07:25:25Z

And again today: https://gitlab.com/fdroid/fdroiddata/-/commit/ad409922be0d86c7b335652039ecf2026f3e3cc5

@nielsvanvelzen I guess this fixes it? https://gist.github.com/obfusk/61046e09cee352ae6dd109911534b12e

nielsvanvelzen · 2023-05-17T07:32:38Z

Thanks @licaon-kter, I realized we didn't have the workaround after we hit release but I know how to fix it this time! We will release another version (2.5.1) soon.

linsui added the bug Something isn't working label May 31, 2021

nielsvanvelzen pinned this issue Jun 16, 2021

h1dden-da3m0n mentioned this issue Jun 19, 2021

Sign APKs with apksigner instead of via Gradle #446

Merged

Maxr1998 closed this as completed in #446 Jun 20, 2021

nielsvanvelzen reopened this Jun 20, 2021

nielsvanvelzen closed this as completed Jun 30, 2021

nielsvanvelzen unpinned this issue Jun 30, 2021

linsui mentioned this issue Jul 3, 2021

F-Droid reproducible build failed Mishiranu/Dashchan#298

Closed

nielsvanvelzen reopened this May 17, 2023

nielsvanvelzen mentioned this issue May 17, 2023

Add workaround for non reproducible baseline profiles in AGP to libre builds #1065

Merged

Maxr1998 closed this as completed in #1065 May 17, 2023

v1tzor mentioned this issue Nov 13, 2023

F-Droid can't build v1tzor/TimePlanner#82

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

F-Droid reproducible build failed #414

F-Droid reproducible build failed #414

linsui commented May 31, 2021

nielsvanvelzen commented May 31, 2021

nielsvanvelzen commented May 31, 2021

linsui commented May 31, 2021

nielsvanvelzen commented Jun 5, 2021

dkanada commented Jun 5, 2021

Maxr1998 commented Jun 5, 2021 •

edited

khimaros commented Jun 12, 2021

IzzySoft commented Jun 12, 2021

nielsvanvelzen commented Jun 18, 2021

CarlosOlivo commented Jun 19, 2021

nielsvanvelzen commented Jun 19, 2021

CarlosOlivo commented Jun 19, 2021

nielsvanvelzen commented Jun 19, 2021

Maxr1998 commented Jun 19, 2021

nielsvanvelzen commented Jun 20, 2021

nielsvanvelzen commented Jun 21, 2021

nielsvanvelzen commented Jun 21, 2021

linsui commented Jun 21, 2021

CarlosOlivo commented Jun 24, 2021

nielsvanvelzen commented Jun 30, 2021

licaon-kter commented May 17, 2023

nielsvanvelzen commented May 17, 2023

F-Droid reproducible build failed #414

F-Droid reproducible build failed #414

Comments

linsui commented May 31, 2021

nielsvanvelzen commented May 31, 2021

nielsvanvelzen commented May 31, 2021

linsui commented May 31, 2021

nielsvanvelzen commented Jun 5, 2021

dkanada commented Jun 5, 2021

Maxr1998 commented Jun 5, 2021 • edited

khimaros commented Jun 12, 2021

IzzySoft commented Jun 12, 2021

nielsvanvelzen commented Jun 18, 2021

CarlosOlivo commented Jun 19, 2021

Comparison of v2.3.0-beta.1

META-INF/JELLYFIN.SF - apksigner

Comparison of v2.3.0-beta.3

META-INF/CERT.SF - Gradle

nielsvanvelzen commented Jun 19, 2021

CarlosOlivo commented Jun 19, 2021

nielsvanvelzen commented Jun 19, 2021

Maxr1998 commented Jun 19, 2021

nielsvanvelzen commented Jun 20, 2021

nielsvanvelzen commented Jun 21, 2021

nielsvanvelzen commented Jun 21, 2021

linsui commented Jun 21, 2021

CarlosOlivo commented Jun 24, 2021

nielsvanvelzen commented Jun 30, 2021

licaon-kter commented May 17, 2023

nielsvanvelzen commented May 17, 2023

Maxr1998 commented Jun 5, 2021 •

edited