doesn't build reproducibly (due to bug in coreLibraryDesugaring) #6486
Comments
|
shasums for the 3 variants: |
|
So far it looks like disabling coreLibraryDesugaring and building w/ openjdk 11 fixes this. |
|
With |
|
Desugaring isn't optional if you're using Java8+ and supporting old Android releases. Sounds like this is a bug worth reporting to Android/Google. Someone else recently reported a bug in an androidx lib, and it seems to have gotten some attention from Google. |
|
we track all known RB bugs at the bottom here
https://f-droid.org/docs/Reproducible_Builds/
|
obfusk
changed the title
|
I couldn't find an issue with google about desugaring related reproducible builds issues so we opened: https://issuetracker.google.com/issues/195968520 |
|
Doesn't downgrade the bugged libraries work as workaround? |
|
the core library desugaring dependency broke reproducible builds right when they were introduced in NewPipe. I tried almost every version. |
|
Is there a way to view this without using a Google account? Forcing a person to sign-in to just view an issue is absolutely moronic. |
Agreed. fwiw: so far, all I can see is that it's been "assigned to an...@google.com". |
|
There has been some activity:
|
Does anyone know how to use this "tip of tree"? I'd be happy to test. @Bubu does that cover the issues you saw or did you see other differences as well? |
|
I don't really know. It's really hard to tell. The diffoscope diff output is 25mb. :-/ Unfortunately I don't currently know how to test latest git version of the desugarer. Meanwhile we recursively forked all our required libs, replaced the Java time api accesses with threetenbp and got rid of this stupid desugarer. The project is reproducible again now. |
The uncompressed text diff is 204Mb :p That seems to be mostly these kind of differences though, which is different -- but could be related, not sure -- from the bug I reported here: |
|
So... the problem seems to be in (older versions of) R8. I've built the latest version 9 times with this patch and all builds were identical. diff --git a/build.gradle b/build.gradle
index 315625a0f..9548d1080 100644
--- a/build.gradle
+++ b/build.gradle
@@ -7,6 +7,7 @@ buildscript {
google()
}
dependencies {
+ classpath 'com.android.tools:r8:2.2.64' // must be before com.android.tools.build:gradle
classpath 'com.android.tools.build:gradle:4.1.3'
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
|
|
I now also tried this, it doesn't change anything for CCTG. (i.e. there's still megabytes of diff output) |
That seems to be a different bug. |
|
@Bubu what happens when you set |
|
I'll test that later (or tomorrow, currently in the middle of 3 other projects) but this isn't really a fix for us as we do want to keep our minsdk at 21. |
I get that. But at least it's some kind of progress (if it works). |
|
tried with this and as it mentioned here not working still |
|
I build the latest version 4 times and all of them are identical. But they are differnt with your release. The only different file is assets/dexopt/baseline.profm. This looks like a known issue https://issuetracker.google.com/issues/231837768 but I always get the same apk. I'd test more build. |
|
The 5th build produced an apk identical with your release. We can enable reproducible build for F-Droid and wait until it builds a reproducible apk. Or you can disable the baseline file. |
~5 build cycles before it gets published seems a bit too long to wait for updates, especially for NewPipe; unless e.g. we were to retry building a few times during the same cycle if RB fails, but that would require modifications to |
I had a similar reproducibility problem (time difference) with zipsigner and I have fixed it but running with: |
|
Could be an option: https://gitlab.com/fdroid/fdroiddata/-/issues/2816#note_1204295547 |
|
@obfusk's https://github.com/obfusk/reproducible-apk-tools#sort-baselinepy can fix the profm problem. @TobiGr Do you want to have a try? |
|
You can also sort the import com.android.tools.profgen.ArtProfileKt
import com.android.tools.profgen.ArtProfileSerializer
import com.android.tools.profgen.DexFile
project.afterEvaluate {
tasks.compileReleaseArtProfile.doLast {
outputs.files.each { file ->
if (file.toString().endsWith(".profm")) {
println("Sorting ${file} ...")
def version = ArtProfileSerializer.valueOf("METADATA_0_0_2")
def profile = ArtProfileKt.ArtProfile(file)
def keys = new ArrayList(profile.profileData.keySet())
def sortedData = new LinkedHashMap()
Collections.sort keys, new DexFile.Companion()
keys.each { key -> sortedData[key] = profile.profileData[key] }
new FileOutputStream(file).with {
write(version.magicBytes$profgen)
write(version.versionBytes$profgen)
version.write$profgen(it, sortedData, "")
}
}
}
}
} |
|
Out of curiosity, what, exactly, is a "baseline.profm"? |
An Not sure what it's actually used for. Google does not seem to provide any documentation regarding these files or their formats. |
|
Here is an apk signed with my keys (NOT NewPipe's) based on |
$ apksigcopier compare app-release-upstream.apk app-release-built-by-obfusk.apk && echo reproducible
reproducibleSo yes :) |
Thanks to obfusk, see https://issuetracker.google.com/issues/231837768 and #6486
I compared the APKs from the last 5 releases on GitHub to the APKs built and published by F-Droid.
Some of them are identical (apart from the signature), but not all of them are.
I've done multiple clean builds of
0.21.4; they were not always identical.I've seen 3 variants, one identical to the GitHub APK, another identical to the F-Droid APK.
The differences I've seen were always in
j$/util/Collection$-EL.class.diffoscope output
cc @eighthave
The text was updated successfully, but these errors were encountered: