Reproducible Build fails for build106.3.1

[IzzySoft]

First, according to the vcsInfo embedded into the APK, it was not built from the tagged commit but rather from 0f9b3df – but regardless whether we build from that commit, or from the tagged one and use git reset --soft to take care for the embedded one, RB fails. Here are the diffs: diffs.zip

Was the APK built from a clean tree – or were there local changes, or artifacts remaining from prior builds? Or maybe some dependencies not pinned? Asking the latter because of:

-  VISIBILITY_RUNTIME Lkotlin/Metadata; d1={ "...\n...\n....\n........2.0.B.¢......¨.." } d2={ "Lcom/bumptech/glide/annotation/ksp/GlideIndexer_a8490dd8_f8f2_34c4_b51f_cd26e9cb7b6f;" "" "<init>" "()V" "app_githubRelease" } k=1 mv={ 2 2 0 } xi=48
+  VISIBILITY_RUNTIME Lkotlin/Metadata; d1={ "...\n...\n....\n........2.0.B.¢......¨.." } d2={ "Lcom/bumptech/glide/annotation/ksp/GlideIndexer_bd0014c5_eea0_3f4c_ae66_680de5087bbd;" "" "<init>" "()V" "app_githubRelease" } k=1 mv={ 2 2 0 } xi=48

-  Class descriptor  : 'Lcom/bumptech/glide/annotation/ksp/GlideIndexer_a8490dd8_f8f2_34c4_b51f_cd26e9cb7b6f;'
+  Class descriptor  : 'Lcom/bumptech/glide/annotation/ksp/GlideIndexer_bd0014c5_eea0_3f4c_ae66_680de5087bbd;'
   Access flags      : 0x0011 (PUBLIC FINAL)

A "dirty build" (local changes/artifacts) looks the most plausible to me, but only you can tell – and I can be wrong there 😉

[Hamza417]

I checked, and the actions logs points to a clean build and yeah the tag doesn't point to the correct commit either. I'll do a clean build and run a RB test.

[Hamza417]

I ran a build locally and RB has failed. The diff points to KSP, possibly because of updating it to 2.2.0?

[IzzySoft]

It somehow looked like a mix of "dirty build" and "unpinned version" of a lib (the GlideIndexer I quoted – the ZIP has more differences). But that KSP update was long before the release, we should both have the same version there.

You use this Github action to build the release APK, correct? The recipe here reflects the Github runner setup. The differing hashes in the diff irritate me a bit; you say the APK was built from a clean tree at 0f9b3df ? Let me check:

• the tagged commit is 7a3678c – and only updates the Readme.
• the tag before that is 791c6ea – which only updates the GH action definition, and even there nothing build or code related
• the tag before that is 0f9b3df – which the APK claims to be built from
• the hash (sha256sum) of the unsigned APK from our first builder is daa5bd6fdbd68bb341d7f2a700ef752402b0f74aeaa7a9b655b374c0b099bc0c
• our second builder however instead has 338e675ee5a9dc25332dd7a58c185b9369d1a0148c330509b70e693cd05696ee

Now that is funny. One should expect those two being identical. So it might be something "flaky" there – but not necessarily in a way that "multiple builds in a loop" produce different APKs (on the first builder, I've run it twice, one time using git reset --soft, as the second builder used, and another time git checkout, and both produced the same APK. That GlideIndexer seems not to be deterministic:

-  Class descriptor  : 'Lcom/bumptech/glide/annotation/ksp/GlideIndexer_a8490dd8_f8f2_34c4_b51f_cd26e9cb7b6f;'
+  Class descriptor  : 'Lcom/bumptech/glide/annotation/ksp/GlideIndexer_3e3b1157_a9aa_3f97_95ea_e792b024ad13;'
   Access flags      : 0x0011 (PUBLIC FINAL)

Not sure what it does, or why it differs. Let me try the shaker here.

Wait, what is that? Did you replace the release? Oof. OK, in that case I can stop, and we wait for the next release then – we cannot get this one RB anymore, as now our repo has a different APK then the one at the replaced release, so zero chance we get the green shield up again. Please, do not replace what has already been distributed. Just make a new release next time please, OK? I'll try the shaker now on the current release, just to see if it works – but as it has the exact same name (and the APK most likely the very same versionCode), it won't be pulled again (and whoever had the APK from the replaced release installed, would not get an update notification anymore anyway).

[IzzySoft]

From the build log, not sure if it's relevant:

w: [ksp] LibraryGlideModules [] are included more than once, keeping only one!

And the shaker didn't help. 6 times in a row, the identical APK – just not yours. Last try was matching the CPU count, no dice either – so I have to give up and let the "failed RB" show, sorry. With some luck, we get it tackled for the next release.

[Hamza417]

Ok, I checked and yeah the issue was indeed by KSP and downgrading it fixed the glide issue. I have checked this APK against the RB build and downgraded the Gradle to 8.10.1 and Kotlin to 2.1.0 (here) KSP to 2.1.0-1.0.29 (here) in this commit: de53453. There is currently an issue here #2509 loosely related to this, highlighting the compatibility issues with Gradle 8.11+ which I used to build the last APK. Some help from #2511 as well, since just there is a series of KSP releases that are broken with recent Gradle releases and not just one.

Although, in my case the RB still fails but not because of the Glide but by unidentical zip info? Not sure why, but at least the main issue has been identified and resolved.

diff1.zip

+ Zip·file·size:·26909297·bytes,·number·of·entries:·2007
- Zip·file·size:·27074586·bytes,·number·of·entries:·2010

And the shaker didn't help. 6 times in a row, the identical APK – just not yours.

Were you able to produce 6 identical builds in your machine with the last build configuration? I was not, despite testing it on my local server and actions as well. No luck.

w: [ksp] LibraryGlideModules [] are included more than once, keeping only one!

That is just a log showing glide modules are being indexed, it is where the GlideIndexer is generated and if the sum of chars in modules is more than if I am right 256 it falls back to just using a UUID for class name. For RB to work all classes should be added in the same order every time so it is best to sort them alphabetically first to ensure that. And it is the job of annotation processors, in this case KSP. The Indexer class itself doesn't sort or order anything, it just relies on the processor to send them in the same order? IDK why it's like that, except it is prone to errors like in this particular case and #399 which was caused by kapt. I should just lock the kotlin and ksp at this point.

[linsui]

Looks like you are comparing an unsigned apk with a signed apk. They can't be identical.

[Hamza417]

Looks like you are comparing an unsigned apk with a signed apk. They can't be identical.

I checked and the difference is indeed caused by cert files. I keep overlooking the small details :)

So, that means build is RB. Should I publish a maintenance release?

[linsui]

Yes, please. :)

[IzzySoft]

Ok, I checked and yeah the issue was indeed by KSP and downgrading it fixed the glide issue.

Good find! Do you plan to file an issue with KSP? Would help not only you (to be able to receive updates), but also other developers using that library.

Were you able to produce 6 identical builds in your machine with the last build configuration?

As I wrote: 6 builds in a loop (so yes, all of them using the very same configuration, running clean on each build for a fresh start) – 6 times the same APK. So something there is bound to machine/OS/setup specifics, not something random, it seems.

it just relies on the processor to send them in the same order?

If that's the case, things like CPU core count could matter (like, depending on what's found first – which might differ with different number of threads).

I should just lock the kotlin and ksp at this point.

That might be a good temporary work-around, yes. But it cannot be the "final resolution". You cannot stick on those versions forever. So ideally, the underlying issue gets fixed at KSP.

Should I publish a maintenance release?

A "Yes, please." From our side as well 😊 And thanks @linsui for chiming in!

[Hamza417]

Good find! Do you plan to file an issue with KSP? Would help not only you (to be able to receive updates), but also other developers using that library.

I'll look into it, there is something new to learn here :)

I have published a new release :) while we are at it, could you update the app's screenshot on both repos? On both the repos, the screenshots showing are a mix of both older and recent ones.

The lists are looking like this at the moment.
IzzyOnDroid

F-Droid

[IzzySoft]

while we are at it, could you update the app's screenshot on both repos?

That's for you to do in your fastlane. Ah, you switched from JPG to PNG while keeping the names – that's the only thing where cleanup fails. I've just removed the JPGs from Inure. What's the second app you refer to? Peri only has PNGs here, 01-12 – same as in the Fastlane of its Github repo.

[Hamza417]

What's the second app you refer to?

Not the second app but on F-Droid as well, it has all old and new images, I think @linsui can help here?

[linsui]

https://gitlab.com/fdroid/admin/-/issues/377#note_2652607403

[linsui]

Reproducible build is fixed, thanks!