Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Okio 2.1.0 / Proguard issue #516

Closed
Tolriq opened this issue Sep 30, 2018 · 33 comments
Closed

Okio 2.1.0 / Proguard issue #516

Tolriq opened this issue Sep 30, 2018 · 33 comments

Comments

@Tolriq
Copy link

Tolriq commented Sep 30, 2018

Took me a while to figure out what was the cause as the issue does not occurs on my CI but updating Okio from 2.0.0 to 2.1.0 cause proguard 6.0.3 on my windows dev machine to goes nuts.

I fear you won't have any idea and I have no idea how to debug that but with Okio 2.1.0 + Moshi 1.7 (And 1.6) + Okhttp 3.11 + some basic okio code in kotlin in my code like

response.body()?.use { body ->
                    return ByteArrayOutputStream().run {
                        this.sink().buffer().use { it.writeAll(body.source()) }
                        toByteArray()?.also { close() }
                    }
                }

Cause proguard to be stuck for infinite time at Optimizing (pass 1/5)... step.
Gradle debug mode does not show any issue during that time, just the usual memory checks without any memory issues.

Reverting to 2.0.0 fix the issue.

R8 from AGP 3.2 have an issue I reported that was unfortunately not fixed in time, and R8 from 3.3a12 does compile but Moshi is no more working (was previously when I helped the Google team fixing the R8 issue) and I have not yet found (searched) the cause but maybe it's tied. (Moshi generates {} instead
of correct objects).

Edit: Tested 2.0.0 with R8 AGP 3.3a12 and same json issue so proguard issue not related to R8 issue that I'll investigate later.

@Tolriq
Copy link
Author

Tolriq commented Oct 4, 2018

So I guess no one have an idea :) Let's hope AGP 3.3 gets final for R8 before OkHTTP require version 2.1 :)

@Egorand
Copy link
Collaborator

Egorand commented Oct 4, 2018

Interesting. Might be worth trying to identify the offending part of that snippet to figure out what exactly breaks ProGuard.

@Tolriq
Copy link
Author

Tolriq commented Oct 4, 2018

I've commented all my custom code that used okio as a test after your remark and the issue is still present.

Okhttp / moshi are the only one using okio

+--- com.squareup.okhttp3:okhttp:3.11.0
|    \--- com.squareup.okio:okio:1.14.0 -> 2.1.0
|         \--- org.jetbrains.kotlin:kotlin-stdlib:1.2.60 -> 1.2.71
|              +--- org.jetbrains.kotlin:kotlin-stdlib-common:1.2.71
|              \--- org.jetbrains:annotations:13.0
+--- com.squareup.okio:okio:2.1.0 (*)
+--- com.squareup.moshi:moshi:1.7.0
|    \--- com.squareup.okio:okio:1.15.0 -> 2.1.0 (*)

@Egorand
Copy link
Collaborator

Egorand commented Oct 4, 2018

So you're suspecting either OkHttp or Moshi? AFAIK neither is supposed to work with Okio 2 (althought it's binary-compatible with Okio 1), so I would probably not force the version by declaring Okio dependency explicitly.

@JayNewstrom
Copy link

I'm also seeing this issue on a project I work on. I'm not using moshi. Only tried upgrading to okio 2.1.0 yesterday. I haven't had time to dig into why it's failing.

@Tolriq
Copy link
Author

Tolriq commented Oct 4, 2018

@Egorand I don't know the cause I just know 2.0.0 works 2.1.0 don't :)

I've seen square/okhttp#4256 where it's said it work :)

And since I'm using Kotlin the new 2.0 kotlin things for my own code are a great addition. I can stay at 2.0 as it works and I can use the Kotlin stuff, but I think it's good to know that there's something odd somewhere that may worth investigating.

@qwertyfinger
Copy link

qwertyfinger commented Oct 12, 2018

Sharing my observations regarding this issue, which I encounter both on macOS and Windows (so I'm not sure it is exclusive to some environment).

Summary:
Okio 2.1.0 + AGP >= 3.2.0 + Proguard with optimizations = Infinite execution of optimization step (build never finishes)

Take anything out of this equation and everything works fine.

Possible solutions in detail:

  1. Use Okio 2.0.0 or 1.x.x.
  2. Stick to AGP 3.1.4 or earlier.
  3. Disable optimizations with -dontoptimize flag in your Proguard config.
  4. Use R8 instead of Proguard:
    a) For AGP 3.2.x: add android.enableR8=true to your gradle.properties.
    b) For AGP 3.3.0-alphaXX: R8 is enabled by default. Note: You can use any AGP >= 3.3.0-alpha01, but the latest (at the moment it's 3.3.0-alpha13) is recommended.

Each solution has its drawbacks, but the most production-friendly are probably using earlier Okio versions if you don't need v.2.1.0 specifically, or disabling optimizations if you can afford to have somewhat bigger APK.

Regarding the origins of this issue and who should be responsible for fixing it: ¯\_(ツ)_/¯

@Tolriq
Copy link
Author

Tolriq commented Oct 12, 2018

Thanks for confirming :)

Maybe @swankjesse or @JakeWharton have ideas about how to debug this :(

R8 is far from production ready seeing the few bugs I have reported to them. And no proguard is not possible for most large applications that do not want multidex.
Only real solution now is 2.0 and 1.x with 2.0 way more fun when working with Kotlin.

I fear that 2.1 or 2.2 or next will be needed by okhttp or moshi before R8 is stable.

@qwertyfinger
Copy link

qwertyfinger commented Oct 12, 2018

@Tolriq I never mentioned "no ProGuard", only "no ProGuard optimization", these are different things. You can still have ProGuard shrinking and obfuscating. Optimization provides relatively small gain in comparison with shrinking, and in most cases you don't need it to avoid multidex.

@Tolriq
Copy link
Author

Tolriq commented Oct 12, 2018

@qwertyfinger For the record on my production app difference between 5 optimization pass and don't optimize is:

  • 238 classes
  • 13 625 methods
  • 15 254 method references

Inlining and merging with Java 8 and lambdas ;)

I'm not sure you can call that small gains ;) (And from real word tests, speeds impact is huge too).
But yes they are not the same thing.

@swankjesse
Copy link
Member

Does it reproduce with Okio 1.16?

@Tolriq
Copy link
Author

Tolriq commented Oct 12, 2018

No it does not. (Sorry for delay but I can no more revert the nice 2.0 kotlin code without conflicts)

@swankjesse
Copy link
Member

Hmmmm... I think our next step is to report an issue to the ProGuard project, which I assume is where the fix will go.

Can you use ps to get the process ID of the hanging ProGuard process, and then jstack to pull a stacktrace from it? That'll help them to diagnose where the problem is.

@Tolriq
Copy link
Author

Tolriq commented Oct 12, 2018

Here's the dump.

Before finding it was okio 2.1 I first tried to see how to report to them but it seems all support is now paid. I suppose you have better contacts :)

okio.txt

@ursusursus
Copy link

ursusursus commented Nov 7, 2018

I am experiencing this as well with 2.1.0, reverting to 2.0.0 helps

btw isnt this a google thing since they reworked the shrinker?

@qwertyfinger
Copy link

@ursusursus Probably not a 'google thing'. In fact, the shrinker Google introduced (R8) doesn't have this issue, only ProGuard does.

@ursusursus
Copy link

@qwertyfinger hmm, I thought it was default in AS 3.2, even the log outputs changed while building. Is it still behind that exprimental flag?

@Tolriq
Copy link
Author

Tolriq commented Nov 7, 2018

R8 is still not default, but no more experimental.

Beware that R8 with AS 3.2 B2 still have some potential major issues and 1.3.33 should be used (or wait for Beta 4;))

Edit: Beta 3 does not include the necessary fixes, better wait for B4.

@ursusursus
Copy link

ursusursus commented Nov 7, 2018

Oh, thanks guys, I even looked at release apks how nice and smaller they are lol, placebo

@Tolriq
Copy link
Author

Tolriq commented Nov 7, 2018

R8 does have impact on APK size, but R8 full mode is really impressive (still experimental and still have some bugs that they are working on, so be cautious for prod).
On my large production application R8 full mode is 1MB less on the dex file compared to R8 normal, and 1.5MB from proguard with 5 optimization pass.

Not talking about compilation speed, they are really making a fantastic job with those tools. (Special thanks to Jinseong at Google for the fast and efficient support and complicated work on understanding and fixing some quite complex issues).

@ursusursus
Copy link

now im confused 😀 r8 normal mode is on in as 3.2, agp 3.2.1?

@JakeWharton
Copy link
Member

JakeWharton commented Nov 7, 2018 via email

@jsjeon
Copy link

jsjeon commented Nov 8, 2018

We enabled R8 normal mode in AS 3.3 canary, then disabled for beta. It's still enabled by default in AS 3.4 canary, and we plan to keep it that way for AS 3.4. At the same time, full mode is available: https://android-developers.googleblog.com/2018/11/r8-new-code-shrinker-from-google-is.html

@swankjesse
Copy link
Member

Tracked by the ProGuard project here. No action for us to take.
https://sourceforge.net/p/proguard/bugs/730/

@Tolriq
Copy link
Author

Tolriq commented Nov 10, 2018

For the others, confirmed as working with proguard 6.1 B1.

To test add in your buildscript

 configurations.all {
        resolutionStrategy {
            force("net.sf.proguard:proguard-base:6.1.0beta1")
        }
    }

@ursusursus
Copy link

ursusursus commented Nov 11, 2018

Question: how a new version of proguard would get to my android gradle build naturally? via new AGP version?

@qwertyfinger
Copy link

@ursusursus Yes, that's correct. But currently all of the AGP 3.2.1, 3.3.0-beta03 and 3.4.0-alpha03 use ProGuard 6.0.3.

@qwertyfinger
Copy link

@Tolriq Seeing as you're aware of all the subtle quirks of R8, are there any dangers in using this latest beta ProGuard version in your opinion?

@Tolriq
Copy link
Author

Tolriq commented Nov 11, 2018

Well it's more that I'm unfortunate to be touched by a few of them in my main application ;)

R8 works great, there's some edge case with over optimization, they fixed a few already, there's still at least one very nasty one that generate native crash and no one have a clue about it for the moment.

About proguard, since beta 1 crash for me on some cases for me I'd say be careful and test :) And if you have issue, just report there, too bad no one gave the bug tracker url when I said I did not found it.

But it's the same thing for every thing that you'll update, you should have proper tests, and use alpha / beta channels on Play Store before production.

@hunter-t
Copy link

Unfortunately, this issue is still here for this config:

  • Android Studio Android Studio 3.4 Beta 3
  • Andoid Gradle Plugin 3.3.0
  • Okio 2.2.2

R8 is just frozen on Optimizing (pass 1/5)...

Had to revert to Okio 2.0.0 in order to resolve this problem.

@Tolriq
Copy link
Author

Tolriq commented Mar 29, 2019

@hunter-t

This is not R8 but proguard that output that.

Either switch to R8 or use the last proguard beta.

@shashachu
Copy link

shashachu commented Apr 5, 2019

Overriding the proguard version doesn't seem to work for me - I used the gradle config in this comment: #516 (comment)

./gradlew buildEnvironment is showing the proper version:

|    +--- net.sf.proguard:proguard-gradle:6.0.3 -> 6.1.0beta1
|    |    \--- net.sf.proguard:proguard-base:6.1.0beta1

Is there some gradle incantation I'm missing?

I'm using AGP 3.3.2, Gradle 4.10.3

@shashachu
Copy link

For future readers: the issue was that we have a buildSrc/build.gradle file. I had to set implementation "net.sf.proguard:proguard-base:6.1.0beta1" inside of that file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants