Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Embed proguard/R8 rules in kotlin-reflect artifact jar #2893

Closed

Conversation

@ZacSweers
Copy link
Contributor

ZacSweers commented Dec 18, 2019

The Android build pipeline can extract embedded proguard configurations from dependencies and merge them automatically. This adds a conservative proguard configuration to the kotlin-reflect JVM artifact in support of that. This focuses mostly on just retaining what's necessary for kotlin-reflect's own functionality to operate, but could be expanded if community feedback discovers other good candidate rules.

With this in place - most Android projects using R8 or Proguard should Just Work™️ with kotlin-reflect.

@ZacSweers

This comment has been minimized.

The Android build pipeline can extract embedded proguard configurations from dependencies and merge them automatically. This adds a conservative proguard configuration to the kotlin-reflect JVM artifact in support of that. This focuses mostly on just retaining what's necessary for kotlin-reflect's own functionality to operate, but could be expanded if community feedback discovers other good candidate rules.

With this in place - most Android projects using R8 or Proguard should Just Work™️ with kotlin-reflect.
@ZacSweers ZacSweers force-pushed the ZacSweers:z/kotlinreflectProguardRules branch from 4c3c64a to 7bb0f84 Dec 18, 2019
@@ -0,0 +1,14 @@
##---------------Begin: Kotlin-reflect ----------
# Keep Metadata annotations so they can be parsed at runtime.
-keep class kotlin.Metadata { *; }

This comment has been minimized.

Copy link
@jsjeon

jsjeon Dec 18, 2019

Contributor

I wonder this is not actually needed. Recently, I started working on synthesizing metadata from scratch (for R8-processed Kotlin library), and kept adding more tests. During that, I found that, even without this rule, R8 will still keep metadata (annotation) associated to classes.

More conversion/details are here: https://r8-review.googlesource.com/c/r8/+/46240/1/src/test/java/com/android/tools/r8/kotlin/metadata/MetadataRenameInParametertypeTest.java#70

This comment has been minimized.

Copy link
@jsjeon

jsjeon Dec 18, 2019

Contributor

Ah, one caveat is, though, R8 with this fix (https://r8-review.googlesource.com/c/r8/+/44220) will keep metadata annotation without definition. For prior versions of R8, it may be the case that we still need this rule.

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Dec 23, 2019

Author Contributor

Yeah I think it's best to keep this as a conservative rule. We've had reports in Moshi that Metadata is stripped in android builds without a custom keep rule for some users, though unsure what version they're using or if they're just using proguard.

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 6, 2020

Author Contributor

@jsjeon what version of R8 would that fix be in? I'm not sure how to check the latest release history or what's in them off-hand there

This comment has been minimized.

Copy link
@jsjeon

jsjeon Jan 8, 2020

Contributor

Ah, it's even difficult for me to check which version includes it. :\ Based on the timeline, I guess AGP 3.6 will include that.

However, that fix I mentioned was about keeping annotations associated with classes/members if the definition of annotations is missing. Therefore, if annotation class Metadata is given (via kotlin reflect artifact jar of course), i.e., not missing, it is still the case that R8 will take it away if it thinks it's not used.


I was about to mention the case like https://issuetracker.google.com/issues/146534384 where kotlin.Metadata itself is retrieved at runtime. In that case, yeah, this rule is necessary.

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 14, 2020

Author Contributor

Sounds like it's safest to just keep this for now


# Keep generic signatures and annotations at runtime.
# R8 requires InnerClasses and EnclosingMethod if you keepattributes Signature.
-keepattributes InnerClasses,Signature,*Annotation*,EnclosingMethod

This comment has been minimized.

Copy link
@jsjeon

jsjeon Dec 18, 2019

Contributor

Do we need broad *Annotation* or just RuntimeVisibleAnnotation (because metadata is runtime visible) ?

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Dec 23, 2019

Author Contributor

I didn't know that RuntimeVisibleAnnotations was a thing! Historically any project reading annotations at runtime has always used *Annotation*, but maybe that's just a bespoke pattern that isn't needed. Let me try in our codebase

This comment has been minimized.

Copy link
@udalov

udalov Jan 3, 2020

Member

There are also at least RuntimeVisibleParameterAnnotations and RuntimeVisibleTypeAnnotations, so maybe the pattern should be RuntimeVisible*Annotations, if that works...

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 6, 2020

Author Contributor

That seems to work b843558

-keep class kotlin.Metadata { *; }

# Keep kotlin-reflect internals.
-keep class kotlin.reflect.jvm.** { *; }

This comment has been minimized.

Copy link
@jsjeon

jsjeon Dec 18, 2019

Contributor

This seems way too broad. At least, we can try public APIs only.

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Dec 23, 2019

Author Contributor

Let me try it in our project. To be totally honest I'm not convinced this is truly needed, but it's frequently suggested as a fix in the community. The last comprehensive list we were using internally was based on this:

square/moshi#883

Though all of those are public classes as well, so it would work. @udalov any advice on what scope should be kept?

This comment has been minimized.

Copy link
@udalov

udalov Jan 3, 2020

Member

The list from the issue mentioned above is actually the combination of all services and implementations used in kotlin-reflect -- see META-INF/services in kotlin-reflect.jar. Shouldn't they be handled correctly in some other way by Proguard/R8? E.g. is it possible to "keep" the directory META-INF/services, and everything it references, to avoid enumerating classes used there?

Other than that, we do need to fully keep the contents of packages kotlin.reflect.full and kotlin.reflect.jvm, but not their subpackages (i.e. kotlin.reflect.jvm.internal).

This comment has been minimized.

Copy link
@JakeWharton

JakeWharton Jan 3, 2020

Contributor

Types declared inside META-INF/services/ are handled automatically provided that the code which calls into ServiceLoader with that service type is not being removed. You shouldn't need to do anything for them.

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 6, 2020

Author Contributor

They're handled automatically by R8, but not proguard iirc. Can we have rules that are only picked up by one or the other?

This comment has been minimized.

Copy link
@JakeWharton

JakeWharton Jan 6, 2020

Contributor

Yes. See kotlinx.coroutines for an example that targets R8 1.6 and newer only.

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 6, 2020

Author Contributor

Done in 08616fb

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 6, 2020

Author Contributor

Also tweaked to just keep service-loaded APIs, only in proguard or R8 <1.6 c15aac4

This comment has been minimized.

Copy link
@jsjeon

jsjeon Jan 8, 2020

Contributor

Thanks for taking care of this thoroughly! Those specific rules that aim for only interface and services look great!

@@ -0,0 +1,14 @@
##---------------Begin: Kotlin-reflect ----------

This comment has been minimized.

Copy link
@udalov

udalov Jan 3, 2020

Member

Are these Begin/End tags required?

Also, please consider renaming the file itself to kotlin-reflect.pro

This comment has been minimized.

Copy link
@JakeWharton

JakeWharton Jan 3, 2020

Contributor

Are these Begin/End tags required?

Not needed. They will be added automatically by R8 now for blaming purposes.

https://issuetracker.google.com/issues/134372926

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 6, 2020

Author Contributor

By R8 yes. Do we care about it in a proguard context? They're not strictly required but the blaming purpose is nice

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 6, 2020

Author Contributor

Removed in c6f922b

Copy link
Contributor Author

ZacSweers left a comment

Took a significant pass at all the feedback and updated. The structure is now

/META-INF
  /com.android.tools
    /proguard/kotlin-reflect.pro       // Proguard-only, with custom serviceloader rules
    /r8-from-1.6.0/kotlin-reflect.pro  // R8 1.6+ with serviceloader handling
    /r8-upto-1.6.0/kotlin-reflect.pro  // R8 <1.6 with customservice loader rules
  /proguard/kotlin-reflect.pro         // Default proguard rules location, with custom serviceloader rules

Let me know what you think

-keep class kotlin.Metadata { *; }

# Keep implementations of service loaded interfaces
# R8 will automatically handle these these in 1.6+

This comment has been minimized.

Copy link
@jsjeon

jsjeon Jan 8, 2020

Contributor

Nit: this file is not quite related to R8. :) (and below too)

This comment has been minimized.

Copy link
@ZacSweers

ZacSweers Jan 14, 2020

Author Contributor

D'oh, fixed in 5091302

@ZacSweers

This comment has been minimized.

Copy link
Contributor Author

ZacSweers commented Jan 14, 2020

@udalov anything else needed for this PR?

@udalov

This comment has been minimized.

Copy link
Member

udalov commented Jan 17, 2020

@ZacSweers Thank you for the contribution!

Merged manually in 482874f. The changes will be available in 1.4-M1.

@udalov udalov closed this Jan 17, 2020
@ZacSweers

This comment has been minimized.

Copy link
Contributor Author

ZacSweers commented Jan 17, 2020

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.