New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental=false using multiDex #161

Closed
franciscodr opened this Issue Apr 15, 2015 · 18 comments

Comments

Projects
None yet
3 participants
@franciscodr

franciscodr commented Apr 15, 2015

We are using android-sdk-plugin in a new Android project.

In order to avoid 65K methods error, we have set up our project to use a multidex configuration and we have disabled the proguardCache, setting proguardCache in Android := Seq.empty

Here more info about our config:

proguardScala in Android := true,
useProguard in Android := true,
proguardCache in Android := Seq.empty

....

dexMaxHeap in Android := "2048m",
dexMulti in Android := true,
dexMinimizeMainFile in Android := true,
dexMainFileClasses in Android := multiDexClasses

Since we use multiDex, incremental=false is always shown in console and the compilation time is too long.

Are we missing anything in our config for using multiDex with incremental?

Thanks!

@pfn

This comment has been minimized.

Show comment
Hide comment
@pfn

pfn Apr 15, 2015

Member

I was led to believe by the contributor of the multidex support that incremental dex is not supported by multidex. If this is not the case, I would appreciate a tested pull request to rectify this.

Member

pfn commented Apr 15, 2015

I was led to believe by the contributor of the multidex support that incremental dex is not supported by multidex. If this is not the case, I would appreciate a tested pull request to rectify this.

@dant3

This comment has been minimized.

Show comment
Hide comment
@dant3

dant3 Apr 15, 2015

Member

If you wonder, there was an opened issue about Dex tool support of multidex + incremental. It is mutably exclusive options currently, and dex -help states that (Or did I missed some major update?). Earlier versions of Dex with multidex were refusing to run if you'll pass --incremental. Things might have changed, but looking at the gradle users group I see that its unlikely - people have same complaints there.

----- Reply message -----
От: "Perry" notifications@github.com
Кому: "pfn/android-sdk-plugin" android-sdk-plugin@noreply.github.com
Тема: [android-sdk-plugin] Incremental=false using multiDex (#161)
Дата: ср, апр. 15, 2015 21:00

I was led to believe by the contributor of the multidex support that incremental dex is not supported by multidex. If this is not the case, I would appreciate a tested pull request to rectify this.


Reply to this email directly or view it on GitHub.

Member

dant3 commented Apr 15, 2015

If you wonder, there was an opened issue about Dex tool support of multidex + incremental. It is mutably exclusive options currently, and dex -help states that (Or did I missed some major update?). Earlier versions of Dex with multidex were refusing to run if you'll pass --incremental. Things might have changed, but looking at the gradle users group I see that its unlikely - people have same complaints there.

----- Reply message -----
От: "Perry" notifications@github.com
Кому: "pfn/android-sdk-plugin" android-sdk-plugin@noreply.github.com
Тема: [android-sdk-plugin] Incremental=false using multiDex (#161)
Дата: ср, апр. 15, 2015 21:00

I was led to believe by the contributor of the multidex support that incremental dex is not supported by multidex. If this is not the case, I would appreciate a tested pull request to rectify this.


Reply to this email directly or view it on GitHub.

@dant3

This comment has been minimized.

Show comment
Hide comment
@dant3

dant3 Apr 16, 2015

Member

From latest build tools:

> $ANDROID_SDK/build-tools/22.0.1/dx --help
usage:
  dx --dex [--debug] [--verbose] [--positions=<style>] [--no-locals]
...
    --multi-dex: allows to generate several dex files if needed. This option is 
    exclusive with --incremental, causes --num-threads to be ignored and only
    supports folder or archive output.
...

Hopefully one day it will work (and I think we'll know about that from loud announce of change in gradle plugin, there is no incremental multidex currently too). But currently it works like this:

> $ANDROID_SDK/build-tools/22.0.1/dx --dex --multi-dex --incremental my_project/build/intermediates/classes-proguard/debug/classes.jar 
--incremental is not supported with --multi-dex
usage:
  dx --dex [--debug] [--verbose] [--positions=<style>] [--no-locals]
...
Member

dant3 commented Apr 16, 2015

From latest build tools:

> $ANDROID_SDK/build-tools/22.0.1/dx --help
usage:
  dx --dex [--debug] [--verbose] [--positions=<style>] [--no-locals]
...
    --multi-dex: allows to generate several dex files if needed. This option is 
    exclusive with --incremental, causes --num-threads to be ignored and only
    supports folder or archive output.
...

Hopefully one day it will work (and I think we'll know about that from loud announce of change in gradle plugin, there is no incremental multidex currently too). But currently it works like this:

> $ANDROID_SDK/build-tools/22.0.1/dx --dex --multi-dex --incremental my_project/build/intermediates/classes-proguard/debug/classes.jar 
--incremental is not supported with --multi-dex
usage:
  dx --dex [--debug] [--verbose] [--positions=<style>] [--no-locals]
...
@franciscodr

This comment has been minimized.

Show comment
Hide comment
@franciscodr

franciscodr Apr 16, 2015

Thanks @pfn and @dant3 for the clarifications about the use of multiDex. I didn't know that multidex and incremental were mutually exclusive options. I hope one day it will work too.

I will close the issue now.

franciscodr commented Apr 16, 2015

Thanks @pfn and @dant3 for the clarifications about the use of multiDex. I didn't know that multidex and incremental were mutually exclusive options. I hope one day it will work too.

I will close the issue now.

@pfn

This comment has been minimized.

Show comment
Hide comment
@pfn

pfn Apr 16, 2015

Member

I've heard using native-multidex support instead of the compat multidex
/might/ be faster?

On Thu, Apr 16, 2015 at 2:54 AM, Francisco Diaz notifications@github.com
wrote:

Thanks @pfn https://github.com/pfn and @dant3 https://github.com/dant3
for the clarifications about the use of multiDex. I didn't know that
multidex and incremental were mutually exclusive options. I hope one day
it will work too.

I will close the issue now.


Reply to this email directly or view it on GitHub
#161 (comment)
.

Member

pfn commented Apr 16, 2015

I've heard using native-multidex support instead of the compat multidex
/might/ be faster?

On Thu, Apr 16, 2015 at 2:54 AM, Francisco Diaz notifications@github.com
wrote:

Thanks @pfn https://github.com/pfn and @dant3 https://github.com/dant3
for the clarifications about the use of multiDex. I didn't know that
multidex and incremental were mutually exclusive options. I hope one day
it will work too.

I will close the issue now.


Reply to this email directly or view it on GitHub
#161 (comment)
.

@dant3

This comment has been minimized.

Show comment
Hide comment
@dant3

dant3 Apr 16, 2015

Member

Perhaps, at runtime, as support library will stop the world to unzip your additional dex files on disk, then add them to classpath before processing further. I doubt there is a compile-time difference - you still have to run dex --multi-dex

----- Reply message -----
От: "Perry" notifications@github.com
Кому: "pfn/android-sdk-plugin" android-sdk-plugin@noreply.github.com
Копия: "Viacheslav Blinov" vblinov@unison.com
Тема: [android-sdk-plugin] Incremental=false using multiDex (#161)
Дата: чт, апр. 16, 2015 19:29

I've heard using native-multidex support instead of the compat multidex

/might/ be faster?

On Thu, Apr 16, 2015 at 2:54 AM, Francisco Diaz notifications@github.com

wrote:

Thanks @pfn https://github.com/pfn and @dant3 https://github.com/dant3

for the clarifications about the use of multiDex. I didn't know that

multidex and incremental were mutually exclusive options. I hope one day

it will work too.

I will close the issue now.

Reply to this email directly or view it on GitHub

#161 (comment)

.


Reply to this email directly or view it on GitHub.

Member

dant3 commented Apr 16, 2015

Perhaps, at runtime, as support library will stop the world to unzip your additional dex files on disk, then add them to classpath before processing further. I doubt there is a compile-time difference - you still have to run dex --multi-dex

----- Reply message -----
От: "Perry" notifications@github.com
Кому: "pfn/android-sdk-plugin" android-sdk-plugin@noreply.github.com
Копия: "Viacheslav Blinov" vblinov@unison.com
Тема: [android-sdk-plugin] Incremental=false using multiDex (#161)
Дата: чт, апр. 16, 2015 19:29

I've heard using native-multidex support instead of the compat multidex

/might/ be faster?

On Thu, Apr 16, 2015 at 2:54 AM, Francisco Diaz notifications@github.com

wrote:

Thanks @pfn https://github.com/pfn and @dant3 https://github.com/dant3

for the clarifications about the use of multiDex. I didn't know that

multidex and incremental were mutually exclusive options. I hope one day

it will work too.

I will close the issue now.

Reply to this email directly or view it on GitHub

#161 (comment)

.


Reply to this email directly or view it on GitHub.

@pfn

This comment has been minimized.

Show comment
Hide comment
@pfn

pfn Apr 22, 2015

Member

@dant3 https://github.com/saturday06/gradle-android-scala-plugin#52-option-2-use-multidex -- the claim is that "incremental multidex is significantly faster" -- do you know why they might say this?

Member

pfn commented Apr 22, 2015

@dant3 https://github.com/saturday06/gradle-android-scala-plugin#52-option-2-use-multidex -- the claim is that "incremental multidex is significantly faster" -- do you know why they might say this?

@dant3

This comment has been minimized.

Show comment
Hide comment
@dant3

dant3 Apr 22, 2015

Member

@pfn from https://developer.android.com/tools/building/multidex.html#dev-build:

For the development flavor, set a minimum SDK version of 21. This setting generates multidex output much faster using the ART-supported format. For the release flavor, set a minimum SDK version which matches your actual minimum support level. This setting generates a multidex APK that is compatible with more devices, but takes longer to build.

Member

dant3 commented Apr 22, 2015

@pfn from https://developer.android.com/tools/building/multidex.html#dev-build:

For the development flavor, set a minimum SDK version of 21. This setting generates multidex output much faster using the ART-supported format. For the release flavor, set a minimum SDK version which matches your actual minimum support level. This setting generates a multidex APK that is compatible with more devices, but takes longer to build.

@pfn

This comment has been minimized.

Show comment
Hide comment
@pfn

pfn Apr 22, 2015

Member

@dant3 so does this mean that is already supported in this plugin? excellent, thanks (flavors can be simulated by using the flavorOf decorator, or just manually copy()ing an existing Project)

Member

pfn commented Apr 22, 2015

@dant3 so does this mean that is already supported in this plugin? excellent, thanks (flavors can be simulated by using the flavorOf decorator, or just manually copy()ing an existing Project)

@dant3

This comment has been minimized.

Show comment
Hide comment
@dant3

dant3 Apr 22, 2015

Member

I'm not entirely sure, there might be some magic behind the scene of gradle plugin. Anyway you can try it, in the leanest possible way just by changing sdk version for debug build type.

Member

dant3 commented Apr 22, 2015

I'm not entirely sure, there might be some magic behind the scene of gradle plugin. Anyway you can try it, in the leanest possible way just by changing sdk version for debug build type.

@pfn

This comment has been minimized.

Show comment
Hide comment
@pfn

pfn Apr 22, 2015

Member

I haven't tried multidex at all, as I focus on using proguard-cache as much
as possible.

On Wed, Apr 22, 2015 at 9:02 AM, Viacheslav Blinov <notifications@github.com

wrote:

I'm not entirely sure, there might be some magic behind the scene of
gradle plugin. Anyway you can try it, in the leanest possible way just by
changing sdk version for debug build type.


Reply to this email directly or view it on GitHub
#161 (comment)
.

Member

pfn commented Apr 22, 2015

I haven't tried multidex at all, as I focus on using proguard-cache as much
as possible.

On Wed, Apr 22, 2015 at 9:02 AM, Viacheslav Blinov <notifications@github.com

wrote:

I'm not entirely sure, there might be some magic behind the scene of
gradle plugin. Anyway you can try it, in the leanest possible way just by
changing sdk version for debug build type.


Reply to this email directly or view it on GitHub
#161 (comment)
.

@dant3

This comment has been minimized.

Show comment
Hide comment
@dant3

dant3 Apr 22, 2015

Member

I'll take a look at that as soon as I'll have a chance.

Member

dant3 commented Apr 22, 2015

I'll take a look at that as soon as I'll have a chance.

@dant3

This comment has been minimized.

Show comment
Hide comment
@dant3

dant3 Apr 22, 2015

Member

So, after having a quick check with gradle project which has about 80k of methods after proguard I see clean build time reduce from 90 seconds to 70 seconds which is already something. The dx used in the same way as always:

$ANDROID_SDK/build-tools/21.1.2/dx -JXmx2048M --dex --verbose --no-optimize --multi-dex --output $MY_PROJECT/build/intermediates/dex/debug --input-list=$MY_PROJECT/build/intermediates/tmp/dex/debug/inputList.txt

However I don't see any build time change after quick check with sbt by changing minimal sdk version. That more interesting here, is I don't see anything special in call to dx tool, it simply does not have any options on how to create dex file, such that it could know about existance of api level 21 and art. So it goes to the magic done in gradle plugin... I'll dig into it's sources

Member

dant3 commented Apr 22, 2015

So, after having a quick check with gradle project which has about 80k of methods after proguard I see clean build time reduce from 90 seconds to 70 seconds which is already something. The dx used in the same way as always:

$ANDROID_SDK/build-tools/21.1.2/dx -JXmx2048M --dex --verbose --no-optimize --multi-dex --output $MY_PROJECT/build/intermediates/dex/debug --input-list=$MY_PROJECT/build/intermediates/tmp/dex/debug/inputList.txt

However I don't see any build time change after quick check with sbt by changing minimal sdk version. That more interesting here, is I don't see anything special in call to dx tool, it simply does not have any options on how to create dex file, such that it could know about existance of api level 21 and art. So it goes to the magic done in gradle plugin... I'll dig into it's sources

@dant3

This comment has been minimized.

Show comment
Hide comment
@dant3

dant3 Apr 22, 2015

Member

So somewhy the magic that I see in TaskManager you referenced does not works on my huge gradle-based project, but I can clearly see it in action following the build definition referenced in gradle scala android plugin.

It looks like the magical "incremental mode" happens here:
https://android.googlesource.com/platform/tools/base/+/master/build-system/gradle-core/src/main/groovy/com/android/build/gradle/internal/TaskManager.groovy#2114

So it means they actually predex libraries if your sdk is at least 21. The block you mentioned should not run for non-legacy multidex as far as I can see, and legacy definition here is:
https://android.googlesource.com/platform/tools/base/+/master/build-system/builder/src/main/java/com/android/builder/core/VariantConfiguration.java#1900

So we just need to add predexing for similar build configuration and we're done.

Member

dant3 commented Apr 22, 2015

So somewhy the magic that I see in TaskManager you referenced does not works on my huge gradle-based project, but I can clearly see it in action following the build definition referenced in gradle scala android plugin.

It looks like the magical "incremental mode" happens here:
https://android.googlesource.com/platform/tools/base/+/master/build-system/gradle-core/src/main/groovy/com/android/build/gradle/internal/TaskManager.groovy#2114

So it means they actually predex libraries if your sdk is at least 21. The block you mentioned should not run for non-legacy multidex as far as I can see, and legacy definition here is:
https://android.googlesource.com/platform/tools/base/+/master/build-system/builder/src/main/java/com/android/builder/core/VariantConfiguration.java#1900

So we just need to add predexing for similar build configuration and we're done.

@pfn

This comment has been minimized.

Show comment
Hide comment
@pfn

pfn Jun 19, 2015

Member

@franciscodr If you're still interested, 1.4.2-SNAPSHOT (which you have to publish local) has the v21+ multidex optimizations. It's helping my dev cycle dramatically, and now works with proguard/proguardCache as well.

Member

pfn commented Jun 19, 2015

@franciscodr If you're still interested, 1.4.2-SNAPSHOT (which you have to publish local) has the v21+ multidex optimizations. It's helping my dev cycle dramatically, and now works with proguard/proguardCache as well.

@pfn

This comment has been minimized.

Show comment
Hide comment
@pfn
Member

pfn commented Jun 19, 2015

@franciscodr

This comment has been minimized.

Show comment
Hide comment
@franciscodr

franciscodr Jun 24, 2015

Thanks @pfn, I'll take a look at it.

franciscodr commented Jun 24, 2015

Thanks @pfn, I'll take a look at it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment