"lein droid build" fails on Windows with JDK7

[oakes]

I'm on Windows 7 x64, and "lein droid build" fails. It initially couldn't find /platform-tools/aapt and /platform-tools/dx, because on windows they are "aapt.exe" and "dx.bat". I made symbolic links to both using mklink, but it still can't run dx.bat because it says it is "not a valid Win32 application". I believe this is because ProcessBuilder can't run *.bat files directly.

I'm not sure if there is a simple way to normalize all the pathnames in the code to make Windows work -- it may be more effort than it is worth. For now, I am booting up an Ubuntu VM, but creating the DEX file is taking forever (over 24 hours) because I have hundreds of *.java files in my project, so I was hoping to try compiling it in my host OS for speed.

[oakes]

I solved this by simply modifying the path name in the lein-droid plugin, so I'll close this issue for now.

[alexander-yakushev]

Hello Zach,

Sorry for the delay in my response. I'm glad you was able to solve this issue. I guess I could add a few parameters to the plugin so you could configure Android binaries paths without touching the plugin.
Could you please share how have you dealt with the second problem (running .bat files)?

[oakes]

I just modified the path string in the code to include .bat at the end, so it wasn't really a solution. I just wanted to get it running on my machine.

Unfortunately, creating the DEX file still took way too long. I can leave it running for hours and it will never finish. When I compile a project in Eclipse, the dexing process takes a few seconds. Do you know why this is?

[alexander-yakushev]

That's strange. Is it just an Java Android application, or you manage to build Clojure Android project in Eclipse?

It would be very useful if you run something like DEBUG=1 lein droid create-dex and post the output here. Also, could you please try to build your project using Android's default Ant script and tell if it take so long as well?

[oakes]

In Eclipse, it was a normal Java Android application. My project has about 700 Java files, so for some reason it's bogging lein-droid down, but not Eclipse. I just tried building with Ant, and it completed in 24 seconds, so about the same as Eclipse.

Here is the output when I prepend the create-dex command with "DEBUG=1":

/home/oakes/android-sdk-linux/platform-tools/dx --dex --verbose-dump --output /home/oakes/lein-droid-test/target/classes.dex /home/oakes/lein-droid-test/target/classes /home/oakes/android-sdk-linux/tools/support/annotations.jar /home/oakes/.m2/repository/android/tools.nrepl/0.2.0-bigstack/tools.nrepl-0.2.0-bigstack.jar /home/oakes/.m2/repository/android/clojure/1.4.0/clojure-1.4.0.jar /home/oakes/.m2/repository/neko/neko/2.0.0-beta1/neko-2.0.0-beta1.jar

[alexander-yakushev]

Hm, obviously long DEXing time is influenced by processing Clojure jars. It is not yet clear if the number of your java files also matter.

Can I ask you to create a sample Clojure Android app (lein droid new foo org.foo.bar) and measure its build time? Thanks in advance.

[oakes]

Hi, sorry for the late response, I'm in the U.S. =) Yes a normal lein droid project builds just fine, even if I add a few Java files. I created a test project with all ~700 Java files inside, so please clone it and try it out:

https://github.com/oakes/lein-droid-test

[alexander-yakushev]

Now that's very strange. I've downloaded your sample project and tried running dexer on it with and w/o Clojure files. Dexing without Clojure is the same as in Ant's version, and it took me exactly 24 seconds as well. Dexing a full project takes 88 seconds which is pretty much expected (DEXing a sample project without Java files takes 54 seconds on my machine).

I also tried DEXing the project with two Android SDK revisions - 18 and 20.0.3. Times for both are the same. What SDK version are you using?

I thought this might be SDK problem on Windows but since you say on Ubuntu it takes hours too, that's probably not a reason. One more thing - how long does it take to DEX a sample Clojure/Android application without those Java files?

[oakes]

I was using 20.0.1, though I just upgraded to 20.0.3 and there is no change -- the full project still hangs. I even removed my SDK and re-downloaded everything, and it just never stops. Dexing a new Clojure/Android app without Java files takes about 30 seconds.

Well I guess I'll have to keep troubleshooting, I'm not sure what I'm doing wrong.

[oakes]

I created a 32-bit Ubuntu VM to see if perhaps it was a 64-bit issue, but creating the DEX still ran for about a half hour before I force-quit it. This is the craziest issue I've had in a while.

[alexander-yakushev]

You are right, this is insane:).

I'll try to doublecheck everything tomorrow (it's past midnight here already), but I really have no clue what's going on.

By the way, do you use latest leiningen preview10? It seems like the newest version kind of messed profiles a bit, thus debug build uses release configuration. I have to check this one tomorrow as well.

[oakes]

I was indeed using preview10. I downgraded to preview9 but no change on Ubuntu x86 or x64. Downgrading to beta1 of the plugin doesn't make a difference either.

BTW my computer uses an 8-core AMD bulldozer CPU, but I just tried it on my other computer (a netbook with an Intel Atom CPU running Ubuntu x86) and it DEXed for about 15 minutes before I gave up.

It's OK if you run out of ideas. It may not be an issue in the plugin itself, who knows.

[alexander-yakushev]

Now that's actually something. I fixed the debug build issue and pushed it as 0.1.0-beta3. Now when I'm using lein droid doall, dex fails with OutOfMemory exception (most likely due to --no-optimize flag). When I remove that flag, build finishes successfully in ~2 minutes. The release build still gets done in ~1 minute.

Could you execute lein clean; DEBUG=1 lein droid release command with the newest lein-droid? I'd like you to notice which namespaces are being compiled and what arguments are passed to dexer.

[oakes]

Here's the output with beta3:

/home/oakes/android-sdk-linux/platform-tools/dx --dex --output /home/oakes/lein-droid-test/target/classes.dex /home/oakes/lein-droid-test/target/classes /home/oakes/android-sdk-linux/tools/support/annotations.jar /home/oakes/.m2/repository/android/clojure/1.4.0/clojure-1.4.0.jar /home/oakes/.m2/repository/neko/neko/2.0.0-beta1/neko-2.0.0-beta1.jar

[oakes]

Actually, I'll just paste the whole output:

Compiling Clojure files...
Project classpath: (/home/oakes/android-sdk-linux/tools/support/annotations.jar /home/oakes/android-sdk-linux/platforms/android-14/android.jar /home/oakes/lein-droid-test/test /home/oakes/lein-droid-test/src/clojure /home/oakes/lein-droid-test/dev-resources /home/oakes/lein-droid-test/resources /home/oakes/lein-droid-test/target/classes /home/oakes/.m2/repository/android/clojure/1.4.0/clojure-1.4.0.jar /home/oakes/.m2/repository/neko/neko/2.0.0-beta1/neko-2.0.0-beta1.jar)
Build type: release, dynamic compilation: disabled, remote REPL: disabled.
Compiling 668 source files to /home/oakes/lein-droid-test/target/classes
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Compiling neko.init.options
Compiling clojure.core.protocols
Compiling clojure.string
Compiling clojure.core
Compiling clojure.java.io
Compilation succeeded.
Creating DEX....
/home/oakes/android-sdk-linux/platform-tools/dx --dex --output /home/oakes/lein-droid-test/target/classes.dex /home/oakes/lein-droid-test/target/classes /home/oakes/android-sdk-linux/tools/support/annotations.jar /home/oakes/.m2/repository/android/clojure/1.4.0/clojure-1.4.0.jar /home/oakes/.m2/repository/neko/neko/2.0.0-beta1/neko-2.0.0-beta1.jar

I left it running for about 5 minutes but I had to end it so I could go to work.

[oakes]

Well, Alex, I figured out the problem: I never checked my JDK version. On every platform, I had JDK 7 in my path. Replacing it with JDK 6 appears to fix the problem. Ubuntu x86 appears to be faster than x64. Sorry for taking up so much time!

(BTW, I also get the OutOfMemoryError with beta3 now.)

[alexander-yakushev]

No problem Zach! It is useful to know for future that JDK 7 can cause such problems.
As for OutOfMemory thing, a patch is coming that allows to optionally skip --no-optimize flag for large projects. I'll try to release it today.

[alexander-yakushev]

So I added two more options to project.clj file. You can put :java-only true to the top level to exclude Clojure compilation. Also :force-dex-optimize true in the :android map will skip --no-optimize flag in dexer arguments. These options are available in 0.1.0-beta4, the example usage is shown in sample/project.clj. Hope this helps.