lein fails on Windows 10 with 'CreateProcess error=206' when classpath too long

[timgilbert]

I'm trying to run a fairly complex project with a lot of dependencies from leiningen 2.8.1 and JDK 1.8.0_172 on Windows 10, and I believe I'm hitting this Windows error. lein run just tells me The system cannot execute the specified program, but when I lein repl I get this:

Exception in thread "Thread-2" java.io.IOException: Cannot run program "java" (in directory "C:\workframe\ocp"): CreateProcess error=206, The filename or extension is too long
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at java.lang.Runtime.exec(Runtime.java:620)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
        at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
        at leiningen.core.eval$sh.invokeStatic(eval.clj:179)
        at leiningen.core.eval$sh.doInvoke(eval.clj:173)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:646)
        at clojure.core$apply.invoke(core.clj:641)
        at leiningen.core.eval$fn__4134.invokeStatic(eval.clj:262)
        at leiningen.core.eval$fn__4134.invoke(eval.clj:260)
        at clojure.lang.MultiFn.invoke(MultiFn.java:233)
        at leiningen.core.eval$eval_in_project.invokeStatic(eval.clj:366)
        at leiningen.core.eval$eval_in_project.invoke(eval.clj:356)
        at leiningen.repl$server$fn__5864.invoke(repl.clj:244)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:646)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1881)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1881)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:650)
        at clojure.core$bound_fn_STAR_$fn__4671.doInvoke(core.clj:1911)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: CreateProcess error=206, The filename or extension is too long
        at java.lang.ProcessImpl.create(Native Method)
        at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)
        at java.lang.ProcessImpl.start(ProcessImpl.java:137)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 31 more

I think the root of the problem is that the number of jars on my classpath causes the java command-line to exceed its length of 32K characters:

> lein classpath | measure -Character

Lines Words Characters Property
----- ----- ---------- --------
                 41538

I am able to work around this problem for now by compiling an uberjar and executing that - the uberjar essentially acts as a "pathing jar" as described in the above StackOverflow answer. Still, it would be nice if lein itself had some way to work around this problem so I wasn't needing to compile code all the time.

[technomancy]

As far as I understand this is a limitation of Java and the OS; I don't think there's anything Leiningen can do about hard-coded OS limitations?

[timgilbert]

Definitely lein can't do anything about the command-line length limit. It might be possible to work around this though; for instance gradle has a pathingJar option which creates a jar file whose manifest contains the paths to all of the other jars. It's built into gradle now but basically works like this:

https://tuhrig.de/gradles-bootrun-and-windows-command-length-limit/

I suppose this might be better handled by a plugin or something, though it seems a little odd to require a plugin just to run a complex project on Windows.

EDIT: some more discussion from the gradle project here: gradle/gradle#1989

[aardvark]

So what should be solution for this case? Should long classpath handling be delegated to plugin or to leiningen-core.
As of now this is only impacting Windows platform but only because arg limit in Windows CLI is lower than Linux (or MacOS).

This is general Java problem and mostly it is development stage problem. Final artifacts be it jar or war almost always have classpath inside and not passed from command line.
In 'pure' Java this is mostly fixed by tools. Both Eclipse and IntelliJ use classpath file approach.
How this should be handled by Leiningen?

[timgilbert]

FWIW, I've developed a fix for this in tools.deps.alpha here:

https://dev.clojure.org/jira/browse/TDEPS-120

The actual code needed to create a pathing jar is not terribly complex, and this shouldn't be too hard to implement in leiningen. I will give it a look when I've got more time.

[timgilbert]

Ok, well it's early days yet but I wrote this plugin which seems to fix the problem by putting a hook around (get-classpath) and creating a pathing jar based on the classpath's MD5 hash:

https://github.com/timgilbert/lein-classpath-jar

This is a sufficient workaround for me, at least, so I'm going to mark this issue as closed for now.

[alexandroid]

Has anyone tried "Enable Win32 long paths" setting in policy editor in Windows 10? (described here)