Mitigate long classpath on Windows for JavaExec

[lacasseio]

Expected Behavior

JavaExec (task and inline method) can execute without any special handling on Windows as it does on Linux and macOS.

Current Behavior

Windows has a limit in term of long path and command-line arguments size. Given a really long classpath, a JavaExec task would typically succeed on macOS and fail on Windows. This causes a lot of pain to users and requires to use house-made solution or rely on 3rd party plugins.

Context

[lacasseio]

Possible Solutions

Solution 1: Modify JavaExecHandleBuilder to always pass all argument as arg file (JDK 9+)

Pro

• Seemlessly work for Windows out of the box

Con

• Sensitive information passed as command-line arguments are now in a file on-disk
• The process command-line will only contain the arg file
• Induce a performance hit for Linux and macOS where long command-line isn't an issue

Note

• Care must be taken to make sure no arg file is already present on the command line

Solution 2: Modify JavaExecHandleBuilder to always pass only the classpath as arg file (JDK 9+)

Pro

• Address the most common scenario for Windows
• All arguments of interest are still on the process command-line

Con

• Induce a performance hit for Linux and macOS where long command-line isn't an issue

Note

• Care must be taken to make sure no arg file is already present on the command line

Solution 3: Modify JavaExecHandleBuilder as noted in the previous solutions but only for Windows (JDK 9+)

Pro

• Narrow the fix to only Windows

Con

• Given Windows will behave differently than the other system, other issues may arise from such as adding an arg file on Linux would work but not on Windows

Solution 4: Modify JavaExecHandleBuilder as noted in the previous solution but enable through a flag (JDK 9+)

Pro

• User will control the behaviour as required

Con

• Requires user interaction for proper configuration on Windows

Solution 5: Provide methods to compose the intended behaviour

Provide a method like argFile(Object... args) or pathingJar(FileCollection) that can be passed as arguments or as classpath respectively to generate the arg file and pathing Jar as well as providing the right argument or classpath to enable the feature.

Pro

• User will control the behaviour as required
• The solution fits nicely to address the issue with argument files as well as pathing Jar

Con

• Requires user interaction for proper configuration on Windows

Solution 6: Use Windows long path notation (e.g. \?)

Need to confirm if this works with Java

Pro

• Solve the issue without the negative effect of the arg file and pathing Jar

Con

• Only newer Windows version can benefit from this technique
• Changing the path notation may have an unexpected effect on the runtime.

Open Questions

Do we care about solving the issue for JDK 8?

Most of the solution rely on JDK 9 feature. We could expand by using the pathing Jar trick but may have a negative effect in certain use cases. Allowing users to decide how to solve the issue composing the behaviour as noted in solution 5 would help.

Is modifying the JavaExecHandleBuilder too wide?

JavaExecHandleBuilder is used in numerous places in Gradle and may impact more than what we are seeing at the moment. At the same time, solving the issue just for JavaExec task would be a poor decision as Project#javaexec is equally important. At the same time, we already handle arg files for worker API. Another one-off solution feels like a bad approach.

[lacasseio]

Possible DSLs could look like this:

Use arg file or pathing Jar for classpath

As a flag:

project.javaexec {
    classpath = configurations.runtime
    useArgFileClasspath() // or usePathingJarClasspath()
}

tasks.withType(JavaExec).configureEach {
    useArgFileClasspath() // or usePathingJarClasspath()

or, as a composable method

project.javaexec {
    classpath = asArgFile(configurations.runtime) // or asPathingJar(configurations.runtime)
}

tasks.withType(JavaExec).configureEach {
    classpath = asArgFile(classpath) // or asPathingJar(classpath)
}

Modeling arg file

project.javaexec {
    argumentProviders.add(argFile('some', 'argument', 'that', 'will', 'be', 'in', 'argument', 'file'))
}

tasks.withType(JavaExec).configureEach {
    argumentProvider.add(argFile('some', 'argument', 'that', 'will', 'be', 'in', 'argument', 'file'))
}

[donat]

@lacasseio FYI Eclipse JDT has a few tricks to shorten the classpath, some of which work with older Java versions: https://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/tree/org.eclipse.jdt.launching/launching/org/eclipse/jdt/internal/launching/ClasspathShortener.java

[donat]

I think the best solution is to change the behavior only on Windows and change the classpath only if it's too long (to prevent performance issues when the classpath is not that long).

Also, why do we need to change the DSL? Can't we just convert the long classpaths (to a classpath-only jar for instance) behind the scenes when necessary?

[lacasseio]

It may have some impact and I guess some users may not want the "new default" we would be pushing. It would be good if there was a way to disable it if they prefer another way to shorten the classpath.

[big-guy]

+1 for doing this transparently. I think this should work for any version of Java we would support via JavaExec/javaexec, but this can be delivered in increments (just argfile, then manifest jar).

I would open this up to not just too long of a classpath, but too long of a command-line, since it would manifest itself as the same sort of problem and we need to know if the full command-line is too long and not just the classpath. It looks like even Linux/macOS have limits: https://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/tree/org.eclipse.jdt.launching/launching/org/eclipse/jdt/internal/launching/ClasspathShortener.java#n267

So some increments could be...

1. If the command-line is too long, we need to fail with a useful error. Add coverage for several versions of Java with JavaExec and javaexec.
2. If the command-line is too long and we're only Java 9+, generate an argfile. We can still fail on <Java 9
   • If there is already an argfile in the arguments, we need to honor it/preserve the order.
   • We need to make sure we follow the rules for generating an argfile properly: https://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-4856361B-8BFD-4964-AE84-121F5F6CF111
   • If we're worried about leaving behind the contents of the command-line, we could delete the argfile after the command finishes.
3. If the command-line is too long and we're only <Java 9, we should attempt to shorten the classpath with a manifest jar.
   • If shortening the classpath still doesn't put the command-line under the limit, we should fail with a useful message.
   • I think generating the manifest jar in the same location as the argfile is OK.

I think this limits the changes to only builds where the java command would have failed already. I think this should be done in a cross-cutting way so all forked Java executions take advantage of this change, but it's out of scope to try to fix this in the Scala/Groovy/Kotlin compilers (if they're forking java).

Here's another source to pull inspiration from: https://github.com/microsoft/vscode-java-debug/pull/532/files

[lacasseio]

Then here is what I propose as the next steps:

• Detect when the JavaExec and javaexec would fail on each major operating system and provide a useful error message (from Gradle instead of coming from the OS).
• For JDK 9+, when no argfile are already in use and the limit is attained, push only the classpath to an argfile if it would bring the command within the OS limit, else fail. We will leave the other argument on the command line at this stage given those may be more important for identifying the command.
• For JDK 8-, use a pathing Jar if it would bring the command within the OS limit, else fail. This should provide the same behaviour as the previous step.
• For JDK 9+ with an argfile already in use, use a pathing JAR only if it would bring the command within the OS limit, else fail.
• (Optional, for optimization purpose) If the classpath is within environment variable OS limit and the CLASSPATH environment variable is not in use, pass the classpath via the environment variable.
• (Optional, for exec with lots of args) If removing the classpath still doesn't bring the command line within the OS limit, we are using JDK 9+ and no argfile are used, push classpath and arguments to an argfile. This would be quite disruptive in term of observation of the process command line and should be used as a last resort.

[lacasseio]

@big-guy On OS where we don't know the limit, should we assume the most restrictive known limit?

[lacasseio]

Actually, I will scope the usage of the environment variable in and use that first before the pathing Jar, the side effect of the pathing Jar is quite troublesome for the users.

[mhemani]

Recently, I started developing a new Grails web application using the newly released Grails 4.0.0, which depends on Gradle 5.1.1.

When I added the Elasticsearch 2.4.2 dependency, I ran into the issue:
Caused by: java.io.IOException: Cannot run program "C:\Program Files\Java\jdk1.8.0_74\bin\java.exe". CreateProcess error=206, File name too long.

I have searched and tried several workarounds offered by many, but all has failed so far, with Grails 4, Grade 5 and JDK 8 on Win10 (IntelliJ).

Is there a possible temporary workaround for the stack / environments that I am using, while we wait for this issue to get resolved?

[lacasseio]

A better error message detection will be provided for Windows in Gradle 6.0. We are rolling out command line shortening via CLASSPATH environment variable on Windows. The fixes will be localized only for Windows, please comment if you are seeing the same issue on Linux and macOS. Both of those OS have a really high limit so we see no need to address them in the near term.

[lacasseio]

We ended up reverting the CLASSPATH shortening as it has shown little improvement over the situation. We went ahead with generating a pathing Jar internally at execution.

[lacasseio]

We decided to not generate any JDK 9 argfile at this stage as the pathing JAR is good enough.

[arihunta]

Just want to ask, regarding this solution: is there a way to force the creation of the pathing jar? I find that I run into this problem when the %APP_HOME% environment variable in the Windows installDist start scripts expands to something long at run-time. So the class-path is short enough, but if someone copies the app to a folder with a long name, then tons of %APP_HOME%/lib/something.jars blow it up.

[lacasseio]

That is a very good point, we addressed the issue strictly for JavaExec and project.javaexec. We didn't change anything regarding the distribution plugin and the start script. Could you open an issue for that?

[RationalRank]

Tried Gradle V 6.0 and still facing this issue for the kotlin compiler in windows.
The absolute paths of the transformed JAR files are sent in the classpath.

2019-10-21T17:35:08.036+0530 [DEBUG] [org.gradle.api.Task] Using ':app:kaptGenerateStubsDebugKotlin' logger
2019-10-21T17:35:08.036+0530 [DEBUG] [org.gradle.api.Task] [KOTLIN] Kotlin compiler class: org.jetbrains.kotlin.cli.jvm.K2JVMCompiler
2019-10-21T17:35:08.037+0530 [DEBUG] [org.gradle.api.Task] [KOTLIN] Kotlin compiler classpath: C:\Users\Ranjith\.gradle\caches\modules-2\files-2.1\org.jetbrains.kotlin\kotlin-compiler-embeddable\1.3.41\6b1d4385d65894e07a0d14a5949f5417a408f0b7\kotlin-compiler-embeddable-1.3.41.jar, C:\Users\Ranjith\.gradle\caches\modules-2\files-2.1\org.jetbrains.kotlin\kotlin-reflect\1.3.41\8fb58b8954661de666e321478bf4178c18ce8018\kotlin-reflect-1.3.41.jar, C:\Users\Ranjith\.gradle\caches\modules-2\files-2.1\org.jetbrains.kotlin\kotlin-stdlib\1.3.41\e24bd38de28a326cce8b1f0d61e809e9a92dad6a\kotlin-stdlib-1.3.41.jar, C:\Users\Ranjith\.gradle\caches\modules-2\files-2.1\org.jetbrains.kotlin\kotlin-script-runtime\1.3.41\bcc3380041bbba171119c22d7024961b60da69e0\kotlin-script-runtime-1.3.41.jar, C:\Users\Ranjith\.gradle\caches\modules-2\files-2.1\org.jetbrains.intellij.deps\trove4j\1.0.20181211\216c2e14b070f334479d800987affe4054cd563f\trove4j-1.0.20181211.jar, C:\Users\Ranjith\.gradle\caches\modules-2\files-2.1\org.jetbrains.kotlin\kotlin-stdlib-common\1.3.41\2ecf4aa059427d7186312fd1736afedf7972e7f7\kotlin-stdlib-common-1.3.41.jar, C:\Users\Ranjith\.gradle\caches\modules-2\files-2.1\org.jetbrains\annotations\13.0\919f0dfe192fb4e063e7dacadee7f8bb9a2672a9\annotations-13.0.jar, C:\Program Files\Java\jdk1.8.0_201\lib\tools.jar
2019-10-21T17:35:08.038+0530 [DEBUG] [org.gradle.api.Task] [KOTLIN] Kotlin compiler args: -Xallow-no-source-files -classpath E:\official\work\product_root\product\scanlibrary\build\intermediates\compile_library_classes\debug\classes.jar;C:\Users\Ranjith\.gradle\caches\transforms-2\files-2.1\0f75e58d3f16ee100bc9ce124a00a7b3\jetified-signature-pad-1.2.1\jars\classes.jar;.....

[lacasseio]

This case seems to be related to the Kotlin compiler, I suggest you raise the issue with JetBrains to they use ExecOperations service to work around the issue, if the compiler is a Java process. In the event where it wouldn't be a Java process, they would have to handle this case by themselves.

[sschuberth]

Just want to ask, regarding this solution: is there a way to force the creation of the pathing jar? I find that I run into this problem when the %APP_HOME% environment variable in the Windows installDist start scripts expands to something long at run-time. So the class-path is short enough, but if someone copies the app to a folder with a long name, then tons of %APP_HOME%/lib/something.jars blow it up.

@arihunta, see #1989 (comment) for a work-around that does not require a pathing JAR.

[sschuberth]

That is a very good point, we addressed the issue strictly for JavaExec and project.javaexec. We didn't change anything regarding the distribution plugin and the start script. Could you open an issue for that?

@lacasseio isn't that already tracked as part of #1989?

[beikov]

My workaround is to create an argument file:

            def argFile = getTemporaryDir().createTempFile("gradle-test-arg-file", "")
            argFile.withWriter{ out ->
                testArgs.each {out.println it}
            }

            jvmArgs = ["@${argFile.absolutePath}"]
            classpath = files()