Support Java Projects with Dependencies

[JoelProminic]

In order to properly support real Java projects, we need to make sure that the language server can handle dependencies. Ideally, the dependencies should be configured by the Ant, Maven, or Gradle scripts, but we can use manual configuration if that is not feasible. The dependencies should include:

1. Dependencies on local projects (Ant and Gradle). Note that Gradle depends on settings.gradlel in the parent directory of a project, which may be a problem for bookmarks in the App Store build
2. Dependencies on local JAR files (Ant and Gradle)
3. Dependencies on remote Maven/Gradle plugins (Maven and Gradle)
4. Dependencies on local Maven/Gradle plugins (Maven and Gradle). This requires access to ~/.m2, which may be a problem for the App Store build.

Another option for populating the dependencies is to import the dependencies from the Eclipse or IntelliJ configuration files. If this is complicated, we can save it for a later release.

[JoelProminic]

I talked to @joshtynjala about how the language server works currently.

The language server expects dependencies to be configured in either pom.xml or build.gradle. It should install the dependencies it needs for code intelligence automatically. This behavior comes from the upstream jdt-language-server project. With that in mind, if we run into sandboxing issues with dependency resolution on macOS, I probably won't be able to change anything on my side.

So, hopefully the language server will automatically detect the dependencies for Maven or Gradle projects. I will plan to get some familiarity with jdt-language-server when I have a chance.

On whether build.gradle or pom.xml has precedence:

I'd need to dig into the source code of the upstream jdt-language-server to find out whether the Maven or Gradle build file takes precedence. I'll try to figure that out when I have a moment.

For Eclipse project files:

I should mention that the Java jdt-language-server is build on top of Eclipse's internals. After getting the project settings from pom.xml or build.gradle, the language server creates these Eclipse project files (.project and .classpath) automatically. I don't know if this information will affect us in any way, but I thought I'd point it out just so that everyone is aware and not surprised.

I don't think this should be a problem, but I'll keep it in mind.

[JoelProminic]

I created some example Java applications to test dependencies:
2019_05_07__test_java_dependencies.zip

Before you open the projects, run the following commands to make sure all of the local repositories are set up. This also shows how to run the applications. If you run the JAR by itself, it gives errors for the dependencies - we will need to revisit this for Build & Run.

Maven Example:

cd maven
cd DependenciesTest2
mvn install
cd ../DependenciesTest1
mvn install:install-file -Dfile=lib/commons-codec-1.12.jar -DgroupId=test -DartifactId=commons-codec -Dversion=1.0 -Dpackaging=jar
mvn clean package exec:java

Grails Example:

cd gradle
cd GradleTest3
gradle clean publishToMavenLocal
cd ../GradleTest1
gradle run

Open all projects in the "maven" directory for Maven, and in the "gradle" directory for Gradle.

I found that the Maven dependencies worked properly with the non-sandbox build - I did not see any errors in DependenciesTests1.java. With the App Store build, I got errors for the local repository as expected.

However, none of the dependencies worked for Gradle. I saw errors for every external dependency in GradleTest1.java.

[JoelProminic]

From this page, it looks like build.gradle is checked before pom.xml.

---

Build types

The server looks for project/build descriptors in order to correctly configure the Java support (compiler level, classpath). The project import mechanism will lookup build descriptors in the following order:

1. Search for *.gradle files: if a Gradle project is detected, the import will be delegated to Eclipse Buildship.
2. Search for pom.xml files: if a Maven project is detected, the import will be delegated to Eclipse m2e.
3. Search for Eclipse IDE .project files: the projects will be imported as simple Eclipse IDE projects.

Depending on the project type, some specific behavior can be expected:

• Gradle
  -- sources of dependencies are automatically downloaded.
  -- changes to build.gradle dependencies require an explicit project configuration update.
  -- please note that Android projects are not supported at the moment.
• Maven
  -- sources of dependencies are automatically downloaded.
  -- Maven errors are reported on the pom.xml.
  -- Changes to pom.xml dependencies automatically updates the projects classpath.
• Eclipse IDE
  -- please note that WTP based projects can not be configured properly, as the JLS is missing all the WTP plugins

---

The Gradle import is delegade to buildship. I didn't have time to review this thoroughly, but it looks promising.

[JoelProminic]

I think I figured this out. It seems that for Gradle projects, we need to manually tell Gradle to update the .classpath file. To do this, you need to add the 'eclipse' plugin:

plugins {
    id 'java'
    id 'application'  // allows 'run' task.
    id 'eclipse'      // allows dependencies to be exported to .classpath
}

Then you can update the .classpath with:

gradle eclipse
# If we run into trouble getting this to update completely, we can do a clean first
gradle cleanEclipse eclipse

Once the classpath is updated, the language server needed to be restarted in order to see the changes. Currently this requires me to close and reopen the project, or to restart Moonshine.

I see two solutions for this:

1. Add a Project > Refresh Gradle Classpath menu action. This will run the above commands and restart the language server.
2. Run this command each time you start the language server for a Gradle project, and then add an action to restart the language server

I think 1 is preferable unless we think we will need 2 in general.

If Moonshine gets an error on "gradle eclipse", it should report "Unable to regenerate classpath for Gradle project. Please check that you have included the 'eclipse' plugin, and verify that your dependencies are correct.

The 'eclipse' plugin should also be added to the Java Gradle template.

Here are my updated projects:
2019_05_09__test_java_dependencies.zip

[rat-moonshine]

From the last discussion with Joel on this,

1. Add Gradle Path configuration to Settings menu
2. Run "gradle eclipse" before restarting language server, for Java Gradle only
3. Add "Build with Gradle" to Project menu for Java Gradle projects. This requires adding a setting to define the default Gradle task, similar to Maven
4. Add Project > Refresh Gradle Classpath
5. Add Gradle in Moonshine SDK Installer

Q. I see you have suggested a menu option “Project > Refresh Gradle Classpath” - this might be applicable in case when Java mapped after opening the project to sidebar? Or, since the language-server autostart to all the opened projects, we need to run the command against Gradle project before starting the server against the project.

A. If we include the "gradle eclipse" logic as part of starting the language server, this action would basically just be restarting the language server. I just want Moonshine to call "gradle eclipse" before starting the language server, for Java Gradle projects only.

[rat-moonshine]

• Updated build.gradle.template to build for eclipse plugins
• Added Gradle plugin to Moonshine settings (typical path setup C:\Program Files (x86)\gradle-4.9)
• Added check and update Gradle classpath prior to start language-server to Gradle project (output can be seen to the console)
• Once classpath update completes language-server will start on its usual course

The above needs to be tested with new Gradle project template as template file updated.

on macOS until we download Gradle through Moonshine SDK Installer, having downloaded Gradle in a browser may require to remove its extended attributes manually prior test.

[piotrzarzycki21]

I see in your commit that you have used NativeProcess - please spend some time and describe what I was asking in details in this issue #518.

[rat-moonshine]

I see in your commit that you have used NativeProcess - please spend some time and describe what I was asking in details in this issue #518.

Yes, I just used the existing NativeProcess API that already exists to the class. Yeh, I'll need to spend some time on #518 .

[rat-moonshine]

Initial Gradle download link added to Moonshine SDK Installer (Moonshine-IDE/Moonshine-SDK-Installer#9) - macOS download link yet to be updated.

[JoelProminic]

@rat-moonshine and I got this error when testing the App Store build:

1. Make sure you don't have any 'java' instances running in Activity Monitor
2. Open Moonshine
3. Open a Gradle project
4. Moonshine completes 'gradle eclipse' successfully
5. You can close and reopen the project - there no problem shown
6. Close Moonshine
7. You should see 'java' instances still running in Activity Monitor which has started upon language-server start
8. Reopen Moonshine and try opening the Gradle project
9. This time console thrown error running 'gradle eclipse'
10. In Activity Monitor close all the 'java' instances
11. Reopen Gradle project in Moonshine
12. 'gradle eclipse' completed successfully

The error looks like this:

: FAILURE: Build failed with an exception.
: * What went wrong:
: Could not create service of type ScriptPluginFactory using BuildScopeServices.createScriptPluginFactory().
: > Could not create service of type FileHasher using BuildSessionScopeServices.createFileSnapshotter().
: * Try:
: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
: * Get more help at https://help.gradle.org
: BUILD FAILED in 0s

From my preliminary investigation, I think this is related to how we handle the Gradle daemons. I will create a separate issue for this.

[JoelProminic]

I couldn't reproduce the results from the above procedure with the non-sandbox build on macOS, but I did notice a separate issue.

Currently, Gradle Home is not populated automatically, but "gradle eclipse" runs anyway. This triggers this daemon:

$ ps -ef | grep GradleDaemon
  504  3258  3252   0  2:58PM ??         0:18.01 /Users/<user>/Downloads/MoonshineSDKs/Java/openjdk-1.8.0/bin/java -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError -Xmx512m -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -cp /Users/<user>/.gradle/wrapper/dists/gradle-5.0-bin/pu5208521seraqlersebvqk/gradle-5.0/lib/gradle-launcher-5.0.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 5.0

However the Gradle version is wrong in this case - 5.0 instead of 5.4.1. It looks to me like it is using an old instance of Gradle created with the Gradle wrapper, but I don't understand why that shows up here. This behavior may be specific to my workstation.

If I set Gradle Home manually then it loads another daemon with the correct version:

$ ps -ef | grep GradleDaemon
  504  3258     1   0  2:58PM ??         0:18.33 /Users/<user>/Downloads/MoonshineSDKs/Java/openjdk-1.8.0/bin/java -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError -Xmx512m -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -cp /Users/<user>/.gradle/wrapper/dists/gradle-5.0-bin/pu5208521seraqlersebvqk/gradle-5.0/lib/gradle-launcher-5.0.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 5.0
  504  3684     1   0  3:01PM ??         0:14.21 /Users/<user>/Downloads/MoonshineSDKs/Java/openjdk-1.8.0/bin/java -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError -Xms256m -Xmx512m -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -cp /Users/<user>/Downloads/MoonshineSDKs/Gradle/gradle-5.4.1-bin/lib/gradle-launcher-5.4.1.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 5.4.1

[rat-moonshine]

@piotrzarzycki21 , it looks like JavaImporter does the configuration parsing process for Java project to Moonshine.

I also see MavenBuildOptions references in AS3ProjectVO and in FlashDeveloperImporter. Do we need them here anyway for some other purpose (?)

[piotrzarzycki21]

I also see MavenBuildOptions references in AS3ProjectVO and in FlashDeveloperImporter. Do we need them here anyway for some other purpose (?)

Yes we need, cause ActionScript project like Flex/Royale can be build by Maven. All examples in Royale repository have pom.xml and are working with Maven.

[rat-moonshine]

Added new Gradle "Build Actions" to gradle project settings.

Gradle templates adjusted to work with 'gradle eclipse'. I start receiving 'gradle clean build' error after running 'gradle eclipse'. Thanks to @piotrzarzycki21 he suggested how to add mainClassName property to the build.gradle file, which is now part of the regular template. I was having this error when the mainClassName not declared to the build.gradle file:

C:\NewJavaGradleProject>gradle clean build
> Task :startScripts FAILED
FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':startScripts'.
> No value has been specified for property 'mainClassName'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org

BUILD FAILED in 2s

Following menu items are now also working to Gradle project:

• Build
• Build & Run
• Clean
• Refresh Gradle Classpath

Tested gradle build with 2019_05_09__test_java_dependencies.zip example files. Following things I required to add in their gradle.build file prior build tests:

plugins {
    id 'java'
    id 'application'  // added
    id 'eclipse'      // added
}
mainClassName = '$projectFileName' // added

When GradleTest2 and GradleTest3 worked expectedly for me, GradleTest1 terminate by following error (I think this is different bug):

: C:\Users\devsena\Desktop\2019_05_07__test_java_dependencies\test_java_dependencies\gradle\GradleTest1\src\main\java\GradleTest1.java:24: error: illegal character: '\u2039'
: 			String encodedText = StringEscapeUtils.escapeXml11​(originalText);

GRADLE_HOME now adds to the local environment variable population list.

[JoelProminic]

From some of my other testing, I noticed that the App Store version of Moonshine is using a different path for the Maven repository:

: Execution failed for task ':GradleTest1:compileJava'.
: > Could not resolve all files for configuration ':GradleTest1:compileClasspath'.
:    > Could not find test:gradle-test-3:1.0.
:      Searched in the following locations:
:        - file:/Users/<user>/Library/Containers/com.moonshine-ide/Data/.m2/repository/test/gradle-test-3/1.0/gradle-test-3-1.0.pom
:        - file:/Users/<user>/Library/Containers/com.moonshine-ide/Data/.m2/repository/test/gradle-test-3/1.0/gradle-test-3-1.0.jar

So, it uses ~/Library/Containers/com.moonshine-ide/Data/.m2/repository instead of ~/.m2/repository. I confirmed that if I run all of the Maven and Gradle commands from my test_java_dependencies demo above in Moonshine, then the language server and build work properly. This will allow the App Store Moonshine to be usable with dependencies, as long as the user remembers to run all Maven commands from Moonshine.

However, Build & Run fails for Maven because the JAR does not have all of its dependencies. I can instead run DependenciesTest1 with the "clean package exec:java" Build Actions (this requires additional configuration in pom.xml).

: Exception in thread "main" java.lang.NoClassDefFoundError: test/HelloWorld2
: 	at DependenciesTest1.main(DependenciesTest1.java:14)
: Caused by: java.lang.ClassNotFoundException: test.HelloWorld2
: 	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
: 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
: 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
: 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
: 	... 1 more

[JoelProminic]

I did some tests to verify the behavior for when the Gradle project did not have the "eclipse" plugin. This works fine for a single project, but I noticed that it seemed to run for all Gradle projects, which gave some confusing results:

1. Comment out "id 'eclipse'" in build.gradle for a Java Gradle project
2. Close out all other projects
3. Restart Moonshine
4. Test Project > Refresh Gradle Classpath - this triggers an error only
5. Open another Gradle project
6. Select the first project and try Project > Refresh Gradle Classpath. BUG: moonshine shows both a error and a valid result
7. Close the other Gradle project
8. Select the first project and try Project > Refresh Gradle Classpath. BUG: moonshine shows both a error and a valid result

[rat-moonshine]

I did some tests to verify the behavior for when the Gradle project did not have the "eclipse" plugin. This works fine for a single project, but I noticed that it seemed to run for all Gradle projects, which gave some confusing results

This should have fixed now. Please, check.

[rat-moonshine]

Do we have anything left to close this issue, @JoelProminic (?)

[JoelProminic]

Yes, this looks fine. Here is an example of how the message looks when the Eclipse plugin is missing:

: 3 actionable tasks: 2 executed, 1 up-to-date
NewJavaGradleProject: Unable to regenerate classpath for Gradle project. Please check that you have included the 'eclipse' plugin, and verify that your dependencies are correct. 
shellError while updating Gradle classpath to get full insights. 
 
* Get more help at https://help.gradle.org 
 
BUILD FAILED in 0s 

[JoelProminic]

Here is another way to handle the local JAR dependencies for Gradle. I confirmed that this also worked in Moonshine.

repositories {
    ...
    flatDir {   // this is an alternative way to handle the local JAR files
        dirs 'lib'
    }
}
...
dependencies {
	...
	// Local JAR dependency
	//compile files('lib/commons-codec-1.12.jar')
	compile name: "commons-codec-1.12"
	...
}