New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native Image From Compiled Groovy Class #346

Closed
cfsilence opened this Issue Apr 18, 2018 · 19 comments

Comments

5 participants
@cfsilence

cfsilence commented Apr 18, 2018

I'm trying to slightly modify the HelloWorld.java native image example to work with Groovy instead:

HiWorld.groovy:

class HiWorld {
  static void main(String[] args) {
        println 'hi!'
  }
}
groovyc HiWorld.groovy

Which results in a HiWorld.class file. When trying to run this class file with: java HiWorld it results in the following error:

Error: Could not find or load main class HiWorld

Which makes sense - Groovy isn't on the class path (see here). So, running it with: java -cp ".:/$GROOVY_HOME/lib/*" HiWorld makes things work. However, the native-image -cp flag seems to work differently?

native-image -cp ".:/$GROOVY_HOME/lib/*" HiWorld
Error: Invalid Path entry /home/toddsharp/.sdkman/candidates/groovy/current/lib/*

Can we not use wildcards in the -cp argument to native-image?

@chumer

This comment has been minimized.

Member

chumer commented Apr 18, 2018

Can we not use wildcards in the -cp argument to native-image?

I don't think so. We try to emulate the java command with native-image but we are not using the same code.

@chumer chumer added the feature label Apr 18, 2018

@cfsilence

This comment has been minimized.

cfsilence commented Apr 18, 2018

As a followup, I tried:

$ native-image -cp ".:$(echo /$GROOVY_HOME/lib/*.jar | tr ' ' ':')" HiWorld

Which failed with:

Build on Server(pid: 10370, port: 26681)
   classlist:   1,559.13 ms
       setup:      84.69 ms
error: could not find non-optional target method: public static jline.Terminal com.oracle.svm.jline.subst.Target_jline_TerminalFactory.create(java.lang.String)
Error: Processing image build request failed

Even though this works:

$ java -cp ".:$(echo /$GROOVY_HOME/lib/*.jar | tr ' ' ':')" HiWorld
hi!
@cstancu

This comment has been minimized.

Member

cstancu commented Apr 18, 2018

Does your JAVA_HOME point to Java 9 or later? Can you try with Java 8? We don't currently support Java versions greater than 8 for SubstrateVM. The problem here seems to be that the signature for jline.TerminalFactory.create() has changed starting with Java 9 and our substitution mechanism doesn't know that. We are working on Java 9 and later support.

@cstancu cstancu added the substrate label Apr 18, 2018

@dougxc

This comment has been minimized.

Member

dougxc commented Apr 18, 2018

I think the native-image command should detect the Java version is it running on and fail fast if it's not a supported version. (/cc @olpaw )

@cfsilence

This comment has been minimized.

cfsilence commented Apr 18, 2018

@cstancu

$ java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-12)
GraalVM 1.0.0-rc1 (build 25.71-b01-internal-jvmci-0.42, mixed mode)
@cstancu

This comment has been minimized.

Member

cstancu commented Apr 18, 2018

Then it is probably a classpath issue. Can you post the output when you run native-image with the --verbose flag.

@cfsilence

This comment has been minimized.

cfsilence commented Apr 18, 2018

$ native-image --verbose -cp ".:$(echo /$GROOVY_HOME/lib/*.jar | tr ' ' ':')" HiWorld
Build on Server(pid: 10370, port: 26681)
SendBuildRequest [
-task=com.oracle.svm.hosted.NativeImageGeneratorRunner
-imagecp
/home/toddsharp/opt/graalvm/jre/lib/svm/builder/pointsto.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/builder/objectfile.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/builder/svm.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/jvmci-hotspot.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/graal.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/jvmci-api.jar:/home/toddsharp/opt/graalvm/jre/lib/boot/graal-sdk.jar:/home/toddsharp/opt/graalvm/jre/lib/boot/graaljs-scriptengine.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/library-support.jar:/home/toddsharp/Projects/scratch/graalvm:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ant-1.9.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ant-antlr-1.9.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ant-junit-1.9.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ant-launcher-1.9.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/bsf-2.4.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/commons-cli-1.2.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/commons-logging-1.2.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/gpars-1.2.1.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-ant-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-bsf-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-console-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-docgenerator-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-groovydoc-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-groovysh-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-jmx-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-json-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-jsr223-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-nio-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-servlet-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-sql-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-swing-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-templates-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-test-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-testng-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-xml-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/hamcrest-core-1.3.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ivy-2.4.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jansi-1.11.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jcommander-1.47.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jline-2.12.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jsp-api-2.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jsr166y-1.7.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/junit-4.12.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/multiverse-core-0.7.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/qdox-1.12.1.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/servlet-api-2.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/testng-6.8.13.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/xmlpull-1.1.3.1.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/xstream-1.4.7.jar
-H:Path=/home/toddsharp/Projects/scratch/graalvm
-H:CLibraryPath=/home/toddsharp/opt/graalvm/jre/lib/svm/clibraries/linux-amd64
-H:Class=HiWorld
-H:Name=hiworld
]
   classlist:   1,390.74 ms
       setup:      96.83 ms
error: could not find non-optional target method: public static jline.Terminal com.oracle.svm.jline.subst.Target_jline_TerminalFactory.create(java.lang.String)
Error: Processing image build request failed
@cstancu

This comment has been minimized.

Member

cstancu commented Apr 18, 2018

Thanks! That doesn't show the build server -cp which is the one that might cause the problem. You probably had the server already started from a previous build request. Can you add --no-server too? Or alternatively you can shutdown all the build servers first with --server-shutdown-all. (Btw, if you want to see options related to the build server you can run native-image --help-extra).

@cfsilence

This comment has been minimized.

cfsilence commented Apr 18, 2018

How is this?

$ native-image --verbose --no-server -cp ".:$(echo /$GROOVY_HOME/lib/*.jar | tr ' ' ':')" HiWorld
Executing [
/home/toddsharp/opt/graalvm/bin/java \
-Xbootclasspath/a:/home/toddsharp/opt/graalvm/jre/lib/boot/graal-sdk.jar:/home/toddsharp/opt/graalvm/jre/lib/boot/graaljs-scriptengine.jar \
-cp \
/home/toddsharp/opt/graalvm/jre/lib/svm/builder/pointsto.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/builder/objectfile.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/builder/svm.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/jvmci-hotspot.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/graal.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/jvmci-api.jar \
-server \
-d64 \
-noverify \
-XX:+UnlockExperimentalVMOptions \
-XX:+EnableJVMCI \
-XX:-UseJVMCIClassLoader \
-XX:+UseJVMCICompiler \
-Dgraal.CompileGraalWithC1Only=false \
-XX:CICompilerCount=4 \
-Dgraal.VerifyGraalGraphs=false \
-Dgraal.VerifyGraalGraphEdges=false \
-Dgraal.VerifyGraalPhasesSize=false \
-Dgraal.VerifyPhases=false \
-Dgraal.EagerSnippets=true \
-Xss10m \
-Xms1g \
-Xmx6690845488 \
-Duser.country=US \
-Duser.language=en \
-Dsubstratevm.version=68c7c1073a86a3d541ffb82434acc664f3096079:substratevm \
-Dgraalvm.version=1.0.0-rc1 \
-Dorg.graalvm.version=1.0.0-rc1 \
-Dcom.oracle.graalvm.isaot=true \
-Djvmci.class.path.append=/home/toddsharp/opt/graalvm/jre/lib/jvmci/graal.jar \
com.oracle.svm.hosted.NativeImageGeneratorRunner \
-imagecp \
/home/toddsharp/opt/graalvm/jre/lib/boot/graal-sdk.jar:/home/toddsharp/opt/graalvm/jre/lib/boot/graaljs-scriptengine.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/builder/pointsto.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/builder/objectfile.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/builder/svm.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/jvmci-hotspot.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/graal.jar:/home/toddsharp/opt/graalvm/jre/lib/jvmci/jvmci-api.jar:/home/toddsharp/opt/graalvm/jre/lib/svm/library-support.jar:/home/toddsharp/Projects/scratch/graalvm:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ant-1.9.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ant-antlr-1.9.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ant-junit-1.9.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ant-launcher-1.9.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/bsf-2.4.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/commons-cli-1.2.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/commons-logging-1.2.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/gpars-1.2.1.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-ant-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-bsf-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-console-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-docgenerator-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-groovydoc-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-groovysh-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-jmx-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-json-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-jsr223-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-nio-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-servlet-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-sql-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-swing-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-templates-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-test-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-testng-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/groovy-xml-2.4.10.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/hamcrest-core-1.3.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/ivy-2.4.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jansi-1.11.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jcommander-1.47.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jline-2.12.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jsp-api-2.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/jsr166y-1.7.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/junit-4.12.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/multiverse-core-0.7.0.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/qdox-1.12.1.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/servlet-api-2.4.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/testng-6.8.13.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/xmlpull-1.1.3.1.jar:/home/toddsharp/.sdkman/candidates/groovy/current/lib/xstream-1.4.7.jar \
-H:Path=/home/toddsharp/Projects/scratch/graalvm \
-H:CLibraryPath=/home/toddsharp/opt/graalvm/jre/lib/svm/clibraries/linux-amd64 \
-H:Class=HiWorld \
-H:Name=hiworld
]
   classlist:   4,774.47 ms
       setup:     636.20 ms
error: could not find non-optional target method: public static jline.Terminal com.oracle.svm.jline.subst.Target_jline_TerminalFactory.create(java.lang.String)
Error: Image building with exit status 1
@cstancu

This comment has been minimized.

Member

cstancu commented Apr 18, 2018

That's the complete verbose output. Thank you!

@cstancu

This comment has been minimized.

Member

cstancu commented Apr 18, 2018

The classpath looks good, i.e., the jars that are needed by native-image are all there. I will install groovy and try to replicate the problem.

@cstancu

This comment has been minimized.

Member

cstancu commented Apr 18, 2018

I replicated the problem and will investigate further. It most likely has to do with some features that the Groovy implementation uses (e.g., like reflection, static initializers that are either too lazy or too eager) which have a special treatment on SubstrateVM (e.g., reflection configuration files and/or some substitutions). We take a similar approach with Scala and Kotlin.

@olpaw it looks like we don't currently support wildcards in the -cp argument to native-image; we expect a list of absolute paths divided by the system-dependent path-separator character.

@cfsilence

This comment has been minimized.

cfsilence commented Apr 18, 2018

Thanks a lot! I'm really impressed with what your team has done with Graal so far!

@cstancu cstancu self-assigned this Apr 18, 2018

@cstancu

This comment has been minimized.

Member

cstancu commented Apr 19, 2018

The problem is that groovy-2.4.15, the latest stable version that I installed through sdkman as I guess you did too, brings in jline-2.12, quite an old version. In SubstrateVM we currently support a later version of jline, 2.14.5, used by some other components. The changes between jline-2.12 and jline-2.14.5 range from simple method signature changes to new classes.

JLine support involves patching some reflective code. FYI The substitution code is in JLineSubstitutions.java. We match the original method using its signature. In this case the signature of jline.TerminalFactory.create() has changed, like I suspected initially when I thought you might be using Java 9.

I pulled in the latest groovy, 3.0.0-alpha-2, and this problem goes away. 3.0.0-alpha-2 uses jline-2.14.6 (which is greater than 2.14.5 but it doesn't really matter as long as the code that we patch didn't change significantly). However, now I hit a bug in SubstrateVM. I'll have to fix that first.

@cfsilence

This comment has been minimized.

cfsilence commented Apr 19, 2018

Installed groovy 3.0.0-alpha-2 and hit this (Probably the same thing you're seeing):

$ native-image -cp ".:$(echo /$GROOVY_HOME/lib/*.jar | tr ' ' ':')" HiWorld
Warning: Native image server limit exceeded. Use options --server{-list,-shutdown[-all]} to fix the problem.
   classlist:   9,498.28 ms
       (cap):   2,202.05 ms
       setup:   4,330.03 ms
    analysis:  38,210.05 ms
fatal error: java.lang.IndexOutOfBoundsException: index 4
        at java.util.concurrent.atomic.AtomicReferenceArray.checkedByteOffset(AtomicReferenceArray.java:78)
        at java.util.concurrent.atomic.AtomicReferenceArray.get(AtomicReferenceArray.java:125)
        at com.oracle.graal.pointsto.flow.context.object.AnalysisObject.getInstanceFieldTypeStore(AnalysisObject.java:211)
        at com.oracle.graal.pointsto.flow.context.object.AnalysisObject.getInstanceFieldFlow(AnalysisObject.java:197)
        at com.oracle.graal.pointsto.flow.LoadFieldTypeFlow$LoadInstanceFieldTypeFlow.onObservedUpdate(LoadFieldTypeFlow.java:157)
        at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:345)
        at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:387)
        at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:498)
        at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:172)
        at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Error: Image building with exit status 1

Edit: I can provide verbose output if that's different from what you're seeing.

@cstancu

This comment has been minimized.

Member

cstancu commented Apr 19, 2018

Indeed that's the issue I see too. It will be fixed soon.

@cstancu

This comment has been minimized.

Member

cstancu commented Apr 20, 2018

That issue is fixed by ea925f2. To get the latest changes you need to build the native-image from source as described in the substratevm README.md.
However, now we get to the real issues that I expected: groovy makes heavy use of reflection and dynamic class loading. Even if the minimal test doesn't make use of that functionality the static analysis finds that code as reachable (even if the test class is annotated with @CompileStatic). It also doesn't help that groovy doesn't package the runtime separate from the compiler.
One workaround is to use the -H:+ReportUnsupportedElementsAtRuntime option together with annotating the test class with @CompileStatic:

import groovy.transform.CompileStatic

@CompileStatic
class HiWorldStatic {
    static void main(String[] args) {
        println 'hi static!'
  }
}
$ ./native-image -H:+ReportUnsupportedElementsAtRuntime --no-server -cp ".:$(echo $GROOVY_HOME/lib/*.jar | tr ' ' ':')" HiWorldStatic

This works and produces a working executable hiworldstatic:

$ ./hiworldstatic 
hi static!

-H:+ReportUnsupportedElementsAtRuntime forces the analysis to ignore unsupported features and instead only report them at runtime if really reachable.

Edit: I spoke too soon. To get that last example working you need a small substitution patch that needs to make its way through our CI system first.

@olpaw

This comment has been minimized.

Member

olpaw commented Apr 20, 2018

@olpaw it looks like we don't currently support wildcards in the -cp argument to native-image; we expect a list of absolute paths divided by the system-dependent path-separator character

That's correct. Currently we don't have wildcards support for classpath entries. I created #355 for that.

@cstancu

This comment has been minimized.

Member

cstancu commented Apr 20, 2018

The patch for groovy is in. You should be able to run the example as I described above. In the future we will look into eliminating the need to use -H:+ReportUnsupportedElementsAtRuntime.

More generally for groovy we should be able to run most workloads as long as you use static compilation, i.e., either @CompileStatic for methods/types or per application using a compiler configuration like:

import groovy.transform.CompileStatic

withConfig(configuration) {
    ast(CompileStatic)
}

Support for dynamic behavior is limited as described in the substratevm LIMITATIONS.md

@cstancu cstancu closed this Jun 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment