Provide configuration for native compilation using graalvm native-image

[fafriat]

Issue #, if available:
Lambda RIC use of reflection prevents simple native image compilation (with graale-vm native-image) and cause Error at run #205

Description of changes:
Java Lambda RIC is difficult to include in lambda code for compilation because of the use of reflection and native code with jni.
Including these config files will help a lot all those wishing to compile their code with graalvm native-image.
For details on configuration native-image see: https://www.graalvm.org/reference-manual/native-image/BuildConfiguration/

In particular: "The generated configuration files can be supplied to the native-image tool by placing them in a META-INF/native-image/ directory on the class path, for example, in a JAR file used in the image build. This directory (or any of its subdirectories) is searched for files with the names jni-config.json, reflect-config.json, proxy-config.json and resource-config.json, which are then automatically included in the build. Not all of those files must be present. When multiple files with the same name are found, all of them are included."

These files are very small so the inconvenience for the others users would be negligible.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[fafriat]

I wrote a blog explaining howto build a custom runtime here.
Project code can be found on github here.
In case you want to verify that this configuration is working...

[msailes]

Hi Frank,

Thanks for the PR and the blog post, I found it really interesting. I must admit, I'm still catching up with the advances with Java and GraalVM. It's great to have this support from the community.

Some questions from me from what I've read of the build config reference

• Would we be best to put the various configuration files within a groupid and artifactid folder structure to avoid conflicts?
• I presume you've generated these files with the native-image-agent did you notice any different configurations being generated between different executions of the code?
• Have those different configurations been merged in this PR?

Thanks,

Mark

[fafriat]

Hi Mark,

Thank you, I'm still catching up also...

1. Good point, in fact I did not use native-image.properties configuration file at all and it makes sense to use it because

• all the provided json files can be put everywhere (not only like did) if configured in it (so using the same folder as native-image.properties is a good idea).
• Some people could use an uber jar even if it is not recommended (micronaut framework for example is not using an uber jar) so it makes sense to use the recommended way to put the file inside subdirectories to avoid overlapping
• it give the opportunity to add the default commandline argument I needed to use in the Dockerfile: --initialize-at-build-time=jdk.xml.internal.SecuritySupport

2. Yes, I used the agent, but also some part was manual (ex: jni configuration for musl lib) but I did not notice any problem at runtime after compilation with the provided configuration.

Finally the native-image.properties could contain something like that:

Args = --initialize-at-build-time=jdk.xml.internal.SecuritySupport \
-H:ReflectionConfigurationResources=${.}/reflect-config.json \
-H:JNIConfigurationResources=${.}/jni-config.json \
-H:ResourceConfigurationResources=${.}/resource-config.json

As the proxy-config.json is empty I did not include it.
What do you think ?

[msailes]

I'm guessing this is a mistake?

[fafriat]

yes sure sorry... only related to my example, forgot it.

[fafriat]

I just updated the PR branch using native-image.properties...

Also I tried the config in my github project in a branch native-config https://github.com/fafriat/hello-lambda/tree/native-config, if you want to also try...
so far it works fine.

[fafriat]

Hi, any news ?

[msailes]

Hi Frank,

I'm waiting for a couple of people to return from annual leave to discuss this further.

Thanks for your patience.

Mark

[carlzogh]

Hey @fafriat - thanks for your contribution and sorry I'm late here!

Changes look good to me, tested and validated working. I'd like us to also add relevant documentation for this in the README/USAGE docs for this library as part of the added support. Is this something you'd also be willing to contribute?

[fafriat]

Hi,

I would be very happy to contribute to the documentation.
If you are ok I could modify the file aws-lambda-java-libs/aws-lambda-java-runtime-interface-client/README.md and add a section about native compilation and add it to this Merge Request ?

[carlzogh]

That would be awesome, thanks @fafriat! :)

[fafriat]

Please, what will be the released version ?

[fafriat]

Please review the update README.md...
Thanks !

[ilopmar]

I've discovered that this JNI configuration triggers the following error during native image generating with a Micronaut application:

[application:25]     universe:   2,074.36 ms,  4.87 GB
Fatal error:java.lang.NoClassDefFoundError
       at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
       at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
       at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:603)
       at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
       at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:488)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:370)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:529)
       at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:119)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus.main(NativeImageGeneratorRunner.java:561)
Caused by: java.lang.NoClassDefFoundError: io/netty/internal/tcnative/SSLPrivateKeyMethod
        at jdk.internal.vm.ci/jdk.vm.ci.hotspot.CompilerToVM.getDeclaredMethods(Native Method)
       at jdk.internal.vm.ci/jdk.vm.ci.hotspot.HotSpotResolvedObjectTypeImpl.getDeclaredMethods(HotSpotResolvedObjectTypeImpl.java:987)
       at com.oracle.svm.jni.access.JNIAccessibleMethod.anyMatchIgnoreReturnType(JNIAccessibleMethod.java:139)
       at com.oracle.svm.jni.access.JNIAccessibleMember.findHidingSubclasses(JNIAccessibleMember.java:103)
       at com.oracle.svm.jni.access.JNIAccessibleMember.findHidingSubclasses(JNIAccessibleMember.java:110)
       at com.oracle.svm.jni.access.JNIAccessibleMember.setHidingSubclasses(JNIAccessibleMember.java:83)
        at com.oracle.svm.jni.access.JNIAccessibleMethod.finishBeforeCompilation(JNIAccessibleMethod.java:135)
       at com.oracle.svm.jni.access.JNIAccessFeature.beforeCompilation(JNIAccessFeature.java:350)
        at com.oracle.svm.hosted.NativeImageGenerator.lambda$doRun$2(NativeImageGenerator.java:619)
       at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:70)
       at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:619)
       at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:476)
       at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
       at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
       at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
       at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
       at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
       at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLPrivateKeyMethod
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 18 more
Error: Image build request failed with exit status 1

More info: micronaut-projects/micronaut-aws#1024 (comment)

[ilopmar]

Actually I was wrong about this. The thing that triggers everything is this resource configuration (already included in this PR):

{
  "resources":{
  "includes":[
    {"pattern":"\\Qaws-lambda-runtime-interface-client.glibc.so\\E"}, 
    {"pattern":"\\Qaws-lambda-runtime-interface-client.musl.so\\E"}
  ]},
  "bundles":[]
}

More info: micronaut-projects/micronaut-aws#1024 (comment)

[fafriat]

these native libraries are necessary for running the RIC...
I cannot understand why there would be this strange side effect...

Except that without this configuration nothing is working nor failing ??!?

[msailes]

Hi @fafriat,

We're working on getting all the libraries to support GraalVM. Can I bring your changes into my branch so that we can test everything together?

Thanks,

Mark

[fafriat]

Hi Mark @msailes,

Feel free to bring everything :-) would be great!

Thanks,

Frank

[msailes]

Combining with #255

Thank you so much for starting this great work.