New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.0.0 RC7 breaks support for Micronaut/Netty: No instances are allowed in the image heap #713

Open
graemerocher opened this Issue Oct 4, 2018 · 25 comments

Comments

4 participants
@graemerocher

graemerocher commented Oct 4, 2018

With RC7 and attempting to build a native image that previously worked in RC6 results in:

error: No instances are allowed in the image heap for a class that is initialized or reinitialzied at image runtime: sun.security.provider.NativePRNG
Detailed message:
Error: No instances are allowed in the image heap for a class that is initialized or reinitialzied at image runtime: sun.security.provider.NativePRNG
Trace: 	object io.netty.handler.ssl.util.ThreadLocalInsecureRandom
	method io.netty.handler.ssl.util.ThreadLocalInsecureRandom.current()
Call path from entry point to io.netty.handler.ssl.util.ThreadLocalInsecureRandom.current(): 
	at io.netty.handler.ssl.util.ThreadLocalInsecureRandom.current(ThreadLocalInsecureRandom.java:35)
	at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:110)
	at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:88)
	at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:79)
	at io.micronaut.http.server.netty.ssl.NettyServerSslBuilder.build(NettyServerSslBuilder.java:67)
	at io.micronaut.http.server.netty.NettyHttpServer.<init>(NettyHttpServer.java:207)
	at 

Further trace:

com.oracle.svm.reflect.proxies.Proxy_io_micronaut_http_server_netty_NettyHttpServer_constructor_io_micronaut_http_server_netty_configuration_NettyHttpServerConfiguration_io_micronaut_context_ApplicationContext_io_micronaut_web_router_Router_io_micronaut_http_server_binding_RequestArgumentSatisfier_io_micronaut_http_codec_MediaTypeCodecRegistry_io_micronaut_http_server_netty_types_NettyCustomizableResponseTypeHandlerRegistry_io_micronaut_web_router_resource_StaticResourceResolver_java_util_concurrent_ExecutorService_java_util_concurrent_ThreadFactory_io_micronaut_scheduling_executor_ExecutorSelector_io_micronaut_http_server_netty_ssl_NettyServerSslBuilder_java_util_List_430.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at sun.security.x509.OtherName.getGNI(OtherName.java:132)
	at sun.security.x509.OtherName.equals(OtherName.java:184)
	at io.micronaut.caffeine.cache.UnboundedLocalCache$EntrySetView.contains(UnboundedLocalCache.java:729)
	at java.util.AbstractCollection.containsAll(AbstractCollection.java:318)
	at java.util.RegularEnumSet.containsAll(RegularEnumSet.java:199)
	at com.oracle.svm.core.amd64.AMD64CPUFeatureAccess.verifyHostSupportsArchitecture(AMD64CPUFeatureAccess.java:158)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:138)
	at com.oracle.svm.core.code.CEntryPointCallStubs.com_002eoracle_002esvm_002ecore_002eJavaMainWrapper_002erun_0028int_002corg_002egraalvm_002enativeimage_002ec_002etype_002eCCharPointerPointer_0029(generated:0)

Error: Processing image build request failed

Are there any new flags that should be passed to avoid this? Or am I missing something

@graemerocher

This comment has been minimized.

graemerocher commented Oct 4, 2018

Possibly relates to #712

@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

Can you try pushing the initialization of ThreadLocalInsecureRandom to runtime using --delay-class-initialization-to-runtime=ThreadLocalInsecureRandom?

@graemerocher

This comment has been minimized.

graemerocher commented Oct 4, 2018

@cstancu If I do that it now fails with:

error: No instances are allowed in the image heap for a class that is initialized or reinitialzied at image runtime: sun.security.provider.NativePRNG
Detailed message:
Error: No instances are allowed in the image heap for a class that is initialized or reinitialzied at image runtime: sun.security.provider.NativePRNG
Trace: 	object java.security.SecureRandom
	field sun.security.jca.JCAUtil$CachedSecureRandomHolder.instance

Error: Processing image build request failed

If I try add java.security.SecureRandom then it fails with:

error: Class is already initialized, so it is too late to register delaying class initialization: java.security.SecureRandom
Error: Processing image build request failed
@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

You cannot delay SecureRandom, we need that in the image. You need --rerun-class-initialization-at-runtime=sun.security.jca.JCAUtil$CachedSecureRandomHolder. Notice that this is --rerun-class-initialization-at-runtime instead of --delay-class-initialization-to-runtime since this is a JDK class and might be already initialized during image building.

@graemerocher

This comment has been minimized.

graemerocher commented Oct 4, 2018

I made the following change:

--- a/app-with-discovery/build-native-image.sh
+++ b/app-with-discovery/build-native-image.sh
@@ -8,4 +8,5 @@ native-image --class-path build/libs/app-with-discovery-0.1-all.jar \
                         -H:Class=app.with.discovery.Application \
                         -H:+ReportUnsupportedElementsAtRuntime \
                         -H:+AllowVMInspection \
-                        --delay-class-initialization-to-runtime=io.netty.handler.codec.http.HttpObjectEncoder,io.netty.handler.codec.http.websocketx.WebSocket00FrameEncoder,io.netty.handler.codec.http.HttpObjectDecoder
+                        --delay-class-initialization-to-runtime=io.netty.handler.codec.http.HttpObjectEncoder,io.netty.handler.codec.http.websocketx.WebSocket00FrameEncoder,io.netty.handler.codec.http.HttpObjectDecoder,io.netty.handler.ssl.util.ThreadLocalInsecureRandom \
+                        --rerun-class-initialization-at-runtime=sun.security.jca.JCAUtil$CachedSecureRandomHolder

But it still fails with the same error:

error: No instances are allowed in the image heap for a class that is initialized or reinitialzied at image runtime: sun.security.provider.NativePRNG
Detailed message:
Error: No instances are allowed in the image heap for a class that is initialized or reinitialzied at image runtime: sun.security.provider.NativePRNG
Trace: 	object java.security.SecureRandom
	field sun.security.jca.JCAUtil$CachedSecureRandomHolder.instance
@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

Can you point me to the repo/branch to replicate this?

@graemerocher

This comment has been minimized.

graemerocher commented Oct 4, 2018

@cstancu you can replicate with the steps in #714 just using RC7 instead of RC6

@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

Trying to build I fail with

$ ./gradlew assemble

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':shadowJar'.
> Could not resolve all dependencies for configuration ':runtimeClasspath'.
   > Could not find io.micronaut:graal:.
     Required by:
         project :

@cstancu cstancu self-assigned this Oct 4, 2018

@cstancu cstancu added the substrate label Oct 4, 2018

@graemerocher

This comment has been minimized.

graemerocher commented Oct 4, 2018

@cstancu are you behind a proxy? I cleaned my Maven and Gradle caches and it seems to work fine. You need to able to resolve snapshots from https://oss.sonatype.org/content/repositories/snapshots

@graemerocher

This comment has been minimized.

graemerocher commented Oct 4, 2018

@cstancu maybe the step to install the Graal SVM dependency is missing?

mvn install:install-file -Dfile=${JAVA_HOME}/jre/lib/svm/builder/svm.jar \
                           -DgroupId=com.oracle.substratevm \
                           -DartifactId=svm \
                           -Dversion=GraalVM-1.0.0-rc6 \
                           -Dpackaging=jar

If you install the RC7 SVM jar you will need to update the reference in the build.gradle

@mavilein

This comment has been minimized.

mavilein commented Oct 4, 2018

We are hitting the same issue. We have a reproduction available here. This is the error we are seeing:

[main:498]     analysis:  14,269.80 ms
error: No instances are allowed in the image heap for a class that is initialized or reinitialzied at image runtime: sun.security.provider.NativePRNG
Detailed message:
Error: No instances are allowed in the image heap for a class that is initialized or reinitialzied at image runtime: sun.security.provider.NativePRNG
Trace: 	object java.security.SecureRandom
	object sun.security.ssl.SSLContextImpl$DefaultSSLContext
	object javax.net.ssl.SSLContext
	field javax.net.ssl.SSLContext.defaultContext

Error: Processing image build request failed

I am also happy to jump on a call if this easier for you @cstancu

@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

@graemerocher I already had the SVM dependency installed. What I needed was to clean the maven and gradle caches. Now ./gradlew assemble builds successfully, however the second step java -cp build/libs/app-with-discovery-0.1-all.jar io.micronaut.graal.reflect.GraalClassLoadingAnalyzer doesn't produce the build/reflect.json:

$ ./gradlew assemble

BUILD SUCCESSFUL in 1s
10 actionable tasks: 10 up-to-date
$ java -cp build/libs/app-with-discovery-0.1-all.jar io.micronaut.graal.reflect.GraalClassLoadingAnalyzer
$ ls build/
classes  distributions  generated  libs  resources  scripts  scriptsShadow  tmp

@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

@graemerocher never mind, it looks like the GraalClassLoadingAnalyzer tool needs the java from GraalVM.

@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

@graemerocher the problem is the $ sign on the command line, you need to add the class name between '':-rerun-class-initialization-at-runtime='sun.security.jca.JCAUtil$CachedSecureRandomHolder'. Then it works, I can build the image, however when I run it I get io.micronaut.http.client.exceptions.HttpClientException: Connect Error: Connection refused: localhost/127.0.0.1:8500. Do I need a server running?

@graemerocher

This comment has been minimized.

graemerocher commented Oct 4, 2018

@cstancu great news, yes the example needs consul running. If you want to try it further you can do

docker run -p 8500:8500 consul
@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

@mavilein you need --rerun-class-initialization-at-runtime=javax.net.ssl.SSLContext to get past that error and you also need --enable-all-security-services since your code is using the JCA. With this the image builds but it fails at run time due to missing reflection configurations. Please open another issue if you still have problems.

@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

@graemerocher I tried your app further but it looks like the image is missing some reflection config: io.micronaut.http.codec.CodecException: Error encoding object [io.micronaut.discovery.consul.client.v1.NewServiceEntry@7f2f1d2c4420] to JSON: Class com.fasterxml.jackson.databind.PropertyNamingStrategy$UpperCamelCaseStrategy has no default (no arg) constructor.

@cstancu

This comment has been minimized.

Member

cstancu commented Oct 4, 2018

As a resolution for this issue:

  • I'll make --rerun-class-initialization-at-runtime='sun.security.jca.JCAUtil$CachedSecureRandomHolder' a default via SecurityServicesFeature. In general any class that has a static reference to a SecureRandom needs to be re-initialized at run time because the SecureRandom implementations open the /dev/random and /dev/urandom files which are used as sources for entropy.
  • --delay-class-initialization-to-runtime=ThreadLocalInsecureRandom seems to be needed by netty ssl.
  • I'll make --rerun-class-initialization-at-runtime=javax.net.ssl.SSLContext a default for when --enable-all-security-services is used (which is enabled automatically when --enable-https is used).
@graemerocher

This comment has been minimized.

graemerocher commented Oct 5, 2018

@cstancu btw when I add --rerun-class-initialization-at-runtime=javax.net.ssl.SSLContext a default for when --enable-all-security-services to the app above (just as a test to the impact on it). The application fails to build with:

fatal error: java.lang.NullPointerException
	at com.oracle.svm.hosted.code.CompileQueue.defaultCompileFunction(CompileQueue.java:893)
	at com.oracle.svm.hosted.code.CompileQueue.doCompile(CompileQueue.java:882)
	at com.oracle.svm.hosted.code.CompileQueue$CompileTask.run(CompileQueue.java:257)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:174)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	a
@cstancu

This comment has been minimized.

Member

cstancu commented Oct 5, 2018

@graemerocher I cannot replicate the NPE trace that you posted above. I pulled in your latest changes to the micronaut-graal-experiments/app-with-discovery and tried with rc7. Maybe you were using rc6?

@archerz

This comment has been minimized.

archerz commented Oct 30, 2018

@graemerocher @cstancu Are there any updates on this issue?

@graemerocher

This comment has been minimized.

graemerocher commented Oct 30, 2018

@archerz are you seeing issues with Micronaut 1.0 related to this?

@archerz

This comment has been minimized.

archerz commented Oct 30, 2018

@graemerocher With 1.0.0.RC2. I'm using Graal 1.0.0-rc8.

The command I'm using is:

./gradlew assemble
java -cp build/libs/appbilling-api-all.jar io.micronaut.graal.reflect.GraalClassLoadingAnalyzer
native-image --class-path build/libs/app-all.jar \
			 -H:ReflectionConfigurationFiles=build/reflect.json \
			 -H:EnableURLProtocols=http \
			 -H:IncludeResources="logback.xml|application.yml|META-INF/services/*.*" \
			 -H:Name=app \
			 -H:Class=com.app.App \
			 -H:+ReportUnsupportedElementsAtRuntime \
			 -H:+AllowVMInspection \
			 --rerun-class-initialization-at-runtime=io.netty.handler.ssl.util.ThreadLocalInsecureRandom \
			 --delay-class-initialization-to-runtime= \
			    io.netty.handler.codec.http.HttpObjectEncoder, \
			    io.netty.handler.ssl.util.ThreadLocalInsecureRandom, \
			    io.netty.handler.codec.http.websocketx.WebSocket00FrameEncoder

And the error:

error: unsupported features in 2 methods
Detailed message:
Error: Detected a direct/mapped ByteBuffer in the image heap. A direct ByteBuffer has a pointer to unmanaged C memory, and C memory from the image generator is not available at image run time. A mapped ByteBuffer references a file descriptor, which is no longer open and mapped at run time. The object was probably created by a class initializer and is reachable from a static field. By default, all class initialization is done during native image building.You can manually delay class initialization to image run time by using the option --delay-class-initialization-to-runtime=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace:  object io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf
        object io.netty.buffer.UnreleasableByteBuf
        method io.netty.handler.codec.http.HttpObjectEncoder.encodeChunkedContent(ChannelHandlerContext, Object, long, List)
Call path from entry point to io.netty.handler.codec.http.HttpObjectEncoder.encodeChunkedContent(ChannelHandlerContext, Object, long, List): 
        at io.netty.handler.codec.http.HttpObjectEncoder.encodeChunkedContent(HttpObjectEncoder.java:196)
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:169)
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
        at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
        at java.lang.Thread.run(Thread.java:748)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:476)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:199)
        at com.oracle.svm.core.code.CEntryPointCallStubs.com_002eoracle_002esvm_002ecore_002eposix_002ethread_002ePosixJavaThreads_002epthreadStartRoutine_0028com_002eoracle_002esvm_002ecore_002ethread_002eJavaThreads_0024ThreadStartData_0029(generated:0)
Error: No instances are allowed in the image heap for a class that is initialized or reinitialized at image runtime: sun.security.provider.NativePRNG
Trace:  object io.netty.handler.ssl.util.ThreadLocalInsecureRandom
        method io.netty.handler.ssl.util.ThreadLocalInsecureRandom.current()
Call path from entry point to io.netty.handler.ssl.util.ThreadLocalInsecureRandom.current(): 
        at io.netty.handler.ssl.util.ThreadLocalInsecureRandom.current(ThreadLocalInsecureRandom.java:35)
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:110)
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:88)
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:79)
        at io.micronaut.http.server.netty.ssl.NettyServerSslBuilder.build(NettyServerSslBuilder.java:67)
        at io.micronaut.http.server.netty.NettyHttpServer.<init>(NettyHttpServer.java:208)
        at com.oracle.svm.reflect.proxies.Proxy_io_micronaut_http_server_netty_NettyHttpServer_constructor_io_micronaut_http_server_netty_configuration_NettyHttpServerConfiguration_io_micronaut_context_ApplicationContext_io_micronaut_web_router_Router_io_micronaut_http_server_binding_RequestArgumentSatisfier_io_micronaut_http_codec_MediaTypeCodecRegistry_io_micronaut_http_server_netty_types_NettyCustomizableResponseTypeHandlerRegistry_io_micronaut_web_router_resource_StaticResourceResolver_java_util_concurrent_ExecutorService_java_util_concurrent_ThreadFactory_io_micronaut_scheduling_executor_ExecutorSelector_io_micronaut_http_server_netty_ssl_NettyServerSslBuilder_java_util_List_522.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.security.x509.OtherName.getGNI(OtherName.java:132)
        at sun.security.x509.OtherName.equals(OtherName.java:184)
        at java.util.HashMap.getNode(HashMap.java:579)
        at java.util.HashMap.get(HashMap.java:557)
        at com.oracle.svm.jni.access.JNIReflectionDictionary.getFieldNameByID(JNIReflectionDictionary.java:150)
        at com.oracle.svm.jni.functions.JNIFunctions.ToReflectedField(JNIFunctions.java:839)
        at com.oracle.svm.core.code.CEntryPointCallStubs.com_002eoracle_002esvm_002ejni_002efunctions_002eJNIFunctions_002eToReflectedField_0028com_002eoracle_002esvm_002ejni_002enativeapi_002eJNIEnvironment_002ccom_002eoracle_002esvm_002ejni_002enativeapi_002eJNIObjectHandle_002ccom_002eoracle_002esvm_002ejni_002enativeapi_002eJNIFieldId_0029(generated:0)

Error: Processing image build request failed
@graemerocher

This comment has been minimized.

graemerocher commented Oct 30, 2018

@archerz Try 1.0.0 final create a new project and use 1.0 RC7 of Graal. For RC8 came out the same day as 1.0.0 of Micronaut so we have some updates to do for that due to some changes with META-INF/services handling

@graemerocher

This comment has been minimized.

graemerocher commented Oct 30, 2018

Although some folks have got Micronaut working with RC8, think you need to pass -H:-UseServiceLoaderFeature additional flag to the compiler. See https://gitlab.com/puravida-software/hello-micronaut-graalvm/blob/master/.gitlab-ci.yml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment