Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Helidon MP Graal native-image #716

Closed
20 tasks done
tomas-langer opened this issue May 21, 2019 · 25 comments
Closed
20 tasks done

Helidon MP Graal native-image #716

tomas-langer opened this issue May 21, 2019 · 25 comments
Assignees
@tomas-langer
Labels
enhancement New feature or request epic graal MP
Projects
Backlog Helidon 2.0
Milestone
2.0.0

Comments

@tomas-langer
Copy link
Member

tomas-langer commented May 21, 2019 •
edited by m0mus

This task is tracking work related to GraalVM support in Helidon MP.

  • PoC

Now the PoC is done, let's start adding these features to Helidon. The checked features are implemented and working. I will attach PR numbers to these once I can.

13th January 2020: merged into master
10th January 2020:
PR #1259 (modules tested with this PR):

  • CDI (Weld + Helidon modifications)
  • MP Config
  • Server (WebServer, Jersey, Jersey integration with CDI)
  • Override config from MP Server (Helidon config)
  • Tracing (MP Tracing + Jaeger Tracer)
  • Access Log
  • Fault Tolerance - Fallback, Timeout, and Retry tested
  • Metrics 2.0
  • Health Check
  • REST Client (no support for default methods in interfaces due to issue with MethodHandle)
  • Jersey client
  • Other media types (JSON-P, JSON-B)

16th January 2020 merged #1295

Merged:

TODOs:

gRPC is a separate epic: #1294
@tomas-langer tomas-langer self-assigned this May 21, 2019
@tomas-langer tomas-langer added this to Needs triage in Backlog via automation May 21, 2019
@tomas-langer
Copy link
Member Author

I have created a "pure" CDI example that only starts Weld and uses a single ApplicationScoped bean.
The current blocker:
The beans.xml processing for beans discovery fails (as Weld tried to locate the file on the file system, where it cannot be).

Next step:
Disable runtime scanning
Use only Jandex for beans discovery (if possible)

@m0mus m0mus moved this from Needs triage to Normal priority in Backlog May 23, 2019
@dansiviter dansiviter mentioned this issue Jun 17, 2019
Closed
@tomas-langer tomas-langer moved this from Normal priority to In Progress in Backlog Sep 12, 2019
@tomas-langer
Copy link
Member Author

tomas-langer commented Sep 15, 2019 •
edited

  • Weld
  • Helidon WebServer
  • Jersey support for WebServer
  • Jersey
  • Jersey client
  • Jersey integration with CDI
  • Interceptor support (conceptual for Metrics)
  • REST Client - issue with MethodHandle
  • Metrics 2.0
  • Healthchecks
  • Tracing
  • OpenAPI
  • Fault Tolerance - Fallback, Timeout, and Retry tested
  • Security (MP JWT Auth + others)
  • MP Config (conceptual)
  • Override config from MP Server (Helidon config)
  • Other media types (JSON-P, JSON-B etc.)
  • JPA
  • JTA

@tomas-langer
Copy link
Member Author

REST client - the implementation caches config instance that was used during CDI extension. As a result we could not use config passed to the MP Server.
Workaround: The MP configuration uses delegate pattern. We can replace the delegate before starting the CDI container fully. All components that hold the Config instance will get values from the new configuration.

@tomas-langer
Copy link
Member Author

REST client - uses method handles, cannot be compiled using native-image

@tomas-langer
Copy link
Member Author

Fault tolerance - Unsafe issue:

Warning: RecomputeFieldValue.ArrayIndexScale automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayIndexScale(Class) was detected in the static initializer of rx.internal.util.unsafe.ConcurrentCircularArrayQueue. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.arrayIndexScale(Class) for the array index scale computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store. 
Warning: RecomputeFieldValue.ArrayBaseOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayBaseOffset(Class) was detected in the static initializer of rx.internal.util.unsafe.ConcurrentCircularArrayQueue. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.arrayBaseOffset(Class) for the array base offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store. 
Warning: RecomputeFieldValue.ArrayIndexScale automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayIndexScale(Class) was detected in the static initializer of rx.internal.util.unsafe.SpscUnboundedArrayQueue. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.arrayIndexScale(Class) for the array index scale computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store. 
Warning: RecomputeFieldValue.ArrayIndexScale automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayIndexScale(Class) was detected in the static initializer of rx.internal.util.unsafe.ConcurrentSequencedCircularArrayQueue. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.arrayIndexScale(Class) for the array index scale computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.

@tomas-langer
Copy link
Member Author

Jersey - issue with injection using @Context - other features worked fine

1. com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface io.helidon.security.SecurityContext, interface org.glassfish.hk2.api.ProxyCtl] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=<comma-separated-config-files> and -H:DynamicProxyConfigurationResources=<comma-separated-config-resources> options.
2. java.lang.IllegalArgumentException: While attempting to create a Proxy for io.helidon.security.SecurityContext in scope org.glassfish.jersey.process.internal.RequestScoped an error occured while creating the proxy

        at org.jvnet.hk2.internal.ProxyUtilities.generateProxy(ProxyUtilities.java:211)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2074)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.resolve(ContextInjectionResolverImpl.java:103)
        at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:988)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:998)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:207)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$2.getInstance(AbstractCdiBeanSupplier.java:88)
        at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:103)
        at org.glassfish.jersey.ext.cdi1x.internal.GenericCdiBeanSupplier.get(GenericCdiBeanSupplier.java:42)
        at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:53)
        at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:129)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:46)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.lambda$getAllServiceHolders$0(AbstractHk2InjectionManager.java:136)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getAllServiceHolders(AbstractHk2InjectionManager.java:140)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getAllServiceHolders(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.internal.inject.Providers.getServiceHolders(Providers.java:299)
        at org.glassfish.jersey.internal.inject.Providers.getAllRankedProviders(Providers.java:182)
        at org.glassfish.jersey.server.ProcessingProvidersConfigurator.postInit(ProcessingProvidersConfigurator.java:89)
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$2(ApplicationHandler.java:349)
        at java.util.Arrays$ArrayList.forEach(Arrays.java:3880)
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:349)
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:293)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:232)
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:292)
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:259)
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:246)
        at io.helidon.webserver.jersey.JerseySupport.<init>(JerseySupport.java:126)
        at io.helidon.webserver.jersey.JerseySupport.<init>(JerseySupport.java:80)
        at io.helidon.webserver.jersey.JerseySupport$Builder.build(JerseySupport.java:436)
        at io.helidon.microprofile.server.ServerCdiExtension.createJerseySupport(ServerCdiExtension.java:144)
        at io.helidon.microprofile.server.ServerCdiExtension.addApplicationsInstances(ServerCdiExtension.java:110)
        at io.helidon.microprofile.server.ServerCdiExtension.routing(ServerCdiExtension.java:103)
        at io.helidon.microprofile.server.ServerCdiExtension.startServer(ServerCdiExtension.java:69)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95)
        at org.jboss.weld.injection.MethodInvocationStrategy$SpecialParamPlusBeanManagerStrategy.invoke(MethodInvocationStrategy.java:187)
        at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:330)
        at org.jboss.weld.event.ExtensionObserverMethodImpl.sendEvent(ExtensionObserverMethodImpl.java:123)
        at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:308)
        at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:286)
        at javax.enterprise.inject.spi.ObserverMethod.notify(ObserverMethod.java:124)
        at org.jboss.weld.util.Observers.notify(Observers.java:166)
        at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:285)
        at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:273)
        at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:177)
        at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:171)
        at org.jboss.weld.bootstrap.events.AbstractContainerEvent.fire(AbstractContainerEvent.java:53)
        at org.jboss.weld.bootstrap.events.AbstractDeploymentContainerEvent.fire(AbstractDeploymentContainerEvent.java:35)
        ... 7 more
Caused by: com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface io.helidon.security.SecurityContext, interface org.glassfish.hk2.api.ProxyCtl] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=<comma-separated-config-files> and -H:DynamicProxyConfigurationResources=<comma-separated-config-resources> options.
        at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:102)
        at com.oracle.svm.reflect.proxy.DynamicProxySupport.getProxyClass(DynamicProxySupport.java:113)
        at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:54)
        at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:719)
        at org.jvnet.hk2.internal.ProxyUtilities$3.run(ProxyUtilities.java:120)
        at java.security.AccessController.doPrivileged(AccessController.java:69)
        at org.jvnet.hk2.internal.ProxyUtilities.secureCreate(ProxyUtilities.java:115)
        at org.jvnet.hk2.internal.ProxyUtilities.generateProxy(ProxyUtilities.java:194)
        ... 70 more

@tomas-langer
Copy link
Member Author

Security - impacted by failure to inject @Context

@tomas-langer
Copy link
Member Author

JPA - currently fully dependent on compile time configuration. Working on a simple example

@tomas-langer
Copy link
Member Author

JPA - example with Hibernate. Fails on injection of transaction manager. Cannot easily workaround right now, as class JtaTransactionSupport is final and cannot be proxied.

@ljnelson ljnelson mentioned this issue Sep 26, 2019
Merged
@ljnelson
Copy link
Member

See #1061.

@ljnelson ljnelson mentioned this issue Sep 26, 2019
Closed
@tomas-langer
Copy link
Member Author

Security + Jersey server
Found necessary Proxy configuration + reflection configuration to support the injection used in the example.
Still todo - make sure all injectable objects can be injected (from Jersey)

@tomas-langer
Copy link
Member Author

Fallback in Fault tolerance now works.
Still todo - all FT annotations should be tested

@tomas-langer
Copy link
Member Author

REST client - the method handle must be used to invoke a default method on an interface that is used with java.lang.reflect.Proxy.
There is no other way available in java to do so.

@tomas-langer
Copy link
Member Author

JPA - resolved issue with proxies.
Next issue - ForkJoinPool used in image

@tomas-langer
Copy link
Member Author

JPA - resolved issue with ForkJoinPool
Next issue - reflection in XML parsing - probably needs to be in runtime, as the persistence unit configuration may be runtime specific (e.g. not available compile time). Needs further investigation

@tomas-langer
Copy link
Member Author

JPA ForkJoinPool substituted
Next issue - Hibernate cannot find ID annotation in entity class

@tomas-langer tomas-langer added this to To do in Helidon 2.0 via automation Oct 9, 2019
@tomas-langer tomas-langer added this to the 2.0.0 milestone Oct 9, 2019
@tomas-langer tomas-langer moved this from To do to "Big" changes in Helidon 2.0 Oct 9, 2019
@tomas-langer
Copy link
Member Author

Jersey - all injections supported by JAX-RS work in native image.
Created an issue for Jersey to get rid of unwanted dependencies on XML parsing and image processing: eclipse-ee4j/jersey#4293

@tomas-langer
Copy link
Member Author

JPA - resolved issue with ID annotation (all annotated fields, even private, are now added for reflection)
Next issue - finding out all classes that need to be added to reflection configuration

@tomas-langer
Copy link
Member Author

JPA - after adding all the reflection configuration, the runtime now fails with

Caused by: java.lang.IllegalArgumentException: Could not create type
	at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:154)
	at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:365)
	at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:174)
	at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:376)
	at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState.load(ByteBuddyState.java:175)
	at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState.loadProxy(ByteBuddyState.java:99)
	at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyProxyHelper.buildProxy(ByteBuddyProxyHelper.java:54)
	at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyProxyFactory.postInstantiate(ByteBuddyProxyFactory.java:61)
	at org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFactory(PojoEntityTuplizer.java:163)
	at org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:155)
	at org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:59)
	... 71 more

@tomas-langer
Copy link
Member Author

JPA - fixed problem above, disabled byte code manipulation. Now I get a connection pool.
Still problems with (most likely) reflection configuration resulting in NPE

@tomas-langer
Copy link
Member Author

JPA now works (disabled drop/create tables in Hibernate to fix the problem)

@tomas-langer
Copy link
Member Author

Now trying Fault tolerance annotations (other than @Fallback)

@tomas-langer
Copy link
Member Author

Rest client now works with the exception of usage of default methods from interfaces in a subset of configuration options of an annotation (not expected to be used frequently anyway).

@tomas-langer
Copy link
Member Author

tomas-langer commented Nov 21, 2019 •
edited

There are three issues with native-image that we know of that impact us:

  1. Default methods on interfaces in combination with a Proxy - method handle issue: workaround by substitution of the offending method
  2. Java 11 - Xerces resource bundle not found: workaround by including the resource bundle in our sources
  3. Java 11 - NPE when trying to get annotations of a package, impacts JSON-B: workaround not yet done (root cause is that ClassLoaders.bootLoader() returns null - also causes another NPE when using REST client)

@tomas-langer tomas-langer moved this from "Big" changes to In progress in Helidon 2.0 Nov 27, 2019
@tomas-langer
Copy link
Member Author

tomas-langer commented Dec 1, 2019 •
edited

Now the POC works, let's start adding these features to Helidon. The checked features are implemented and working. I will attach PR numbers to these once I can.

  • Weld
  • Jersey
  • Server
  • Jersey client
  • Jersey integration with CDI
  • REST Client - issue with MethodHandle
  • Metrics 2.0
  • Healthchecks
  • Tracing
  • Access Log
  • OpenAPI
  • Fault Tolerance - Fallback, Timeout, and Retry tested
  • Security (MP JWT Auth + others)
  • MP Config
  • Override config from MP Server (Helidon config)
  • Other media types (JSON-P, JSON-B etc.)
  • JPA
  • JTA

@m0mus m0mus added enhancement New feature or request epic labels Dec 12, 2019
@tomas-langer tomas-langer moved this from In Progress to Legendary in Backlog Dec 12, 2019
@tomas-langer tomas-langer mentioned this issue Dec 26, 2019
Merged
@m0mus m0mus added the 2.0-M1 label Jan 10, 2020
This was referenced Jan 13, 2020
Closed
Closed
Open
@m0mus m0mus closed this as completed Jun 18, 2020
Backlog automation moved this from Legendary to Closed Jun 18, 2020
Helidon 2.0 automation moved this from In progress to Done Jun 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomas-langer tomas-langer

Labels
enhancement New feature or request epic graal MP
Projects
Backlog
  
Closed
Helidon 2.0
  
Done
Milestone
2.0.0
Development

No branches or pull requests
3 participants
@ljnelson @m0mus @tomas-langer