feat: Quarkus native build support #2536
Conversation
|
Tested on OCP 4.7.19, hosted on IBM Cloud, for a simple YAML Integration: The build time (including Maven dependencies download) is around 6m36s: Operator Pod, running the native build:
Integration Pod (Native):
Integration Pod (JVM):
|
|
@astefanutti adding js to the mix is gogin to require much more time and resources, I remember it required ~12 GB btw, we can probably add the required additional-build-args as part of the build step for the js dsl |
Yes, I've noticed the build time increases by a couple of minutes. I agree the |
|
Whao nice to see. So all the extra memory and cpu you pay for the native build, is maybe saved after 1000 years of running ;) |
|
I've just redone the same test after upgrading to GraalVM 21.2.0, using the Operator Pod:
We'll have to do some more testing to see whether that's consistent. There may also be some ways to speed up the build, like scaling vertically. |
Yeah, this could be because the type of the body, should be registered for reflection, as far as I remember adding something like: Should fix the issue |
may be worth adding a log or something about resource usage when doing native builds :D |
Or if we get support for csimple with camel-k, then it can be compiled as plain java code, like the Java DSL. |
I wanted to add a quantum node to the cluster to achieve a quantum Quarkus native build, but I did not found the option. Maybe it's somewhere and somewhere else at the same time, like I need to solve the wave function 😄. |
|
I'm still getting: But maybe I've added the build property in the wrong place... |
There is a chance that java.lang. classes are not included as not found by jandex, @ppalaga may know more |
|
Here is the exception that's thrown at start time, with a Java Integration: $ kamel run examples/rest/RestDSL.java
ERROR [org.apa.cam.qua.mai.CamelMainRuntime] (main) Failed to start application: org.apache.camel.RuntimeCamelException: org.joor.ReflectException: Error while compiling RestDSL
at org.apache.camel.k.support.SourcesSupport.load(SourcesSupport.java:170)
at org.apache.camel.k.support.SourcesSupport.loadSources(SourcesSupport.java:84)
at org.apache.camel.k.listener.SourcesConfigurer.accept(SourcesConfigurer.java:80)
at org.apache.camel.k.listener.AbstractPhaseListener.accept(AbstractPhaseListener.java:32)
at org.apache.camel.k.quarkus.Application$ListenerAdapter.invokeListeners(Application.java:135)
at org.apache.camel.k.quarkus.Application$ListenerAdapter.beforeConfigure(Application.java:97)
at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:531)
at org.apache.camel.quarkus.main.CamelMain.initCamelContext(CamelMain.java:116)
at org.apache.camel.quarkus.main.CamelMain.doInit(CamelMain.java:86)
at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
at org.apache.camel.quarkus.main.CamelMain.startEngine(CamelMain.java:137)
at org.apache.camel.quarkus.main.CamelMainRuntime.start(CamelMainRuntime.java:49)
at org.apache.camel.quarkus.core.CamelBootstrapRecorder.start(CamelBootstrapRecorder.java:45)
at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot-173480958.deploy_0(CamelBootstrapProcessor$boot-173480958.zig:101)
at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot-173480958.deploy(CamelBootstrapProcessor$boot-173480958.zig:40)
at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:799)
at io.quarkus.runtime.Application.start(Application.java:101)
at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:101)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:66)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:42)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:119)
at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:29)
Caused by: org.joor.ReflectException: Error while compiling RestDSL
at org.joor.Compile.compile(Compile.java:158)
at org.joor.Reflect.compile(Reflect.java:104)
at org.joor.Reflect.compile(Reflect.java:79)
at org.apache.camel.dsl.java.joor.JavaRoutesBuilderLoader.doLoadRouteBuilder(JavaRoutesBuilderLoader.java:50)
at org.apache.camel.dsl.support.RouteBuilderLoaderSupport.loadRoutesBuilder(RouteBuilderLoaderSupport.java:59)
at org.apache.camel.impl.engine.DefaultRoutesLoader.findRoutesBuilders(DefaultRoutesLoader.java:100)
at org.apache.camel.spi.RoutesLoader.findRoutesBuilders(RoutesLoader.java:66)
at org.apache.camel.k.support.SourcesSupport.load(SourcesSupport.java:163)
... 21 more
Caused by: java.lang.NullPointerException
at org.joor.Compile.compile(Compile.java:66)
... 28 moreI'm not sure JOOR is supposed to work in combination of native builds. |
no, only xml, yaml, js are supposed to work in native mode |
Yep, that's perhaps the case. Could you try to register via |
Thanks, I'm going to try |
|
So generating the following file during the project generation [
{
"name" : "java.lang.String",
"allPublicMethods" : true
}
]And adding the $ kamel run examples/languages/routes.yaml --logs
Integration "routes" created
Condition "IntegrationPlatformAvailable" is "True" for Integration routes: astefanu/camel-k
Integration routes in phase "Initialization"
Integration routes in phase "Building Kit"
Condition "IntegrationKitAvailable" is "False" for Integration routes: creating a new integration kit
Integration routes subresource kit-c43sgj6vs72714dt3jsg (Integration Kit) changed phase to "Build Submitted"
Integration routes subresource kit-c43sgj6vs72714dt3jsg (Build) changed phase to "Scheduling"
Integration routes subresource kit-c43sgj6vs72714dt3jsg (Build) changed phase to "Pending"
Integration routes subresource kit-c43sgj6vs72714dt3jsg (Build) changed phase to "Running"
Integration routes subresource kit-c43sgj6vs72714dt3jsg (Integration Kit) changed phase to "Build Running"
Integration routes subresource kit-c43sgj6vs72714dt3jsg (Build) changed phase to "Succeeded"
Integration routes subresource kit-c43sgj6vs72714dt3jsg (Integration Kit) changed phase to "Ready"
Condition "IntegrationKitAvailable" is "True" for Integration routes: kit-c43sgj6vs72714dt3jsg
Integration routes in phase "Deploying"
Condition "DeploymentAvailable" is "True" for Integration routes: deployment name is routes
Condition "CronJobAvailable" is "False" for Integration routes: different controller strategy used (deployment)
Condition "KnativeServiceAvailable" is "False" for Integration routes: different controller strategy used (deployment)
Integration routes in phase "Running"
Condition "Ready" is "False" for Integration routes
[1] Monitoring pod routes-66886d9564-swf9n
[1] 2021-08-02 10:30:48,198 INFO [org.apa.cam.k.Runtime] (main) Apache Camel K Runtime 1.8.0
[1] 2021-08-02 10:30:48,199 INFO [org.apa.cam.qua.cor.CamelBootstrapRecorder] (main) Bootstrap runtime: org.apache.camel.quarkus.main.CamelMainRuntime
[1] 2021-08-02 10:30:48,199 INFO [org.apa.cam.k.lis.SourcesConfigurer] (main) Loading routes from: SourceDefinition{name='camel-k-embedded-flow', language='yaml', type='source', location='file:/etc/camel/sources/camel-k-embedded-flow.yaml', }
[1] 2021-08-02 10:30:48,203 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Routes startup summary (total:1 started:1)
[1] 2021-08-02 10:30:48,203 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Started route1 (timer://tick)
[1] 2021-08-02 10:30:48,203 INFO [org.apa.cam.imp.eng.AbstractCamelContext] (main) Apache Camel 3.11.0 (camel-1) started in 1ms (build:0ms init:1ms start:0ms)
[1] 2021-08-02 10:30:48,204 INFO [io.quarkus] (main) camel-k-integration 1.6.0-SNAPSHOT native (powered by Quarkus 2.0.0.Final) started in 0.012s.
[1] 2021-08-02 10:30:48,204 INFO [io.quarkus] (main) Profile prod activated.
[1] 2021-08-02 10:30:48,204 INFO [io.quarkus] (main) Installed features: [camel-bean, camel-core, camel-k-core, camel-k-runtime, camel-log, camel-support-common, camel-timer, camel-yaml-dsl, cdi]
Condition "Ready" is "True" for Integration routes
[1] 2021-08-02 10:30:49,205 INFO [info] (Camel (camel-1) thread #0 - timer://tick) Exchange[ExchangePattern: InOnly, BodyType: String, Body: HELLO YAML !!!]
[1] 2021-08-02 10:30:54,203 INFO [info] (Camel (camel-1) thread #0 - timer://tick) Exchange[ExchangePattern: InOnly, BodyType: String, Body: HELLO YAML !!!]
[1] 2021-08-02 10:30:59,203 INFO [info] (Camel (camel-1) thread #0 - timer://tick) Exchange[ExchangePattern: InOnly, BodyType: String, Body: HELLO YAML !!!]The native build duration seems stable around 4m30s: $ kubectl get builds.camel.apache.org
NAME PHASE AGE STARTED DURATION ATTEMPTS
kit-c43s8si2qa460fs1tumg Succeeded 4m53s 4m53s 4m28.894542898s The |
This is a topic we have since the beginning of the camel-quarkus project :) however based on experience, is not always possible to determine the classes that have to be registered for reflection so I think @ppalaga I think that if the class the pattern given to the |
|
I have filed apache/camel-quarkus#2969 and apache/camel-quarkus#2970 |
|
There is a PR apache/camel-quarkus#2973 for apache/camel-quarkus#2970 |
|
It turns out Mandrel does not include any polyglot support, so the I've successfully built a YAML Integration with the On the pros side, the Mandrel image is almost half the size that of GraalVM, that is around 780MB/323MB (uncompressed/compressed). That adds about 300MB (uncompressed) compared to the existing operator image. The cons is that it doesn't support building polyglot integrations, but the toll is so important w.r.t. memory requirement and build duration, that we may not want to support it anyway. Noted that Java would not be supported in any case as JOOR is used for compilation at start time. |
|
@astefanutti does that mean that the js-dsl extension is not that important for you anymore? |
|
@ppalaga I'd say polyglot native build has too high a price at the moment to be included: +700MB operator/builder image size, x2 memory requirement, x2 build duration. I'm inclined to favour Mandrel, as it targets Quarkus specifically, and works with the YAML DSL. |
|
yeah focus on native for kamelets via yaml - and then later java pre-compiled (not to use joor) and also pre compiled csimple language to reduce overhead |
astefanutti
added
kind/feature
|
I think it's ready. Users can use the $ kamel run -t quarkus.package-type=nativeIt's possible to run both standard and native builds in parallel, and have a rollout deployment to the native one when it's ready, by running: $ kamel run -t quarkus.package-type=fast-jar -t quarkus.package-type=nativeA native build of a simple REST integration completes successfully in about 5 minutes on a typical node, inside a Pod with 4GiB memory limit. The good news is native build e2e tests pass in GH Actions hosted VMs, that have only 7GB of memory. The native integration base image is set to For the implementation / technical details, this PR:
A couple of improvements may be done later:
|
|
Is there documentation included, if not I think we should have a new page in the docs about this (also if we have in some trait, then this functionality does not stand out, and it may be worth to have a new "build" page (or better name) that introduces this, and can link to the trait for more details etc). |
Right, it's only documented in the Quarkus trait page at the moment, so it's a bit "hidden". I need to think a bit to find what would be the best place. There are a couple of other build-related pages in the Configuration section, so that may be a good place, and have all them grouped under a Build section with a new page about the native build. |
This PR adds Quarkus native build support.
I've tested it successfully on Minikube, CRC and OpenShift @ IBM Cloud, with a simple YAML Integration:
The first build duration is about 6m36s (including Maven dependencies download):
Also the build image size is reduced to 73.2MB when using the
quay.io/quarkus/quarkus-distroless-imagebase image.TODO:
Fixes #991.
Release Note