java.lang.ClassNotFoundException: io.quarkus.bootstrap.runner.QuarkusEntryPoint after k8s node update

[catshout]

Hi everybody.

Our k8s cluster has been updated yesterday to v1.22.6. After the update the Camel-K routes don't work anymore. We did create a sample route like

// camel-k: language=java

import org.apache.camel.builder.RouteBuilder;

public class Route extends RouteBuilder {
  @Override
  public void configure() throws Exception {

      // Write your routes here, for example:
      from("timer:java?period=1000")
        .routeId("java")
        .setBody()
          .simple("Hello Camel K from ${routeId}")
        .to("log:info");

  }
}

and did get the following error

kamel run Route.java -n integration-dev --dev
Integration "route" created
Progress: integration "route" in phase Initialization
Progress: integration "route" in phase Building Kit
Condition "IntegrationPlatformAvailable" is "True" for Integration route: integration-dev/camel-k
Integration "route" in phase "Initialization"
Integration "route" in phase "Building Kit"
Condition "IntegrationKitAvailable" is "False" for Integration route: creating a new integration kit
Integration Kit "kit-c8o9vr4ssjrf6und501g", created by Integration "route", changed phase to "Build Submitted"
Build "kit-c8o9vr4ssjrf6und501g", created by Integration "route", changed phase to "Scheduling"
Build "kit-c8o9vr4ssjrf6und501g", created by Integration "route", changed phase to "Pending"
Build "kit-c8o9vr4ssjrf6und501g", created by Integration "route", changed phase to "Running"
Integration Kit "kit-c8o9vr4ssjrf6und501g", created by Integration "route", changed phase to "Build Running"
Build "kit-c8o9vr4ssjrf6und501g", created by Integration "route", changed phase to "Succeeded"
Integration Kit "kit-c8o9vr4ssjrf6und501g", created by Integration "route", changed phase to "Ready"
Progress: integration "route" in phase Deploying
Condition "IntegrationKitAvailable" is "True" for Integration route: kit-c8o9vr4ssjrf6und501g
Integration "route" in phase "Deploying"
Progress: integration "route" in phase Running
Condition "CronJobAvailable" is "False" for Integration route: different controller strategy used (deployment)
Condition "DeploymentAvailable" is "True" for Integration route: deployment name is route
Condition "ServiceAvailable" is "False" for Integration route: no http service required
Condition "ExposureAvailable" is "False" for Integration route: no host or service defined
Condition "Ready" is "False" for Integration route: 0/1 updated replicas
Integration "route" in phase "Running"
Condition "CronJobAvailable" is "False" for Integration route: different controller strategy used (deployment)
Condition "DeploymentAvailable" is "True" for Integration route: deployment name is route
Condition "ServiceAvailable" is "False" for Integration route: no http service required
Condition "ExposureAvailable" is "False" for Integration route: no host or service defined
Condition "Ready" is "False" for Integration route: 0/1 updated replicas
Integration "route" in phase "Running"
Condition "Ready" is "False" for Integration route: 0/1 ready replicas
(combined from similar events): Condition "Ready" is "False" for Integration route
Integration "route" in phase "Error"
[1] Monitoring pod route-5b448fd659-ghgdr
[1] exec java -cp ./resources:/etc/camel/application.properties:/etc/camel/conf.d/_resources:/etc/camel/resources:/etc/camel/sources/Route.java:dependencies/app/camel-k-integration-1.8.0.jar:dependencies/lib/boot/io.quarkus.quarkus-bootstrap-runner-2.6.0.Final.jar:dependencies/lib/boot/io.quarkus.quarkus
-development-mode-spi-2.6.0.Final.jar:dependencies/lib/boot/io.smallrye.common.smallrye-common-io-1.8.0.jar:dependencies/lib/boot/org.glassfish.jakarta.json-1.1.6.jar:dependencies/lib/boot/org.graalvm.sdk.graal-sdk-21.3.0.jar:dependencies/lib/boot/org.jboss.logging.jboss-logging-3.4.2.Final.jar:dependenc
ies/lib/boot/org.jboss.logmanager.jboss-logmanager-embedded-1.0.9.jar:dependencies/lib/boot/org.wildfly.common.wildfly-common-1.5.4.Final-format-001.jar:dependencies/lib/main/io.quarkus.arc.arc-2.6.0.Final.jar:dependencies/lib/main/io.quarkus.quarkus-arc-2.6.0.Final.jar:dependencies/lib/main/io.quarkus.q
uarkus-core-2.6.0.Final.jar:dependencies/lib/main/io.quarkus.quarkus-fs-util-0.0.3.jar:dependencies/lib/main/io.quarkus.quarkus-jsonp-2.6.0.Final.jar:dependencies/lib/main/io.quarkus.quarkus-logging-json-2.6.0.Final.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-annotation-1.8.0.jar:depende
ncies/lib/main/io.smallrye.common.smallrye-common-classloader-1.8.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-constraint-1.8.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-expression-1.8.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-function-1.8.0.jar:d
ependencies/lib/main/io.smallrye.config.smallrye-config-2.7.0.jar:dependencies/lib/main/io.smallrye.config.smallrye-config-common-2.7.0.jar:dependencies/lib/main/io.smallrye.config.smallrye-config-core-2.7.0.jar:dependencies/lib/main/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:dependencies/lib/ma
in/jakarta.el.jakarta.el-api-3.0.3.jar:dependencies/lib/main/jakarta.enterprise.jakarta.enterprise.cdi-api-2.0.2.jar:dependencies/lib/main/jakarta.inject.jakarta.inject-api-1.0.jar:dependencies/lib/main/jakarta.interceptor.jakarta.interceptor-api-1.2.5.jar:dependencies/lib/main/jakarta.transaction.jakart
a.transaction-api-1.3.3.jar:dependencies/lib/main/org.apache.camel.camel-api-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-base-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-base-engine-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-bean-3.14.0.jar:dependencies/lib/main/org.a
pache.camel.camel-componentdsl-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-core-catalog-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-core-engine-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-core-languages-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-core-model
-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-core-processor-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-core-reifier-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-dsl-support-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-endpointdsl-3.14.0.jar:dependencies/lib/
main/org.apache.camel.camel-java-joor-dsl-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-log-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-main-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-management-api-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-microprofile-co
nfig-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-support-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-timer-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-tooling-model-3.14.0.jar:dependencies/lib/main/org.apache.camel.camel-util-3.14.0.jar:dependencies/lib/main/org.apache
.camel.camel-util-json-3.14.0.jar:dependencies/lib/main/org.apache.camel.k.camel-k-core-1.11.0.jar:dependencies/lib/main/org.apache.camel.k.camel-k-core-api-1.11.0.jar:dependencies/lib/main/org.apache.camel.k.camel-k-core-support-1.11.0.jar:dependencies/lib/main/org.apache.camel.k.camel-k-runtime-1.11.0.
jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-bean-2.6.0.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-core-2.6.0.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-java-joor-dsl-2.6.0.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-log
-2.6.0.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-timer-2.6.0.jar:dependencies/lib/main/org.eclipse.microprofile.config.microprofile-config-api-2.0.jar:dependencies/lib/main/org.eclipse.microprofile.context-propagation.microprofile-context-propagation-api-1.2.jar:dependencies/lib/ma
in/org.jboss.logging.jboss-logging-annotations-2.2.1.Final.jar:dependencies/lib/main/org.jboss.slf4j.slf4j-jboss-logmanager-1.1.0.Final.jar:dependencies/lib/main/org.jboss.spec.javax.xml.bind.jboss-jaxb-api_2.3_spec-2.0.0.Final.jar:dependencies/lib/main/org.jboss.threads.jboss-threads-3.4.2.Final.jar:dep
endencies/lib/main/org.jooq.joor-0.9.14.jar:dependencies/lib/main/org.slf4j.slf4j-api-1.7.30.jar:dependencies/quarkus-app-dependencies.txt:dependencies/quarkus-run.jar:dependencies/quarkus/generated-bytecode.jar:dependencies/quarkus/quarkus-application.dat io.quarkus.bootstrap.runner.QuarkusEntryPoint
[1] Error: Could not find or load main class io.quarkus.bootstrap.runner.QuarkusEntryPoint
[1] Caused by: java.lang.ClassNotFoundException: io.quarkus.bootstrap.runner.QuarkusEntryPoint
(combined from similar events): Condition "Ready" is "False" for Integration route: back-off 10s restarting failed container=integration pod=route-5b448fd659-ghgdr_integration-dev(96dfe073-f965-4d66-abe3-2009b7a2458d)
Condition "Ready" is "False" for Integration route: back-off 10s restarting failed container=integration pod=route-5b448fd659-ghgdr_integration-dev(96dfe073-f965-4d66-abe3-2009b7a2458d)
Condition "Ready" is "False" for Integration route
Run integration terminating

Any ideas what might be the reason for and how to fix it?

[catshout]

@squakez @astefanutti we did a lot of examination, with no success at all. Do you have an idea where we could search for the root cause of this behavior?

Camel Spring-Boot routes are running well in sam k8s Cluster as other Java programs. The nodes are still running on cgroup v1 level. What can we check further?

Many thanks for any hint in this case.

[catshout]

We did proof that Camel-K and the Integration are running on AWS cluster 1.21 but not on 1.22. So again .. any help to debug the issue would help.

[oscerd]

The only thing comes to my mind, is we updated the Kubernetes API to version 1.21.4 and it's in 1.8.2, so there wasn't any kind of work on 1.22.x

[oscerd]

#2655

[catshout]

@oscerd I did try Camel-K on GKE with Node version v1.22.6-gke.300, runs fine. Just with Camel-K 1.8.0, same as on our client's environment.

Any further hints where to find additional debug information? k8s logs may be?

[catshout]

Finally we did find the reason, that in our (managed) cluster environment the RBAC has been changed so that the Operator wasn't allowed to load dependencies further.

For an improvement here it would be hepful to see such information within the Operator logs.

[tadayosi]

@catshout What was changed in the RBAC? Depending on the info, we might be able to find a way to improve the operator so that it can show some helpful message in the log.

[github-actions]

This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!