NoClassDefFoundError when using S3 source connector with STS jar. #1536

dezka · 2023-06-16T16:32:21Z

Version 3.18.2 of the connector, Kafka Connect 2.8, Java 11, Kafka 2.8.

config:

{
  "connector.class": "org.apache.camel.kafkaconnector.awss3source.CamelAwss3sourceSourceConnector",
  "camel.kamelet.aws-s3-source.deleteAfterRead": "true",
  "camel.kamelet.aws-s3-source.bucketNameOrArn": "mybucket",
  "topics": "mytopic",
  "tasks.max": "1",
  "name": "s3-source-test",
  "camel.kamelet.aws-s3-source.prefix": "test/",
  "camel.kamelet.aws-s3-source.useDefaultCredentialsProvider": "true",
  "camel.kamelet.aws-s3-source.region": "us-east-1"
}

I am intending to use the Web Identity Token method of authing into AWS. The token file is present on the Kafka Connect instances and I am currently using it with other connectors. However I seem to be missing another JAR dependency here and I can't figure out which one.

Files:

LICENSE.txt                                         camel-core-processor-3.18.2.jar                     http-client-spi-2.17.250.jar                        netty-codec-http-4.1.79.Final.jar
NOTICE.txt                                          camel-core-reifier-3.18.2.jar                       httpclient-4.5.13.jar                               netty-codec-http2-4.1.79.Final.jar
README.adoc                                         camel-direct-3.18.2.jar                             httpcore-4.4.15.jar                                 netty-common-4.1.79.Final.jar
annotations-13.0.jar                                camel-dsl-support-3.18.2.jar                        jackson-annotations-2.13.3.jar                      netty-handler-4.1.79.Final.jar
annotations-2.17.223.jar                            camel-health-3.18.2.jar                             jackson-core-2.13.3.jar                             netty-nio-client-2.17.250.jar
apache-client-2.17.250.jar                          camel-kafka-3.18.2.jar                              jackson-databind-2.13.3.jar                         netty-resolver-4.1.79.Final.jar
apicurio-registry-common-1.3.2.Final.jar            camel-kafka-connector-3.18.2.jar                    jackson-dataformat-avro-2.13.3.jar                  netty-transport-4.1.79.Final.jar
apicurio-registry-rest-client-1.3.2.Final.jar       camel-kamelet-3.18.2.jar                            jackson-datatype-jdk8-2.10.2.jar                    netty-transport-classes-epoll-4.1.79.Final.jar
apicurio-registry-utils-converter-1.3.2.Final.jar   camel-main-3.18.2.jar                               jakarta.activation-1.2.2.jar                        netty-transport-native-unix-common-4.1.79.Final.jar
apicurio-registry-utils-serde-1.3.2.Final.jar       camel-management-api-3.18.2.jar                     jakarta.activation-api-1.2.2.jar                    okhttp-3.14.9.jar
arns-2.17.250.jar                                   camel-seda-3.18.2.jar                               jakarta.xml.bind-api-2.3.3.jar                      okio-1.17.2.jar
auth-2.17.250.jar                                   camel-support-3.18.2.jar                            javax.annotation-api-1.3.2.jar                      profiles-2.17.250.jar
avro-1.11.0.jar                                     camel-util-3.18.2.jar                               jaxb-core-2.3.0.jar                                 protobuf-java-3.13.0.jar
aws-core-2.17.250.jar                               camel-xml-io-util-3.18.2.jar                        jaxb-impl-2.3.3.jar                                 protocol-core-2.17.250.jar
aws-query-protocol-2.17.250.jar                     camel-xml-jaxb-3.18.2.jar                           jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar            reactive-streams-1.0.3.jar
aws-xml-protocol-2.17.250.jar                       camel-xml-jaxp-3.18.2.jar                           jctools-core-3.3.0.jar                              regions-2.17.250.jar
camel-api-3.18.2.jar                                camel-yaml-dsl-3.18.2.jar                           json-utils-2.17.250.jar                             retrofit-2.9.0.jar
camel-aws-s3-source-kafka-connector-3.18.2.jar      camel-yaml-dsl-common-3.18.2.jar                    kotlin-reflect-1.3.20.jar                           s3-2.17.250.jar
camel-aws2-s3-3.18.2.jar                            camel-yaml-dsl-deserializers-3.18.2.jar             kotlin-stdlib-1.3.20.jar                            sdk-core-2.17.250.jar
camel-base-3.18.2.jar                               commons-codec-1.15.jar                              kotlin-stdlib-common-1.3.20.jar                     snakeyaml-engine-2.3.jar
camel-base-engine-3.18.2.jar                        commons-compress-1.21.jar                           medeia-validator-core-1.1.1.jar                     sts-2.17.250.jar
camel-bean-3.18.2.jar                               commons-logging-1.2.jar                             medeia-validator-jackson-1.1.1.jar                  third-party-jackson-core-2.17.250.jar
camel-core-engine-3.18.2.jar                        connect-json-2.6.0.jar                              metrics-spi-2.17.250.jar                            utils-2.17.223.jar
camel-core-languages-3.18.2.jar                     converter-jackson-2.9.0.jar                         netty-buffer-4.1.79.Final.jar
camel-core-model-3.18.2.jar                         eventstream-1.0.1.jar                               netty-codec-4.1.79.Final.jar

Here is the exception I'm getting:

java.lang.NoClassDefFoundError: software/amazon/awssdk/utils/cache/CachedSupplier$StaleValueBehavior
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:110)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:59)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider$BuilderImpl.build(InstanceProfileCredentialsProvider.java:377)
	at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.lambda$createChain$0(DefaultCredentialsProvider.java:104)
	at software.amazon.awssdk.utils.Lazy.getValue(Lazy.java:50)
	at software.amazon.awssdk.utils.Lazy.close(Lazy.java:71)
	at software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
	at software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
	at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.close(LazyAwsCredentialsProvider.java:50)
	at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.close(DefaultCredentialsProvider.java:128)
	at software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:70)
	at software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:87)
	at software.amazon.awssdk.utils.AttributeMap.lambda$close$0(AttributeMap.java:87)
	at java.base/java.util.HashMap$Values.forEach(HashMap.java:977)
	at software.amazon.awssdk.utils.AttributeMap.close(AttributeMap.java:87)
	at software.amazon.awssdk.core.client.config.SdkClientConfiguration.close(SdkClientConfiguration.java:79)
	at software.amazon.awssdk.core.internal.http.HttpClientDependencies.close(HttpClientDependencies.java:80)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient.close(AmazonSyncHttpClient.java:73)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.close(BaseSyncClientHandler.java:88)
	at software.amazon.awssdk.services.s3.DefaultS3Client.close(DefaultS3Client.java:11047)
	at org.apache.camel.component.aws2.s3.AWS2S3Endpoint.doStop(AWS2S3Endpoint.java:141)
	at org.apache.camel.support.service.BaseService.stop(BaseService.java:160)
	at org.apache.camel.support.service.ServiceHelper.stopService(ServiceHelper.java:162)
	at org.apache.camel.support.service.ServiceHelper.stopAndShutdownServices(ServiceHelper.java:257)
	at org.apache.camel.support.service.ServiceHelper.stopAndShutdownServices(ServiceHelper.java:215)
	at org.apache.camel.impl.engine.RouteService.doShutdown(RouteService.java:302)
	at org.apache.camel.support.ChildServiceSupport.shutdown(ChildServiceSupport.java:113)
	at org.apache.camel.support.service.ServiceHelper.stopAndShutdownService(ServiceHelper.java:233)
	at org.apache.camel.impl.engine.AbstractCamelContext.shutdownServices(AbstractCamelContext.java:3629)
	at org.apache.camel.impl.engine.AbstractCamelContext.shutdownServices(AbstractCamelContext.java:3654)
	at org.apache.camel.impl.engine.AbstractCamelContext.doStop(AbstractCamelContext.java:3434)
	at org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:152)
	at org.apache.camel.support.service.BaseService.stop(BaseService.java:160)
	at org.apache.camel.impl.engine.AbstractCamelContext.stop(AbstractCamelContext.java:2703)
	at org.apache.camel.main.SimpleMain.doStop(SimpleMain.java:58)
	at org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain.doStop(CamelKafkaConnectMain.java:67)
	at org.apache.camel.support.service.BaseService.stop(BaseService.java:160)
	at org.apache.camel.kafkaconnector.CamelSourceTask.stop(CamelSourceTask.java:303)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.close(WorkerSourceTask.java:173)
	at org.apache.kafka.connect.runtime.WorkerTask.doClose(WorkerTask.java:167)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:193)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:237)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)",

The text was updated successfully, but these errors were encountered:

oscerd · 2023-06-16T16:34:14Z

What is the class missing? NoClassDefFoundError is the exception, what is the message?

dezka · 2023-06-16T16:35:09Z

Sorry, missed it on the OP, updated it to include the exception.

oscerd · 2023-06-16T16:42:33Z

The error is this one:

utils-2.17.223.jar

The StaleValueBehavior is not present in 2.17.223: https://javadoc.io/doc/software.amazon.awssdk/utils/2.17.223/software/amazon/awssdk/utils/cache/CachedSupplier.html

But it's present in 2.17.250: https://javadoc.io/doc/software.amazon.awssdk/utils/2.17.250/software/amazon/awssdk/utils/cache/CachedSupplier.html

oscerd · 2023-06-16T16:48:08Z

There should have been something wrong with 3.18.2, because utils is misaligned with the other jars. Try with 3.20.3, where they are all aligned to 2.18.40

oscerd · 2023-06-16T16:52:45Z

or alternatively you should remove the 2.17.233 version of utils jar and use 2.17.250

dezka · 2023-06-16T17:40:47Z

👍 thanks. Using 3.20.3 works for me. Is this a problem that only presented itself because I'm trying to use the STS jar? Or did 3.18.2 already have this issue?

Is 3.20.3 production ready?

oscerd · 2023-06-16T17:48:41Z

I think so. It should be related to sts.

This project is mostly followed by volunteers. It's just a release with a good level of trust

dezka · 2023-06-16T18:28:45Z

Cool. Thanks for all the help!

dezka closed this as completed Jun 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NoClassDefFoundError when using S3 source connector with STS jar. #1536

NoClassDefFoundError when using S3 source connector with STS jar. #1536

dezka commented Jun 16, 2023 •

edited

oscerd commented Jun 16, 2023

dezka commented Jun 16, 2023

oscerd commented Jun 16, 2023

oscerd commented Jun 16, 2023

oscerd commented Jun 16, 2023

dezka commented Jun 16, 2023

oscerd commented Jun 16, 2023

dezka commented Jun 16, 2023

NoClassDefFoundError when using S3 source connector with STS jar. #1536

NoClassDefFoundError when using S3 source connector with STS jar. #1536

Comments

dezka commented Jun 16, 2023 • edited

oscerd commented Jun 16, 2023

dezka commented Jun 16, 2023

oscerd commented Jun 16, 2023

oscerd commented Jun 16, 2023

oscerd commented Jun 16, 2023

dezka commented Jun 16, 2023

oscerd commented Jun 16, 2023

dezka commented Jun 16, 2023

dezka commented Jun 16, 2023 •

edited