Not able to run aws s3 source connector #95
Comments
|
I read you're using the s3 confluent connector. It's not clear what are you trying to do, but mixing confluent connector and camel connector from this project is not tested or recommended |
|
No, I have added the |
|
Also in your pom you need to explicitly add the camel-aws-s3 dependency from camel 3.0.1 |
|
Let me explicitly add try the build. Thanks, I will update you with how it goes. |
|
Now, I am getting the following error: |
|
Can you please show your pom file? |
1 similar comment
|
Can you please show your pom file? |
|
As I am using the stimzi, I followed the documentation to deploy the connector to the my strimzi cluster. This is my docker File: SO I run this docker file and push the image to the internal gitlab registry. This is my connector deployment yaml file: I use the follwing command to deploy the connector: |
|
I'll try to reproduce |
|
Thank you!! |
|
Strimzi documentation, if it's useful: https://strimzi.io/docs/master/#kafka-connect-str |
|
So, I see the difference in the deployment of the |
|
@oscerd : Were you able to re-generate the error? |
|
I'll have a look tomorrow or this week |
|
Ok thanks |
|
Hello @ruchirvaninasdaq, are you 100% sure that when you have built the project with please note that Can you post the list of the content of the directory |
|
Hello @valdar : The pom file here ( camel-kafka-connector/core/pom.xml Line 82 in e507ea7 <scope>provided</scope>. I will try removing and rebuild again and update you with results.
Thanks |
|
I know that on master it has that scope, we should probably be more precise in the documentation. The fact is that we are also, slowly, producing one Kafka connector for each camel-component pre-packaged with all the needed dependencies and pre-configured to be simpler to use, see #15. So in the near future, all of this will not be necessary anymore |
|
These jars are in the folder: |
|
@valdar : I am still getting this error: |
|
You need to add camel-api jar. |
|
Is there a maven repo for that? |
|
and I just noticed that I have this in my plugins |
|
The class are all there, so your classpath is clearly not complete. |
|
By the way is probably something related to difference between openshift and kubernetes approach. |
|
@ruchirvaninasdaq I think now you have the correct jars, but for some reason, they are not picked up in the Kafka connect classpath. I see that the Strimzi doc mention this as a recipe to make a custom image for kafka connectors: I see 2 differences with your Dockerfile:
|
|
@valdar & @oscerd : Just to verify that I have all the right plugins files within my connect pod. I logged into it and verify. |
|
I few things I think may be relevant ... last time I built an image I had some problems with this.
I haven't built an image with latest patches from @valdar, so things may be different now, but might be good to check those items as well. |
|
Thanks @orpiske . I will try these 2 steps and post results after that. |
|
We need to document this. Thanks @ruchirvaninasdaq |
|
Do you want me create document around deployment on strimzi with kubereates? I have all the example files. |
|
You're welcome to work on that, you can take a look at the docs we have and start with that as a base document. Thanks. |
|
I think we can leave this open untill we fix the documentation. |
|
oscerd open the new issue: #96 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am trying to run the aws s3 source connector. I have followed this documetation to install connector:
https://camel.apache.org/camel-kafka-connector/latest/try-it-out-on-openshift-with-strimzi.html.
I have installed the connectors correctly as I can list it connecor:
This is the json for source connector:
Now when I am trying to print status for the connector, I get following error:
{"name":"s3-connector-camel","connector":{"state":"RUNNING","worker_id":"100.80.22.122:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"100.80.22.122:8083","trace":"org.apache.kafka.connect.errors.ConnectException: Failed to create and start Camel context\n\tat org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:98)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:199)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.IllegalArgumentException: Error configuring property: camel.component.aws-s3.configuration.secret-key because cannot find component with name aws-s3. Make sure you have the component on the classpath\n\tat org.apache.camel.main.BaseMainSupport.lambda$autoConfigurationFromProperties$14(BaseMainSupport.java:895)\n\tat org.apache.camel.main.BaseMainSupport.computeProperties(BaseMainSupport.java:1084)\n\tat org.apache.camel.main.BaseMainSupport.autoConfigurationFromProperties(BaseMainSupport.java:892)\n\tat org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:545)\n\tat org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:422)\n\tat org.apache.camel.main.Main.doInit(Main.java:108)\n\tat org.apache.camel.support.service.ServiceSupport.init(ServiceSupport.java:80)\n\tat org.apache.camel.support.service.ServiceSupport.start(ServiceSupport.java:108)\n\tat org.apache.camel.main.MainSupport.run(MainSupport.java:77)\n\tat org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:214)\n\t... 3 more\n"}],"type":"source"}The text was updated successfully, but these errors were encountered: