Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Define how to package the connectors #127

Closed
oscerd opened this issue Apr 16, 2020 · 7 comments · Fixed by #131
Closed

Define how to package the connectors #127

oscerd opened this issue Apr 16, 2020 · 7 comments · Fixed by #131

Comments

@oscerd
Copy link
Contributor

oscerd commented Apr 16, 2020

We should do a zip for each connector, we can take as example this https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.1.0.Final/debezium-connector-postgres-1.1.0.Final-plugin.tar.gz

Lets collect ideas here.

ref: https://issues.redhat.com/browse/DBZ-1656
@omarsmak
Copy link
Member

+1. I think each connector should independent as much as possible

@gunnarmorling
Copy link

👍 In Debezium, we create both, *.zip and *.tar.gz, so people can choose whichever they prefer. They are self-contained and can be extracted into the KC plug-in path.

@orpiske
Copy link
Contributor

orpiske commented Apr 16, 2020

+1 for creating separate zip and tar.gz files.

I think it would be great if we could package them in a way that we can just drop the connector code into the plugin path and the connector dependencies in whatever is loaded into the KC classpath. I'm thinking about making it easier - for the user - to avoid potential class path issues and that annoying DelegatingClassLoader error.

@jpechane
Copy link

@orpiske +1
In fact it would be doable even now just with Maven assembly plug-in as all deps could be downloaded automatically.

The only issue I see now is that the dependencies are not exactly correct.

We have an example here - https://github.com/debezium/debezium-examples/tree/master/camel-kafka-connect it is a demo of Debezium to CoAP server. The connector distro is build using https://github.com/debezium/debezium-examples/tree/master/camel-kafka-connect/coap-connector-dist

If you look at the deps list

mvn dependency:tree
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------< io.debezium.examples.camel:coap-connector-dist >-----------
[INFO] Building CoAP Server 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ coap-connector-dist ---
[INFO] io.debezium.examples.camel:coap-connector-dist:jar:1.0.0-SNAPSHOT
[INFO] \- org.apache.camel.kafkaconnector:camel-coap-kafka-connector:jar:0.0.1-SNAPSHOT:compile
[INFO]    +- org.apache.camel:camel-coap:jar:3.0.1:compile
[INFO]    |  +- org.apache.camel:camel-support:jar:3.0.1:compile
[INFO]    |  |  +- org.apache.camel:camel-api:jar:3.0.1:compile
[INFO]    |  |  +- org.apache.camel:camel-management-api:jar:3.0.1:compile
[INFO]    |  |  +- org.apache.camel:camel-util:jar:3.0.1:compile
[INFO]    |  |  +- org.apache.camel:camel-util-json:jar:3.0.1:compile
[INFO]    |  |  \- org.slf4j:slf4j-api:jar:1.7.29:compile
[INFO]    |  +- org.eclipse.californium:californium-core:jar:2.0.0-M16:compile
[INFO]    |  |  +- org.eclipse.californium:californium-legal:jar:2.0.0-M16:runtime
[INFO]    |  |  +- org.eclipse.californium:element-connector:jar:2.0.0-M16:compile
[INFO]    |  |  \- org.eclipse.californium:element-connector:test-jar:tests:2.0.0-M16:compile
[INFO]    |  +- org.eclipse.californium:scandium:jar:2.0.0-M16:compile
[INFO]    |  \- org.eclipse.californium:element-connector-tcp:jar:2.0.0-M16:compile
[INFO]    |     +- io.netty:netty-handler:jar:4.1.25.Final:compile
[INFO]    |     |  +- io.netty:netty-buffer:jar:4.1.25.Final:compile
[INFO]    |     |  |  \- io.netty:netty-common:jar:4.1.25.Final:compile
[INFO]    |     |  \- io.netty:netty-codec:jar:4.1.25.Final:compile
[INFO]    |     \- io.netty:netty-transport:jar:4.1.25.Final:compile
[INFO]    |        \- io.netty:netty-resolver:jar:4.1.25.Final:compile
[INFO]    \- org.apache.camel.kafkaconnector:camel-kafka-connector:jar:0.0.1-SNAPSHOT:compile
[INFO]       +- org.apache.camel:camel-core:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-core-engine:jar:3.0.1:compile
[INFO]       |  |  +- org.apache.camel:spi-annotations:jar:3.0.1:compile
[INFO]       |  |  +- org.apache.camel:camel-base:jar:3.0.1:compile
[INFO]       |  |  \- org.apache.camel:camel-jaxp:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-bean:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-browse:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-caffeine-lrucache:jar:3.0.1:compile
[INFO]       |  |  \- com.github.ben-manes.caffeine:caffeine:jar:2.8.0:compile
[INFO]       |  +- org.apache.camel:camel-controlbus:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-dataformat:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-dataset:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-direct:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-directvm:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-file:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-language:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-log:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-mock:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-ref:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-rest:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-saga:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-scheduler:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-seda:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-stub:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-timer:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-validator:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-vm:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-xpath:jar:3.0.1:compile
[INFO]       |  \- org.apache.camel:camel-xslt:jar:3.0.1:compile
[INFO]       +- org.apache.camel:camel-main:jar:3.0.1:compile
[INFO]       \- org.apache.camel:camel-elasticsearch-rest:jar:3.0.1:compile
[INFO]          +- org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:7.3.2:compile
[INFO]          |  +- org.elasticsearch:elasticsearch:jar:7.3.2:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-core:jar:7.3.2:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-secure-sm:jar:7.3.2:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-x-content:jar:7.3.2:compile
[INFO]          |  |  |  +- org.yaml:snakeyaml:jar:1.17:compile
[INFO]          |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.8.11:compile
[INFO]          |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.8.11:compile
[INFO]          |  |  |  \- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.11:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-geo:jar:7.3.2:compile
[INFO]          |  |  +- org.apache.lucene:lucene-core:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-analyzers-common:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-backward-codecs:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-grouping:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-highlighter:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-join:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-memory:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-misc:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-queries:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-queryparser:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-sandbox:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-spatial:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-spatial-extras:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-spatial3d:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-suggest:jar:8.1.0:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-cli:jar:7.3.2:compile
[INFO]          |  |  |  \- net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile
[INFO]          |  |  +- com.carrotsearch:hppc:jar:0.8.1:compile
[INFO]          |  |  +- joda-time:joda-time:jar:2.10.2:compile
[INFO]          |  |  +- com.tdunning:t-digest:jar:3.2:compile
[INFO]          |  |  +- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile
[INFO]          |  |  +- org.apache.logging.log4j:log4j-api:jar:2.11.1:compile
[INFO]          |  |  \- org.elasticsearch:jna:jar:4.5.1:compile
[INFO]          |  +- org.elasticsearch.client:elasticsearch-rest-client:jar:7.3.2:compile
[INFO]          |  |  +- org.apache.httpcomponents:httpasyncclient:jar:4.1.4:compile
[INFO]          |  |  \- org.apache.httpcomponents:httpcore-nio:jar:4.4.11:compile
[INFO]          |  +- org.elasticsearch.plugin:parent-join-client:jar:7.3.2:compile
[INFO]          |  +- org.elasticsearch.plugin:aggs-matrix-stats-client:jar:7.3.2:compile
[INFO]          |  +- org.elasticsearch.plugin:rank-eval-client:jar:7.3.2:compile
[INFO]          |  \- org.elasticsearch.plugin:lang-mustache-client:jar:7.3.2:compile
[INFO]          |     \- com.github.spullara.mustache.java:compiler:jar:0.9.3:compile
[INFO]          +- org.elasticsearch.client:elasticsearch-rest-client-sniffer:jar:7.3.2:compile
[INFO]          |  +- org.apache.httpcomponents:httpclient:jar:4.5.8:compile
[INFO]          |  +- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO]          |  +- commons-codec:commons-codec:jar:1.11:compile
[INFO]          |  +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO]          |  \- com.fasterxml.jackson.core:jackson-core:jar:2.8.11:compile
[INFO]          \- com.fasterxml.jackson.core:jackson-databind:jar:2.10.1:compile
[INFO]             \- com.fasterxml.jackson.core:jackson-annotations:jar:2.10.1:compile
[INFO] ------------------------------------------------------------------------

there are two majo issues - ES introduced via `org.apache.camel.kafkaconnector:camel-kafka-connector` and the dependency on `org.apache.camel:camel-core`, maybe `org.apache.camel:camel-core-engine` would be enough or many of the compoenents that are dependency of the `camel-core` should be excluded.
When done you'd have a nicely trimmed per-connector installation package.

@oscerd
Copy link
Contributor Author

oscerd commented Apr 20, 2020

The elasticsearch stuff is part of the camel component, it cannot be excluded directly, if you exclude elasticsearch, you'll need to declare it explicitly, with the exact same version used by the component.

@oscerd
Copy link
Contributor Author

oscerd commented Apr 20, 2020

ah, no, it's in all connectors. This must be fixed.

@oscerd
Copy link
Contributor Author

oscerd commented Apr 20, 2020

camel-elasticsearch-rest is not there anymore.

@oscerd oscerd mentioned this issue Apr 20, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Packaging the connectors as zip and tar.gz apache/camel-kafka-connector
5 participants
@gunnarmorling @jpechane @omarsmak @orpiske @oscerd