Skip to content
Permalink
Browse files
AWS* Examples: Moved to 0.9.0
  • Loading branch information
oscerd committed Apr 6, 2021
1 parent c1f394b commit 1279ee0b33624e5396082d92d60177b0437ccf11
Showing 21 changed files with 114 additions and 114 deletions.
@@ -36,10 +36,10 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-sjms2-kafka-connector/0.8.0/camel-sjms2-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-sjms2-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.8.0/camel-aws-s3-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws-s3-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-sjms2-kafka-connector/0.9.0/camel-sjms2-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-sjms2-kafka-connector-0.9.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.9.0/camel-aws-s3-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws-s3-kafka-connector-0.9.0-package.tar.gz
```

These are the basic connectors. For camel-sjms2 we have a bunch of provided dependencies we need to add in our path, so run the following commands:
@@ -24,8 +24,8 @@ Download the connector package tar.gz and extract the content to a directory. In
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.9.0/camel-aws2-iam-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.9.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -24,8 +24,8 @@ Download the connector package tar.gz and extract the content to a directory. In
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.9.0/camel-aws2-iam-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.9.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -25,8 +25,8 @@ Download the connector package tar.gz and extract the content to a directory. In
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.9.0/camel-aws2-iam-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.9.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -25,8 +25,8 @@ Download the connector package tar.gz and extract the content to a directory. In
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.9.0/camel-aws2-iam-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.9.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -30,8 +30,8 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-firehose-kafka-connector/0.8.0/camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-firehose-kafka-connector/0.9.0/camel-aws2-kinesis-firehose-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-kinesis-firehose-kafka-connector-0.9.0-package.tar.gz
```

On AWS console create a Kinesis firehose delivery stream named firehose-stream and choose to send it to an S3 bucket from the available list (in the same region)
@@ -146,8 +146,8 @@ So we need to do something like this:

```
> cd my-connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-firehose-kafka-connector/0.8.0/camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-firehose-kafka-connector/0.9.0/camel-aws2-kinesis-firehose-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-kinesis-firehose-kafka-connector-0.9.0-package.tar.gz
```

Now we can start the build
@@ -172,7 +172,7 @@ You should see something like this:

[source,json,options="nowrap"]
----
[{"class":"org.apache.camel.kafkaconnector.CamelSinkConnector","type":"sink","version":"0.8.0"},{"class":"org.apache.camel.kafkaconnector.CamelSourceConnector","type":"source","version":"0.8.0"},{"class":"org.apache.camel.kafkaconnector.aws2kinesisfirehose.CamelAws2kinesisfirehoseSinkConnector","type":"sink","version":"0.8.0"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"2.5.0"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"2.5.0"},{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"1"}]
[{"class":"org.apache.camel.kafkaconnector.CamelSinkConnector","type":"sink","version":"0.9.0"},{"class":"org.apache.camel.kafkaconnector.CamelSourceConnector","type":"source","version":"0.9.0"},{"class":"org.apache.camel.kafkaconnector.aws2kinesisfirehose.CamelAws2kinesisfirehoseSinkConnector","type":"sink","version":"0.9.0"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"2.5.0"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"2.5.0"},{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"1"}]
----

### Set the AWS credential as secret (optional)
@@ -30,8 +30,8 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-kafka-connector/0.8.0/camel-aws2-kinesis-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kinesis-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-kafka-connector/0.9.0/camel-aws2-kinesis-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-kinesis-kafka-connector-0.9.0-package.tar.gz
```

On AWS console create a Kinesis stream delivery stream named streamTest.
@@ -92,7 +92,7 @@ After this we'll need to perform a get-records operation
"Data": "aGVsbG8gdGhlcmU=",
"PartitionKey": "partition-1",
"ApproximateArrivalTimestamp": 1610729857.904,
"SequenceNumber": "4961458467700449568901978308705626930.8.0414429070721026"
"SequenceNumber": "4961458467700449568901978308705626930.9.0414429070721026"
},
{
"Data": "aGVsbG8gdGhlcmU=",
@@ -28,7 +28,7 @@ and set the `plugin.path` property to your choosen location
You'll need to build your connector starting from an archetype:

```
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.8.0
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.9.0
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
@@ -42,34 +42,34 @@ You'll need to build your connector starting from an archetype:
[INFO]
[INFO] --- maven-archetype-plugin:3.2.0:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.8.0] found in catalog remote
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.9.0] found in catalog remote
Define value for property 'groupId': org.apache.camel.kafkaconnector
Define value for property 'artifactId': aws2-kinesis-extended
Define value for property 'version' 1.0-SNAPSHOT: : 0.8.0
Define value for property 'version' 1.0-SNAPSHOT: : 0.9.0
Define value for property 'package' org.apache.camel.kafkaconnector: :
Define value for property 'camel-kafka-connector-name': camel-aws2-kinesis-kafka-connector
[INFO] Using property: camel-kafka-connector-version = 0.8.0
[INFO] Using property: camel-kafka-connector-version = 0.9.0
Confirm properties configuration:
groupId: org.apache.camel.kafkaconnector
artifactId: aws2-kinesis-extended
version: 0.8.0
version: 0.9.0
package: org.apache.camel.kafkaconnector
camel-kafka-connector-name: camel-aws2-kinesis-kafka-connector
camel-kafka-connector-version: 0.8.0
camel-kafka-connector-version: 0.9.0
Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.8.0
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.9.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: artifactId, Value: aws2-kinesis-extended
[INFO] Parameter: version, Value: 0.8.0
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: packageInPathFormat, Value: org/apache/camel/kafkaconnector
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: version, Value: 0.8.0
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: camel-kafka-connector-name, Value: camel-aws2-kinesis-kafka-connector
[INFO] Parameter: camel-kafka-connector-version, Value: 0.8.0
[INFO] Parameter: camel-kafka-connector-version, Value: 0.9.0
[INFO] Parameter: artifactId, Value: aws2-kinesis-extended
[INFO] Project created from Archetype in dir: /home/oscerd/workspace/miscellanea/aws2-kinesis-extended
[INFO] ------------------------------------------------------------------------
@@ -255,7 +255,7 @@ so that each one is in its own subfolder
In this case we need to extend an existing connector and add a transform, so we need to leverage the archetype

```
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.8.0
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.9.0
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
@@ -269,34 +269,34 @@ In this case we need to extend an existing connector and add a transform, so we
[INFO]
[INFO] --- maven-archetype-plugin:3.2.0:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.8.0] found in catalog remote
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.9.0] found in catalog remote
Define value for property 'groupId': org.apache.camel.kafkaconnector
Define value for property 'artifactId': aws2-kinesis-extended
Define value for property 'version' 1.0-SNAPSHOT: : 0.8.0
Define value for property 'version' 1.0-SNAPSHOT: : 0.9.0
Define value for property 'package' org.apache.camel.kafkaconnector: :
Define value for property 'camel-kafka-connector-name': camel-aws2-kinesis-kafka-connector
[INFO] Using property: camel-kafka-connector-version = 0.8.0
[INFO] Using property: camel-kafka-connector-version = 0.9.0
Confirm properties configuration:
groupId: org.apache.camel.kafkaconnector
artifactId: aws2-kinesis-extended
version: 0.8.0
version: 0.9.0
package: org.apache.camel.kafkaconnector
camel-kafka-connector-name: camel-aws2-kinesis-kafka-connector
camel-kafka-connector-version: 0.8.0
camel-kafka-connector-version: 0.9.0
Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.8.0
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.9.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: artifactId, Value: aws2-kinesis-extended
[INFO] Parameter: version, Value: 0.8.0
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: packageInPathFormat, Value: org/apache/camel/kafkaconnector
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: version, Value: 0.8.0
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: camel-kafka-connector-name, Value: camel-aws2-kinesis-kafka-connector
[INFO] Parameter: camel-kafka-connector-version, Value: 0.8.0
[INFO] Parameter: camel-kafka-connector-version, Value: 0.9.0
[INFO] Parameter: artifactId, Value: aws2-kinesis-extended
[INFO] Project created from Archetype in dir: /home/oscerd/workspace/miscellanea/aws2-kinesis-extended
[INFO] ------------------------------------------------------------------------
@@ -398,7 +398,7 @@ You should see something like this:

[source,json,options="nowrap"]
----
[{"class":"org.apache.camel.kafkaconnector.CamelSinkConnector","type":"sink","version":"0.8.0"},{"class":"org.apache.camel.kafkaconnector.CamelSourceConnector","type":"source","version":"0.8.0"},{"class":"org.apache.camel.kafkaconnector.aws2kinesis.CamelAws2kinesisSinkConnector","type":"sink","version":"0.8.0"},{"class":"org.apache.camel.kafkaconnector.aws2kinesis.CamelAws2kinesisSourceConnector","type":"source","version":"0.8.0"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"2.5.0"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"2.5.0"},{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"1"}]
[{"class":"org.apache.camel.kafkaconnector.CamelSinkConnector","type":"sink","version":"0.9.0"},{"class":"org.apache.camel.kafkaconnector.CamelSourceConnector","type":"source","version":"0.9.0"},{"class":"org.apache.camel.kafkaconnector.aws2kinesis.CamelAws2kinesisSinkConnector","type":"sink","version":"0.9.0"},{"class":"org.apache.camel.kafkaconnector.aws2kinesis.CamelAws2kinesisSourceConnector","type":"source","version":"0.9.0"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"2.5.0"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"2.5.0"},{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"1"}]
----

### Set the AWS credential as secret (optional)
@@ -24,8 +24,8 @@ Download the connector package tar.gz and extract the content to a directory. In
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kms-kafka-connector/0.8.0/camel-aws2-kms-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kms-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kms-kafka-connector/0.9.0/camel-aws2-kms-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-kms-kafka-connector-0.9.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -24,8 +24,8 @@ Download the connector package tar.gz and extract the content to a directory. In
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kms-kafka-connector/0.8.0/camel-aws2-kms-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kms-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kms-kafka-connector/0.9.0/camel-aws2-kms-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-aws2-kms-kafka-connector-0.9.0-package.tar.gz
----

=== Configuring Kafka Connect

0 comments on commit 1279ee0

Please sign in to comment.