Skip to content
Permalink
Browse files
AWS Examples: Moved to tar.gz
  • Loading branch information
oscerd committed Feb 25, 2021
1 parent b64c695 commit 616bd3d26b7b66c98f25a0c4f3091dab09820acd
Showing 25 changed files with 104 additions and 104 deletions.
@@ -36,10 +36,10 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-sjms2-kafka-connector/0.8.0/camel-sjms2-kafka-connector-0.8.0-package.zip
> unzip camel-sjms2-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.8.0/camel-aws-s3-kafka-connector-0.8.0-package.zip
> unzip camel-aws-s3-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-sjms2-kafka-connector/0.8.0/camel-sjms2-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-sjms2-kafka-connector-0.8.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.8.0/camel-aws-s3-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws-s3-kafka-connector-0.8.0-package.tar.gz
```

These are the basic connectors. For camel-sjms2 we have a bunch of provided dependencies we need to add in our path, so run the following commands:
@@ -19,13 +19,13 @@ $KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --rep

=== Download the connector package

Download the connector package zip and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`

[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-iam-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -19,13 +19,13 @@ $KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --rep

=== Download the connector package

Download the connector package zip and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`

[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-iam-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -20,13 +20,13 @@ $KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --rep

=== Download the connector package

Download the connector package zip and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`

[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-iam-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -20,13 +20,13 @@ $KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --rep

=== Download the connector package

Download the connector package zip and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`

[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-iam-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-iam-kafka-connector/0.8.0/camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-iam-kafka-connector-0.8.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -30,8 +30,8 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-firehose-kafka-connector/0.8.0/camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-firehose-kafka-connector/0.8.0/camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.tar.gz
```

On AWS console create a Kinesis firehose delivery stream named firehose-stream and choose to send it to an S3 bucket from the available list (in the same region)
@@ -146,8 +146,8 @@ So we need to do something like this:

```
> cd my-connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-firehose-kafka-connector/0.8.0/camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-firehose-kafka-connector/0.8.0/camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kinesis-firehose-kafka-connector-0.8.0-package.tar.gz
```

Now we can start the build
@@ -30,8 +30,8 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-kafka-connector/0.8.0/camel-aws2-kinesis-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-kinesis-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-kafka-connector/0.8.0/camel-aws2-kinesis-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kinesis-kafka-connector-0.8.0-package.tar.gz
```

On AWS console create a Kinesis stream delivery stream named streamTest.
@@ -374,7 +374,7 @@ Now we need to build the connector:
> mvn clean package
```

And move the zip package in target to my-connectors folder and unzipped it.
And move the tar.gz package in target to my-connectors folder and untar.gzped it.

Now we can start the build

@@ -19,13 +19,13 @@ $KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --rep

=== Download the connector package

Download the connector package zip and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`

[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kms-kafka-connector/0.8.0/camel-aws2-kms-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-kms-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kms-kafka-connector/0.8.0/camel-aws2-kms-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kms-kafka-connector-0.8.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -19,13 +19,13 @@ $KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --rep

=== Download the connector package

Download the connector package zip and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`

[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kms-kafka-connector/0.8.0/camel-aws2-kms-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-kms-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kms-kafka-connector/0.8.0/camel-aws2-kms-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-kms-kafka-connector-0.8.0-package.tar.gz
----

=== Configuring Kafka Connect
@@ -29,8 +29,8 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-lambda-kafka-connector/0.8.0/camel-aws2-lambda-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-sns-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-lambda-kafka-connector/0.8.0/camel-aws2-lambda-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-sns-kafka-connector-0.8.0-package.tar.gz
```

### Deploying the AWS Lambda function
@@ -161,8 +161,8 @@ So we need to do something like this:

```
> cd my-connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-lambda-kafka-connector/0.8.0/camel-aws2-lambda-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-lambda-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-lambda-kafka-connector/0.8.0/camel-aws2-lambda-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-lambda-kafka-connector-0.8.0-package.tar.gz
```

Now we can start the build
@@ -29,8 +29,8 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-s3-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
```

Now we need to setup a MinIO instance
@@ -28,8 +28,8 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-s3-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
```

Now it's time to setup the connectors
@@ -145,8 +145,8 @@ So we need to do something like this:

```
> cd my-connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-s3-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
```

Now we can start the build
@@ -31,9 +31,9 @@ Since this is example is related to a new feature you'll need to build the lates
```
> cd <ckc_project>
> mvn clean package
> cp <ckc_project>/connectors/camel-aws2-s3-kafka-connector/target/camel-aws2-s3-kafka-connector-0.8.0-SNAPSHOT-package.zip /home/oscerd/connectors/
> cp <ckc_project>/connectors/camel-aws2-s3-kafka-connector/target/camel-aws2-s3-kafka-connector-0.8.0-SNAPSHOT-package.tar.gz /home/oscerd/connectors/
> cd /home/oscerd/connectors/
> unzip camel-aws2-s3-kafka-connector-0.8.0-SNAPSHOT-package.zip
> untar.gz camel-aws2-s3-kafka-connector-0.8.0-SNAPSHOT-package.tar.gz
```

Now it's time to setup the connectors
@@ -28,8 +28,8 @@ In this example we'll use `/home/oscerd/connectors/`

```
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-s3-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
```

Now it's time to setup the connectors
@@ -157,8 +157,8 @@ So we need to do something like this:

```
> cd my-connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.zip
> unzip camel-aws2-s3-kafka-connector-0.8.0-package.zip
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
```

Now we can start the build
@@ -102,7 +102,7 @@ Now we need to edit the POM
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-zipfile</artifactId>
<artifactId>camel-tar.gzfile</artifactId>
<version>3.4.2</version>
</dependency>
</dependencies>
@@ -111,20 +111,20 @@ Now we need to edit the POM
.
```

In the dependencies section you'll need to uncomment the aws2-s3 connector dependency and adding the camel-zipfile component
In the dependencies section you'll need to uncomment the aws2-s3 connector dependency and adding the camel-tar.gzfile component

Now we need to build the connector:

```
> mvn clean package
```

In this example we'll use `/home/oscerd/connectors/` as plugin.path, but we'll need the generated zip from the previois build
In this example we'll use `/home/oscerd/connectors/` as plugin.path, but we'll need the generated tar.gz from the previois build

```
> cd /home/oscerd/connectors/
> cp /home/workspace/miscellanea/aws2s3/target/aws2s3-1.0-SNAPSHOT-package.zip .
> unzip aws2s3-1.0-SNAPSHOT-package.zip
> cp /home/workspace/miscellanea/aws2s3/target/aws2s3-1.0-SNAPSHOT-package.tar.gz .
> untar.gz aws2s3-1.0-SNAPSHOT-package.tar.gz
```

Now it's time to setup the connectors
@@ -145,9 +145,9 @@ camel.component.aws2-s3.accessKey=xxxx
camel.component.aws2-s3.secretKey=yyyy
camel.component.aws2-s3.region=eu-west-1

camel.sink.endpoint.keyName=${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.zip
camel.sink.endpoint.keyName=${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.tar.gz

camel.beans.aggregate=#class:org.apache.camel.processor.aggregate.zipfile.ZipAggregationStrategy
camel.beans.aggregate=#class:org.apache.camel.processor.aggregate.tar.gzfile.ZipAggregationStrategy
camel.aggregation.size=10
camel.aggregation.timeout=5000
```
@@ -173,7 +173,7 @@ Kafka to S3 message 4
Kafka to S3 message 5
```

You should see (after the timeout has been reached) a file with date-exchangeId.zip name containing the following multiple files. Those files will contain the messages.
You should see (after the timeout has been reached) a file with date-exchangeId.tar.gz name containing the following multiple files. Those files will contain the messages.

```
Kafka to S3 message 1
@@ -326,7 +326,7 @@ Now we need to edit the POM
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-zipfile</artifactId>
<artifactId>camel-tar.gzfile</artifactId>
<version>3.5.0</version>
</dependency>
</dependencies>
@@ -341,7 +341,7 @@ Now we need to build the connector:
> mvn clean package
```

And move the zip package in targe to my-connectors folder and unzipped it.
And move the tar.gz package in targe to my-connectors folder and untar.gzped it.

Now we can start the build

@@ -415,8 +415,8 @@ oc exec -i `oc get pods --field-selector status.phase=Running -l strimzi.io/name
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"topics": "s3-topic",
"camel.sink.path.bucketNameOrArn": "camel-kafka-connector",
"camel.sink.endpoint.keyName": "${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.zip",
"camel.beans.aggregate": "#class:org.apache.camel.processor.aggregate.zipfile.ZipAggregationStrategy",
"camel.sink.endpoint.keyName": "${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.tar.gz",
"camel.beans.aggregate": "#class:org.apache.camel.processor.aggregate.tar.gzfile.ZipAggregationStrategy",
"camel.aggregation.size": "10",
"camel.aggregation.timeout": "5000",
"camel.component.aws2-s3.accessKey": "xxx",
@@ -447,8 +447,8 @@ spec:
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: s3-topic
camel.sink.path.bucketNameOrArn: camel-kafka-connector
camel.sink.endpoint.keyName: ${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.zip
camel.beans.aggregate: #class:org.apache.camel.processor.aggregate.zipfile.ZipAggregationStrategy
camel.sink.endpoint.keyName: ${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.tar.gz
camel.beans.aggregate: #class:org.apache.camel.processor.aggregate.tar.gzfile.ZipAggregationStrategy
camel.aggregation.size: 10
camel.aggregation.timeout: 5000
camel.component.aws2-s3.accessKey: xxxx
@@ -461,7 +461,7 @@ If you followed the optional step for secret credentials you can run the followi

[source,bash,options="nowrap"]
----
oc apply -f config/openshift/aws2-s3-sink-with-zip-aggregation.yaml
oc apply -f config/openshift/aws2-s3-sink-with-tar.gz-aggregation.yaml
----

You can check the status of the connector using
@@ -484,7 +484,7 @@ Kafka to S3 message 4
Kafka to S3 message 5
```

You should see (after the timeout has been reached) a file with date-exchangeId.zip name containing the following multiple files. Those files will contain the messages.
You should see (after the timeout has been reached) a file with date-exchangeId.tar.gz name containing the following multiple files. Those files will contain the messages.

```
Kafka to S3 message 1
@@ -28,8 +28,8 @@ camel.component.aws2-s3.accessKey=xxxx
camel.component.aws2-s3.secretKey=yyyy
camel.component.aws2-s3.region=eu-west-1

camel.sink.endpoint.keyName=${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.zip
camel.sink.endpoint.keyName=${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.tar.gz

camel.beans.aggregate=#class:org.apache.camel.processor.aggregate.zipfile.ZipAggregationStrategy
camel.beans.aggregate=#class:org.apache.camel.processor.aggregate.tar.gzfile.ZipAggregationStrategy
camel.aggregation.size=10
camel.aggregation.timeout=5000
@@ -13,8 +13,8 @@ spec:
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: s3-topic
camel.sink.path.bucketNameOrArn: camel-kafka-connector
camel.sink.endpoint.keyName: ${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.zip
camel.beans.aggregate: '#class:org.apache.camel.processor.aggregate.zipfile.ZipAggregationStrategy'
camel.sink.endpoint.keyName: ${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}.tar.gz
camel.beans.aggregate: '#class:org.apache.camel.processor.aggregate.tar.gzfile.ZipAggregationStrategy'
camel.aggregation.size: 10
camel.aggregation.timeout: 5000
camel.component.aws2-s3.accessKey: ${file:/opt/kafka/external-configuration/aws-credentials/aws2-s3-cred.properties:accessKey}

0 comments on commit 616bd3d

Please sign in to comment.