Skip to content
Permalink
Browse files
FTP Examples: Moved to 0.10.0
  • Loading branch information
oscerd committed May 26, 2021
1 parent 1171c51 commit 1e86a41fcbfcdac643ae97bb12c0f8ec9c2ef14b
Show file tree
Hide file tree
Showing 6 changed files with 50 additions and 50 deletions.
@@ -16,9 +16,9 @@ Run the following command:

```
docker run -d -v /my/data/directory:/home/vsftpd -p 20:20 -p 21:21 -p 21100-21110:21100-21110 -e FTP_USER=admin -e FTP_PASS=password -e PASV_ADDRESS=127.0.0.1 -e PASV_MIN_PORT=21100 -e PASV_MAX_PORT=21110 --name vsftpd --restart=always fauria/vsftpd
9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe6039b038370a12
9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe60.10.038370a12
```
take note of the container id. In our case it is 9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe6039b038370a12
take note of the container id. In our case it is 9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe60.10.038370a12

### Running Kafka

@@ -42,8 +42,8 @@ Download the connector package tar.gz and extract the content to a directory. In
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-ftp-kafka-connector/0.9.0/camel-ftp-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-ftp-kafka-connector-0.9.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-ftp-kafka-connector/0.10.0/camel-ftp-kafka-connector-0.10.0-package.tar.gz
> untar.gz camel-ftp-kafka-connector-0.10.0-package.tar.gz
----

Now it's time to setup the connector
@@ -83,7 +83,7 @@ In a different terminal we need to send something to mytopic
Now we need to connect to the ftp server and check the file

```
> docker exec -it 9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe6039b038370a12 bash
> docker exec -it 9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe60.10.038370a12 bash
[root@9534a8d7a87c /]# cd /home/vsftpd/admin
[root@9534a8d7a87c admin]# cat mydata-20201111.txt
Hello from ckc
@@ -16,10 +16,10 @@ Run the following command:

```
docker run -d -v /my/data/directory:/home/vsftpd -p 20:20 -p 21:21 -p 21100-21110:21100-21110 -e FTP_USER=admin -e FTP_PASS=password -e PASV_ADDRESS=127.0.0.1 -e PASV_MIN_PORT=21100 -e PASV_MAX_PORT=21110 --name vsftpd --restart=always fauria/vsftpd
9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe6039b038370a12
9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe60.10.038370a12
```

take note of the container id. In our case it is 9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe6039b038370a12
take note of the container id. In our case it is 9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe60.10.038370a12

### Running Kafka

@@ -40,7 +40,7 @@ and set the `plugin.path` property to your choosen location
You'll need to build your connector starting from an archetype:

```
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.9.0
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.10.0
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
@@ -54,34 +54,34 @@ You'll need to build your connector starting from an archetype:
[INFO]
[INFO] --- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.9.0] found in catalog remote
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.10.0] found in catalog remote
Define value for property 'groupId': org.apache.camel.kafkaconnector
Define value for property 'artifactId': ftp-extended
Define value for property 'version' 1.0-SNAPSHOT: : 0.9.0
Define value for property 'version' 1.0-SNAPSHOT: : 0.10.0
Define value for property 'package' org.apache.camel.kafkaconnector: :
Define value for property 'camel-kafka-connector-name': camel-ftp-kafka-connector
[INFO] Using property: camel-kafka-connector-version = 0.9.0
[INFO] Using property: camel-kafka-connector-version = 0.10.0
Confirm properties configuration:
groupId: org.apache.camel.kafkaconnector
artifactId: ftp-extended
version: 0.9.0
version: 0.10.0
package: com.github.oscerd
camel-kafka-connector-name: camel-ftp-kafka-connector
camel-kafka-connector-version: 0.9.0
camel-kafka-connector-version: 0.10.0
Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.9.0
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.10.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: artifactId, Value: ftp-extended
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: version, Value: 0.10.0
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: packageInPathFormat, Value: org/apache/camel/kafkaconnector
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: version, Value: 0.10.0
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: camel-kafka-connector-name, Value: camel-ftp-kafka-connector
[INFO] Parameter: camel-kafka-connector-version, Value: 0.9.0
[INFO] Parameter: camel-kafka-connector-version, Value: 0.10.0
[INFO] Parameter: artifactId, Value: ftp-extended
[INFO] Project created from Archetype in dir: /home/workspace/miscellanea/ftp-extended
[INFO] ------------------------------------------------------------------------
@@ -163,8 +163,8 @@ In this example we'll use `/home/oscerd/connectors/` as plugin.path, but we'll n

```
> cd /home/oscerd/connectors/
> cp /home/workspace/miscellanea/ftp-extended/target/ftp-extended-0.9.0-package.tar.gz .
> untar.gz ftp-extended-0.9.0-package.tar.gz
> cp /home/workspace/miscellanea/ftp-extended/target/ftp-extended-0.10.0-package.tar.gz .
> untar.gz ftp-extended-0.10.0-package.tar.gz
```

Now it's time to setup the connector
@@ -200,7 +200,7 @@ $KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.prop
Now we need to connect to the ftp server and add some stuff to the demos folder

```
> docker exec -it 9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe6039b038370a12 bash
> docker exec -it 9534a8d7a87c5f0525079824f692552fe6306fcea2e0e2a0fe60.10.038370a12 bash
[root@9534a8d7a87c /]# cd /home/vsftpd
[root@9534a8d7a87c vsftpd]# touch test.txt
[root@9534a8d7a87c vsftpd]# echo "Ckc rocks" > test.txt
@@ -212,7 +212,7 @@ In another terminal, using kafkacat, you should be able to see the headers.
```
> ./kafkacat -b localhost:9092 -t mytopic -f 'Headers: %h: Message value: %s\n'
% Auto-selecting Consumer mode (use -P or -C to override)
Headers: CamelHeader.CamelFileAbsolute=false,CamelHeader.CamelFileAbsolutePath=test.txt,CamelHeader.CamelFileHost=127.0.0.1,CamelHeader.CamelFileLastModified=16050.9.00000,CamelHeader.CamelFileLength=10,CamelHeader.CamelFileName=test.txt,CamelHeader.CamelFileNameConsumed=test.txt,CamelHeader.CamelFileNameOnly=test.txt,CamelHeader.CamelFileParent=/,CamelHeader.CamelFilePath=/test.txt,CamelHeader.CamelFileRelativePath=test.txt,CamelHeader.CamelFtpReplyCode=226,CamelHeader.CamelFtpReplyString=226 Transfer complete.
Headers: CamelHeader.CamelFileAbsolute=false,CamelHeader.CamelFileAbsolutePath=test.txt,CamelHeader.CamelFileHost=127.0.0.1,CamelHeader.CamelFileLastModified=16050.10.00000,CamelHeader.CamelFileLength=10,CamelHeader.CamelFileName=test.txt,CamelHeader.CamelFileNameConsumed=test.txt,CamelHeader.CamelFileNameOnly=test.txt,CamelHeader.CamelFileParent=/,CamelHeader.CamelFilePath=/test.txt,CamelHeader.CamelFileRelativePath=test.txt,CamelHeader.CamelFtpReplyCode=226,CamelHeader.CamelFtpReplyString=226 Transfer complete.
,CamelProperty.CamelBatchSize=1,CamelProperty.CamelUnitOfWorkProcessSync=true,CamelProperty.CamelBatchComplete=true,CamelProperty.CamelBatchIndex=0,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&pollingConsumerQueueSize=1000: Message value: Ckc rocks
% Reached end of topic mytopic [0] at offset 1
```
@@ -57,8 +57,8 @@ Download the connector package tar.gz and extract the content to a directory. In
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-ftps-kafka-connector/0.9.0/camel-ftps-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-ftps-kafka-connector-0.9.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-ftps-kafka-connector/0.10.0/camel-ftps-kafka-connector-0.10.0-package.tar.gz
> untar.gz camel-ftps-kafka-connector-0.10.0-package.tar.gz
----

Now it's time to setup the connector
@@ -54,7 +54,7 @@ and set the `plugin.path` property to your choosen location
You'll need to build your connector starting from an archetype:

```
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.9.0
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.10.0
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
@@ -68,34 +68,34 @@ You'll need to build your connector starting from an archetype:
[INFO]
[INFO] --- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.9.0] found in catalog remote
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.10.0] found in catalog remote
Define value for property 'groupId': org.apache.camel.kafkaconnector
Define value for property 'artifactId': ftps-extended
Define value for property 'version' 1.0-SNAPSHOT: : 0.9.0
Define value for property 'version' 1.0-SNAPSHOT: : 0.10.0
Define value for property 'package' org.apache.camel.kafkaconnector: :
Define value for property 'camel-kafka-connector-name': camel-ftps-kafka-connector
[INFO] Using property: camel-kafka-connector-version = 0.9.0
[INFO] Using property: camel-kafka-connector-version = 0.10.0
Confirm properties configuration:
groupId: org.apache.camel.kafkaconnector
artifactId: ftps-extended
version: 0.9.0
version: 0.10.0
package: com.github.oscerd
camel-kafka-connector-name: camel-ftps-kafka-connector
camel-kafka-connector-version: 0.9.0
camel-kafka-connector-version: 0.10.0
Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.9.0
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.10.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: artifactId, Value: ftps-extended
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: version, Value: 0.10.0
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: packageInPathFormat, Value: org/apache/camel/kafkaconnector
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: version, Value: 0.10.0
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: camel-kafka-connector-name, Value: camel-ftps-kafka-connector
[INFO] Parameter: camel-kafka-connector-version, Value: 0.9.0
[INFO] Parameter: camel-kafka-connector-version, Value: 0.10.0
[INFO] Parameter: artifactId, Value: ftps-extended
[INFO] Project created from Archetype in dir: /home/workspace/miscellanea/ftps-extended
[INFO] ------------------------------------------------------------------------
@@ -177,8 +177,8 @@ In this example we'll use `/home/oscerd/connectors/` as plugin.path, but we'll n

```
> cd /home/oscerd/connectors/
> cp /home/workspace/miscellanea/ftps-extended/target/ftps-extended-0.9.0-package.tar.gz .
> untar.gz ftps-extended-0.9.0-package.tar.gz
> cp /home/workspace/miscellanea/ftps-extended/target/ftps-extended-0.10.0-package.tar.gz .
> untar.gz ftps-extended-0.10.0-package.tar.gz
```

Now it's time to setup the connector
@@ -60,8 +60,8 @@ Download the connector package tar.gz and extract the content to the plugin.path
[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-sftp-kafka-connector/0.9.0/camel-sftp-kafka-connector-0.9.0-package.tar.gz
> untar.gz camel-sftp-kafka-connector-0.9.0-package.tar.gz
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-sftp-kafka-connector/0.10.0/camel-sftp-kafka-connector-0.10.0-package.tar.gz
> untar.gz camel-sftp-kafka-connector-0.10.0-package.tar.gz
----

Open the SFTP sink configuration file
@@ -41,7 +41,7 @@ and set the `plugin.path` property to your choosen location
You'll need to build your connector starting from an archetype:

```
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.9.0
> mvn archetype:generate -DarchetypeGroupId=org.apache.camel.kafkaconnector.archetypes -DarchetypeArtifactId=camel-kafka-connector-extensible-archetype -DarchetypeVersion=0.10.0
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
@@ -55,34 +55,34 @@ You'll need to build your connector starting from an archetype:
[INFO]
[INFO] --- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.9.0] found in catalog remote
[INFO] Archetype repository not defined. Using the one from [org.apache.camel.kafkaconnector.archetypes:camel-kafka-connector-extensible-archetype:0.10.0] found in catalog remote
Define value for property 'groupId': org.apache.camel.kafkaconnector
Define value for property 'artifactId': sftp-extended
Define value for property 'version' 1.0-SNAPSHOT: : 0.9.0
Define value for property 'version' 1.0-SNAPSHOT: : 0.10.0
Define value for property 'package' com.github.oscerd: :
Define value for property 'camel-kafka-connector-name': camel-sftp-kafka-connector
[INFO] Using property: camel-kafka-connector-version = 0.9.0
[INFO] Using property: camel-kafka-connector-version = 0.10.0
Confirm properties configuration:
groupId: org.apache.camel.kafkaconnector
artifactId: sftp-extended
version: 0.9.0
version: 0.10.0
package: com.github.oscerd
camel-kafka-connector-name: camel-sftp-kafka-connector
camel-kafka-connector-version: 0.9.0
camel-kafka-connector-version: 0.10.0
Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.9.0
[INFO] Using following parameters for creating project from Archetype: camel-kafka-connector-extensible-archetype:0.10.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: artifactId, Value: sftp-extended
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: version, Value: 0.10.0
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: packageInPathFormat, Value: org/apache/camel/kafkaconnector
[INFO] Parameter: package, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: version, Value: 0.9.0
[INFO] Parameter: version, Value: 0.10.0
[INFO] Parameter: groupId, Value: org.apache.camel.kafkaconnector
[INFO] Parameter: camel-kafka-connector-name, Value: camel-sftp-kafka-connector
[INFO] Parameter: camel-kafka-connector-version, Value: 0.9.0
[INFO] Parameter: camel-kafka-connector-version, Value: 0.10.0
[INFO] Parameter: artifactId, Value: sftp-extended
[INFO] Project created from Archetype in dir: /home/workspace/miscellanea/sftp-extended
[INFO] ------------------------------------------------------------------------
@@ -164,8 +164,8 @@ In this example we'll use `/home/oscerd/connectors/` as plugin.path, but we'll n

```
> cd /home/oscerd/connectors/
> cp /home/workspace/miscellanea/sftp-extended/target/sftp-extended-0.9.0-package.tar.gz .
> untar.gz sftp-extended-0.9.0-package.tar.gz
> cp /home/workspace/miscellanea/sftp-extended/target/sftp-extended-0.10.0-package.tar.gz .
> untar.gz sftp-extended-0.10.0-package.tar.gz
```

Now it's time to setup the connector
@@ -213,7 +213,7 @@ In another terminal, using kafkacat, you should be able to see the headers.
```
> ./kafkacat -b localhost:9092 -t mytopic -f 'Headers: %h: Message value: %s\n'
% Auto-selecting Consumer mode (use -P or -C to override)
Headers: CamelHeader.CamelFileAbsolute=false,CamelHeader.CamelFileAbsolutePath=demos/file.txt,CamelHeader.CamelFileHost=localhost,CamelHeader.CamelFileLastModified=1604560.9.000,CamelHeader.CamelFileLength=29,CamelHeader.CamelFileName=file.txt,CamelHeader.CamelFileNameConsumed=file.txt,CamelHeader.CamelFileNameOnly=file.txt,CamelHeader.CamelFileParent=demos,CamelHeader.CamelFilePath=demos//file.txt,CamelHeader.CamelFileRelativePath=file.txt,CamelHeader.CamelFtpReplyCode=0,CamelHeader.CamelFtpReplyString=OK,CamelProperty.CamelBatchSize=1,CamelProperty.CamelUnitOfWorkProcessSync=true,CamelProperty.CamelBatchComplete=true,CamelProperty.CamelBatchIndex=0,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&pollingConsumerQueueSize=1000: Message value: Test file content
Headers: CamelHeader.CamelFileAbsolute=false,CamelHeader.CamelFileAbsolutePath=demos/file.txt,CamelHeader.CamelFileHost=localhost,CamelHeader.CamelFileLastModified=1604560.10.000,CamelHeader.CamelFileLength=29,CamelHeader.CamelFileName=file.txt,CamelHeader.CamelFileNameConsumed=file.txt,CamelHeader.CamelFileNameOnly=file.txt,CamelHeader.CamelFileParent=demos,CamelHeader.CamelFilePath=demos//file.txt,CamelHeader.CamelFileRelativePath=file.txt,CamelHeader.CamelFtpReplyCode=0,CamelHeader.CamelFtpReplyString=OK,CamelProperty.CamelBatchSize=1,CamelProperty.CamelUnitOfWorkProcessSync=true,CamelProperty.CamelBatchComplete=true,CamelProperty.CamelBatchIndex=0,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&pollingConsumerQueueSize=1000: Message value: Test file content
% Reached end of topic mytopic [0] at offset 1
```

0 comments on commit 1e86a41

Please sign in to comment.