connect-aws-s3-source

S3 Source connector

Objective

Quickly test S3 Source connector.

AWS Setup

• Make sure you have an AWS account.
• Set up AWS Credentials

This project assumes ~/.aws/credentials is set, see docker-compose.ymlfile for connect:

    connect:
    <snip>
    volumes:
        - $HOME/.aws/credentials:/root/.aws/credentials:ro

How to run

Simply run:

$ ./s3-source.sh <AWS_BUCKET_NAME>

Note: you can also export these values as environment variable

Details of what the script is doing

Steps from connect-aws-s3-sink

The connector is created with:

$ curl -X PUT \
     -H "Content-Type: application/json" \
     --data '{
               "tasks.max": "1",
                    "connector.class": "io.confluent.connect.s3.source.S3SourceConnector",
                    "s3.region": "us-east-1",
                    "s3.bucket.name": "'"$AWS_BUCKET_NAME"'",
                    "format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
                    "confluent.license": "",
                    "confluent.topic.bootstrap.servers": "broker:9092",
                    "confluent.topic.replication.factor": "1",
                    "transforms": "AddPrefix",
                    "transforms.AddPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter",
                    "transforms.AddPrefix.regex": ".*",
                    "transforms.AddPrefix.replacement": "copy_of_$0"
          }' \
     http://localhost:8083/connectors/s3-source/config | jq .

Verifying topic copy_of_s3_topic

$ docker exec broker kafka-console-consumer -bootstrap-server broker:9092 --topic copy_of_s3_topic --from-beginning --max-messages 9

Results:

value1

value2

value3

value4

value5

value6

value7

value8

value9
Processed a total of 9 messages

N.B: Control Center is reachable at http://127.0.0.1:9021