Kafka-Connect-ZooKeeper

A Zookeeper Kafka connector implementation for both sink and the source

Instructions for building

---

$ git clone https://github.com/s8sg/kafka-connect-zookeeper
$ (cd kafka-connect-zookeeper && mvn clean package)

Instructions for running

---

1. Run zookeeper somehow

   Here are instructions on how to run it using docker.
   $ docker run -d --name zookeeper -p 2181:2181 confluent/zookeeper
   

   We are going to use te same zk for Kafka and for the ZK connector

2. Download and run kafka 0.9.0 locally by following the instructions at http://kafka.apache.org/documentation.html#quickstart

   $ curl -o kafka_2.11-0.9.0.0.tgz http://www.us.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
   $ tar xvfz kafka_2.11-0.9.0.0.tgz
   $ cd kafka_2.11-0.9.0.0
   $ ./bin/kafka-server-start.sh config/server.properties
   

Configure

3. Configure The Source and Sink properties
   connect-zk-source.properties

Name	Description
zk-hosts	Single ZooKeeper Host, or Comma Seperated List of ZooKeeper Hosts in a Cluster
zk-nodes	Single ZooKeeper Data Node, or Comma Seperated List of Zookeeper Nodes

connect-zk-sink.properties

Name	Description
zk-hosts	Single ZooKeeper Host, or Comma Seperated List of ZooKeeper Hosts in a Cluster
zk-node	ZooKeeper Data Node

Run ZK Connect Source

1. Run your connect-zookeeper-source plugin

   $ export CLASSPATH=/path/to/kafka-connect-zookeeper/target/kafka-connect-zookeeper-1.0.jar
   $ kafka_2.11-0.9.0.0/bin/connect-standalone.sh kafka-connect-zookeeper/connect-standalone.properties  kafka-connect-zookeeper/config/connect-zk-source.properties
   

2. Write stuff to zookeeper node test-data-source (the connector will read from, as configured in connect-zk-source.properties) You could set the data using zkcli.sh. The current repo comes with an utility (zk_util.py) to upload data to zk node

   $ pip install kazoo
   $ python zk_util.py upload localhost:2181 /test/test-data-source this_is_a_test_data
   

3. Read the data out from the kafka topic named 'test' (that is the that this connector will write to, as configured in connect-zk-source.properties)

   $ kafka_2.11-0.9.0.0/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic test
   {"schema":{"type":"string","optional":false},"payload":"this_is_a_test_data"}
   

Run ZK Connect Sink

1. Run your connect-zookeeper-sink plugin

   $ export CLASSPATH=/path/to/kafka-connect-zookeeper/target/kafka-connect-zookeeper-1.0.jar
   $ kafka_2.11-0.9.0.0/bin/connect-standalone.sh kafka-connect-zookeeper/connect-standalone.properties  kafka-connect-zookeeper/config/connect-zk-sink.properties
   

2. Check that the zookeeper-sink plugin has written the data to the zookeeper node test-data-sink

   $ python zk_util.py download localhost:2181 /test/test-data-sink
   this_is_a_test_data
   

TODO

---

• Write Test Cases

Note

---

• This repo is under active devolpment
• For any contribution or suggestions, please create PR or Issues