Kafka Connect Mirror Connector

kafka-connect-mirror is a Kafka Connector for mirroring data between Kafka clusters.

Building

mvn clean package 

Quick Start

connect standalone

./bin/connect-standalone.sh config/connect-standalone.properties  mirror-connector.properties

connect distributed

./bin/connect-standalone.sh config/connect-distributed.properties
 curl -X POST -H "Content-Type: application/json" -d @mirror-connector.json  http://localhost:8083/connectors

Configuration

topic.whitelist Whitelist of topics to be mirrored.

• Type: list
• Default: ""
• Importance: high

topic.blacklist Topics to exclude from mirroring.

• Type: list
• Default: ""
• Importance: high

topic.regex Regex of topics to mirror.

• Type: string
• Default: null
• Valid Values: '.' for all topics
• Importance: high

topic.poll.interval.ms Frequency in ms to poll for new or removed topics, which may result in updated task configurations to start polling for data in added topics/partitions or stop polling for data in removed topics.

• Type: int
• Default: 180000
• Valid Values: [0,...]
• Importance: low

topic.rename.format A format string to rename the topics in the destination cluster. the format string should contain '${topic}' that we bill replaced with the source topic name For example, with'${topic}_mirror' format the topic 'test' will be renamed at the destination cluster to 'test_mirror'.

• Type: string
• Default: ${topic}
• Importance: high

topic.preserve.partitions Ensure that messages mirrored from the source cluster use the same partition in the destination cluster. [if source topic have more partitions than destination topic, some partitions will be not mirrored.]

• Type: boolean
• Default: true
• Importance: low