cdc

A mysql binlog parser, which has running stable for more then 36 months in our production environment on more than 10 mysql instances. It transforms binlog event(insert/delete/update) into json format data into rabbitmq or other mq(such as kafka), Based on this project: mysql-binlog-connector-java.

What is cdc?

change data capture, Key Features:

• High availability, cluster deployment
• Table level filter
• Supports almost all mysql field parsing
• Automatically save binlog file/position, smooth upgrade and restart
• Configure centralized management
• Operational status monitoring
• Dynamically loading table configuration files
• support split table config
• Mysql binlog data order Guaranteed
• Support a easy way to tap it into other mq (sucn as kafka,nsq,redis...)

dependency

• Java 8 or above is required.

• etcd 3.0 or above is required.

• mysql 5.6+

  • open mysql binlog log

    [mysqld]  
    log-bin=mysql-bin
    binlog-format=ROW
    server_id=666

  • authorize mysql slave read permission

    CREATE USER root IDENTIFIED BY '123'; 
    GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'root'@'%';
    FLUSH PRIVILEGES;

Download

Maven

<dependency>
  <groupId>com.rong360</groupId>
  <artifactId>cdc</artifactId>
  <version>1.4.0</version>
</dependency>

Quick start

• install etcd, start etcd

$ ./bin/etcd

• Set the database configuration in etcd

etcdctl put cdc/master/admin/config/app/mysql/host 10.0.0.2
etcdctl put cdc/master/admin/config/app/mysql/port 3306
etcdctl put cdc/master/admin/config/app/mysql/username admin
etcdctl put cdc/master/admin/config/app/mysql/password 123

• Start program

CdcClient client = new CdcClient("http://127.0.0.1:2379", "", "");
client.setInstance("admin");//"admin" is one of the ETCD configuration prefix elements
client.setWatchAllTable(true);
client.start();

Documentation

English

See DOUCUMENT for details.

简体中文

请见详细文档。

Connect with us

zhangtao@rong360.com

liuchi@rong360.com

Q&A