Skip to content
/ qin-cdc Public

Real-time data synchronization component, supports MySQL -> StarRocks, MySQL, Doris, Kafka, etc.), used to replace Aliyun DTS

License

Notifications You must be signed in to change notification settings

sqlpub/qin-cdc

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

qin-cdc data sync

Simple, efficient, real-time, stable, scalable, highly available, AI, open source

LICENSE Release

English | 简体中文

Support sync database

source

  1. mysql
  2. TODO sqlserver
  3. TODO mongo
  4. TODO oracle

target

  1. mysql
  2. starrocks
  3. doris
  4. kafka json
  5. TODO kafka canal
  6. kafka aliyun_dts_canal

Quick start

1. Install

Download the latest release and extract it.

2. Create a synchronization account

CREATE USER 'qin_cdc'@'%' IDENTIFIED BY 'xxxxxx';
GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'qin_cdc'@'%';

3. Create a configuration file

mysql-to-starrocks.toml

# name is required and must be globally unique when multiple instances are running
name = "mysql2starrocks"

[input]
type = "mysql"

[input.config.source]
host = "127.0.0.1"
port = 3306
username = "root"
password = "root"

[input.config.source.options]
#start-gtid = "3ba13781-44eb-2157-88a5-0dc879ec2221:1-123456"
#server-id = 1001

[[transforms]]
type = "rename-column"
[transforms.config]
match-schema = "mysql_test"
match-table = "tb1"
columns = ["col_1", "col_2"]
rename-as = ["col_11", "col_22"]

[[transforms]]
type = "delete-column"
[transforms.config]
match-schema = "mysql_test"
match-table = "tb1"
columns = ["phone"]

[output]
type = "starrocks"

[output.config.target]
host = "127.0.0.1"
port = 9030
load-port = 8040 # support fe httpPort:8030 or be httpPort:8040
username = "root"
password = ""

[input.config.target.options]
batch-size = 1000
batch-interval-ms = 1000
parallel-workers = 4

[[output.config.routers]]
source-schema = "mysql_test"
source-table = "tb1"
target-schema = "sr_test"
target-table = "ods_tb1"

[[output.config.routers]]
source-schema = "mysql_test"
source-table = "tb2"
target-schema = "sr_test"
target-table = "ods_tb2"
# mapper column, optional, if empty, same name mapping
# [output.config.routers.columns-mapper]
# source-columns = []
# target-columns = []

4. View Help

[sr@ ~]$ ./qin-cdc-linux-xxxxxx -h

5. Startup

[sr@ ~]$ ./qin-cdc-linux-xxxxxx -config mysql-to-starrocks.toml -log-file mysql2starrocks.log -level info -daemon

6. View logs

[sr@ ~]$ tail -f mysql2starrocks.log

TODO AI functional points

  1. Intelligent data synchronization and migration
  2. Data security and monitoring
  3. Intelligent operation and maintenance management
  4. User experience optimization
  5. Automated data mapping
  6. Natural language processing

About

Real-time data synchronization component, supports MySQL -> StarRocks, MySQL, Doris, Kafka, etc.), used to replace Aliyun DTS

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published