Tapdata is a Change Data Capture(CDC) based, Real-Time data integration platform that enables data to be synchronized in real-time among various systems such as databases, SaaS services, applications, and files. The synchronization tasks can be easily built through drag-and-drop operations, from table creation to full and incremental synchronization, all processes are fully automated.
For more details, please read docs
Based on different distribution strategies and functionalities, Tapdata has three distribution versions: Community Edition, Cloud Service, and Enterprise Edition. All content under this open-source project corresponds to the Community Edition.
For the differences between the three versions, you can refer to: https://tapdata.io/
1. Key Features
2. Use Cases
3. Quick Start
4. Examples
5. License
6. Contact Us
7. Build From Source
8. Contributing
- Synchronize data from traditional RDBMS to modern databases such as MongoDB, Elasticsearch (ES), or Redis to support new business uses.
- Consolidate data from various databases into a unified data warehouse.
- Stream database changes to Kafka.
- Perform data synchronization between heterogeneous databases.
- Use MongoDB to build your unified data hub.
To run the complete service, the basic resource requirements are:
- At least 5GB of available memory
- At least 20GB of available disk space
- At least 1 free CPU core
RUN docker run -d -p 3030:3030 ghcr.io/tapdata/tapdata:latest
, wait for 3 minutes, then you can get it from http://localhost:3030/, if everything is ok, you can see login page like this:
default username is: admin@admin.com, default password is admin
Tapdata service is available in cloud service, you can use fully-managed service, or deploy engine to your private network
Try on https://cloud.tapdata.io/, support google and github account login, free trial, NO credit card needed, start your real-time data journey immediately.
-
Login tapdata platform
-
In the left navigation panel, click Connections
-
On the right side of the page, click Create
-
In the pop-up dialog, search and select MySQL
-
On the page that you are redirected to, follow the instructions below to fill in the connection information for MySQL
- Click Test, make sure all test pass, then click Save
-
Create MySQL and MongoDB data source
-
In the left navigation panel, click Data Pipelines -> Data Replications
-
On the right side of the page, click Create
-
Drag and drop MySQL and MongoDB data sources onto the canvas
-
Drag a line from the MySQL data source to MongoDB
-
Configure the MySQL data source and select the data tables you want to synchronize
-
Click the Save button in the upper right corner, then click the Start button
-
Observe the indicators and events on the task page until data is in sync
-
Create MySQL and PostgreSQL data source
-
In the left navigation panel, click Data Pipelines -> Data Transformation
-
On the right side of the page, click Create
-
Drag and drop MySQL and PostgreSQL data sources onto the canvas
-
Drag a line from the MySQL data source to PostgreSQL
-
Click the plus sign on the connection line and select Field Rename
- Click Field Rename node, change i_price to price, i_data to data in config form
-
Click the Save button in the upper right corner, then click the Start button
-
Observe the indicators and events on the task page until data is in sync
Materialized view is a special feature of tapdata, You can give full play to the characteristics of MongoDB document database and create the data model you need, try enjoy it !
In this example, I will make a view using 2 tables in MySQL: order and product, make product as a embedded document of order, here is the step:
-
Create MySQL and MongoDB data source
-
In the left navigation panel, click Data Pipelines -> Data Transformation
-
On the right side of the page, click Create
-
Click mysql data source in left up side, then drag and drop order table and product table onto the canvas
-
Drag and drop "Master-slave merge" node in left bottom side onto the canvas
-
Drag a line from the order table to Master-slave merge
-
Drag a line from the product table to Master-slave merge
-
Drag and drop MongoDB data source onto the canvas, and drag a line from the "Master-slave merge" node to MongoDB node
- Click "Master-slave merge" node, then drag product table into order table in the right side in "Table Name"
-
Click "Master-slave merge" node, then click product table, config Data write model to "Match and Merge", Field write path to "product", Association Conditions to "order_id" => "order_id", then you can see Schema in bottom changed
-
Click MongoDB node, and config target table name as order_with_product, update condition field config as "order_id"
-
Click the Save button in the upper right corner, then click the Start button
-
Observe the indicators and events on the task page until data is in sync
-
Check collection order_with_product in MongoDB, and you will see the data model
Using the data verification feature, you can quickly check whether the synchronized data is consistent and accurate
-
In the left navigation panel, click Data Pipelines -> Data Validation
-
On the right side of the page, click Task Consistency Validation
-
Choose 1 task, and valid type choose "All Fields Validation", it means system will check all fields for all record
-
Click Save, then click Execute in the task list
-
Wait validation task finished, click Result in the task list, and check the validation result
Tapdata Community is under the Apache 2.0 license. See the LICENSE file for details.