From 3ef9fdd87351e280968e3223ebeaf17265aa1e13 Mon Sep 17 00:00:00 2001 From: Ran Date: Thu, 11 Jun 2020 16:25:29 +0800 Subject: [PATCH 1/3] tools: add ecosystem tools overview and use case in 3.0 --- TOC.md | 3 +- ecosystem-tool-user-case.md | 39 +++++++ ecosystem-tool-user-guide.md | 191 ++++++++--------------------------- 3 files changed, 82 insertions(+), 151 deletions(-) create mode 100644 ecosystem-tool-user-case.md diff --git a/TOC.md b/TOC.md index 298eb0315b552..37b7ad3aefa2c 100644 --- a/TOC.md +++ b/TOC.md @@ -313,7 +313,8 @@ - [Error Handling](/tidb-binlog/handle-tidb-binlog-errors.md) - [FAQ](/tidb-binlog/tidb-binlog-faq.md) + Tools - - [Tools User Guide](/ecosystem-tool-user-guide.md) + - [Overview](/ecosystem-tool-user-guide.md) + - [Use Cases](/ecosystem-tool-user-case.md) - [Mydumper](/mydumper-overview.md) - [Syncer](/syncer-overview.md) - [Loader](/loader-overview.md) diff --git a/ecosystem-tool-user-case.md b/ecosystem-tool-user-case.md new file mode 100644 index 0000000000000..e74d5947178e3 --- /dev/null +++ b/ecosystem-tool-user-case.md @@ -0,0 +1,39 @@ +--- +title: TiDB Ecosystem Tools Use Cases +summary: Learn the common use cases of TiDB ecosystem tools and how to choose the tools. +category: reference +--- + +# TiDB Ecosystem Tools Use Cases + +This document introduces the common use cases of TiDB ecosystem tools and how to choose the right tool for your scenario. + +## Import data from CSV to TiDB + +If you need to import the compatible CSV files exported by other tools to TiDB, use [TiDB Lightning](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md). + +## Import full data from MySQL/Aurora + +If you need to import full data from MySQL or Aurora, use [Dumpling](/export-or-backup-using-dumpling.md) first to export data as SQL dump files, and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import data into the TiDB cluster. + +## Migrate data from MySQL/Aurora + +If you need to migrate both full data and incremental data from MySQL/Aurora, use [TiDB Data Migration](https://pingcap.com/docs-cn/tidb-data-migration/stable/overview/) (DM) to perform the full and incremental data migration. + +If the full data volume is large (at the TB level), you can first use [Dumpling](/export-or-backup-using-dumpling.md) and [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to perform the full data migration, and then use DM to perform the incremental data migration. + +## Back up and restore TiDB cluster + +If you need to back up a TiDB cluster, use [Dumpling](/export-or-backup-using-dumpling.md). + +If you need to restore data to a TiDB cluster, use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md). + +## Migrate data from TiDB + +If you need to migrate data from a TiDB cluster to MySQL or to another TiDB cluster, use [Dumpling](/export-or-backup-using-dumpling.md) to export full data from TiDB as SQL dump files, and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import data to MySQL or another TiDB cluster. + +If you also need to migrate incremental data, use [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md). + +## TiDB incremental data subscription + +If you need to subscribe to TiDB's incremental changes, use [TiDB Binlog](/tidb-binlog/binlog-slave-client.md). \ No newline at end of file diff --git a/ecosystem-tool-user-guide.md b/ecosystem-tool-user-guide.md index ab323a0397812..c02f152d5b94e 100644 --- a/ecosystem-tool-user-guide.md +++ b/ecosystem-tool-user-guide.md @@ -1,190 +1,81 @@ --- -title: TiDB Ecosystem Tools User Guide +title: TiDB Ecosystem Tools Overview category: reference aliases: ['/docs/v3.0/reference/tools/user-guide/','/docs/v3.0/how-to/migrate/from-mysql/','/docs/v3.0/how-to/migrate/incrementally-from-mysql/','/docs/v3.0/how-to/migrate/overview/'] --- -# TiDB Ecosystem Tools User Guide +# TiDB Ecosystem Tools Overview -The TiDB ecosystem has a wealth of tools for data migration, backup & restore for users with different use cases to choose from. +This document introduces the functionalities of TiDB ecosystem tools and their relationship. -- Some of the functionalities of these tools might overlap. For example, TiDB Loader, TiDB Lightning and TiDB DM can all do full data loading. -- Some of the tools might have evolved. For example, TiDB Binlog will be evolved to CDC (Change Data Capture). -- Some of the tools are designed to support specific TiDB versions and the others might be deprecated as user requirements change. +## Full data export -This guide is specifically designed to help you better understand these tools and therefore make an informed decision while choosing these tools to support your business. +[Dumpling](/export-or-backup-using-dumpling.md) is a tool for the logical full data export from MySQL or TiDB. -## Data import (restore or data replication) +The following are the basics of Dumpling: -### Full data import tools +- Input: MySQL/TiDB cluster +- Output: SQL/CSV file +- Supported TiDB versions: all versions +- Kubernetes support: No -#### TiDB Lightning +## Full data import -[TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) (Lightning) is a tool used for the fast full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Mydumper or CSV data source. +[TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) (Lightning) is a tool used for the full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Dumpling or CSV data source. -TiDB Lightning supports two back ends: "Importer" and "TiDB". It determines how tidb-lightning delivers data into the target cluster. The two back ends are as follows: +TiDB Lightning supports two modes: -1. The default one is [`Importer` back end](/tidb-lightning/tidb-lightning-overview.md). When using `Importer` as the back end, the cluster cannot provide normal services during the import process. It is used for a large amount of data importing (TB). -2. The second one is [`TiDB` back end](/tidb-lightning/tidb-lightning-tidb-backend.md) (just work as [Loader](#tidb-loader-to-be-deprecated)). It is much slower than `Importer` back end model. But the cluster could serve the application during the import process. It is used to handle tens/hundreds of GB data. +- `importer`: This mode uses tikv-importer as the backend, which is usually for importing a large amount of data (at the TB level). During the import, the cluster cannot provide services. +- `tidb`: This mode uses TiDB/MySQL as the backend, which is slower than the `importer` mode but can be performed online. It also supports importing data to MySQL. The following are the basics of TiDB Lightning: - Input data source: - - The output file of Mydumper - - CSV file + - The output file of Dumpling + - Other compatible CSV file - Supported TiDB versions: v2.1 or later - Kubernetes support: Yes. See [Quickly restore data into a TiDB cluster in Kubernetes using TiDB Lightning](https://pingcap.com/docs/tidb-in-kubernetes/stable/restore-data-using-tidb-lightning/) for details. -#### BR (beta) - -[BR](https://pingcap.com/docs/dev/reference/tools/br/br/) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/Loader/Lightning, BR is more suitable for scenarios of huge data volume. - -The following are the basics of BR: - -- Input data source: The output file of BR -- Supported TiDB versions: v3.1 or later -- Kubernetes support: Yes. The document is WIP. - -#### TiDB Loader (to be deprecated) - > **Note:** -> -> TiDB Loader is to be deprecated and replaced with [Lightning](/tidb-lightning/tidb-lightning-tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-back-end). - -[TiDB Loader](/loader-overview.md) is a lightweight full-data importing tool for TiDB. It reads the output file of Mydumper and loads the data into TiDB. - -The following are the basics of Loader: - -- Input data source: Mydumper’s output file -- Supported TiDB versions: all versions -- Kubernetes support: Yes. See [Backup and restore](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-and-restore-using-helm-charts/) for details. - -### Incremental data import tools - -#### Syncer (deprecated) - -[Syncer](/syncer-overview.md) is a tool used to import data incrementally. It acts as a MySQL slave to read binlog from MySQL/MariaDB master and replicate the binlog to the downstream. It is recommended to use [TiDB Data Migration](#tidb-data-migration) to replace Syncer. - -The following are the basics of Syncer: - -- Input data source: MySQL/MariaDB binlog service -- Supported TiDB versions: all versions -- Kubernetes support: No - -### Full and incremental data import tools - -#### TiDB Data Migration - -[TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process. - -For the full data migration, it uses an embedded Loader and an embedded Mydumper. For the incremental data migration, it uses Syncer as its kernel. - -The following are the basics of DM: - -- Input data source: MySQL/MariaDB master host/port -- Supported TiDB versions: all versions -- Kubernetes support: No, under development (the estimated time is 2020 Q2) - -## Data export (backup) - -### Full data export tools - -#### Mydumper +> +> The Loader tool is no longer maintained. For scenarios related to Loader, it is recommended that you use the [`tidb` mode of TiDB Lighting](/tidb-lightning/tidb-lightning-tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend) instead. -[Mydumper](/mydumper-overview.md) is a tool to create a logical full backup for TiDB. - -The following are the basics of Mydumper: - -- Input/Output - - Input: TiDB/MySQL host:port - - Output: schema and insert statements file -- Supported TiDB versions: all versions -- Kubernetes support: Yes. See [Backup and Restore](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-and-restore-using-helm-charts/) for details. +## Backup and restore -#### BR (beta) +[Dumpling](/export-or-backup-using-dumpling.md) can be used to back up the TiDB cluster to SQL/CSV files. See [Full data export](#full-data-export). -[BR](https://pingcap.com/docs/dev/reference/tools/br/br/) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/loader, BR is more suitable for scenarios of huge data volume. +[TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) can be used to restore the SQL/CSV files exported by Dumpling to the TiDB cluster. See [Full data import](#full-data-import). -The following are the basics of BR: +## Incremental data replication -- Input/Output - - Input: TiDB cluster - - Output: Full backup file -- Supported TiDB versions: v3.1 or v4.0 -- Kubernetes support: Yes. The document is WIP. - -### Incremental data export tools - -#### TiDB Binlog - -[TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) is a system that collects binlog for TiDB clusters and provides tools for near real-time sync and backup. +[TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) is a tool that collects binlog for TiDB clusters and provides near real-time sync and backup. It can be used for incremental data replication between TiDB clusters, such as making a TiDB cluster the secondary cluster of the primary TiDB cluster. The following are the basics of TiDB Binlog: -- Input/Output: - - Input: TiDB Cluster - - Output: MySQL, TiDB, Kafka or incremental backup files +- Input: TiDB cluster +- Output: TiDB cluster, MySQL, Kafka or incremental backup files - Supported TiDB versions: v2.1 or later - Kubernetes support: Yes. See [TiDB Binlog Cluster Operations](https://pingcap.com/docs/tidb-in-kubernetes/stable/deploy-tidb-binlog/) and [TiDB Binlog Drainer Configurations in Kubernetes](https://pingcap.com/docs/tidb-in-kubernetes/stable/configure-tidb-binlog-drainer/) for details. -#### CDC (Beta, under development, ETA May/June 2020 with TiDB 4.0) - -[CDC](https://pingcap.com/docs/dev/reference/tools/ticdc/overview/) (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in row changed order. - -- Input/Output: - - Input: TiDB Cluster - - Output: MySQL, TiDB, Kafka or incremental backup files -- Supported TiDB versions: v4.0 -- Kubernetes support: On the development road map, ETA Q2 2020 - -## Recommended tools for TiDB 3.0 +## Data migration -- MySQL full data backup: use Mydumper -- MySQL full data import to TiDB: - - TB scale: use TiDB Lightning - - Sub-TB scale: use DM -- MySQL incremental data sync to TiDB: use DM -- TiDB full data backup: use Mydumper -- TiDB full data restore: use TiDB Lightning -- TiDB incremental backup & restore: use TiDB-Binlog +[TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB to TiDB. -For the recommended tools for other TiDB versions, see [Recommended tools for TiDB versions](https://pingcap.com/docs/dev/reference/tools/user-guide/#recommended-tools-for-tidb-versions). - -## Tools evolution roadmap - -- TiDB Full Data Backup: - - Mydumper -> BR - - Mydumper -> [dumpling](https://github.com/pingcap/dumpling) (under development, replace Lighting in lightweight scenarios) -- TiDB Full Data Restore: - - Loader -> Lightning -> BR -- MySQL Data Migration: - - Mydumper/Loader + Syncer -> DM (in the next step, we will integrate Lightning into DM) -- TiDB Incremental Data Migration: - - TiDB Binlog -> CDC - -## Full-path data migration solution for TiDB 3.0 - -TiDB 3.0 is the recommended version and is also the most widely adopted version. In addition, TiDB 3.1 GA and 4.0 GA will be released this year. - -For TiDB 3.0 versions, this section covers how to migrate data from MySQL to TiDB, between TiDB clusters, and from TiDB to MySQL for each version, as well as how to back up and restore data. - -### Migrating MySQL data to TiDB - -If the MySQL data volume is in TBs: - -- Use Mydumper to export MySQL full data as a backup -- Use Lightning to import the full MySQL backup data into TiDB cluster -- Use DM to replicate incremental MySQL data to TiDB - -If the MySQL data volume is in GBs: +The following are the basics of DM: -- Use DM to migrate MySQL data to TiDB for both full and incremental data import +- Input: MySQL/MariaDB +- Output: TiDB cluster +- Supported TiDB versions: all versions +- Kubernetes support: No, under development -### Data replication between TiDB/MySQL clusters +If the data volume is below the TB level, it is recommended to migrate data from MySQL/MariaDB to TiDB directly using DM. The migration process includes the full data import and export and the incremental data replication. -You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster. +If the data volume is at the TB level, take the following steps: -### Full backup and restore of the data in TiDB/MySQL clusters +1. Use [Dumpling](/export-or-backup-using-dumpling.md) to export the full data from MySQL/MariaDB. +2. Use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import the data exported in Step 1 to the TiDB cluster. +3. Use DM to migrate the incremental data from MySQL/MariaDB to TiDB. -- Use the Mydumper tool for full data backup -- Use the Lightning tool with `tidb` backend for full data restore +> **Note:** +> +> The Syncer tool is no longer maintained. For scenarios related to Syncer, it is recommended that you use DM's incremental task mode instead. From 4f885e7250e0e3a06065600d5dcee0bfdb98c013 Mon Sep 17 00:00:00 2001 From: Ran Date: Thu, 11 Jun 2020 16:25:51 +0800 Subject: [PATCH 2/3] Update ecosystem-tool-user-case.md --- ecosystem-tool-user-case.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ecosystem-tool-user-case.md b/ecosystem-tool-user-case.md index e74d5947178e3..e2f38c1ea57b6 100644 --- a/ecosystem-tool-user-case.md +++ b/ecosystem-tool-user-case.md @@ -36,4 +36,4 @@ If you also need to migrate incremental data, use [TiDB Binlog](/tidb-binlog/tid ## TiDB incremental data subscription -If you need to subscribe to TiDB's incremental changes, use [TiDB Binlog](/tidb-binlog/binlog-slave-client.md). \ No newline at end of file +If you need to subscribe to TiDB's incremental changes, use [TiDB Binlog](/tidb-binlog/binlog-slave-client.md). From 51b36570f23e4394c5419e368aaec4b7da489bb9 Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Thu, 11 Jun 2020 17:53:54 +0800 Subject: [PATCH 3/3] Update ecosystem-tool-user-case.md --- ecosystem-tool-user-case.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ecosystem-tool-user-case.md b/ecosystem-tool-user-case.md index e2f38c1ea57b6..953abac056df0 100644 --- a/ecosystem-tool-user-case.md +++ b/ecosystem-tool-user-case.md @@ -18,7 +18,7 @@ If you need to import full data from MySQL or Aurora, use [Dumpling](/export-or- ## Migrate data from MySQL/Aurora -If you need to migrate both full data and incremental data from MySQL/Aurora, use [TiDB Data Migration](https://pingcap.com/docs-cn/tidb-data-migration/stable/overview/) (DM) to perform the full and incremental data migration. +If you need to migrate both full data and incremental data from MySQL/Aurora, use [TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/overview/) (DM) to perform the full and incremental data migration. If the full data volume is large (at the TB level), you can first use [Dumpling](/export-or-backup-using-dumpling.md) and [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to perform the full data migration, and then use DM to perform the incremental data migration.