Skip to content

reference: refine the ecosystem tools user guide (#2759) #2793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 11, 2020
Merged

reference: refine the ecosystem tools user guide (#2759) #2793

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ca0a0d5
cherry pick #2759 to release-3.1
ran-huang Jun 11, 2020
c551589
resolve conflict
ran-huang Jun 11, 2020
75e9b12
Update TOC.md
ran-huang Jun 11, 2020
1a531e2
Update TOC.md
ran-huang Jun 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions TOC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -331,8 +331,8 @@
- [Error Handling](/tidb-binlog/handle-tidb-binlog-errors.md)
- [FAQ](/tidb-binlog/tidb-binlog-faq.md)
+ Tools
- [Tools User Guide](/ecosystem-tool-user-guide.md)
- [Download](/download-ecosystem-tools.md)
- [Overview](/ecosystem-tool-user-guide.md)
- [Use Cases](/ecosystem-tool-user-case.md)
+ Backup & Restore (BR)
- [Use BR](/br/backup-and-restore-tool.md)
- [BR Use Cases](/br/backup-and-restore-use-cases.md)
Expand Down
39 changes: 39 additions & 0 deletions ecosystem-tool-user-case.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
title: TiDB Ecosystem Tools Use Cases
summary: Learn the common use cases of TiDB ecosystem tools and how to choose the tools.
category: reference
---

# TiDB Ecosystem Tools Use Cases

This document introduces the common use cases of TiDB ecosystem tools and how to choose the right tool for your scenario.

## Import data from CSV to TiDB

If you need to import the compatible CSV files exported by other tools to TiDB, use [TiDB Lightning](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md).

## Import full data from MySQL/Aurora

If you need to import full data from MySQL or Aurora, use [Dumpling](/export-or-backup-using-dumpling.md) first to export data as SQL dump files, and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import data into the TiDB cluster.

## Migrate data from MySQL/Aurora

If you need to migrate both full data and incremental data from MySQL/Aurora, use [TiDB Data Migration](https://pingcap.com/docs-cn/tidb-data-migration/stable/overview/) (DM) to perform the full and incremental data migration.

If the full data volume is large (at the TB level), you can first use [Dumpling](/export-or-backup-using-dumpling.md) and [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to perform the full data migration, and then use DM to perform the incremental data migration.

## Back up and restore TiDB cluster

If you need to back up a TiDB cluster or restore backed up data to the cluster, use [BR](/br/backup-and-restore-tool.md) (Backup & Restore).

In addition, BR can also be used to perform [incremental backup](/br/backup-and-restore-tool.md#back-up-incremental-data) and [incremental restore](/br/backup-and-restore-tool.md#restore-incremental-data) of TiDB cluster data.

## Migrate data from TiDB

If you need to migrate data from a TiDB cluster to MySQL or to another TiDB cluster, use [Dumpling](/export-or-backup-using-dumpling.md) to export full data from TiDB as SQL dump files, and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import data to MySQL or another TiDB cluster.

If you also need to migrate incremental data, use [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md).

## TiDB incremental data subscription

If you need to subscribe to TiDB's incremental changes, use [TiDB Binlog](/tidb-binlog/binlog-slave-client.md).
194 changes: 43 additions & 151 deletions ecosystem-tool-user-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,194 +1,86 @@
---
title: TiDB Ecosystem Tools User Guide
title: TiDB Ecosystem Tools Overview
category: reference
aliases: ['/docs/v3.1/reference/tools/user-guide/','/docs/v3.1/how-to/migrate/from-mysql/','/docs/v3.1/how-to/migrate/incrementally-from-mysql/','/docs/v3.1/how-to/migrate/overview/']
---

# TiDB Ecosystem Tools User Guide
# TiDB Ecosystem Tools Overview

The TiDB ecosystem has a wealth of tools for data migration, backup & restore for users with different use cases to choose from.
This document introduces the functionalities of TiDB ecosystem tools and their relationship.

- Some of the functionalities of these tools might overlap. For example, TiDB Loader, TiDB Lightning and TiDB DM can all do full data loading.
- Some of the tools might have evolved. For example, TiDB Binlog will be evolved to CDC (Change Data Capture).
- Some of the tools are designed to support specific TiDB versions and the others might be deprecated as user requirements change.
## Full data export

This guide is specifically designed to help you better understand these tools and therefore make an informed decision while choosing these tools to support your business.
[Dumpling](/export-or-backup-using-dumpling.md) is a tool for the logical full data export from MySQL or TiDB.

## Data import (restore or data replication)
The following are the basics of Dumpling:

### Full data import tools
- Input: MySQL/TiDB cluster
- Output: SQL/CSV file
- Supported TiDB versions: all versions
- Kubernetes support: No

#### TiDB Lightning
## Full data import

[TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) (Lightning) is a tool used for the fast full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Mydumper or CSV data source.
[TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) (Lightning) is a tool used for the full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Dumpling or CSV data source.

TiDB Lightning supports two back ends: "Importer" and "TiDB". It determines how tidb-lightning delivers data into the target cluster. The two back ends are as follows:
TiDB Lightning supports two modes:

1. The default one is [`Importer` back end](/tidb-lightning/tidb-lightning-overview.md). When using `Importer` as the back end, the cluster cannot provide normal services during the import process. It is used for a large amount of data importing (TB).
2. The second one is [`TiDB` back end](/tidb-lightning/tidb-lightning-tidb-backend.md) (just work as [Loader](#tidb-loader-to-be-deprecated)). It is much slower than `Importer` back end model. But the cluster could serve the application during the import process. It is used to handle tens/hundreds of GB data.
- `importer`: This mode uses tikv-importer as the backend, which is usually for importing a large amount of data (at the TB level). During the import, the cluster cannot provide services.
- `tidb`: This mode uses TiDB/MySQL as the backend, which is slower than the `importer` mode but can be performed online. It also supports importing data to MySQL.

The following are the basics of TiDB Lightning:

- Input data source:
- The output file of Mydumper
- CSV file
- The output file of Dumpling
- Other compatible CSV file
- Supported TiDB versions: v2.1 or later
- Kubernetes support: Yes. See [Quickly restore data into a TiDB cluster in Kubernetes using TiDB Lightning](https://pingcap.com/docs/tidb-in-kubernetes/stable/restore-data-using-tidb-lightning/) for details.

#### BR (beta)

[BR](/br/backup-and-restore-tool.md) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/Loader/Lightning, BR is more suitable for scenarios of huge data volume.

The following are the basics of BR:

- Input data source: The output file of BR
- Supported TiDB versions: v3.1 or later
- Kubernetes support: Yes. The document is WIP.

#### TiDB Loader (to be deprecated)

> **Note:**
>
> TiDB Loader is to be deprecated and replaced with [Lightning](/tidb-lightning/tidb-lightning-tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-back-end).

[TiDB Loader](/loader-overview.md) is a lightweight full-data importing tool for TiDB. It reads the output file of Mydumper and loads the data into TiDB.

The following are the basics of Loader:

- Input data source: Mydumper’s output file
- Supported TiDB versions: all versions
- Kubernetes support: Yes. See [Backup and restore](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-and-restore-using-helm-charts/) for details.

### Incremental data import tools

#### Syncer (deprecated)

[Syncer](/syncer-overview.md) is a tool used to import data incrementally. It acts as a MySQL slave to read binlog from MySQL/MariaDB master and replicate the binlog to the downstream. It is recommended to use [TiDB Data Migration](#tidb-data-migration) to replace Syncer.

The following are the basics of Syncer:

- Input data source: MySQL/MariaDB binlog service
- Supported TiDB versions: all versions
- Kubernetes support: No

### Full and incremental data import tools

#### TiDB Data Migration

[TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process.

For the full data migration, it uses an embedded Loader and an embedded Mydumper. For the incremental data migration, it uses Syncer as its kernel.

The following are the basics of DM:

- Input data source: MySQL/MariaDB master host/port
- Supported TiDB versions: all versions
- Kubernetes support: No, under development (the estimated time is 2020 Q2)
>
> The Loader tool is no longer maintained. For scenarios related to Loader, it is recommended that you use the [`tidb` mode of TiDB Lighting](/tidb-lightning/tidb-lightning-tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend) instead.

## Data export (backup)
## Backup and restore

### Full data export tools

#### Mydumper

[Mydumper](/mydumper-overview.md) is a tool to create a logical full backup for TiDB.

The following are the basics of Mydumper:

- Input/Output
- Input: TiDB/MySQL host:port
- Output: schema and insert statements file
- Supported TiDB versions: all versions
- Kubernetes support: Yes. See [Backup and Restore](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-and-restore-using-helm-charts/) for details.

#### BR (beta)

[BR](/br/backup-and-restore-tool.md) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/loader, BR is more suitable for scenarios of huge data volume.
[Backup & Restore](/br/backup-and-restore-tool.md) (BR) is a command-line tool for distributed backup and restore of the TiDB cluster data. BR can effectively back up and restore TiDB clusters of huge data volume.

The following are the basics of BR:

- Input/Output
- Input: TiDB cluster
- Output: Full backup file
- Supported TiDB versions: v3.1 or v4.0
- Kubernetes support: Yes. The document is WIP.

### Incremental data export tools
- [Input and output data source](/br/backup-and-restore-tool.md#types-of-backup-files): SST + `backupmeta` file
- Supported TiDB versions: v3.1 and v4.0
- Kubernetes support: Yes. See [Back up Data to S3-Compatible Storage Using BR](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-to-aws-s3-using-br/) and [Restore Data from S3-Compatible Storage Using BR](https://pingcap.com/docs/tidb-in-kubernetes/stable/restore-from-aws-s3-using-br/) for details.

#### TiDB Binlog
## Incremental data replication

[TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) is a system that collects binlog for TiDB clusters and provides tools for near real-time sync and backup.
[TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) is a tool that collects binlog for TiDB clusters and provides near real-time sync and backup. It can be used for incremental data replication between TiDB clusters, such as making a TiDB cluster the secondary cluster of the primary TiDB cluster.

The following are the basics of TiDB Binlog:

- Input/Output:
- Input: TiDB Cluster
- Output: MySQL, TiDB, Kafka or incremental backup files
- Input: TiDB cluster
- Output: TiDB cluster, MySQL, Kafka or incremental backup files
- Supported TiDB versions: v2.1 or later
- Kubernetes support: Yes. See [TiDB Binlog Cluster Operations](https://pingcap.com/docs/tidb-in-kubernetes/stable/deploy-tidb-binlog/) and [TiDB Binlog Drainer Configurations in Kubernetes](https://pingcap.com/docs/tidb-in-kubernetes/stable/configure-tidb-binlog-drainer/) for details.

#### CDC (Beta, under development, ETA May/June 2020 with TiDB 4.0)

[CDC](https://pingcap.com/docs/dev/reference/tools/ticdc/overview/) (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in row changed order.

- Input/Output:
- Input: TiDB Cluster
- Output: MySQL, TiDB, Kafka or incremental backup files
- Supported TiDB versions: v4.0
- Kubernetes support: On the development road map, ETA Q2 2020

## Recommended tools for TiDB 3.1

- MySQL full data backup: use Mydumper
- MySQL full data import to TiDB:
- TB scale: use TiDB Lightning
- Sub-TB scale: use DM
- MySQL incremental data sync to TiDB: use DM
- TiDB full data backup: use BR
- TiDB full data restore: use BR
- TiDB incremental backup & restore: use TiDB-Binlog
## Data migration

For the recommended tools for other TiDB versions, see [Recommended tools for TiDB versions](https://pingcap.com/docs/dev/reference/tools/user-guide/#recommended-tools-for-tidb-versions).
[TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB to TiDB.

## Tools evolution roadmap

- TiDB Full Data Backup:
- Mydumper -> BR
- Mydumper -> [dumpling](https://github.com/pingcap/dumpling) (under development, replace Lighting in lightweight scenarios)
- TiDB Full Data Restore:
- Loader -> Lightning -> BR
- MySQL Data Migration:
- Mydumper/Loader + Syncer -> DM (in the next step, we will integrate Lightning into DM)
- TiDB Incremental Data Migration:
- TiDB Binlog -> CDC

## Full-path data migration solution for TiDB 3.1

For TiDB 3.1 versions, this section covers how to migrate data from MySQL to TiDB, between TiDB clusters, and from TiDB to MySQL for each version, as well as how to back up and restore data.

### Migrating MySQL data to TiDB

If the MySQL data volume is in TBs:

- Use Mydumper to export MySQL full data as a backup
- Use Lightning to import the full MySQL backup data into TiDB cluster
- Use DM to replicate incremental MySQL data to TiDB

If the MySQL data volume is in GBs:

- Use DM to migrate MySQL data to TiDB for both full and incremental data import

### Data replication between TiDB/MySQL clusters

You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster.
The following are the basics of DM:

### Full backup and restore of the data in TiDB/MySQL clusters
- Input: MySQL/MariaDB
- Output: TiDB cluster
- Supported TiDB versions: all versions
- Kubernetes support: No, under development

To restore data to a TiDB cluster:
If the data volume is below the TB level, it is recommended to migrate data from MySQL/MariaDB to TiDB directly using DM. The migration process includes the full data import and export and the incremental data replication.

- Use the BR tool for both full data backup and full data restore
If the data volume is at the TB level, take the following steps:

To restore data to a MySQL cluster:
1. Use [Dumpling](/export-or-backup-using-dumpling.md) to export the full data from MySQL/MariaDB.
2. Use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import the data exported in Step 1 to the TiDB cluster.
3. Use DM to migrate the incremental data from MySQL/MariaDB to TiDB.

- Use the Mydumper tool for full data backup
- Use the Lightning tool with `tidb` backend for full data restore
> **Note:**
>
> The Syncer tool is no longer maintained. For scenarios related to Syncer, it is recommended that you use DM's incremental task mode instead.