From ca0a0d524fbb3e5523491df5e80888fbb397389f Mon Sep 17 00:00:00 2001 From: Ran Date: Thu, 11 Jun 2020 15:08:29 +0800 Subject: [PATCH 1/4] cherry pick #2759 to release-3.1 Signed-off-by: sre-bot --- TOC.md | 116 +++++++++++++++++++++++++++ ecosystem-tool-user-case.md | 39 ++++++++++ ecosystem-tool-user-guide.md | 147 ++++++++++++----------------------- 3 files changed, 204 insertions(+), 98 deletions(-) create mode 100644 ecosystem-tool-user-case.md diff --git a/TOC.md b/TOC.md index e6bb10df0efe4..480581b950e0e 100644 --- a/TOC.md +++ b/TOC.md @@ -88,6 +88,7 @@ - [Identify Slow Queries](/identify-slow-queries.md) - [Identify Expensive Queries](/identify-expensive-queries.md) + Scale +<<<<<<< HEAD - [Scale using Ansible](/scale-tidb-using-ansible.md) + Upgrade - [Upgrade to TiDB 3.1](/upgrade-tidb-using-ansible.md) @@ -95,6 +96,121 @@ - [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - [Troubleshoot TiDB Lightning](/troubleshoot-tidb-lightning.md) +======= + + [Use TiUP (Recommended)](/scale-tidb-using-tiup.md) + + [Use TiDB Ansible](/scale-tidb-using-ansible.md) + + [Use TiDB Operator](https://pingcap.com/docs/tidb-in-kubernetes/stable/scale-a-tidb-cluster/) + + Backup and Restore + + [Use Mydumper and TiDB Lightning](/backup-and-restore-using-mydumper-lightning.md) + + [Use Dumpling for Export or Backup](/export-or-backup-using-dumpling.md) + + Use BR Tool + + [Use BR Tool](/br/backup-and-restore-tool.md) + + [BR Use Cases](/br/backup-and-restore-use-cases.md) + + [BR storages](/br/backup-and-restore-storages.md) + + [Daily Checklist](/daily-check.md) + + [Maintain TiCDC](/ticdc/manage-ticdc.md) + + [Maintain TiFlash](/tiflash/maintain-tiflash.md) + + [Maintain TiDB Using TiUP](/maintain-tidb-using-tiup.md) + + [Maintain TiDB Using Ansible](/maintain-tidb-using-ansible.md) ++ Monitor and Alert + + [Monitoring Framework](/tidb-monitoring-framework.md) + + [Monitor a TiDB Cluster](/monitor-a-tidb-cluster.md) + + [TiDB Cluster Alert Rules](/alert-rules.md) + + [TiFlash Alert Rules](/tiflash/tiflash-alert-rules.md) ++ Troubleshoot + + [Identify Slow Queries](/identify-slow-queries.md) + + [Identify Expensive Queries](/identify-expensive-queries.md) + + [Statement Summary Tables](/statement-summary-tables.md) + + [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) + + [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) + + [Troubleshoot TiFlash](/tiflash/troubleshoot-tiflash.md) ++ Performance Tuning + + Software Tuning + + Configuration + + [TiKV Tuning](/tune-tikv-performance.md) + + [TiKV Follower Read](/follower-read.md) + + [TiFlash Tuning](/tiflash/tune-tiflash-performance.md) + + [Coprocessor Cache](/coprocessor-cache.md) + + SQL Tuning + + [SQL Tuning with `EXPLAIN`](/query-execution-plan.md) + + SQL Optimization + + [SQL Optimization Process](/sql-optimization-concepts.md) + + Logic Optimization + + [Join Reorder](/join-reorder.md) + + Physical Optimization + + [Statistics](/statistics.md) + + Control Execution Plan + + [Optimizer Hints](/optimizer-hints.md) + + [SQL Plan Management](/sql-plan-management.md) + + [Access Tables Using `IndexMerge`](/index-merge.md) ++ Tutorials + + [Geo-Redundant Deployment](/geo-redundancy-deployment.md) + + Best Practices + + [Java Application Development](/best-practices/java-app-best-practices.md) + + [Use HAProxy](/best-practices/haproxy-best-practices.md) + + [Highly Concurrent Write](/best-practices/high-concurrency-best-practices.md) + + [Grafana Monitoring](/best-practices/grafana-monitor-best-practices.md) + + [PD Scheduling](/best-practices/pd-scheduling-best-practices.md) + + [TiKV Performance Tuning with Massive Regions](/best-practices/massive-regions-best-practices.md) + + [Use Placement Rules](/configure-placement-rules.md) ++ TiDB Ecosystem Tools + + [Overview](/ecosystem-tool-user-guide.md) + + [Use Cases](/ecosystem-tool-user-case.md) + + [Download](/download-ecosystem-tools.md) + + Backup & Restore (BR) + + [BR FAQ](/br/backup-and-restore-faq.md) + + [Use BR Tool](/br/backup-and-restore-tool.md) + + [BR Use Cases](/br/backup-and-restore-use-cases.md) + + TiDB Binlog + + [Overview](/tidb-binlog/tidb-binlog-overview.md) + + [Deploy](/tidb-binlog/deploy-tidb-binlog.md) + + [Maintain](/tidb-binlog/maintain-tidb-binlog-cluster.md) + + [Configure](/tidb-binlog/tidb-binlog-configuration-file.md) + + [Pump](/tidb-binlog/tidb-binlog-configuration-file.md#pump) + + [Drainer](/tidb-binlog/tidb-binlog-configuration-file.md#drainer) + + [Upgrade](/tidb-binlog/upgrade-tidb-binlog.md) + + [Monitor](/tidb-binlog/monitor-tidb-binlog-cluster.md) + + [Reparo](/tidb-binlog/tidb-binlog-reparo.md) + + [Binlog Slave Client](/tidb-binlog/binlog-slave-client.md) + + [TiDB Binlog Relay Log](/tidb-binlog/tidb-binlog-relay-log.md) + + [Bidirectional Replication Between TiDB Clusters](/tidb-binlog/bidirectional-replication-between-tidb-clusters.md) + + [Glossary](/tidb-binlog/tidb-binlog-glossary.md) + + Troubleshoot + + [Troubleshoot](/tidb-binlog/troubleshoot-tidb-binlog.md) + + [Handle Errors](/tidb-binlog/handle-tidb-binlog-errors.md) + + [FAQ](/tidb-binlog/tidb-binlog-faq.md) + + TiDB Lightning + + [Overview](/tidb-lightning/tidb-lightning-overview.md) + + [Tutorial](/get-started-with-tidb-lightning.md) + + [Deploy](/tidb-lightning/deploy-tidb-lightning.md) + + [Configure](/tidb-lightning/tidb-lightning-configuration.md) + + Key Features + + [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md) + + [Table Filter](/tidb-lightning/tidb-lightning-table-filter.md) + + [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) + + [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) + + [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md) + + [Monitor](/tidb-lightning/monitor-tidb-lightning.md) + + [Troubleshoot](/troubleshoot-tidb-lightning.md) + + [FAQ](/tidb-lightning/tidb-lightning-faq.md) + + [Glossary](/tidb-lightning/tidb-lightning-glossary.md) + + TiCDC + + [Overview](/ticdc/ticdc-overview.md) + + [Configure Sink URI](/ticdc/sink-url.md) + + [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) + + [Column and DDL Type Codes](/ticdc/column-ddl-type-codes.md) + + sync-diff-inspector + + [Overview](/sync-diff-inspector/sync-diff-inspector-overview.md) + + [Data Check for Tables with Different Schema/Table Names](/sync-diff-inspector/route-diff.md) + + [Data Check in Sharding Scenarios](/sync-diff-inspector/shard-diff.md) + + [Data Check for TiDB Upstream/Downstream Clusters](/sync-diff-inspector/upstream-downstream-diff.md) + + [Loader](/loader-overview.md) + + [Mydumper](/mydumper-overview.md) + + [Syncer](/syncer-overview.md) + + TiSpark + + [Quick Start](/get-started-with-tispark.md) + + [User Guide](/tispark-overview.md) +>>>>>>> 37cbeea... reference: refine the ecosystem tools user guide (#2759) + Reference + SQL - [MySQL Compatibility](/mysql-compatibility.md) diff --git a/ecosystem-tool-user-case.md b/ecosystem-tool-user-case.md new file mode 100644 index 0000000000000..11c21150b1679 --- /dev/null +++ b/ecosystem-tool-user-case.md @@ -0,0 +1,39 @@ +--- +title: TiDB Ecosystem Tools Use Cases +summary: Learn the common use cases of TiDB ecosystem tools and how to choose the tools. +category: reference +--- + +# TiDB Ecosystem Tools Use Cases + +This document introduces the common use cases of TiDB ecosystem tools and how to choose the right tool for your scenario. + +## Import data from CSV to TiDB + +If you need to import the compatible CSV files exported by other tools to TiDB, use [TiDB Lightning](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md). + +## Import full data from MySQL/Aurora + +If you need to import full data from MySQL or Aurora, use [Dumpling](/export-or-backup-using-dumpling.md) first to export data as SQL dump files, and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import data into the TiDB cluster. + +## Migrate data from MySQL/Aurora + +If you need to migrate both full data and incremental data from MySQL/Aurora, use [TiDB Data Migration](https://pingcap.com/docs-cn/tidb-data-migration/stable/overview/) (DM) to perform the full and incremental data migration. + +If the full data volume is large (at the TB level), you can first use [Dumpling](/export-or-backup-using-dumpling.md) and [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to perform the full data migration, and then use DM to perform the incremental data migration. + +## Back up and restore TiDB cluster + +If you need to back up a TiDB cluster or restore backed up data to the cluster, use [BR](/br/backup-and-restore-tool.md) (Backup & Restore). + +In addition, BR can also be used to perform [incremental backup](/br/backup-and-restore-tool.md#back-up-incremental-data) and [incremental restore](/br/backup-and-restore-tool.md#restore-incremental-data) of TiDB cluster data. + +## Migrate data from TiDB + +If you need to migrate data from a TiDB cluster to MySQL or to another TiDB cluster, use [Dumpling](/export-or-backup-using-dumpling.md) to export full data from TiDB as SQL dump files, and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import data to MySQL or another TiDB cluster. + +If you also need to migrate incremental data, use [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md). + +## TiDB incremental data subscription + +If you need to subscribe to TiDB's incremental changes, use [TiDB Binlog](/tidb-binlog/binlog-slave-client.md). diff --git a/ecosystem-tool-user-guide.md b/ecosystem-tool-user-guide.md index 30479823d15de..4a47f15762e04 100644 --- a/ecosystem-tool-user-guide.md +++ b/ecosystem-tool-user-guide.md @@ -1,132 +1,68 @@ --- -title: TiDB Ecosystem Tools User Guide +title: TiDB Ecosystem Tools Overview category: reference aliases: ['/docs/v3.1/reference/tools/user-guide/','/docs/v3.1/how-to/migrate/from-mysql/','/docs/v3.1/how-to/migrate/incrementally-from-mysql/','/docs/v3.1/how-to/migrate/overview/'] --- -# TiDB Ecosystem Tools User Guide +# TiDB Ecosystem Tools Overview -The TiDB ecosystem has a wealth of tools for data migration, backup & restore for users with different use cases to choose from. +This document introduces the functionalities of TiDB ecosystem tools and their relationship. -- Some of the functionalities of these tools might overlap. For example, TiDB Loader, TiDB Lightning and TiDB DM can all do full data loading. -- Some of the tools might have evolved. For example, TiDB Binlog will be evolved to CDC (Change Data Capture). -- Some of the tools are designed to support specific TiDB versions and the others might be deprecated as user requirements change. +## Full data export -This guide is specifically designed to help you better understand these tools and therefore make an informed decision while choosing these tools to support your business. +[Dumpling](/export-or-backup-using-dumpling.md) is a tool for the logical full data export from MySQL or TiDB. -## Data import (restore or data replication) +The following are the basics of Dumpling: -### Full data import tools +- Input: MySQL/TiDB cluster +- Output: SQL/CSV file +- Supported TiDB versions: all versions +- Kubernetes support: No -#### TiDB Lightning +## Full data import -[TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) (Lightning) is a tool used for the fast full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Mydumper or CSV data source. +[TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) (Lightning) is a tool used for the full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Dumpling or CSV data source. -TiDB Lightning supports two back ends: "Importer" and "TiDB". It determines how tidb-lightning delivers data into the target cluster. The two back ends are as follows: +TiDB Lightning supports two modes: -1. The default one is [`Importer` back end](/tidb-lightning/tidb-lightning-overview.md). When using `Importer` as the back end, the cluster cannot provide normal services during the import process. It is used for a large amount of data importing (TB). -2. The second one is [`TiDB` back end](/tidb-lightning/tidb-lightning-tidb-backend.md) (just work as [Loader](#tidb-loader-to-be-deprecated)). It is much slower than `Importer` back end model. But the cluster could serve the application during the import process. It is used to handle tens/hundreds of GB data. +- `importer`: This mode uses tikv-importer as the backend, which is usually for importing a large amount of data (at the TB level). During the import, the cluster cannot provide services. +- `tidb`: This mode uses TiDB/MySQL as the backend, which is slower than the `importer` mode but can be performed online. It also supports importing data to MySQL. The following are the basics of TiDB Lightning: - Input data source: - - The output file of Mydumper - - CSV file + - The output file of Dumpling + - Other compatible CSV file - Supported TiDB versions: v2.1 or later - Kubernetes support: Yes. See [Quickly restore data into a TiDB cluster in Kubernetes using TiDB Lightning](https://pingcap.com/docs/tidb-in-kubernetes/stable/restore-data-using-tidb-lightning/) for details. -#### BR (beta) - -[BR](/br/backup-and-restore-tool.md) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/Loader/Lightning, BR is more suitable for scenarios of huge data volume. - -The following are the basics of BR: - -- Input data source: The output file of BR -- Supported TiDB versions: v3.1 or later -- Kubernetes support: Yes. The document is WIP. - -#### TiDB Loader (to be deprecated) - > **Note:** -> -> TiDB Loader is to be deprecated and replaced with [Lightning](/tidb-lightning/tidb-lightning-tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-back-end). - -[TiDB Loader](/loader-overview.md) is a lightweight full-data importing tool for TiDB. It reads the output file of Mydumper and loads the data into TiDB. - -The following are the basics of Loader: - -- Input data source: Mydumper’s output file -- Supported TiDB versions: all versions -- Kubernetes support: Yes. See [Backup and restore](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-and-restore-using-helm-charts/) for details. +> +> The Loader tool is no longer maintained. For scenarios related to Loader, it is recommended that you use the `tidb` mode of TiDB Lighting instead. -### Incremental data import tools +## Backup and restore -#### Syncer (deprecated) - -[Syncer](/syncer-overview.md) is a tool used to import data incrementally. It acts as a MySQL slave to read binlog from MySQL/MariaDB master and replicate the binlog to the downstream. It is recommended to use [TiDB Data Migration](#tidb-data-migration) to replace Syncer. - -The following are the basics of Syncer: - -- Input data source: MySQL/MariaDB binlog service -- Supported TiDB versions: all versions -- Kubernetes support: No - -### Full and incremental data import tools - -#### TiDB Data Migration - -[TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process. - -For the full data migration, it uses an embedded Loader and an embedded Mydumper. For the incremental data migration, it uses Syncer as its kernel. - -The following are the basics of DM: - -- Input data source: MySQL/MariaDB master host/port -- Supported TiDB versions: all versions -- Kubernetes support: No, under development (the estimated time is 2020 Q2) - -## Data export (backup) - -### Full data export tools - -#### Mydumper - -[Mydumper](/mydumper-overview.md) is a tool to create a logical full backup for TiDB. - -The following are the basics of Mydumper: - -- Input/Output - - Input: TiDB/MySQL host:port - - Output: schema and insert statements file -- Supported TiDB versions: all versions -- Kubernetes support: Yes. See [Backup and Restore](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-and-restore-using-helm-charts/) for details. - -#### BR (beta) - -[BR](/br/backup-and-restore-tool.md) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/loader, BR is more suitable for scenarios of huge data volume. +[Backup & Restore](/br/backup-and-restore-tool.md) (BR) is a command-line tool for distributed backup and restore of the TiDB cluster data. BR can effectively back up and restore TiDB clusters of huge data volume. The following are the basics of BR: -- Input/Output - - Input: TiDB cluster - - Output: Full backup file -- Supported TiDB versions: v3.1 or v4.0 -- Kubernetes support: Yes. The document is WIP. - -### Incremental data export tools +- [Input and output data source](/br/backup-and-restore-tool.md#types-of-backup-files): SST + `backupmeta` file +- Supported TiDB versions: v3.1 and v4.0 +- Kubernetes support: Yes. See [Back up Data to S3-Compatible Storage Using BR](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-to-aws-s3-using-br/) and [Restore Data from S3-Compatible Storage Using BR](https://pingcap.com/docs/tidb-in-kubernetes/stable/restore-from-aws-s3-using-br/) for details. -#### TiDB Binlog +## Incremental data replication -[TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) is a system that collects binlog for TiDB clusters and provides tools for near real-time sync and backup. +[TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) is a tool that collects binlog for TiDB clusters and provides near real-time sync and backup. It can be used for incremental data replication between TiDB clusters, such as making a TiDB cluster the secondary cluster of the primary TiDB cluster. The following are the basics of TiDB Binlog: - Input/Output: - - Input: TiDB Cluster - - Output: MySQL, TiDB, Kafka or incremental backup files + - Input: TiDB cluster + - Output: TiDB cluster, MySQL, Kafka or incremental backup files - Supported TiDB versions: v2.1 or later - Kubernetes support: Yes. See [TiDB Binlog Cluster Operations](https://pingcap.com/docs/tidb-in-kubernetes/stable/deploy-tidb-binlog/) and [TiDB Binlog Drainer Configurations in Kubernetes](https://pingcap.com/docs/tidb-in-kubernetes/stable/configure-tidb-binlog-drainer/) for details. +<<<<<<< HEAD #### CDC (Beta, under development, ETA May/June 2020 with TiDB 4.0) [CDC](https://pingcap.com/docs/dev/reference/tools/ticdc/overview/) (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in row changed order. @@ -183,12 +119,27 @@ If the MySQL data volume is in GBs: You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster. ### Full backup and restore of the data in TiDB/MySQL clusters +======= +### Data migration -To restore data to a TiDB cluster: +[TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB to TiDB. -- Use the BR tool for both full data backup and full data restore +The following are the basics of DM: -To restore data to a MySQL cluster: +- Input: MySQL/MariaDB +- Output: TiDB cluster +- Supported TiDB versions: all versions +- Kubernetes support: No, under development +>>>>>>> 37cbeea... reference: refine the ecosystem tools user guide (#2759) + +If the data volume is below the TB level, it is recommended to migrate data from MySQL/MariaDB to TiDB directly using DM. The migration process includes the full data import and export and the incremental data replication. + +If the data volume is at the TB level, take the following steps: -- Use the Mydumper tool for full data backup -- Use the Lightning tool with `tidb` backend for full data restore +1. Use [Dumpling](/export-or-backup-using-dumpling.md) to export the full data from MySQL/MariaDB. +2. Use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import the data exported in Step 1 to the TiDB cluster. +3. Use DM to migrate the incremental data from MySQL/MariaDB to TiDB. + +> **Note:** +> +> The Syncer tool is no longer maintained. For scenarios related to Syncer, it is recommended that you use DM's incremental task mode instead. From c55158968a090d3922c2d60b31ec6ff06f25c025 Mon Sep 17 00:00:00 2001 From: Ran Date: Thu, 11 Jun 2020 15:58:01 +0800 Subject: [PATCH 2/4] resolve conflict --- TOC.md | 124 +---------------------------------- ecosystem-tool-user-guide.md | 63 +----------------- 2 files changed, 4 insertions(+), 183 deletions(-) diff --git a/TOC.md b/TOC.md index 480581b950e0e..039e5649bcb96 100644 --- a/TOC.md +++ b/TOC.md @@ -88,7 +88,6 @@ - [Identify Slow Queries](/identify-slow-queries.md) - [Identify Expensive Queries](/identify-expensive-queries.md) + Scale -<<<<<<< HEAD - [Scale using Ansible](/scale-tidb-using-ansible.md) + Upgrade - [Upgrade to TiDB 3.1](/upgrade-tidb-using-ansible.md) @@ -96,121 +95,6 @@ - [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - [Troubleshoot TiDB Lightning](/troubleshoot-tidb-lightning.md) -======= - + [Use TiUP (Recommended)](/scale-tidb-using-tiup.md) - + [Use TiDB Ansible](/scale-tidb-using-ansible.md) - + [Use TiDB Operator](https://pingcap.com/docs/tidb-in-kubernetes/stable/scale-a-tidb-cluster/) - + Backup and Restore - + [Use Mydumper and TiDB Lightning](/backup-and-restore-using-mydumper-lightning.md) - + [Use Dumpling for Export or Backup](/export-or-backup-using-dumpling.md) - + Use BR Tool - + [Use BR Tool](/br/backup-and-restore-tool.md) - + [BR Use Cases](/br/backup-and-restore-use-cases.md) - + [BR storages](/br/backup-and-restore-storages.md) - + [Daily Checklist](/daily-check.md) - + [Maintain TiCDC](/ticdc/manage-ticdc.md) - + [Maintain TiFlash](/tiflash/maintain-tiflash.md) - + [Maintain TiDB Using TiUP](/maintain-tidb-using-tiup.md) - + [Maintain TiDB Using Ansible](/maintain-tidb-using-ansible.md) -+ Monitor and Alert - + [Monitoring Framework](/tidb-monitoring-framework.md) - + [Monitor a TiDB Cluster](/monitor-a-tidb-cluster.md) - + [TiDB Cluster Alert Rules](/alert-rules.md) - + [TiFlash Alert Rules](/tiflash/tiflash-alert-rules.md) -+ Troubleshoot - + [Identify Slow Queries](/identify-slow-queries.md) - + [Identify Expensive Queries](/identify-expensive-queries.md) - + [Statement Summary Tables](/statement-summary-tables.md) - + [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - + [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - + [Troubleshoot TiFlash](/tiflash/troubleshoot-tiflash.md) -+ Performance Tuning - + Software Tuning - + Configuration - + [TiKV Tuning](/tune-tikv-performance.md) - + [TiKV Follower Read](/follower-read.md) - + [TiFlash Tuning](/tiflash/tune-tiflash-performance.md) - + [Coprocessor Cache](/coprocessor-cache.md) - + SQL Tuning - + [SQL Tuning with `EXPLAIN`](/query-execution-plan.md) - + SQL Optimization - + [SQL Optimization Process](/sql-optimization-concepts.md) - + Logic Optimization - + [Join Reorder](/join-reorder.md) - + Physical Optimization - + [Statistics](/statistics.md) - + Control Execution Plan - + [Optimizer Hints](/optimizer-hints.md) - + [SQL Plan Management](/sql-plan-management.md) - + [Access Tables Using `IndexMerge`](/index-merge.md) -+ Tutorials - + [Geo-Redundant Deployment](/geo-redundancy-deployment.md) - + Best Practices - + [Java Application Development](/best-practices/java-app-best-practices.md) - + [Use HAProxy](/best-practices/haproxy-best-practices.md) - + [Highly Concurrent Write](/best-practices/high-concurrency-best-practices.md) - + [Grafana Monitoring](/best-practices/grafana-monitor-best-practices.md) - + [PD Scheduling](/best-practices/pd-scheduling-best-practices.md) - + [TiKV Performance Tuning with Massive Regions](/best-practices/massive-regions-best-practices.md) - + [Use Placement Rules](/configure-placement-rules.md) -+ TiDB Ecosystem Tools - + [Overview](/ecosystem-tool-user-guide.md) - + [Use Cases](/ecosystem-tool-user-case.md) - + [Download](/download-ecosystem-tools.md) - + Backup & Restore (BR) - + [BR FAQ](/br/backup-and-restore-faq.md) - + [Use BR Tool](/br/backup-and-restore-tool.md) - + [BR Use Cases](/br/backup-and-restore-use-cases.md) - + TiDB Binlog - + [Overview](/tidb-binlog/tidb-binlog-overview.md) - + [Deploy](/tidb-binlog/deploy-tidb-binlog.md) - + [Maintain](/tidb-binlog/maintain-tidb-binlog-cluster.md) - + [Configure](/tidb-binlog/tidb-binlog-configuration-file.md) - + [Pump](/tidb-binlog/tidb-binlog-configuration-file.md#pump) - + [Drainer](/tidb-binlog/tidb-binlog-configuration-file.md#drainer) - + [Upgrade](/tidb-binlog/upgrade-tidb-binlog.md) - + [Monitor](/tidb-binlog/monitor-tidb-binlog-cluster.md) - + [Reparo](/tidb-binlog/tidb-binlog-reparo.md) - + [Binlog Slave Client](/tidb-binlog/binlog-slave-client.md) - + [TiDB Binlog Relay Log](/tidb-binlog/tidb-binlog-relay-log.md) - + [Bidirectional Replication Between TiDB Clusters](/tidb-binlog/bidirectional-replication-between-tidb-clusters.md) - + [Glossary](/tidb-binlog/tidb-binlog-glossary.md) - + Troubleshoot - + [Troubleshoot](/tidb-binlog/troubleshoot-tidb-binlog.md) - + [Handle Errors](/tidb-binlog/handle-tidb-binlog-errors.md) - + [FAQ](/tidb-binlog/tidb-binlog-faq.md) - + TiDB Lightning - + [Overview](/tidb-lightning/tidb-lightning-overview.md) - + [Tutorial](/get-started-with-tidb-lightning.md) - + [Deploy](/tidb-lightning/deploy-tidb-lightning.md) - + [Configure](/tidb-lightning/tidb-lightning-configuration.md) - + Key Features - + [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md) - + [Table Filter](/tidb-lightning/tidb-lightning-table-filter.md) - + [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) - + [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) - + [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md) - + [Monitor](/tidb-lightning/monitor-tidb-lightning.md) - + [Troubleshoot](/troubleshoot-tidb-lightning.md) - + [FAQ](/tidb-lightning/tidb-lightning-faq.md) - + [Glossary](/tidb-lightning/tidb-lightning-glossary.md) - + TiCDC - + [Overview](/ticdc/ticdc-overview.md) - + [Configure Sink URI](/ticdc/sink-url.md) - + [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) - + [Column and DDL Type Codes](/ticdc/column-ddl-type-codes.md) - + sync-diff-inspector - + [Overview](/sync-diff-inspector/sync-diff-inspector-overview.md) - + [Data Check for Tables with Different Schema/Table Names](/sync-diff-inspector/route-diff.md) - + [Data Check in Sharding Scenarios](/sync-diff-inspector/shard-diff.md) - + [Data Check for TiDB Upstream/Downstream Clusters](/sync-diff-inspector/upstream-downstream-diff.md) - + [Loader](/loader-overview.md) - + [Mydumper](/mydumper-overview.md) - + [Syncer](/syncer-overview.md) - + TiSpark - + [Quick Start](/get-started-with-tispark.md) - + [User Guide](/tispark-overview.md) ->>>>>>> 37cbeea... reference: refine the ecosystem tools user guide (#2759) + Reference + SQL - [MySQL Compatibility](/mysql-compatibility.md) @@ -447,12 +331,8 @@ - [Error Handling](/tidb-binlog/handle-tidb-binlog-errors.md) - [FAQ](/tidb-binlog/tidb-binlog-faq.md) + Tools - - [Tools User Guide](/ecosystem-tool-user-guide.md) - - [Download](/download-ecosystem-tools.md) - + Backup & Restore (BR) - - [Use BR](/br/backup-and-restore-tool.md) - - [BR Use Cases](/br/backup-and-restore-use-cases.md) - - [BR FAQ](/br/backup-and-restore-faq.md) + - [Overview](/ecosystem-tool-user-guide.md) + - [Use Cases](/ecosystem-tool-user-case.md) - [Mydumper](/mydumper-overview.md) - [Syncer](/syncer-overview.md) - [Loader](/loader-overview.md) diff --git a/ecosystem-tool-user-guide.md b/ecosystem-tool-user-guide.md index 4a47f15762e04..0d3b659f37baa 100644 --- a/ecosystem-tool-user-guide.md +++ b/ecosystem-tool-user-guide.md @@ -38,7 +38,7 @@ The following are the basics of TiDB Lightning: > **Note:** > -> The Loader tool is no longer maintained. For scenarios related to Loader, it is recommended that you use the `tidb` mode of TiDB Lighting instead. +> The Loader tool is no longer maintained. For scenarios related to Loader, it is recommended that you use the [`tidb` mode of TiDB Lighting](/tidb-lightning/tidb-lightning-tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend) instead. ## Backup and restore @@ -62,65 +62,7 @@ The following are the basics of TiDB Binlog: - Supported TiDB versions: v2.1 or later - Kubernetes support: Yes. See [TiDB Binlog Cluster Operations](https://pingcap.com/docs/tidb-in-kubernetes/stable/deploy-tidb-binlog/) and [TiDB Binlog Drainer Configurations in Kubernetes](https://pingcap.com/docs/tidb-in-kubernetes/stable/configure-tidb-binlog-drainer/) for details. -<<<<<<< HEAD -#### CDC (Beta, under development, ETA May/June 2020 with TiDB 4.0) - -[CDC](https://pingcap.com/docs/dev/reference/tools/ticdc/overview/) (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in row changed order. - -- Input/Output: - - Input: TiDB Cluster - - Output: MySQL, TiDB, Kafka or incremental backup files -- Supported TiDB versions: v4.0 -- Kubernetes support: On the development road map, ETA Q2 2020 - -## Recommended tools for TiDB 3.1 - -- MySQL full data backup: use Mydumper -- MySQL full data import to TiDB: - - TB scale: use TiDB Lightning - - Sub-TB scale: use DM -- MySQL incremental data sync to TiDB: use DM -- TiDB full data backup: use BR -- TiDB full data restore: use BR -- TiDB incremental backup & restore: use TiDB-Binlog - -For the recommended tools for other TiDB versions, see [Recommended tools for TiDB versions](https://pingcap.com/docs/dev/reference/tools/user-guide/#recommended-tools-for-tidb-versions). - -## Tools evolution roadmap - -- TiDB Full Data Backup: - - Mydumper -> BR - - Mydumper -> [dumpling](https://github.com/pingcap/dumpling) (under development, replace Lighting in lightweight scenarios) -- TiDB Full Data Restore: - - Loader -> Lightning -> BR -- MySQL Data Migration: - - Mydumper/Loader + Syncer -> DM (in the next step, we will integrate Lightning into DM) -- TiDB Incremental Data Migration: - - TiDB Binlog -> CDC - -## Full-path data migration solution for TiDB 3.1 - -For TiDB 3.1 versions, this section covers how to migrate data from MySQL to TiDB, between TiDB clusters, and from TiDB to MySQL for each version, as well as how to back up and restore data. - -### Migrating MySQL data to TiDB - -If the MySQL data volume is in TBs: - -- Use Mydumper to export MySQL full data as a backup -- Use Lightning to import the full MySQL backup data into TiDB cluster -- Use DM to replicate incremental MySQL data to TiDB - -If the MySQL data volume is in GBs: - -- Use DM to migrate MySQL data to TiDB for both full and incremental data import - -### Data replication between TiDB/MySQL clusters - -You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster. - -### Full backup and restore of the data in TiDB/MySQL clusters -======= -### Data migration +## Data migration [TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB to TiDB. @@ -130,7 +72,6 @@ The following are the basics of DM: - Output: TiDB cluster - Supported TiDB versions: all versions - Kubernetes support: No, under development ->>>>>>> 37cbeea... reference: refine the ecosystem tools user guide (#2759) If the data volume is below the TB level, it is recommended to migrate data from MySQL/MariaDB to TiDB directly using DM. The migration process includes the full data import and export and the incremental data replication. From 75e9b1271c1c24629cdac94e09d3a147754e74dd Mon Sep 17 00:00:00 2001 From: Ran Date: Thu, 11 Jun 2020 16:02:36 +0800 Subject: [PATCH 3/4] Update TOC.md --- TOC.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/TOC.md b/TOC.md index 039e5649bcb96..5ebb0488b5def 100644 --- a/TOC.md +++ b/TOC.md @@ -333,6 +333,10 @@ + Tools - [Overview](/ecosystem-tool-user-guide.md) - [Use Cases](/ecosystem-tool-user-case.md) + + Backup & Restore (BR) + - [Use BR](/br/backup-and-restore-tool.md) + - [BR Use Cases](/br/backup-and-restore-use-cases.md) + - [BR FAQ](/br/backup-and-restore-faq.md) - [Mydumper](/mydumper-overview.md) - [Syncer](/syncer-overview.md) - [Loader](/loader-overview.md) From 1a531e2d1d15af6f1ea6e8cb7b96e6ddf28b8e78 Mon Sep 17 00:00:00 2001 From: Ran Date: Thu, 11 Jun 2020 16:03:22 +0800 Subject: [PATCH 4/4] Update TOC.md --- TOC.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/TOC.md b/TOC.md index 5ebb0488b5def..855dac2f0d92b 100644 --- a/TOC.md +++ b/TOC.md @@ -333,9 +333,9 @@ + Tools - [Overview](/ecosystem-tool-user-guide.md) - [Use Cases](/ecosystem-tool-user-case.md) - + Backup & Restore (BR) - - [Use BR](/br/backup-and-restore-tool.md) - - [BR Use Cases](/br/backup-and-restore-use-cases.md) + + Backup & Restore (BR) + - [Use BR](/br/backup-and-restore-tool.md) + - [BR Use Cases](/br/backup-and-restore-use-cases.md) - [BR FAQ](/br/backup-and-restore-faq.md) - [Mydumper](/mydumper-overview.md) - [Syncer](/syncer-overview.md)