From dc370898ecf6e2e3d611936a4b9711b46f972cb5 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Wed, 10 Jun 2020 20:15:51 +0800 Subject: [PATCH 01/13] Add data-migration-route.md --- data-migration-route.md | 64 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 data-migration-route.md diff --git a/data-migration-route.md b/data-migration-route.md new file mode 100644 index 0000000000000..ba5cc9249e2d7 --- /dev/null +++ b/data-migration-route.md @@ -0,0 +1,64 @@ +--- +title: Data Migration Path +summary: This document describes the paths you can use to migrate data to TiDB. +category: reference +--- + +# Data Migration Path + +This document describes the paths you can use to migrate data to TiDB, including migrating data from MySQL to TiDB and from CSV/SQL files to TiDB. + +## Migrate from MySQL to TiDB + +To migrate data from MySQL to TiDB, it is recommended to use one of the following methods: + +- [Use Mydumper and TiDB Lightning (full data)](#use-mydumper-and-tidb-lightning-full-data). +- [Use DM (full and incremental data)](#use-dm). + +### Use Mydumper and TiDB Lightning (full data) + +#### Scenarios + +You can use Mydumper and TiDB Lightning to migrate full data when the data size is greater than 1TB. If you need to replicate incremental data, it is recommended to [use TiDB Data Migration (DM)](#use-dm) to create an incremental replication task. + +#### Migration method + +1. Use Mydumper to export the full MySQL data; +2. Use TiDB Lightning to import the full data to TiDB. For details, refer to [Migrate from Mydumper Files]((/migrate-from-mysql-mydumper-files.md)) + +### Use DM + +#### Scenarios + +You use DM to migrate full MySQL data and to replicate incremental data. Note that the size of the full data must be less than 1TB. Otherwise, it is recommended to use Mydumper and TiDB Lightning to import the full data, and then use DM to replicate the incremental data. + +#### Migration method + +For details, refer to [Migrate from MySQL (Amazon Aurora)](/migrate-from-aurora-mysql-database.md) + +## Migrate data from files to TiDB + +You can migrate data from CSV/SQL files to TiDB. + +### Migrate data from CSV files to TiDB + +#### Scenarios + +You can migrate data from heterogeneous databases that are not compatible with MySQL protocol to TiDB. + +#### Migration method + +1. Export full data to CSV files. +2. Import CSV files to TiDB using one of the following methods: + + - Use TiDB Lightning. + + Its import speed is fast. It is recommended to use TiDB Lightning in the case of large amount of data in CSV files. For details, refer to [TiDB Lightning CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md). + + - Use the `LOAD DATA` statement. + + Execute the `LOAD DATA` statement in TiDB to import CSV files. This is more convenient, though manual intervention is required to check the consistency and integrity of the data if an error or interruption occurs during the import. Therefore, it is **not recommended** to use this method in the production environment. For details, refer to [LOAD DATA](/sql-statements/sql-statement-load-data.md). + +### Migrate data from SQL files to TiDB + +Use Mydumper and TiDB Lightning to migrate data from SQL files to TiDB. For details, refer to [Use Mydumper and TiDB Lightning](#use-mydumper-and-tidb-lightning-full-data). \ No newline at end of file From 138d016997bea7a29aa91d311d0d33bf87ec14eb Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 15:23:36 +0800 Subject: [PATCH 02/13] Update data-migration-route.md Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> --- data-migration-route.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/data-migration-route.md b/data-migration-route.md index ba5cc9249e2d7..056539fc784f6 100644 --- a/data-migration-route.md +++ b/data-migration-route.md @@ -12,7 +12,7 @@ This document describes the paths you can use to migrate data to TiDB, including To migrate data from MySQL to TiDB, it is recommended to use one of the following methods: -- [Use Mydumper and TiDB Lightning (full data)](#use-mydumper-and-tidb-lightning-full-data). +- [Use Mydumper and TiDB Lightning](#use-mydumper-and-tidb-lightning-full-data) to migrate full data. - [Use DM (full and incremental data)](#use-dm). ### Use Mydumper and TiDB Lightning (full data) @@ -61,4 +61,4 @@ You can migrate data from heterogeneous databases that are not compatible with M ### Migrate data from SQL files to TiDB -Use Mydumper and TiDB Lightning to migrate data from SQL files to TiDB. For details, refer to [Use Mydumper and TiDB Lightning](#use-mydumper-and-tidb-lightning-full-data). \ No newline at end of file +Use Mydumper and TiDB Lightning to migrate data from SQL files to TiDB. For details, refer to [Use Mydumper and TiDB Lightning](#use-mydumper-and-tidb-lightning-full-data). From 2a8691c9898ce58e889508aa094d8e305d6dc1fb Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 15:24:26 +0800 Subject: [PATCH 03/13] Update data-migration-route.md Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> --- data-migration-route.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-migration-route.md b/data-migration-route.md index 056539fc784f6..94ec6d04608fe 100644 --- a/data-migration-route.md +++ b/data-migration-route.md @@ -13,7 +13,7 @@ This document describes the paths you can use to migrate data to TiDB, including To migrate data from MySQL to TiDB, it is recommended to use one of the following methods: - [Use Mydumper and TiDB Lightning](#use-mydumper-and-tidb-lightning-full-data) to migrate full data. -- [Use DM (full and incremental data)](#use-dm). +- [Use TiDB Data Migration (DM)](#use-dm) to migrate full and incremental data. ### Use Mydumper and TiDB Lightning (full data) From e5333bbe005000e84616f8a3aec250b479073143 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 15:24:45 +0800 Subject: [PATCH 04/13] Update data-migration-route.md Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> --- data-migration-route.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-migration-route.md b/data-migration-route.md index 94ec6d04608fe..523e95907d5e9 100644 --- a/data-migration-route.md +++ b/data-migration-route.md @@ -19,7 +19,7 @@ To migrate data from MySQL to TiDB, it is recommended to use one of the followin #### Scenarios -You can use Mydumper and TiDB Lightning to migrate full data when the data size is greater than 1TB. If you need to replicate incremental data, it is recommended to [use TiDB Data Migration (DM)](#use-dm) to create an incremental replication task. +You can use Mydumper and TiDB Lightning to migrate full data when the data size is greater than 1 TB. If you need to replicate incremental data, it is recommended to [use DM](#use-dm) to create an incremental replication task. #### Migration method From c356ec8f3845eaade0230c4eea79baed144e6a3f Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 15:24:53 +0800 Subject: [PATCH 05/13] Update data-migration-route.md Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> --- data-migration-route.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-migration-route.md b/data-migration-route.md index 523e95907d5e9..446901a060c45 100644 --- a/data-migration-route.md +++ b/data-migration-route.md @@ -23,7 +23,7 @@ You can use Mydumper and TiDB Lightning to migrate full data when the data size #### Migration method -1. Use Mydumper to export the full MySQL data; +1. Use Mydumper to export the full MySQL data. 2. Use TiDB Lightning to import the full data to TiDB. For details, refer to [Migrate from Mydumper Files]((/migrate-from-mysql-mydumper-files.md)) ### Use DM From 4bb610a2409bb90cab31a2ee727ddb14dc78aef8 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 15:25:04 +0800 Subject: [PATCH 06/13] Update data-migration-route.md Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> --- data-migration-route.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-migration-route.md b/data-migration-route.md index 446901a060c45..2576f8d21d375 100644 --- a/data-migration-route.md +++ b/data-migration-route.md @@ -30,7 +30,7 @@ You can use Mydumper and TiDB Lightning to migrate full data when the data size #### Scenarios -You use DM to migrate full MySQL data and to replicate incremental data. Note that the size of the full data must be less than 1TB. Otherwise, it is recommended to use Mydumper and TiDB Lightning to import the full data, and then use DM to replicate the incremental data. +You can use DM to migrate full MySQL data and to replicate incremental data. Note that the size of the full data must be less than 1 TB. Otherwise, it is recommended to use Mydumper and TiDB Lightning to import the full data, and then use DM to replicate the incremental data. #### Migration method From 53e5fbc61d6792488467b61905891d62d22b5e93 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 15:25:24 +0800 Subject: [PATCH 07/13] Update data-migration-route.md Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> --- data-migration-route.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-migration-route.md b/data-migration-route.md index 2576f8d21d375..4e7fa89d17175 100644 --- a/data-migration-route.md +++ b/data-migration-route.md @@ -24,7 +24,7 @@ You can use Mydumper and TiDB Lightning to migrate full data when the data size #### Migration method 1. Use Mydumper to export the full MySQL data. -2. Use TiDB Lightning to import the full data to TiDB. For details, refer to [Migrate from Mydumper Files]((/migrate-from-mysql-mydumper-files.md)) +2. Use TiDB Lightning to import the full data to TiDB. For details, refer to [Migrate from Mydumper Files](/migrate-from-mysql-mydumper-files.md). ### Use DM From 1470b5f2a35b45a605b8629abd061ccc7df9381c Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 15:26:32 +0800 Subject: [PATCH 08/13] Apply suggestions from code review Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> --- data-migration-route.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/data-migration-route.md b/data-migration-route.md index 4e7fa89d17175..0f7067ed6ac2b 100644 --- a/data-migration-route.md +++ b/data-migration-route.md @@ -44,7 +44,7 @@ You can migrate data from CSV/SQL files to TiDB. #### Scenarios -You can migrate data from heterogeneous databases that are not compatible with MySQL protocol to TiDB. +You can migrate data from heterogeneous databases that are not compatible with the MySQL protocol to TiDB. #### Migration method @@ -53,7 +53,7 @@ You can migrate data from heterogeneous databases that are not compatible with M - Use TiDB Lightning. - Its import speed is fast. It is recommended to use TiDB Lightning in the case of large amount of data in CSV files. For details, refer to [TiDB Lightning CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md). + Its import speed is fast. It is recommended to use TiDB Lightning in the case of large amounts of data in CSV files. For details, refer to [TiDB Lightning CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md). - Use the `LOAD DATA` statement. From e333e4038f5106ac1600415effac480c7192c9a1 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 15:27:41 +0800 Subject: [PATCH 09/13] Update data-migration-route.md Co-authored-by: Caitin <34535727+CaitinChen@users.noreply.github.com> --- data-migration-route.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-migration-route.md b/data-migration-route.md index 0f7067ed6ac2b..448d054ef95e9 100644 --- a/data-migration-route.md +++ b/data-migration-route.md @@ -57,7 +57,7 @@ You can migrate data from heterogeneous databases that are not compatible with t - Use the `LOAD DATA` statement. - Execute the `LOAD DATA` statement in TiDB to import CSV files. This is more convenient, though manual intervention is required to check the consistency and integrity of the data if an error or interruption occurs during the import. Therefore, it is **not recommended** to use this method in the production environment. For details, refer to [LOAD DATA](/sql-statements/sql-statement-load-data.md). + Execute the `LOAD DATA` statement in TiDB to import CSV files. This is more convenient, but if an error or interruption occurs during the import, manual intervention is required to check the consistency and integrity of the data. Therefore, it is **not recommended** to use this method in the production environment. For details, refer to [LOAD DATA](/sql-statements/sql-statement-load-data.md). ### Migrate data from SQL files to TiDB From 8e56c573f911a1f888b68e7afe1fd660af76cc4d Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Sun, 14 Jun 2020 22:55:45 +0800 Subject: [PATCH 10/13] rename file and update wording --- data-migration-route.md => migration-overview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) rename data-migration-route.md => migration-overview.md (90%) diff --git a/data-migration-route.md b/migration-overview.md similarity index 90% rename from data-migration-route.md rename to migration-overview.md index 448d054ef95e9..d95838bc61c01 100644 --- a/data-migration-route.md +++ b/migration-overview.md @@ -1,12 +1,12 @@ --- -title: Data Migration Path -summary: This document describes the paths you can use to migrate data to TiDB. +title: Migration Overview +summary: This document describes how to migrate data from databases or data formats (CSV/SQL). category: reference --- -# Data Migration Path +# Migration Overview -This document describes the paths you can use to migrate data to TiDB, including migrating data from MySQL to TiDB and from CSV/SQL files to TiDB. +This document describes how to migrate data to TiDB, including migrating data from MySQL and from CSV/SQL files. ## Migrate from MySQL to TiDB From 29ee83677369db7a46bc2e3a2009a1f3688dc1ca Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Sun, 14 Jun 2020 23:01:40 +0800 Subject: [PATCH 11/13] Update TOC.md --- TOC.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/TOC.md b/TOC.md index 1df5259b08854..5f9b7fb2256ee 100644 --- a/TOC.md +++ b/TOC.md @@ -46,8 +46,9 @@ + [Migrate from Mydumper Files](/migrate-from-mysql-mydumper-files.md) + [Migrate from Amazon Aurora MySQL Using DM](/migrate-from-aurora-mysql-database.md) + Migrate from CSV Files - + [Use TiDB Lightning](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) - + [Use `LOAD DATA` Statement](/sql-statements/sql-statement-load-data.md) + + Migrate from CSV Files + + [Use TiDB Lightning](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) + + [Use `LOAD DATA` Statement](/sql-statements/sql-statement-load-data.md) + [Migrate from SQL Files](/migrate-from-mysql-mydumper-files.md) + Maintain + Upgrade From 52f3c9142cbe360f7838a5685c90538de13771fb Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Sun, 14 Jun 2020 23:04:26 +0800 Subject: [PATCH 12/13] update indentation --- TOC.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/TOC.md b/TOC.md index 5f9b7fb2256ee..1df5259b08854 100644 --- a/TOC.md +++ b/TOC.md @@ -46,9 +46,8 @@ + [Migrate from Mydumper Files](/migrate-from-mysql-mydumper-files.md) + [Migrate from Amazon Aurora MySQL Using DM](/migrate-from-aurora-mysql-database.md) + Migrate from CSV Files - + Migrate from CSV Files - + [Use TiDB Lightning](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) - + [Use `LOAD DATA` Statement](/sql-statements/sql-statement-load-data.md) + + [Use TiDB Lightning](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) + + [Use `LOAD DATA` Statement](/sql-statements/sql-statement-load-data.md) + [Migrate from SQL Files](/migrate-from-mysql-mydumper-files.md) + Maintain + Upgrade From 73f64ae98590e8c63f441cdae81933d8cebab1bc Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Tue, 16 Jun 2020 19:48:26 +0800 Subject: [PATCH 13/13] address comment --- migration-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/migration-overview.md b/migration-overview.md index d95838bc61c01..9c962997d0d22 100644 --- a/migration-overview.md +++ b/migration-overview.md @@ -30,7 +30,7 @@ You can use Mydumper and TiDB Lightning to migrate full data when the data size #### Scenarios -You can use DM to migrate full MySQL data and to replicate incremental data. Note that the size of the full data must be less than 1 TB. Otherwise, it is recommended to use Mydumper and TiDB Lightning to import the full data, and then use DM to replicate the incremental data. +You can use DM to migrate full MySQL data and to replicate incremental data. It is suggested that the size of the full data is less than 1 TB. Otherwise, it is recommended to use Mydumper and TiDB Lightning to import the full data, and then use DM to replicate the incremental data. #### Migration method