From 572fadeb6df8c197933eb7981cd039b929bcd02e Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Wed, 3 Jun 2020 15:18:54 +0800 Subject: [PATCH 01/22] Update backup-and-restore-tool.md --- br/backup-and-restore-tool.md | 199 +++++++++++++++++++++++++++++++++- 1 file changed, 195 insertions(+), 4 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 498f42f0a23be..06c9ad453ecde 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -52,6 +52,7 @@ BackupRequest{ ClusterId, // The cluster ID. StartKey, // The starting key of the backup (backed up). EndKey, // The ending key of the backup (not backed up). + StartVersion, // The version of the last backup snapshot, used for the incremental backup. EndVersion, // The backup snapshot time. StorageBackend, // The path where backup files are stored. RateLimit, // Backup speed (MB/s). @@ -62,6 +63,8 @@ After receiving the backup request, the TiKV node traverses all Region leaders o After finishing backing up the data of the corresponding Region, the TiKV node returns the metadata to BR. BR collects the metadata and stores it in the `backupmeta` file which is used for restoration. +If `StartVersion` is not `0`, the backup is seen as an incremental backup. In addition to KVs, BR also collects DDLs between `[StartVersion, EndVersion)`. During data restoration, these DDLs are restored first. + If checksum is enabled when you execute the backup command, BR calculates the checksum of each backed up table for data check. #### Types of backup files @@ -122,10 +125,20 @@ Explanations for the above command are as follows: * `backup`: the sub-command of `br`. * `full`: the sub-command of `backup`. * `-s` (or `--storage`): the option that specifies the path where the backup files are stored. -* `"local:///tmp/backup"`: the parameter of `-s`. `/tmp/backup` is the path in the local disk where the backup files are stored. +* `"local:///tmp/backup"`: the parameter of `-s`. `/tmp/backup` is the path in the local disk where the backed up files of each TiKV node are stored. * `--pd`: the option that specifies the Placement Driver (PD) service address. * `"${PDIP}:2379"`: the parameter of `--pd`. +> **Note:** +> +> - When the `local` storage is used, the backup data are scattered in the local file system of each node. +> +> - It is **not recommended** to backup to a local disk in the production environment because you **have to** manually aggregate these data to complete the data restoration. For more information, see [Restore Cluster Data](#restore-cluster-data). +> +> - Aggregating these backup data might cause redundancy and bring troubles to operation and maintenance. Even worse, if restoring data without aggregating these data, you can receive a rather confusing error message `SST file not found`. +> +> - It is recommended to mount the NFS disk on each node, or backup to the `S3` object storage. + ### Sub-commands A `br` command consists of multiple layers of sub-commands. Currently, BR has the following three sub-commands: @@ -172,6 +185,12 @@ To back up all the cluster data, execute the `br backup full` command. To get he Back up all the cluster data to the `/tmp/backup` path of each TiKV node and write the `backupmeta` file to this path. +> **Note:** +> +> + If the backup disk and the service disk are different, it has been tested that online backup reduces QPS of the read-only online service by about 15%-25% in case of full-speed backup. If you want to reduce the impact on QPS, use `--ratelimit` to limit the rate. +> +> + If the backup disk and the service disk are the same, the backup competes with the service for I/O resources. This might decrease the QPS of the read-only online service by more than half. Therefore, it is **highly not recommended** to back up the online service data to the TiKV data disk. + {{< copyable "shell-regular" >}} ```shell @@ -246,10 +265,95 @@ The `table` sub-command has two options: * `--db`: specifies the database name * `--table`: specifies the table name. -For descriptions of other options, see [Back up all cluster data](#back-up-all-cluster-data). +For descriptions of other options, see [Back up all cluster data](#back-up-all-the-cluster-data). A progress bar is displayed in the terminal during the backup operation. When the progress bar advances to 100%, the backup is complete. Then the BR also checks the backup data to ensure data safety. +### Back up data to Amazon S3 backend + +If you back up the data to the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access to Amazon S3. + +You can refer to an [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) to create an S3 `Bucket` in the specified `Region`. You can also refer to another [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-folder.html) to create a `Folder` in the `Bucket`. + +Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. And then pass the permission to the TiKV node through BR. + +{{< copyable "shell-regular" >}} + +```shell +export AWS_ACCESS_KEY_ID=${AccessKey} +export AWS_SECRET_ACCESS_KEY=${SecretKey} +``` + +When backing up using BR, explicitly specify the parameters `--s3.region` and `--send-credentials-to-tikv`. `--s3.region` indicates the region where S3 is located, and `--send-credentials-to-tikv` means passing the privilege to access S3 to the TiKV node. + +{{< copyable "shell-regular" >}} + +```shell +br backup full \ + --pd "${PDIP}:2379" \ + --storage "s3://${Bucket}/${Folder}" \ + --s3.region "${region}" \ + --send-credentials-to-tikv=true \ + --log-file backuptable.log +``` + +### Back up incremental data + +If you want to backup incrementally, you only need to specify the **last backup timestamp** `--lastbackupts` during backup. + +The following are some limitations of the incremental backup: + +- The incremental backup needs to be under a different path from the previous full backup. +- No GC (Garbage Collection) happens between the start time of the incremental backup and `lastbackupts`. + +Execute the following command to backup the incremental data between `(LAST_BACKUP_TS, current PD timestamp)`: + +{{< copyable "shell-regular" >}} + +```shell +br backup full\ + --pd ${PDIP}:2379 \ + -s local:///home/tidb/backupdata/incr \ + --lastbackupts ${LAST_BACKUP_TS} +``` + +Execute the `validate` command to get the timestamp of the last backup. For example: + +{{< copyable "shell-regular" >}} + +```shell +LAST_BACKUP_TS=`br validate decode --field="end-version" -s local:///home/tidb/backupdata` +``` + +In the above example, the incremental backup data includes the written data and the DDLs between `(LAST_BACKUP_TS, current PD timestamp]`. When restoring data, BR restores DDLs first, and then restore the written data. + +### Back up Raw KV (experimental feature) + +> **Warning:** +> +> This feature is in the experiment, without being thoroughly tested. It is highly **not recommended** to use this feature in the production environment. + +In some scenarios, TiKV might run independently of TiDB. Given that, BR also supports bypassing the TiDB layer and backing up data in TiKV. + +For example, you can execute the following command to backup all keys between `[0x31, 0x3130303030303030)` of the default CF to `$BACKUP_DIR`: + +{{< copyable "shell-regular" >}} + +```shell +br backup raw --pd $PD_ADDR \ + -s "local://$BACKUP_DIR" \ + --start 31 \ + --end 3130303030303030 \ + --format hex \ + --cf default +``` + +Here, the parameters of `--start` and `--end` are decoded using the method specified by `--format` before being sent to TiKV. Currently, the following methods are available: + +- "raw": The input string is directly encoded as a key in binary format. +- "hex": The default encoding method. The input string is seen as a hexadecimal number. +- "escape": First escape the input string, and then encode it into binary format. + ## Restore cluster data To restore the cluster data, use the `br restore` command. You can add the `full`, `db` or `table` sub-command to specify the scope of your restoration: the whole cluster, a database or a single table. @@ -316,7 +420,7 @@ br restore db \ --log-file restorefull.log ``` -In the above command, `--db` specifies the name of the database to be restored. For descriptions of other options, see [Restore all backup data](#restore-all-backup-data). +In the above command, `--db` specifies the name of the database to be restored. For descriptions of other options, see [Restore all backup data](#restore-all-the-backup-data)). ### Restore a table @@ -337,7 +441,94 @@ br restore table \ --log-file restorefull.log ``` -In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore all backup data](#restore-all-backup-data) and [Restore a database](#restore-a-database). +In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore all backup data](#restore-all-the-backup-data) and [Restore a database](#restore-a-database). + +### Restore data from Amazon S3 backend + +If you restore data from the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access to Amazon S3. + +Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. And then pass the permission to the TiKV node through BR. + +{{< copyable "shell-regular" >}} + +```shell +export AWS_ACCESS_KEY_ID=${AccessKey} +export AWS_SECRET_ACCESS_KEY=${SecretKey} +``` + +When restoring data using BR, explicitly specify the parameters `--s3.region` and `--send-credentials-to-tikv`. `--s3.region` indicates the region where S3 is located, and `--send-credentials-to-tikv` means passing the privilege to access S3 to the TiKV node. + +`Bucket` and `Folder` in the `--storage` parameter represent the S3 bucket and the folder where the data to be restored is located. + +{{< copyable "shell-regular" >}} + +```shell +br restore full \ + --pd "${PDIP}:2379" \ + --storage "s3://${Bucket}/${Folder}" \ + --s3.region "${region}" \ + --send-credentials-to-tikv=true \ + --log-file restorefull.log +``` + +### Restore incremental data + +Restoring incremental data is similar to [restoring full data using BR](#restore-all-the-backup-data). Note that when restoring incremental data, make sure that all the data backed up before `last backup ts` has been restored to the target cluster. + +### Restore Raw KV (experimental feature) + +> **Warning:** +> +> This feature is in the experiment, without being thoroughly tested. It is highly **not recommended** to use this feature in the production environment. + +Similar to [backing up Rwa KV](#back-up-raw-kv-experimental-feature), you can execute the following command to restore Raw KV: + +{{< copyable "shell-regular" >}} + +br restore raw --pd $PD_ADDR \ + -s "local://$BACKUP_DIR" \ + --start 31 \ + --end 3130303030303030 \ + --format hex \ + --cf default + +In the above example. all the backed up keys in the range `[0x31, 0x3130303030303030)` are restored to the TiKV cluster. The coding methods of these keys are identical to that of [keys during the backup process](#back-up-raw-kv-experimental-feature) + +### Online restore (experimental feature) + +> **Warning:** +> +> This feature is in the experiment, without being thoroughly tested. It also relies on the unstable `Placement Rules` of PD. It is highly **not recommended** to use this feature in the production environment. + +During data restoration, writing too much data affects the performance of the online cluster. To avoid this effect as much as possible, BR supports [Placement rules](/configure-placement-rules.md) to isolate resources. In this case, downloading and importing SST are only performed on a few specified nodes (or "restore nodes" for short). Take the following steps to complete the online restore. + +1. Configure PD, and start Placement rules: + + {{< copyable "shell-regular" >}} + + ```shell + echo "config set enable-placement-rules true" | pd-ctl + ``` + +2. Edit the configuration file of the "restore node" in TiKV, and specify "restore" to the `server` configuration item: + + {{< copyable "" >}} + + ``` + [server] + labels = { exclusive = "restore" } + ``` + +3. Start TiKV of the "restore node" and restore the backed up files using BR. Compared with the offline restore, you only need to add the `--online` flag: + + {{< copyable "shell-regular" >}} + + ``` + br restore full \ + -s "local://$BACKUP_DIR" \ + --pd $PD_ADDR \ + --online + ``` ## Best practices From babb1c65c654d69a481e18f3dbad5b0bbf756268 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Wed, 3 Jun 2020 15:36:54 +0800 Subject: [PATCH 02/22] refine wording --- br/backup-and-restore-tool.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 06c9ad453ecde..ebd13bcf88aa3 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -133,11 +133,11 @@ Explanations for the above command are as follows: > > - When the `local` storage is used, the backup data are scattered in the local file system of each node. > -> - It is **not recommended** to backup to a local disk in the production environment because you **have to** manually aggregate these data to complete the data restoration. For more information, see [Restore Cluster Data](#restore-cluster-data). +> - It is **not recommended** to back up to a local disk in the production environment because you **have to** manually aggregate these data to complete the data restoration. For more information, see [Restore Cluster Data](#restore-cluster-data). > > - Aggregating these backup data might cause redundancy and bring troubles to operation and maintenance. Even worse, if restoring data without aggregating these data, you can receive a rather confusing error message `SST file not found`. > -> - It is recommended to mount the NFS disk on each node, or backup to the `S3` object storage. +> - It is recommended to mount the NFS disk on each node, or back up to the `S3` object storage. ### Sub-commands @@ -299,14 +299,14 @@ br backup full \ ### Back up incremental data -If you want to backup incrementally, you only need to specify the **last backup timestamp** `--lastbackupts` during backup. +If you want to back up incrementally, you only need to specify the **last backup timestamp** `--lastbackupts`. The following are some limitations of the incremental backup: - The incremental backup needs to be under a different path from the previous full backup. - No GC (Garbage Collection) happens between the start time of the incremental backup and `lastbackupts`. -Execute the following command to backup the incremental data between `(LAST_BACKUP_TS, current PD timestamp)`: +Execute the following command to back up the incremental data between `(LAST_BACKUP_TS, current PD timestamp)`: {{< copyable "shell-regular" >}} @@ -335,7 +335,7 @@ In the above example, the incremental backup data includes the written data and In some scenarios, TiKV might run independently of TiDB. Given that, BR also supports bypassing the TiDB layer and backing up data in TiKV. -For example, you can execute the following command to backup all keys between `[0x31, 0x3130303030303030)` of the default CF to `$BACKUP_DIR`: +For example, you can execute the following command to back up all keys between `[0x31, 0x3130303030303030)` of the default CF to `$BACKUP_DIR`: {{< copyable "shell-regular" >}} From 897daa9fb2c1b9e0c8dc15c5020cbf103a80cf16 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 15:10:15 +0800 Subject: [PATCH 03/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index ebd13bcf88aa3..94442cea1a549 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -271,7 +271,7 @@ A progress bar is displayed in the terminal during the backup operation. When th ### Back up data to Amazon S3 backend -If you back up the data to the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access to Amazon S3. +If you back up the data to the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access Amazon S3. You can refer to an [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) to create an S3 `Bucket` in the specified `Region`. You can also refer to another [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-folder.html) to create a `Folder` in the `Bucket`. From a8db3b2946f772e6e2819caa18647d8a258d60ef Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 15:10:26 +0800 Subject: [PATCH 04/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 94442cea1a549..453fdf6159e65 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -273,7 +273,7 @@ A progress bar is displayed in the terminal during the backup operation. When th If you back up the data to the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access Amazon S3. -You can refer to an [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) to create an S3 `Bucket` in the specified `Region`. You can also refer to another [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-folder.html) to create a `Folder` in the `Bucket`. +You can refer to the [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) to create an S3 `Bucket` in the specified `Region`. You can also refer to another [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-folder.html) to create a `Folder` in the `Bucket`. Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. And then pass the permission to the TiKV node through BR. From 0254e65e1dacfb20e292a194d5214e1ab23c941f Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 15:16:02 +0800 Subject: [PATCH 05/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 453fdf6159e65..ebd186d2a6cbb 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -275,7 +275,7 @@ If you back up the data to the Amazon S3 backend, instead of `local` storage, yo You can refer to the [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) to create an S3 `Bucket` in the specified `Region`. You can also refer to another [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-folder.html) to create a `Folder` in the `Bucket`. -Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. And then pass the permission to the TiKV node through BR. +Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. Then pass the permission to the TiKV node through BR. {{< copyable "shell-regular" >}} From 96be1efe75e8a3cfedcd72d521b065bc224a4cdc Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 21:34:30 +0800 Subject: [PATCH 06/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index ebd186d2a6cbb..77998b21a0a85 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -301,7 +301,7 @@ br backup full \ If you want to back up incrementally, you only need to specify the **last backup timestamp** `--lastbackupts`. -The following are some limitations of the incremental backup: +The incremental backup has two limitations: - The incremental backup needs to be under a different path from the previous full backup. - No GC (Garbage Collection) happens between the start time of the incremental backup and `lastbackupts`. From 3a8de13f087facc8493e206678997fc71e4ecae5 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 21:34:57 +0800 Subject: [PATCH 07/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 77998b21a0a85..f750e7219d63f 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -306,7 +306,7 @@ The incremental backup has two limitations: - The incremental backup needs to be under a different path from the previous full backup. - No GC (Garbage Collection) happens between the start time of the incremental backup and `lastbackupts`. -Execute the following command to back up the incremental data between `(LAST_BACKUP_TS, current PD timestamp)`: +To back up the incremental data between `(LAST_BACKUP_TS, current PD timestamp]`, execute the following command: {{< copyable "shell-regular" >}} From a81e1187a6474857d11c84f1a4d0c6f02e83d0fe Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 21:35:08 +0800 Subject: [PATCH 08/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index f750e7219d63f..b15da60d48617 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -447,7 +447,7 @@ In the above command, `--table` specifies the name of the table to be restored. If you restore data from the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access to Amazon S3. -Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. And then pass the permission to the TiKV node through BR. +Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. Then pass the permission to the TiKV node through BR. {{< copyable "shell-regular" >}} From b6b1464d48d8feb8ae4eee0bb4ad292bee3ec83e Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 21:35:19 +0800 Subject: [PATCH 09/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index b15da60d48617..e6f79413213b2 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -492,7 +492,7 @@ br restore raw --pd $PD_ADDR \ --format hex \ --cf default -In the above example. all the backed up keys in the range `[0x31, 0x3130303030303030)` are restored to the TiKV cluster. The coding methods of these keys are identical to that of [keys during the backup process](#back-up-raw-kv-experimental-feature) +In the above example, all the backed up keys in the range `[0x31, 0x3130303030303030)` are restored to the TiKV cluster. The coding methods of these keys are identical to that of [keys during the backup process](#back-up-raw-kv-experimental-feature) ### Online restore (experimental feature) From d926b0e0e8ab3911dfb1c1d824070c912282fdbc Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 21:35:46 +0800 Subject: [PATCH 10/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index e6f79413213b2..2685a41b1d71a 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -498,7 +498,7 @@ In the above example, all the backed up keys in the range `[0x31, 0x313030303030 > **Warning:** > -> This feature is in the experiment, without being thoroughly tested. It also relies on the unstable `Placement Rules` of PD. It is highly **not recommended** to use this feature in the production environment. +> This feature is in the experiment, without being thoroughly tested. It also relies on the unstable `Placement Rules` feature of PD. It is highly **not recommended** to use this feature in the production environment. During data restoration, writing too much data affects the performance of the online cluster. To avoid this effect as much as possible, BR supports [Placement rules](/configure-placement-rules.md) to isolate resources. In this case, downloading and importing SST are only performed on a few specified nodes (or "restore nodes" for short). Take the following steps to complete the online restore. From 364d49c80b33f26c0b8646ed6a0a1a91cd1496f9 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 21:41:24 +0800 Subject: [PATCH 11/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 2685a41b1d71a..dcd126ffad781 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -317,7 +317,7 @@ br backup full\ --lastbackupts ${LAST_BACKUP_TS} ``` -Execute the `validate` command to get the timestamp of the last backup. For example: +To get the timestamp of the last backup, execute the `validate` command. For example: {{< copyable "shell-regular" >}} From 06318155e634db4dfc9281b0642328937ed4b237 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 21:44:36 +0800 Subject: [PATCH 12/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index dcd126ffad781..da49257c693f6 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -325,7 +325,7 @@ To get the timestamp of the last backup, execute the `validate` command. For exa LAST_BACKUP_TS=`br validate decode --field="end-version" -s local:///home/tidb/backupdata` ``` -In the above example, the incremental backup data includes the written data and the DDLs between `(LAST_BACKUP_TS, current PD timestamp]`. When restoring data, BR restores DDLs first, and then restore the written data. +In the above example, the incremental backup data includes the newly written data and the DDLs between `(LAST_BACKUP_TS, current PD timestamp]`. When restoring data, BR restores DDLs first and then restores the written data. ### Back up Raw KV (experimental feature) From 3aba2b090edf74c7a6f40a4240520bc8688029be Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 21:57:14 +0800 Subject: [PATCH 13/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index da49257c693f6..861878b966037 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -331,7 +331,7 @@ In the above example, the incremental backup data includes the newly written dat > **Warning:** > -> This feature is in the experiment, without being thoroughly tested. It is highly **not recommended** to use this feature in the production environment. +> This feature is experimental and not thoroughly tested. It is highly **not recommended** to use this feature in the production environment. In some scenarios, TiKV might run independently of TiDB. Given that, BR also supports bypassing the TiDB layer and backing up data in TiKV. From 7c574fbb71d60523ff1a22edc546e2915e7256da Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 22:41:04 +0800 Subject: [PATCH 14/22] Update backup-and-restore-tool.md --- br/backup-and-restore-tool.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 861878b966037..9a791a5139528 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -275,7 +275,7 @@ If you back up the data to the Amazon S3 backend, instead of `local` storage, yo You can refer to the [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) to create an S3 `Bucket` in the specified `Region`. You can also refer to another [AWS Official Document](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-folder.html) to create a `Folder` in the `Bucket`. -Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. Then pass the permission to the TiKV node through BR. +Pass `SecretKey` and `AccessKey` of the account that has privilege to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. Then pass the privilege to the TiKV node through BR. {{< copyable "shell-regular" >}} @@ -447,7 +447,7 @@ In the above command, `--table` specifies the name of the table to be restored. If you restore data from the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access to Amazon S3. -Pass `SecretKey` and `AccessKey` of the account that has permission to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. Then pass the permission to the TiKV node through BR. +Pass `SecretKey` and `AccessKey` of the account that has privilege to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. Then pass the privilege to the TiKV node through BR. {{< copyable "shell-regular" >}} From fdbc4722a62ae181352d9f81aeeef2adb1a27f43 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 22:43:34 +0800 Subject: [PATCH 15/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 9a791a5139528..f8e52d793cdff 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -335,7 +335,7 @@ In the above example, the incremental backup data includes the newly written dat In some scenarios, TiKV might run independently of TiDB. Given that, BR also supports bypassing the TiDB layer and backing up data in TiKV. -For example, you can execute the following command to back up all keys between `[0x31, 0x3130303030303030)` of the default CF to `$BACKUP_DIR`: +For example, you can execute the following command to back up all keys between `[0x31, 0x3130303030303030)` in the default CF to `$BACKUP_DIR`: {{< copyable "shell-regular" >}} From 630b10257278753534875e629a5b3ce68421f0ee Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 22:43:41 +0800 Subject: [PATCH 16/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index f8e52d793cdff..5455d8319c887 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -351,7 +351,7 @@ br backup raw --pd $PD_ADDR \ Here, the parameters of `--start` and `--end` are decoded using the method specified by `--format` before being sent to TiKV. Currently, the following methods are available: - "raw": The input string is directly encoded as a key in binary format. -- "hex": The default encoding method. The input string is seen as a hexadecimal number. +- "hex": The default encoding method. The input string is treated as a hexadecimal number. - "escape": First escape the input string, and then encode it into binary format. ## Restore cluster data From 8346ca752402044959c30ef2307e3b46b3126dbe Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 22:44:20 +0800 Subject: [PATCH 17/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 5455d8319c887..c80915184c260 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -445,7 +445,7 @@ In the above command, `--table` specifies the name of the table to be restored. ### Restore data from Amazon S3 backend -If you restore data from the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access to Amazon S3. +If you restore data from the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access Amazon S3. Pass `SecretKey` and `AccessKey` of the account that has privilege to access the S3 backend to the BR node. Here `SecretKey` and `AccessKey` are passed as environment variables. Then pass the privilege to the TiKV node through BR. From dcaac97c4b7962ea7f2beab7c5d0a325673872db Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 22:45:09 +0800 Subject: [PATCH 18/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index c80915184c260..af73902fecd5e 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -500,7 +500,7 @@ In the above example, all the backed up keys in the range `[0x31, 0x313030303030 > > This feature is in the experiment, without being thoroughly tested. It also relies on the unstable `Placement Rules` feature of PD. It is highly **not recommended** to use this feature in the production environment. -During data restoration, writing too much data affects the performance of the online cluster. To avoid this effect as much as possible, BR supports [Placement rules](/configure-placement-rules.md) to isolate resources. In this case, downloading and importing SST are only performed on a few specified nodes (or "restore nodes" for short). Take the following steps to complete the online restore. +During data restoration, writing too much data affects the performance of the online cluster. To avoid this effect as much as possible, BR supports [Placement rules](/configure-placement-rules.md) to isolate resources. In this case, downloading and importing SST are only performed on a few specified nodes (or "restore nodes" for short). To complete the online restore, take the following steps. 1. Configure PD, and start Placement rules: From a024f1834eb38911b6ad1a5eed0551d266e78b9a Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 5 Jun 2020 22:50:39 +0800 Subject: [PATCH 19/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index af73902fecd5e..448d661abecd0 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -471,6 +471,8 @@ br restore full \ --log-file restorefull.log ``` +In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore a database](#restore-a-database). + ### Restore incremental data Restoring incremental data is similar to [restoring full data using BR](#restore-all-the-backup-data). Note that when restoring incremental data, make sure that all the data backed up before `last backup ts` has been restored to the target cluster. From 471c40f1267d8ae7087d724c9205d9c280748d26 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Mon, 8 Jun 2020 12:15:59 +0800 Subject: [PATCH 20/22] Update br/backup-and-restore-tool.md Co-authored-by: Ran --- br/backup-and-restore-tool.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 448d661abecd0..ee6f146a043e9 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -473,6 +473,8 @@ br restore full \ In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore a database](#restore-a-database). +In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore a database](#restore-a-database). + ### Restore incremental data Restoring incremental data is similar to [restoring full data using BR](#restore-all-the-backup-data). Note that when restoring incremental data, make sure that all the data backed up before `last backup ts` has been restored to the target cluster. From 2d182f18d6fad73f168b2339ce2e952c6ea312bb Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Mon, 8 Jun 2020 12:18:14 +0800 Subject: [PATCH 21/22] revert unnec changes --- br/backup-and-restore-tool.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index ee6f146a043e9..448d661abecd0 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -473,8 +473,6 @@ br restore full \ In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore a database](#restore-a-database). -In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore a database](#restore-a-database). - ### Restore incremental data Restoring incremental data is similar to [restoring full data using BR](#restore-all-the-backup-data). Note that when restoring incremental data, make sure that all the data backed up before `last backup ts` has been restored to the target cluster. From 051d404a4b71b2ff0abf808ddf2cf8ac2f4f6daa Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Mon, 8 Jun 2020 12:44:18 +0800 Subject: [PATCH 22/22] Update br/backup-and-restore-tool.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 448d661abecd0..5e6d8056a9f6c 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -483,7 +483,7 @@ Restoring incremental data is similar to [restoring full data using BR](#restore > > This feature is in the experiment, without being thoroughly tested. It is highly **not recommended** to use this feature in the production environment. -Similar to [backing up Rwa KV](#back-up-raw-kv-experimental-feature), you can execute the following command to restore Raw KV: +Similar to [backing up Raw KV](#back-up-raw-kv-experimental-feature), you can execute the following command to restore Raw KV: {{< copyable "shell-regular" >}}