From 72637ecabbd605a416bd79684be351c9977cc264 Mon Sep 17 00:00:00 2001 From: JmPotato Date: Thu, 30 Jul 2020 12:02:46 +0800 Subject: [PATCH 01/10] Add isolation_level Signed-off-by: JmPotato --- configure-placement-rules.md | 3 +++ location-awareness.md | 41 ++++++++++++++++++++++++++++++++++-- pd-configuration-file.md | 6 ++++++ pd-control.md | 2 ++ 4 files changed, 50 insertions(+), 2 deletions(-) diff --git a/configure-placement-rules.md b/configure-placement-rules.md index 0f5e56cf92891..5a0638b94412d 100644 --- a/configure-placement-rules.md +++ b/configure-placement-rules.md @@ -38,6 +38,7 @@ The following table shows the meaning of each field in a rule: | `Count` | `int`, positive integer | The number of replicas. | | `LabelConstraint` | `[]Constraint` | Filers nodes based on the label. | | `LocationLabels` | `[]string` | Used for physical isolation. | +| `IsolationLevel` | `string` | Used to set the minimum physical isolation level `LabelConstraint` is similar to the function in Kubernetes that filters labels based on these four primitives: `in`, `notIn`, `exists`, and `notExists`. The meanings of these four primitives are as follows: @@ -48,6 +49,8 @@ The following table shows the meaning of each field in a rule: The meaning and function of `LocationLabels` are the same with those earlier than v4.0. For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology: the cluster has multiple zones (Availability Zones), each zone has multiple racks, and each rack has multiple hosts. When performing schedule, PD first tries to place the Region's peers in different zones. If this try fails (such as there are three replicas but only two zones in total), PD guarantees to place these replicas in different racks. If the number of racks is not enough to guarantee isolation, then PD tries the host-level isolation. +The meaning and function of `IsolationLevel` is elaborated in [Cluster topology configuration](/location-awareness.md). For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology with `LocationLabels` and set `IsolationLevel` to `"zone"`. Then PD will ensure that all peers of each Region are placed in different zones when scheduling. If the minimum isolation level restriction of `IsolationLevel` cannot be met (e.g, 3 replicas are configured but there are only 2 data zones in total), PD will not try to make up to meet the restriction. The default value of `IsolationLevel` is an empty string, which means it is disabled. + ## Configure rules The operations in this section are based on [pd-ctl](/pd-control.md), and the commands involved in the operations also support calls via HTTP API. diff --git a/location-awareness.md b/location-awareness.md index 132ab854f7666..a48f31ad04399 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -8,7 +8,7 @@ aliases: ['/docs/dev/location-awareness/','/docs/dev/how-to/deploy/geographic-re ## Overview -PD schedules according to the topology of the TiKV cluster to maximize the TiKV's capability for disaster recovery. +PD schedules according to the topology of the TiKV cluster to maximize the TiKV's capability for disaster recovery. We recommend that TiKV nodes should be physically dispersed as much as possible. For example, TiKV nodes can be distributed in different racks or even different data zones. According to the topology information of TiKV, the PD scheduler will automatically schedule in the background to isolate the replicas of the Region as much as possible, thereby maximizing the data disaster tolerance. Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). @@ -16,7 +16,7 @@ Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-dep TiKV reports the topological information to PD according to the startup parameter or configuration of TiKV. -Assuming that the topology has three structures: zone > rack > host, use lables to specify the following information: +Assuming that the topology has three structures: zone > rack > host, use labels to specify the following information: Startup parameter: @@ -41,12 +41,45 @@ max-replicas = 3 location-labels = ["zone", "rack", "host"] ``` +After the PD cluster is initialized, you need to use the pd-ctl tool to make online changes: + +{{< copyable "shell-regular" >}} + +```bash +pd-ctl config set location-labels zone,rack,host +``` + `location-labels` needs to correspond to the TiKV `labels` name so that PD can understand that the `labels` represents the TiKV topology. > **Note:** > > You must configure `location-labels` for PD and `labels` for TiKV at the same time for `labels` to take effect. +## PD restricts the TiKV topology + +With configuring `location-labels` first, we can further enhance the topological isolation requirements of TiKV clusters through the `isolation-level`. Suppose we divide the cluster topology into three layers through `location-labels` according to the above instructions: zone -> rack -> host, and configure the `isolation-level` as follows. + +{{< copyable "" >}} + +```toml +[replication] +isolation-level = "zone" +``` + +After the PD cluster is initialized, you need to use the pd-ctl tool to make online changes: + +{{< copyable "shell-regular" >}} + +```bash +pd-ctl config set isolation-level zone +``` + +`isolation-level` needs to correspond to one of the `location-labels` name so that PD can understand that this label represents the TiKV topology. + +> **Note:** +> +> `isolation-level` is empty by default, that is, there is no mandatory isolation level restriction. To set it, you must first configure the PD's `location-labels` parameter, and ensure that the value of `isolation-level` must be one of `location-labels`. + ## PD schedules based on the TiKV topology PD makes optimal scheduling according to the topological information. You just need to care about what kind of topology can achieve the desired effect. @@ -89,3 +122,7 @@ In this case, PD will schedule different replicas of each datum to different dat - If the data zone cannot recover within a period of time, PD will remove the replica from this data zone. To sum up, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to reach a certain level of disaster recovery, deploy many machines in different sites according to the topology. The number of machines must be more than the number of `max-replicas`. + +However, with setting `isolation-level` to `"zone"`, PD will strictly ensure that each replica will isolated from each other at the zone level, even if guaranteeing this will not meet the number requirement of `max-replicas`. For example, there are three data zones z1/z2/z3 in the TiKV cluster. Under the requirement of 3 replicas, PD will dispatch the three replicas of the same Region to these three data zones. If there is a power outage in the z1 and it cannot be recovered after a period of time, PD will think that the region replica on z1 is no longer available, but because the `isolation-level` is set to `"zone"`, PD needs to strictly guarantee different region replicas will not fall on the same data zone. Because both z2 and z3 already have copies, PD will not perform any scheduling under the minimum isolation level limit of `isolation-level`, even if there are only two copies at this moment. + +Similarly, when `isolation-level` is `"rack"`, the minimum isolation level will be different racks in the same data zone. Under this configuration, the isolation at the zone level will be guaranteed first if it can be. When the isolation at the zone level cannot be promised, PD will consider to avoid scheduling replicas to be in the same zone with the same rack, and so on. \ No newline at end of file diff --git a/pd-configuration-file.md b/pd-configuration-file.md index 6bb367dc40491..1ca6b96bbc5ea 100644 --- a/pd-configuration-file.md +++ b/pd-configuration-file.md @@ -258,6 +258,12 @@ Configuration items related to replicas + Default value: `[]` + [Cluster topology configuration](/location-awareness.md) +### `isolation-level` + ++ The minimum topological isolation level of a TiKV cluster ++ Default value: `""` ++ [Cluster topology configuration](/location-awareness.md) + ### `strictly-match-label` + Enables the strict check for whether the TiKV label matches PD's `location-labels`. diff --git a/pd-control.md b/pd-control.md index 6066783943c8f..08eb9a2907168 100644 --- a/pd-control.md +++ b/pd-control.md @@ -128,6 +128,7 @@ Usage: { "replication": { "enable-placement-rules": "false", + "isolation-level": "", "location-labels": "", "max-replicas": 3, "strictly-match-label": "false" @@ -169,6 +170,7 @@ Usage: { "max-replicas": 3, "location-labels": "", + "isolation-level": "", "strictly-match-label": "false", "enable-placement-rules": "false" } From e1e1e674721cde749398c517f23511c4c2009668 Mon Sep 17 00:00:00 2001 From: JmPotato Date: Thu, 30 Jul 2020 12:09:43 +0800 Subject: [PATCH 02/10] Typo fix Signed-off-by: JmPotato --- configure-placement-rules.md | 2 +- location-awareness.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/configure-placement-rules.md b/configure-placement-rules.md index 5a0638b94412d..2495b9a2f4e98 100644 --- a/configure-placement-rules.md +++ b/configure-placement-rules.md @@ -49,7 +49,7 @@ The following table shows the meaning of each field in a rule: The meaning and function of `LocationLabels` are the same with those earlier than v4.0. For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology: the cluster has multiple zones (Availability Zones), each zone has multiple racks, and each rack has multiple hosts. When performing schedule, PD first tries to place the Region's peers in different zones. If this try fails (such as there are three replicas but only two zones in total), PD guarantees to place these replicas in different racks. If the number of racks is not enough to guarantee isolation, then PD tries the host-level isolation. -The meaning and function of `IsolationLevel` is elaborated in [Cluster topology configuration](/location-awareness.md). For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology with `LocationLabels` and set `IsolationLevel` to `"zone"`. Then PD will ensure that all peers of each Region are placed in different zones when scheduling. If the minimum isolation level restriction of `IsolationLevel` cannot be met (e.g, 3 replicas are configured but there are only 2 data zones in total), PD will not try to make up to meet the restriction. The default value of `IsolationLevel` is an empty string, which means it is disabled. +The meaning and function of `IsolationLevel` is elaborated in [Cluster topology configuration](/location-awareness.md). For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology with `LocationLabels` and set `IsolationLevel` to `zone`. Then PD will ensure that all peers of each Region are placed in different zones when scheduling. If the minimum isolation level restriction of `IsolationLevel` cannot be met (e.g, 3 replicas are configured but there are only 2 data zones in total), PD will not try to make up to meet the restriction. The default value of `IsolationLevel` is an empty string, which means it is disabled. ## Configure rules diff --git a/location-awareness.md b/location-awareness.md index a48f31ad04399..72fc4feca42de 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -123,6 +123,6 @@ In this case, PD will schedule different replicas of each datum to different dat To sum up, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to reach a certain level of disaster recovery, deploy many machines in different sites according to the topology. The number of machines must be more than the number of `max-replicas`. -However, with setting `isolation-level` to `"zone"`, PD will strictly ensure that each replica will isolated from each other at the zone level, even if guaranteeing this will not meet the number requirement of `max-replicas`. For example, there are three data zones z1/z2/z3 in the TiKV cluster. Under the requirement of 3 replicas, PD will dispatch the three replicas of the same Region to these three data zones. If there is a power outage in the z1 and it cannot be recovered after a period of time, PD will think that the region replica on z1 is no longer available, but because the `isolation-level` is set to `"zone"`, PD needs to strictly guarantee different region replicas will not fall on the same data zone. Because both z2 and z3 already have copies, PD will not perform any scheduling under the minimum isolation level limit of `isolation-level`, even if there are only two copies at this moment. +However, with setting `isolation-level` to `zone`, PD will strictly ensure that each replica will isolated from each other at the zone level, even if guaranteeing this will not meet the number requirement of `max-replicas`. For example, there are three data zones z1/z2/z3 in the TiKV cluster. Under the requirement of 3 replicas, PD will dispatch the three replicas of the same Region to these three data zones. If there is a power outage in the z1 and it cannot be recovered after a period of time, PD will think that the region replica on z1 is no longer available, but because the `isolation-level` is set to `zone`, PD needs to strictly guarantee different region replicas will not fall on the same data zone. Because both z2 and z3 already have copies, PD will not perform any scheduling under the minimum isolation level limit of `isolation-level`, even if there are only two copies at this moment. -Similarly, when `isolation-level` is `"rack"`, the minimum isolation level will be different racks in the same data zone. Under this configuration, the isolation at the zone level will be guaranteed first if it can be. When the isolation at the zone level cannot be promised, PD will consider to avoid scheduling replicas to be in the same zone with the same rack, and so on. \ No newline at end of file +Similarly, when `isolation-level` is `rack`, the minimum isolation level will be different racks in the same data zone. Under this configuration, the isolation at the zone level will be guaranteed first if it can be. When the isolation at the zone level cannot be promised, PD will consider to avoid scheduling replicas to be in the same zone with the same rack, and so on. \ No newline at end of file From 428a15bdc4440c1e26691012cf04515118e1b205 Mon Sep 17 00:00:00 2001 From: Haizhi Geng Date: Thu, 30 Jul 2020 16:08:22 +0800 Subject: [PATCH 03/10] Update configure-placement-rules.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- configure-placement-rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure-placement-rules.md b/configure-placement-rules.md index 2495b9a2f4e98..7a69fb4c8f4a0 100644 --- a/configure-placement-rules.md +++ b/configure-placement-rules.md @@ -49,7 +49,7 @@ The following table shows the meaning of each field in a rule: The meaning and function of `LocationLabels` are the same with those earlier than v4.0. For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology: the cluster has multiple zones (Availability Zones), each zone has multiple racks, and each rack has multiple hosts. When performing schedule, PD first tries to place the Region's peers in different zones. If this try fails (such as there are three replicas but only two zones in total), PD guarantees to place these replicas in different racks. If the number of racks is not enough to guarantee isolation, then PD tries the host-level isolation. -The meaning and function of `IsolationLevel` is elaborated in [Cluster topology configuration](/location-awareness.md). For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology with `LocationLabels` and set `IsolationLevel` to `zone`. Then PD will ensure that all peers of each Region are placed in different zones when scheduling. If the minimum isolation level restriction of `IsolationLevel` cannot be met (e.g, 3 replicas are configured but there are only 2 data zones in total), PD will not try to make up to meet the restriction. The default value of `IsolationLevel` is an empty string, which means it is disabled. +The meaning and function of `IsolationLevel` is elaborated in [Cluster topology configuration](/location-awareness.md). For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology with `LocationLabels` and set `IsolationLevel` to `zone`, then PD ensures that all peers of each Region are placed in different zones during the scheduling. If the minimum isolation level restriction on `IsolationLevel` cannot be met (for example, 3 replicas are configured but there are only 2 data zones in total), PD will not try to make up to meet this restriction. The default value of `IsolationLevel` is an empty string, which means that it is disabled. ## Configure rules From cb9ce0c7a0a925bec9a406a7ecac1196475be19b Mon Sep 17 00:00:00 2001 From: Haizhi Geng Date: Thu, 30 Jul 2020 16:08:39 +0800 Subject: [PATCH 04/10] Update location-awareness.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- location-awareness.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/location-awareness.md b/location-awareness.md index 72fc4feca42de..effbbda9bbadc 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -8,7 +8,7 @@ aliases: ['/docs/dev/location-awareness/','/docs/dev/how-to/deploy/geographic-re ## Overview -PD schedules according to the topology of the TiKV cluster to maximize the TiKV's capability for disaster recovery. We recommend that TiKV nodes should be physically dispersed as much as possible. For example, TiKV nodes can be distributed in different racks or even different data zones. According to the topology information of TiKV, the PD scheduler will automatically schedule in the background to isolate the replicas of the Region as much as possible, thereby maximizing the data disaster tolerance. +PD schedules according to the topology of the TiKV cluster to maximize the TiKV's capability for disaster recovery. It is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data zones. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). @@ -125,4 +125,4 @@ To sum up, PD maximizes the disaster recovery of the cluster according to the cu However, with setting `isolation-level` to `zone`, PD will strictly ensure that each replica will isolated from each other at the zone level, even if guaranteeing this will not meet the number requirement of `max-replicas`. For example, there are three data zones z1/z2/z3 in the TiKV cluster. Under the requirement of 3 replicas, PD will dispatch the three replicas of the same Region to these three data zones. If there is a power outage in the z1 and it cannot be recovered after a period of time, PD will think that the region replica on z1 is no longer available, but because the `isolation-level` is set to `zone`, PD needs to strictly guarantee different region replicas will not fall on the same data zone. Because both z2 and z3 already have copies, PD will not perform any scheduling under the minimum isolation level limit of `isolation-level`, even if there are only two copies at this moment. -Similarly, when `isolation-level` is `rack`, the minimum isolation level will be different racks in the same data zone. Under this configuration, the isolation at the zone level will be guaranteed first if it can be. When the isolation at the zone level cannot be promised, PD will consider to avoid scheduling replicas to be in the same zone with the same rack, and so on. \ No newline at end of file +Similarly, when `isolation-level` is `rack`, the minimum isolation level will be different racks in the same data zone. Under this configuration, the isolation at the zone level will be guaranteed first if it can be. When the isolation at the zone level cannot be promised, PD will consider to avoid scheduling replicas to be in the same zone with the same rack, and so on. From 8e33425c2dff437359181e243bedb78177140c5b Mon Sep 17 00:00:00 2001 From: Haizhi Geng Date: Thu, 30 Jul 2020 16:09:13 +0800 Subject: [PATCH 05/10] Update location-awareness.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- location-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/location-awareness.md b/location-awareness.md index effbbda9bbadc..cbe5a7a26a1ac 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -57,7 +57,7 @@ pd-ctl config set location-labels zone,rack,host ## PD restricts the TiKV topology -With configuring `location-labels` first, we can further enhance the topological isolation requirements of TiKV clusters through the `isolation-level`. Suppose we divide the cluster topology into three layers through `location-labels` according to the above instructions: zone -> rack -> host, and configure the `isolation-level` as follows. +Having configured `location-labels`, you can further enhance the topological isolation requirements on TiKV clusters through the `isolation-level` parameter. Assume that you have made a three-layer cluster topology by configuring `location-labels` according to the instructions above: zone -> rack -> host, and have configured the `isolation-level` as follows: {{< copyable "" >}} From 789013dec3338c85afdd0261ca8030d4d48583dc Mon Sep 17 00:00:00 2001 From: Haizhi Geng Date: Thu, 30 Jul 2020 16:09:21 +0800 Subject: [PATCH 06/10] Update location-awareness.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- location-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/location-awareness.md b/location-awareness.md index cbe5a7a26a1ac..107d3c5e94b66 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -74,7 +74,7 @@ After the PD cluster is initialized, you need to use the pd-ctl tool to make onl pd-ctl config set isolation-level zone ``` -`isolation-level` needs to correspond to one of the `location-labels` name so that PD can understand that this label represents the TiKV topology. +`isolation-level` needs to correspond to one of the `location-labels` names so that PD can understand that this label represents the TiKV topology. > **Note:** > From 9f8ef15f69707450903974e6d8036b4299eeb1b5 Mon Sep 17 00:00:00 2001 From: Haizhi Geng Date: Thu, 30 Jul 2020 16:09:54 +0800 Subject: [PATCH 07/10] Update location-awareness.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- location-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/location-awareness.md b/location-awareness.md index 107d3c5e94b66..6644574681c4c 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -78,7 +78,7 @@ pd-ctl config set isolation-level zone > **Note:** > -> `isolation-level` is empty by default, that is, there is no mandatory isolation level restriction. To set it, you must first configure the PD's `location-labels` parameter, and ensure that the value of `isolation-level` must be one of `location-labels`. +> `isolation-level` is empty by default, that is, there is no mandatory isolation level restriction. To set it, you must first configure the PD's `location-labels` parameter, and ensure that the value of `isolation-level` must be one of the `location-labels` names. ## PD schedules based on the TiKV topology From 4b96c9bfacba33c62dff7c4fe8e22f4edb43ee4f Mon Sep 17 00:00:00 2001 From: Haizhi Geng Date: Thu, 30 Jul 2020 16:10:20 +0800 Subject: [PATCH 08/10] Update location-awareness.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- location-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/location-awareness.md b/location-awareness.md index 6644574681c4c..dd3956cee5949 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -123,6 +123,6 @@ In this case, PD will schedule different replicas of each datum to different dat To sum up, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to reach a certain level of disaster recovery, deploy many machines in different sites according to the topology. The number of machines must be more than the number of `max-replicas`. -However, with setting `isolation-level` to `zone`, PD will strictly ensure that each replica will isolated from each other at the zone level, even if guaranteeing this will not meet the number requirement of `max-replicas`. For example, there are three data zones z1/z2/z3 in the TiKV cluster. Under the requirement of 3 replicas, PD will dispatch the three replicas of the same Region to these three data zones. If there is a power outage in the z1 and it cannot be recovered after a period of time, PD will think that the region replica on z1 is no longer available, but because the `isolation-level` is set to `zone`, PD needs to strictly guarantee different region replicas will not fall on the same data zone. Because both z2 and z3 already have copies, PD will not perform any scheduling under the minimum isolation level limit of `isolation-level`, even if there are only two copies at this moment. +However, if `isolation-level` is set to `zone`, PD will ensure that different replicas of a Region are isolated from each other at the zone level, even if guaranteeing this restriction does not meet the requirement of `max-replicas`. For example, a TiKV cluster is distributed across three data zones z1/z2/z3. Each Region has three replicas as required, and PD distributes the three replicas of the same Region to these three data zones respectively. If a power outage occurs in z1 and cannot be recovered after a period of time, PD determines that the Region replicas on z1 are no longer available. However, because `isolation-level` is set to `zone`, PD needs to strictly guarantee that different replicas of the same Region will not be scheduled on the same data zone. Because both z2 and z3 already have replicas, PD will not perform any scheduling under the minimum isolation level restriction of `isolation-level`, even if there are only two replicas at this moment. Similarly, when `isolation-level` is `rack`, the minimum isolation level will be different racks in the same data zone. Under this configuration, the isolation at the zone level will be guaranteed first if it can be. When the isolation at the zone level cannot be promised, PD will consider to avoid scheduling replicas to be in the same zone with the same rack, and so on. From 28016e7b4fe4f748183c8225d2191ef4079d5950 Mon Sep 17 00:00:00 2001 From: JmPotato Date: Thu, 30 Jul 2020 16:17:21 +0800 Subject: [PATCH 09/10] Address comments Signed-off-by: JmPotato --- configure-placement-rules.md | 3 ++- location-awareness.md | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/configure-placement-rules.md b/configure-placement-rules.md index 7a69fb4c8f4a0..f746f8d968a87 100644 --- a/configure-placement-rules.md +++ b/configure-placement-rules.md @@ -78,7 +78,8 @@ In this way, PD enables this feature after the cluster is successfully bootstrap "end_key": "", "role": "voter", "count": 3, - "location_labels": ["zone", "rack", "host"] + "location_labels": ["zone", "rack", "host"], + "isolation_level": "" } ``` diff --git a/location-awareness.md b/location-awareness.md index dd3956cee5949..577d86098bdea 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -121,8 +121,8 @@ In this case, PD will schedule different replicas of each datum to different dat - If one of the data zones goes down, the high availability of the TiDB cluster is not affected. - If the data zone cannot recover within a period of time, PD will remove the replica from this data zone. -To sum up, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to reach a certain level of disaster recovery, deploy many machines in different sites according to the topology. The number of machines must be more than the number of `max-replicas`. - However, if `isolation-level` is set to `zone`, PD will ensure that different replicas of a Region are isolated from each other at the zone level, even if guaranteeing this restriction does not meet the requirement of `max-replicas`. For example, a TiKV cluster is distributed across three data zones z1/z2/z3. Each Region has three replicas as required, and PD distributes the three replicas of the same Region to these three data zones respectively. If a power outage occurs in z1 and cannot be recovered after a period of time, PD determines that the Region replicas on z1 are no longer available. However, because `isolation-level` is set to `zone`, PD needs to strictly guarantee that different replicas of the same Region will not be scheduled on the same data zone. Because both z2 and z3 already have replicas, PD will not perform any scheduling under the minimum isolation level restriction of `isolation-level`, even if there are only two replicas at this moment. -Similarly, when `isolation-level` is `rack`, the minimum isolation level will be different racks in the same data zone. Under this configuration, the isolation at the zone level will be guaranteed first if it can be. When the isolation at the zone level cannot be promised, PD will consider to avoid scheduling replicas to be in the same zone with the same rack, and so on. +Similarly, when `isolation-level` is set to `rack`, the minimum isolation level applies to different racks in the same data zone. With this configuration, the isolation at the zone level is guaranteed first if possible. When the isolation at the zone level cannot be guaranteed, PD tries to avoid scheduling different replicas to the same rack in the same zone, and so on. + +To sum up, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to reach a certain level of disaster recovery, deploy many machines in different sites according to the topology. The number of machines must be more than the number of `max-replicas`. TiDB also provides a mandatory isolation level configuration `isolation-level` in order to control the topological isolation level of data depending on different cases flexibly. \ No newline at end of file From b058e55e730d2fd8dcfc0429d5a11ad75e630dab Mon Sep 17 00:00:00 2001 From: Haizhi Geng Date: Thu, 30 Jul 2020 17:06:06 +0800 Subject: [PATCH 10/10] Update location-awareness.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- location-awareness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/location-awareness.md b/location-awareness.md index 577d86098bdea..46881778d6f2e 100644 --- a/location-awareness.md +++ b/location-awareness.md @@ -125,4 +125,4 @@ However, if `isolation-level` is set to `zone`, PD will ensure that different re Similarly, when `isolation-level` is set to `rack`, the minimum isolation level applies to different racks in the same data zone. With this configuration, the isolation at the zone level is guaranteed first if possible. When the isolation at the zone level cannot be guaranteed, PD tries to avoid scheduling different replicas to the same rack in the same zone, and so on. -To sum up, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to reach a certain level of disaster recovery, deploy many machines in different sites according to the topology. The number of machines must be more than the number of `max-replicas`. TiDB also provides a mandatory isolation level configuration `isolation-level` in order to control the topological isolation level of data depending on different cases flexibly. \ No newline at end of file +In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. TiDB also provides a mandatory configuration item `isolation-level` to control the topological isolation level of data according to different cases.