From 3d32ffbc290e14fe098b629714fcb89040bae6a3 Mon Sep 17 00:00:00 2001 From: ireneontheway Date: Fri, 4 Sep 2020 10:48:28 +0800 Subject: [PATCH 01/17] Update TOC.md --- TOC.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/TOC.md b/TOC.md index 7b3b47ea8db4b..72dec4d291903 100644 --- a/TOC.md +++ b/TOC.md @@ -507,6 +507,8 @@ + [High Availability FAQs](/faq/high-availability-faq.md) + [High Reliability FAQs](/faq/high-reliability-faq.md) + [Migration FAQs](/faq/migration-tidb-faq.md) ++ Dev Guide + + [Schedule Replicas by Topology Labels](/schedule-replicas-by-topology-labels.md) + [Glossary](/glossary.md) + Release Notes + [All Releases](/releases/release-notes.md) From f0f1dd1cac9fff793421cab65d5557b02fd745eb Mon Sep 17 00:00:00 2001 From: ireneontheway Date: Fri, 4 Sep 2020 11:20:49 +0800 Subject: [PATCH 02/17] update files --- location-awareness.md | 128 ------------------------ pd-configuration-file.md | 4 +- schedule-replicas-by-topology-labels.md | 100 ++++++++++++++++++ 3 files changed, 102 insertions(+), 130 deletions(-) delete mode 100644 location-awareness.md create mode 100644 schedule-replicas-by-topology-labels.md diff --git a/location-awareness.md b/location-awareness.md deleted file mode 100644 index 46881778d6f2e..0000000000000 --- a/location-awareness.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: Cluster Topology Configuration -summary: Learn to configure cluster topology to maximize the capacity for disaster recovery. -aliases: ['/docs/dev/location-awareness/','/docs/dev/how-to/deploy/geographic-redundancy/location-awareness/'] ---- - -# Cluster Topology Configuration - -## Overview - -PD schedules according to the topology of the TiKV cluster to maximize the TiKV's capability for disaster recovery. It is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data zones. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. - -Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). - -## TiKV reports the topological information - -TiKV reports the topological information to PD according to the startup parameter or configuration of TiKV. - -Assuming that the topology has three structures: zone > rack > host, use labels to specify the following information: - -Startup parameter: - -``` -tikv-server --labels zone=,rack=,host= -``` - -Configuration: - -``` toml -[server] -labels = "zone=,rack=,host=" -``` - -## PD understands the TiKV topology - -PD gets the topology of TiKV cluster through the PD configuration. - -``` toml -[replication] -max-replicas = 3 -location-labels = ["zone", "rack", "host"] -``` - -After the PD cluster is initialized, you need to use the pd-ctl tool to make online changes: - -{{< copyable "shell-regular" >}} - -```bash -pd-ctl config set location-labels zone,rack,host -``` - -`location-labels` needs to correspond to the TiKV `labels` name so that PD can understand that the `labels` represents the TiKV topology. - -> **Note:** -> -> You must configure `location-labels` for PD and `labels` for TiKV at the same time for `labels` to take effect. - -## PD restricts the TiKV topology - -Having configured `location-labels`, you can further enhance the topological isolation requirements on TiKV clusters through the `isolation-level` parameter. Assume that you have made a three-layer cluster topology by configuring `location-labels` according to the instructions above: zone -> rack -> host, and have configured the `isolation-level` as follows: - -{{< copyable "" >}} - -```toml -[replication] -isolation-level = "zone" -``` - -After the PD cluster is initialized, you need to use the pd-ctl tool to make online changes: - -{{< copyable "shell-regular" >}} - -```bash -pd-ctl config set isolation-level zone -``` - -`isolation-level` needs to correspond to one of the `location-labels` names so that PD can understand that this label represents the TiKV topology. - -> **Note:** -> -> `isolation-level` is empty by default, that is, there is no mandatory isolation level restriction. To set it, you must first configure the PD's `location-labels` parameter, and ensure that the value of `isolation-level` must be one of the `location-labels` names. - -## PD schedules based on the TiKV topology - -PD makes optimal scheduling according to the topological information. You just need to care about what kind of topology can achieve the desired effect. - -If you use 3 replicas and hope that the TiDB cluster is always highly available even when a data zone goes down, you need at least 4 data zones. - -Assume that you have 4 data zones, each zone has 2 racks, and each rack has 2 hosts. You can start 2 TiKV instances on each host: - -``` -# zone=z1 -tikv-server --labels zone=z1,rack=r1,host=h1 -tikv-server --labels zone=z1,rack=r1,host=h2 -tikv-server --labels zone=z1,rack=r2,host=h1 -tikv-server --labels zone=z1,rack=r2,host=h2 - -# zone=z2 -tikv-server --labels zone=z2,rack=r1,host=h1 -tikv-server --labels zone=z2,rack=r1,host=h2 -tikv-server --labels zone=z2,rack=r2,host=h1 -tikv-server --labels zone=z2,rack=r2,host=h2 - -# zone=z3 -tikv-server --labels zone=z3,rack=r1,host=h1 -tikv-server --labels zone=z3,rack=r1,host=h2 -tikv-server --labels zone=z3,rack=r2,host=h1 -tikv-server --labels zone=z3,rack=r2,host=h2 - -# zone=z4 -tikv-server --labels zone=z4,rack=r1,host=h1 -tikv-server --labels zone=z4,rack=r1,host=h2 -tikv-server --labels zone=z4,rack=r2,host=h1 -tikv-server --labels zone=z4,rack=r2,host=h2 -``` - -In other words, 16 TiKV instances are distributed across 4 data zones, 8 racks and 16 machines. - -In this case, PD will schedule different replicas of each datum to different data zones. - -- If one of the data zones goes down, the high availability of the TiDB cluster is not affected. -- If the data zone cannot recover within a period of time, PD will remove the replica from this data zone. - -However, if `isolation-level` is set to `zone`, PD will ensure that different replicas of a Region are isolated from each other at the zone level, even if guaranteeing this restriction does not meet the requirement of `max-replicas`. For example, a TiKV cluster is distributed across three data zones z1/z2/z3. Each Region has three replicas as required, and PD distributes the three replicas of the same Region to these three data zones respectively. If a power outage occurs in z1 and cannot be recovered after a period of time, PD determines that the Region replicas on z1 are no longer available. However, because `isolation-level` is set to `zone`, PD needs to strictly guarantee that different replicas of the same Region will not be scheduled on the same data zone. Because both z2 and z3 already have replicas, PD will not perform any scheduling under the minimum isolation level restriction of `isolation-level`, even if there are only two replicas at this moment. - -Similarly, when `isolation-level` is set to `rack`, the minimum isolation level applies to different racks in the same data zone. With this configuration, the isolation at the zone level is guaranteed first if possible. When the isolation at the zone level cannot be guaranteed, PD tries to avoid scheduling different replicas to the same rack in the same zone, and so on. - -In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. TiDB also provides a mandatory configuration item `isolation-level` to control the topological isolation level of data according to different cases. diff --git a/pd-configuration-file.md b/pd-configuration-file.md index 950f1099614f3..f3a71388f3354 100644 --- a/pd-configuration-file.md +++ b/pd-configuration-file.md @@ -311,13 +311,13 @@ Configuration items related to replicas + The topology information of a TiKV cluster + Default value: `[]` -+ [Cluster topology configuration](/location-awareness.md) ++ [Cluster topology configuration](/schedule-replicas-by-topology-labels.md) ### `isolation-level` + The minimum topological isolation level of a TiKV cluster + Default value: `""` -+ [Cluster topology configuration](/location-awareness.md) ++ [Cluster topology configuration](/schedule-replicas-by-topology-labels.md) ### `strictly-match-label` diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md new file mode 100644 index 0000000000000..d81746728b3e0 --- /dev/null +++ b/schedule-replicas-by-topology-labels.md @@ -0,0 +1,100 @@ +--- +title: Schedule Replicas by Topology Labels +category: how-to +aliases: ['/docs-cn/dev/how-to/deploy/geographic-redundancy/location-awareness/','/docs-cn/dev/location-awareness/'] +--- + +# Schedule Replicas by Topology Labels + +To improve the high availability and disaster tolerance of TiDB clusters, it is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data zones. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. + +To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for reasonable configuration during deployment. Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). + +## Configure `labels` based on the cluster topology + +### Set the `label` configuration of TiKV + +TiKV supports binding some attributes through command-line flags or configuration files in the form of key-value pairs. The attribute is called a `label`. TiKV reports its labels to PD so we can mark the TiKV location. + +Assume that the topology has three structures: zone > rack > host, use these labels to set the TiKV location. + +Command-line flags: + +{{< copyable "" >}} + +``` +tikv-server --labels zone=,rack=,host= +``` + +Configuration files: + +{{< copyable "" >}} + +```toml +[server] +labels = "zone=,rack=,host=" +``` + +### Set the `location-labels` configuration of PD + +According to the previous description, labels can be any key-value pairs used to describe TiKV attributes, but PD can not distinguish the location-marking labels and the structure among them. Therefore, you need set some configuration to understand the TiKV node topology. + +Set the PD configuration `location-labels` through the PD configuration files. + +{{< copyable "" >}} + +```toml +[replication] +location-labels = ["zone", "rack", "host"] +``` + +After the PD cluster is initialized, you need to use the pd-ctl tool to make online changes: + +{{< copyable "shell-regular" >}} + +```bash +pd-ctl config set location-labels zone,rack,host +``` + +The `location-labels` configuration is an array of strings, and each item corresponds to the key of TiKV `labels`. The sequence of each key represents the layer relationship among different labels. + +> **Note:** +> +> You must configure `location-labels` for PD and `labels` for TiKV at the same time for `labels` to take effect. Otherwise, PD does not schedule according to the topology. + +### Use TiDB Ansible to deploy a cluster + +You can directly set location related configuration in the `inventory.ini` file when using TiDB Ansible to deploy a cluster. `tidb-ansible` generates the corresponding TiKV and PD configuration files during deployment. + +The following example defines a double-level topology of `zone/host`. The TiKV of the cluster is distributed in three zones, with two hosts in each zone. z1 is deployed with two TiKV instances per host, and z2 and z3 are deployed with 1 instance per host. + +``` +[tikv_servers] +# z1 +tikv-1 labels="zone=z1,host=h1" +tikv-2 labels="zone=z1,host=h1" +tikv-3 labels="zone=z1,host=h2" +tikv-4 labels="zone=z1,host=h2" +# z2 +tikv-5 labels="zone=z2,host=h1" +tikv-6 labels="zone=z2,host=h2" +# z3 +tikv-7 labels="zone=z3,host=h1" +tikv-8 labels="zone=z3,host=h2" +[pd_servers:vars] +location_labels = ["zone", "rack"] +``` + +## PD schedules based on topology label + +PD schedules replicas according to the label level to make sure that different replicas of the same data are distributed as much as possible. + +Take the topology in the previous section as an example. + +Assume that the number of cluster replicas is 3 (`max-replicas=3`). As there are 3 zones in total, PD keeps the 3 replicas of each Region placed in z1/z2/z3, so that the TiDB cluster is still available when any data center fails, + +Assume that the number of cluster replicas is 5 (`max-replicas=5`). As there are only 3 zones in total, PD can not guarantee the isolation of each replica at this level. At this time, the PD scheduler adjusts to ensure isolation at the host level. In other words, multiple copies of a Region might be distributed in the same zone, but will not be distributed on the same host. + +Under the premise of the 5-replica configuration, if z3 occurs an overall failure or isolation, and can not be restored after a period of time (controlled by `max-store-down-time`), PD will make up the 5 replicas through scheduling. At this time, only 3 hosts are available, so host level isolation can not be guaranteed. Therefore, multiple copies might be distributed to the same host. + +In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. \ No newline at end of file From 5d2625dc16560a5c9949d0e57b1dade20cebbf90 Mon Sep 17 00:00:00 2001 From: ireneontheway Date: Fri, 4 Sep 2020 11:28:09 +0800 Subject: [PATCH 03/17] Fix links --- schedule-replicas-by-topology-labels.md | 2 +- tidb-scheduling.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index d81746728b3e0..c78f843ea1ba1 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -8,7 +8,7 @@ aliases: ['/docs-cn/dev/how-to/deploy/geographic-redundancy/location-awareness/' To improve the high availability and disaster tolerance of TiDB clusters, it is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data zones. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. -To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for reasonable configuration during deployment. Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). +To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for proper configuration during deployment. Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). ## Configure `labels` based on the cluster topology diff --git a/tidb-scheduling.md b/tidb-scheduling.md index dd0d645dbfb3e..84e9cae9f2225 100644 --- a/tidb-scheduling.md +++ b/tidb-scheduling.md @@ -75,7 +75,7 @@ Scheduling is based on information collection. In short, the PD scheduling compo * Data read/write speed * The number of snapshots that are sent/received (The data might be replicated between replicas through snapshots) * Whether the store is overloaded - * Labels (See [Perception of Topology](/location-awareness.md)) + * Labels (See [Perception of Topology](/schedule-replicas-by-topology-labels.md)) - Information reported by Region leaders: From f9922a014be0ba3550412bbcdfea01a5a65fbd02 Mon Sep 17 00:00:00 2001 From: ireneontheway Date: Fri, 4 Sep 2020 12:03:13 +0800 Subject: [PATCH 04/17] Update configure-placement-rules.md --- configure-placement-rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure-placement-rules.md b/configure-placement-rules.md index f746f8d968a87..b6681e25cb00f 100644 --- a/configure-placement-rules.md +++ b/configure-placement-rules.md @@ -49,7 +49,7 @@ The following table shows the meaning of each field in a rule: The meaning and function of `LocationLabels` are the same with those earlier than v4.0. For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology: the cluster has multiple zones (Availability Zones), each zone has multiple racks, and each rack has multiple hosts. When performing schedule, PD first tries to place the Region's peers in different zones. If this try fails (such as there are three replicas but only two zones in total), PD guarantees to place these replicas in different racks. If the number of racks is not enough to guarantee isolation, then PD tries the host-level isolation. -The meaning and function of `IsolationLevel` is elaborated in [Cluster topology configuration](/location-awareness.md). For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology with `LocationLabels` and set `IsolationLevel` to `zone`, then PD ensures that all peers of each Region are placed in different zones during the scheduling. If the minimum isolation level restriction on `IsolationLevel` cannot be met (for example, 3 replicas are configured but there are only 2 data zones in total), PD will not try to make up to meet this restriction. The default value of `IsolationLevel` is an empty string, which means that it is disabled. +The meaning and function of `IsolationLevel` is elaborated in [Cluster topology configuration](/schedule-replicas-by-topology-labels.md). For example, if you have deployed `[zone,rack,host]` that defines a three-layer topology with `LocationLabels` and set `IsolationLevel` to `zone`, then PD ensures that all peers of each Region are placed in different zones during the scheduling. If the minimum isolation level restriction on `IsolationLevel` cannot be met (for example, 3 replicas are configured but there are only 2 data zones in total), PD will not try to make up to meet this restriction. The default value of `IsolationLevel` is an empty string, which means that it is disabled. ## Configure rules From 821a00b5899383e3a0f42b83c32077ce5d7ec4d7 Mon Sep 17 00:00:00 2001 From: ireneontheway <48651140+ireneontheway@users.noreply.github.com> Date: Mon, 7 Sep 2020 18:07:47 +0800 Subject: [PATCH 05/17] Update TOC.md --- TOC.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/TOC.md b/TOC.md index 72dec4d291903..21f797ef9a43c 100644 --- a/TOC.md +++ b/TOC.md @@ -499,6 +499,7 @@ + [TiCDC Overview](/ticdc/ticdc-overview.md) + [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) + [Table Filter](/table-filter.md) + + [Schedule Replicas by Topology Labels](/schedule-replicas-by-topology-labels.md) + FAQs + [TiDB FAQs](/faq/tidb-faq.md) + [SQL FAQs](/faq/sql-faq.md) @@ -507,8 +508,6 @@ + [High Availability FAQs](/faq/high-availability-faq.md) + [High Reliability FAQs](/faq/high-reliability-faq.md) + [Migration FAQs](/faq/migration-tidb-faq.md) -+ Dev Guide - + [Schedule Replicas by Topology Labels](/schedule-replicas-by-topology-labels.md) + [Glossary](/glossary.md) + Release Notes + [All Releases](/releases/release-notes.md) From faa600b77c476ddaba3032920de0efd041745e90 Mon Sep 17 00:00:00 2001 From: ireneontheway <48651140+ireneontheway@users.noreply.github.com> Date: Mon, 7 Sep 2020 18:14:46 +0800 Subject: [PATCH 06/17] Apply suggestions from code review --- schedule-replicas-by-topology-labels.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index c78f843ea1ba1..df865c42173e1 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -1,12 +1,11 @@ --- title: Schedule Replicas by Topology Labels -category: how-to -aliases: ['/docs-cn/dev/how-to/deploy/geographic-redundancy/location-awareness/','/docs-cn/dev/location-awareness/'] +aliases: ['/docs/dev/how-to/deploy/geographic-redundancy/location-awareness/','/docs/dev/location-awareness/'] --- # Schedule Replicas by Topology Labels -To improve the high availability and disaster tolerance of TiDB clusters, it is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data zones. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. +To improve the high availability and disaster tolerance of TiDB clusters, it is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data centers. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for proper configuration during deployment. Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). @@ -48,7 +47,7 @@ Set the PD configuration `location-labels` through the PD configuration files. location-labels = ["zone", "rack", "host"] ``` -After the PD cluster is initialized, you need to use the pd-ctl tool to make online changes: +If PD cluster is already initialized, you need to use the pd-ctl tool to make online changes: {{< copyable "shell-regular" >}} @@ -97,4 +96,4 @@ Assume that the number of cluster replicas is 5 (`max-replicas=5`). As there are Under the premise of the 5-replica configuration, if z3 occurs an overall failure or isolation, and can not be restored after a period of time (controlled by `max-store-down-time`), PD will make up the 5 replicas through scheduling. At this time, only 3 hosts are available, so host level isolation can not be guaranteed. Therefore, multiple copies might be distributed to the same host. -In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. \ No newline at end of file +In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. From 18a44741a141609bd277bb8d3cdfec57199f8a2d Mon Sep 17 00:00:00 2001 From: ireneontheway <48651140+ireneontheway@users.noreply.github.com> Date: Tue, 8 Sep 2020 10:28:04 +0800 Subject: [PATCH 07/17] Update schedule-replicas-by-topology-labels.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- schedule-replicas-by-topology-labels.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index df865c42173e1..2ec11fa93f4ef 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -1,6 +1,6 @@ --- title: Schedule Replicas by Topology Labels -aliases: ['/docs/dev/how-to/deploy/geographic-redundancy/location-awareness/','/docs/dev/location-awareness/'] +aliases: ['/docs/dev/location-awareness/','/docs/dev/how-to/deploy/geographic-redundancy/location-awareness/','/tidb/dev/location-awareness'] --- # Schedule Replicas by Topology Labels From eb1120e75f1b2edbb57f289125cbebea1cee0dc7 Mon Sep 17 00:00:00 2001 From: ireneontheway Date: Tue, 8 Sep 2020 16:24:29 +0800 Subject: [PATCH 08/17] Update schedule-replicas-by-topology-labels.md --- schedule-replicas-by-topology-labels.md | 44 +++++++++++++++++++++---- 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index 2ec11fa93f4ef..604c8bdd11959 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -7,11 +7,11 @@ aliases: ['/docs/dev/location-awareness/','/docs/dev/how-to/deploy/geographic-re To improve the high availability and disaster tolerance of TiDB clusters, it is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data centers. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. -To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for proper configuration during deployment. Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). +To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for proper configuration during deployment. Before you begin, see [Deploy TiDB Using TiUP](/tidb/dev/production-deployment-using-tiup.md). ## Configure `labels` based on the cluster topology -### Set the `label` configuration of TiKV +### Set the `labels` configuration of TiKV TiKV supports binding some attributes through command-line flags or configuration files in the form of key-value pairs. The attribute is called a `label`. TiKV reports its labels to PD so we can mark the TiKV location. @@ -36,7 +36,7 @@ labels = "zone=,rack=,host=" ### Set the `location-labels` configuration of PD -According to the previous description, labels can be any key-value pairs used to describe TiKV attributes, but PD can not distinguish the location-marking labels and the structure among them. Therefore, you need set some configuration to understand the TiKV node topology. +According to the previous description, labels can be any key-value pairs used to describe TiKV attributes, but PD can not distinguish the location-marking labels and the structure among them. Therefore, you need to set some configuration to understand the TiKV node topology. Set the PD configuration `location-labels` through the PD configuration files. @@ -61,6 +61,33 @@ The `location-labels` configuration is an array of strings, and each item corres > > You must configure `location-labels` for PD and `labels` for TiKV at the same time for `labels` to take effect. Otherwise, PD does not schedule according to the topology. +### Set the `isolation-level` configuration of PD + +Under the premise of configuring `location-labels`, you can further strengthen the topological isolation requirements for TiKV clusters through the `isolation-level` configuration. + +Assume that the topology is divided into three structures through `location-labels` according to the above instructions: zone > rack > host, and the `isolation-level` is configured as follows: + +{{< copyable "" >}} + +```toml +[replication] +isolation-level = "zone" +``` + +If PD cluster is already initialized, you need to use the pd-ctl tool to make online changes: + +{{< copyable "shell-regular" >}} + +```bash +pd-ctl config set isolation-level zone +``` + +The `location-level` configuration is an array of strings, which needs to correspond to a key of `location-labels`. This parameter limits the minimum and mandatory isolation level requirements for TiKV topology clusters. + +> **Note:** +> +> `isolation-level` is empty by default, which means there is no mandatory isolation level restriction. To set it, you need to configure the `location-labels` of PD, and ensure that the value of `isolation-level` must be one of `location-labels`. + ### Use TiDB Ansible to deploy a cluster You can directly set location related configuration in the `inventory.ini` file when using TiDB Ansible to deploy a cluster. `tidb-ansible` generates the corresponding TiKV and PD configuration files during deployment. @@ -80,8 +107,9 @@ tikv-6 labels="zone=z2,host=h2" # z3 tikv-7 labels="zone=z3,host=h1" tikv-8 labels="zone=z3,host=h2" + [pd_servers:vars] -location_labels = ["zone", "rack"] +location_labels = ["zone", "host"] ``` ## PD schedules based on topology label @@ -94,6 +122,10 @@ Assume that the number of cluster replicas is 3 (`max-replicas=3`). As there are Assume that the number of cluster replicas is 5 (`max-replicas=5`). As there are only 3 zones in total, PD can not guarantee the isolation of each replica at this level. At this time, the PD scheduler adjusts to ensure isolation at the host level. In other words, multiple copies of a Region might be distributed in the same zone, but will not be distributed on the same host. -Under the premise of the 5-replica configuration, if z3 occurs an overall failure or isolation, and can not be restored after a period of time (controlled by `max-store-down-time`), PD will make up the 5 replicas through scheduling. At this time, only 3 hosts are available, so host level isolation can not be guaranteed. Therefore, multiple copies might be distributed to the same host. +Under the premise of the 5-replica configuration, if z3 occurs an overall failure or isolation, and can not be restored in a period (controlled by `max-store-down-time`), PD will make up the 5 replicas through scheduling. At this time, only 3 hosts are available, so host level isolation can not be guaranteed. Therefore, multiple copies might be distributed to the same host. But if the value `isolation-level` is set to `zone` instead of empty, this specifies the minimum physical isolation requirements for Region replicas. That is to say, PD ensures that replicas of the same Region are scattered in different zones. Even if following this isolation restriction does not meet the multiple replica requirements of `max-replicas`, PD will not perform corresponding scheduling. + +For example, there are three data centers z1, z2 and z3 in the TiKV cluster. Under the three-replica setting, PD distributes the three replicas of the same Region to these three data centers respectively. At this time, if the entire data center z1 encounters a power outage and cannot be recovered in a period, the PD will consider that the Region copy on z1 is no longer available. But because the `isolation-level` is set to `zone`, PD needs to strictly ensure that different Region replicas will not fall into the same zone. At this time, as both z2 and z3 already have replicas, PD does not perform any scheduling under the restriction of the minimum mandatory isolation level, even if there are only two replicas. + +Similarly, when `isolation-level` is set to `rack`, the minimum isolation level is different racks in the same data center. Under this setting, if isolation can be guaranteed at the zone level, it will be guaranteed first. Only when the zone level isolation can not be completed, will the scheduling in the same zone and the same rack be considered to avoid. -In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. +In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. Also, TiDB provides mandatory isolation level settings such as `isolation-level` to control the topological isolation level according to the scenario more flexibly. From 9a19f7dbee88df0170b3cf6903a727777e6df44c Mon Sep 17 00:00:00 2001 From: ireneontheway Date: Tue, 8 Sep 2020 16:30:49 +0800 Subject: [PATCH 09/17] Update schedule-replicas-by-topology-labels.md --- schedule-replicas-by-topology-labels.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index 604c8bdd11959..f019f18e3e2cf 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -7,7 +7,7 @@ aliases: ['/docs/dev/location-awareness/','/docs/dev/how-to/deploy/geographic-re To improve the high availability and disaster tolerance of TiDB clusters, it is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data centers. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. -To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for proper configuration during deployment. Before you begin, see [Deploy TiDB Using TiUP](/tidb/dev/production-deployment-using-tiup.md). +To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for proper configuration during deployment. Before you begin, see [Deploy TiDB Using TiUP](/production-deployment-using-tiup.md). ## Configure `labels` based on the cluster topology From 860ffad643eda7d506bc9e07593bcffa5e1d6c78 Mon Sep 17 00:00:00 2001 From: ireneontheway <48651140+ireneontheway@users.noreply.github.com> Date: Mon, 14 Sep 2020 12:12:37 +0800 Subject: [PATCH 10/17] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- schedule-replicas-by-topology-labels.md | 91 +++++++++++++------------ 1 file changed, 47 insertions(+), 44 deletions(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index f019f18e3e2cf..aa5b03c0ca9a6 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -1,71 +1,72 @@ --- title: Schedule Replicas by Topology Labels +summary: Learn how to schedule replicas by topology labels. aliases: ['/docs/dev/location-awareness/','/docs/dev/how-to/deploy/geographic-redundancy/location-awareness/','/tidb/dev/location-awareness'] --- # Schedule Replicas by Topology Labels -To improve the high availability and disaster tolerance of TiDB clusters, it is recommended that TiKV nodes are physically distributed as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data centers. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate the replicas of Regions as much as possible, thereby maximizing the capability for disaster recovery. +To improve the high availability and disaster recovery capability of TiDB clusters, it is recommended that TiKV nodes are physically scattered as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data centers. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate each replica of a Region as much as possible, which maximizes the capability of disaster recovery. -To make this mechanism effective, you need to report the topology information of the cluster, especially the TiKV location, to PD for proper configuration during deployment. Before you begin, see [Deploy TiDB Using TiUP](/production-deployment-using-tiup.md). +To make this mechanism effective, you need to properly configure TiKV and PD so that the topology information of the cluster, especially the TiKV location information, is reported to PD during deployment. Before you begin, see [Deploy TiDB Using TiUP](/production-deployment-using-tiup.md) first. ## Configure `labels` based on the cluster topology -### Set the `labels` configuration of TiKV +### Configure `labels` for TiKV -TiKV supports binding some attributes through command-line flags or configuration files in the form of key-value pairs. The attribute is called a `label`. TiKV reports its labels to PD so we can mark the TiKV location. +You can use the command-line flag or set the TiKV configuration file to bind some attributes in the form of key-value pairs. These attributes are called `labels`. TiKV reports its `labels` to PD so users can identify the location of TiKV nodes. -Assume that the topology has three structures: zone > rack > host, use these labels to set the TiKV location. +Assume that the topology has three layers: zone > rack > host, and you can use these labels (zone, rack, host) to set the TiKV location. -Command-line flags: ++ If the TiKV cluster is not started, use the command-line flag: -{{< copyable "" >}} + {{< copyable "" >}} -``` -tikv-server --labels zone=,rack=,host= -``` + ``` + tikv-server --labels zone=,rack=,host= + ``` -Configuration files: ++ If the TiKV cluster is started, configure in the TiKV configuration file: -{{< copyable "" >}} + {{< copyable "" >}} -```toml -[server] -labels = "zone=,rack=,host=" -``` + ```toml + [server] + labels = "zone=,rack=,host=" + ``` -### Set the `location-labels` configuration of PD +### Configure `location-labels` for PD -According to the previous description, labels can be any key-value pairs used to describe TiKV attributes, but PD can not distinguish the location-marking labels and the structure among them. Therefore, you need to set some configuration to understand the TiKV node topology. +According to the description above, the label can be any key-value pair used to describe TiKV attributes. But PD cannot identify the location-related labels and the layer relationship of these labels. Therefore, you need to make the following configuration for PD to understand the TiKV node topology. -Set the PD configuration `location-labels` through the PD configuration files. ++ If the PD cluster is not started, configure `location-labels` in the PD configuration file: -{{< copyable "" >}} + {{< copyable "" >}} -```toml -[replication] -location-labels = ["zone", "rack", "host"] -``` + ```toml + [replication] + location-labels = ["zone", "rack", "host"] + ``` -If PD cluster is already initialized, you need to use the pd-ctl tool to make online changes: ++ If the PD cluster is already initialized, use the pd-ctl tool to make online changes: -{{< copyable "shell-regular" >}} + {{< copyable "shell-regular" >}} -```bash -pd-ctl config set location-labels zone,rack,host -``` + ```bash + pd-ctl config set location-labels zone,rack,host + ``` -The `location-labels` configuration is an array of strings, and each item corresponds to the key of TiKV `labels`. The sequence of each key represents the layer relationship among different labels. +The `location-labels` configuration is an array of strings, and each item corresponds to the key of TiKV `labels`. The sequence of each key represents the layer relationship of different labels. > **Note:** > -> You must configure `location-labels` for PD and `labels` for TiKV at the same time for `labels` to take effect. Otherwise, PD does not schedule according to the topology. +> You must configure `location-labels` for PD and `labels` for TiKV at the same time for the configurations to take effect. Otherwise, PD does not perform scheduling according to the topology. -### Set the `isolation-level` configuration of PD +### Configure `isolation-level` for PD -Under the premise of configuring `location-labels`, you can further strengthen the topological isolation requirements for TiKV clusters through the `isolation-level` configuration. +If `location-labels` has been configured, you can further enhance the topological isolation requirements on TiKV clusters by configuring `isolation-level` in the PD configuration file: -Assume that the topology is divided into three structures through `location-labels` according to the above instructions: zone > rack > host, and the `isolation-level` is configured as follows: +Assume that you have made a three-layer cluster topology by configuring `location-labels` according to the instructions above: zone -> rack -> host, and have configured the `isolation-level` as follows: {{< copyable "" >}} @@ -74,7 +75,7 @@ Assume that the topology is divided into three structures through `location-labe isolation-level = "zone" ``` -If PD cluster is already initialized, you need to use the pd-ctl tool to make online changes: +If the PD cluster is already initialized, you need to use the pd-ctl tool to make online changes: {{< copyable "shell-regular" >}} @@ -82,15 +83,16 @@ If PD cluster is already initialized, you need to use the pd-ctl tool to make on pd-ctl config set isolation-level zone ``` -The `location-level` configuration is an array of strings, which needs to correspond to a key of `location-labels`. This parameter limits the minimum and mandatory isolation level requirements for TiKV topology clusters. +The `location-level` configuration is an array of strings, which needs to correspond to a key of `location-labels`. This parameter limits the minimum and mandatory isolation level requirements on TiKV topology clusters. > **Note:** > -> `isolation-level` is empty by default, which means there is no mandatory isolation level restriction. To set it, you need to configure the `location-labels` of PD, and ensure that the value of `isolation-level` must be one of `location-labels`. +> `isolation-level` is empty by default, which means there is no mandatory restriction on the isolation level. To set it, you need to configure `location-labels` for PD and ensure that the value of `isolation-level` is one of `location-labels` names. -### Use TiDB Ansible to deploy a cluster +
+ Configure a cluster using TiDB Ansible -You can directly set location related configuration in the `inventory.ini` file when using TiDB Ansible to deploy a cluster. `tidb-ansible` generates the corresponding TiKV and PD configuration files during deployment. +When using TiDB Ansible to deploy a cluster, you can directly configure the TiKV location in the `inventory.ini` file. `tidb-ansible` will generate the corresponding TiKV and PD configuration files during deployment. The following example defines a double-level topology of `zone/host`. The TiKV of the cluster is distributed in three zones, with two hosts in each zone. z1 is deployed with two TiKV instances per host, and z2 and z3 are deployed with 1 instance per host. @@ -111,20 +113,21 @@ tikv-8 labels="zone=z3,host=h2" [pd_servers:vars] location_labels = ["zone", "host"] ``` +
## PD schedules based on topology label -PD schedules replicas according to the label level to make sure that different replicas of the same data are distributed as much as possible. +PD schedules replicas according to the label layer to make sure that different replicas of the same data are scattered as much as possible. Take the topology in the previous section as an example. -Assume that the number of cluster replicas is 3 (`max-replicas=3`). As there are 3 zones in total, PD keeps the 3 replicas of each Region placed in z1/z2/z3, so that the TiDB cluster is still available when any data center fails, +Assume that the number of cluster replicas is 3 (`max-replicas=3`). Because there are 3 zones in total, PD ensures that the 3 replicas of each Region are respectively placed in z1, z2, and z3. In this way, the TiDB cluster is still available when one data center fails. -Assume that the number of cluster replicas is 5 (`max-replicas=5`). As there are only 3 zones in total, PD can not guarantee the isolation of each replica at this level. At this time, the PD scheduler adjusts to ensure isolation at the host level. In other words, multiple copies of a Region might be distributed in the same zone, but will not be distributed on the same host. +Then, assume that the number of cluster replicas is 5 (`max-replicas=5`). Because there are only 3 zones in total, PD cannot guarantee the isolation of each replica at the zone level. In this situation, the PD scheduler will ensure replica isolation at the host level. In other words, multiple replicas of a Region might be distributed in the same zone but not on the same host. -Under the premise of the 5-replica configuration, if z3 occurs an overall failure or isolation, and can not be restored in a period (controlled by `max-store-down-time`), PD will make up the 5 replicas through scheduling. At this time, only 3 hosts are available, so host level isolation can not be guaranteed. Therefore, multiple copies might be distributed to the same host. But if the value `isolation-level` is set to `zone` instead of empty, this specifies the minimum physical isolation requirements for Region replicas. That is to say, PD ensures that replicas of the same Region are scattered in different zones. Even if following this isolation restriction does not meet the multiple replica requirements of `max-replicas`, PD will not perform corresponding scheduling. +In the case of the 5-replica configuration, if z3 fails or is isolated as a whole, and cannot be recovered after a period of time (controlled by `max-store-down-time`), PD will make up the 5 replicas through scheduling. At this time, only 3 hosts are available. This means that host-level isolation cannot be guaranteed and that multiple replicas might be scheduled to the same host. But if the `isolation-level` value is set to `zone` instead of being left empty, this specifies the minimum physical isolation requirements for Region replicas. That is to say, PD will ensure that replicas of the same Region are scattered among different zones. PD will not perform corresponding scheduling even if following this isolation restriction does not meet the requirement of `max-replicas` for multiple replicas. -For example, there are three data centers z1, z2 and z3 in the TiKV cluster. Under the three-replica setting, PD distributes the three replicas of the same Region to these three data centers respectively. At this time, if the entire data center z1 encounters a power outage and cannot be recovered in a period, the PD will consider that the Region copy on z1 is no longer available. But because the `isolation-level` is set to `zone`, PD needs to strictly ensure that different Region replicas will not fall into the same zone. At this time, as both z2 and z3 already have replicas, PD does not perform any scheduling under the restriction of the minimum mandatory isolation level, even if there are only two replicas. +For example, a TiKV cluster is distributed across three data zones z1, z2, and z3. Each Region has three replicas as required, and PD distributes the three replicas of the same Region to these three data zones respectively. If a power outage occurs in z1 and cannot be recovered after a period of time, PD determines that the Region replicas on z1 are no longer available. However, because `isolation-level` is set to `zone`, PD needs to strictly guarantee that different replicas of the same Region will not be scheduled on the same data zone. Because both z2 and z3 already have replicas, PD will not perform any scheduling under the minimum isolation level restriction of `isolation-level`, even if there are only two replicas at this moment. Similarly, when `isolation-level` is set to `rack`, the minimum isolation level is different racks in the same data center. Under this setting, if isolation can be guaranteed at the zone level, it will be guaranteed first. Only when the zone level isolation can not be completed, will the scheduling in the same zone and the same rack be considered to avoid. From 667caae6c49d35121741243faae4978874056070 Mon Sep 17 00:00:00 2001 From: ireneontheway Date: Mon, 14 Sep 2020 12:15:08 +0800 Subject: [PATCH 11/17] Update schedule-replicas-by-topology-labels.md --- schedule-replicas-by-topology-labels.md | 1 + 1 file changed, 1 insertion(+) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index aa5b03c0ca9a6..c2ab6175a25b2 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -113,6 +113,7 @@ tikv-8 labels="zone=z3,host=h2" [pd_servers:vars] location_labels = ["zone", "host"] ``` + ## PD schedules based on topology label From 1183679be09e668d47f94e438a94df7556065052 Mon Sep 17 00:00:00 2001 From: ireneontheway <48651140+ireneontheway@users.noreply.github.com> Date: Mon, 14 Sep 2020 14:45:44 +0800 Subject: [PATCH 12/17] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- schedule-replicas-by-topology-labels.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index c2ab6175a25b2..0c87e0aa66274 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -94,7 +94,7 @@ The `location-level` configuration is an array of strings, which needs to corres When using TiDB Ansible to deploy a cluster, you can directly configure the TiKV location in the `inventory.ini` file. `tidb-ansible` will generate the corresponding TiKV and PD configuration files during deployment. -The following example defines a double-level topology of `zone/host`. The TiKV of the cluster is distributed in three zones, with two hosts in each zone. z1 is deployed with two TiKV instances per host, and z2 and z3 are deployed with 1 instance per host. +In the following example, a two-layer topology of `zone/host` is defined. The TiKV nodes of the cluster are distributed among three zones, each zone with two hosts. In z1, two TiKV instances are deployed per host. In z2 and z3, one TiKV instance is deployed per host. ``` [tikv_servers] @@ -132,4 +132,4 @@ For example, a TiKV cluster is distributed across three data zones z1, z2, and z Similarly, when `isolation-level` is set to `rack`, the minimum isolation level is different racks in the same data center. Under this setting, if isolation can be guaranteed at the zone level, it will be guaranteed first. Only when the zone level isolation can not be completed, will the scheduling in the same zone and the same rack be considered to avoid. -In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. Also, TiDB provides mandatory isolation level settings such as `isolation-level` to control the topological isolation level according to the scenario more flexibly. +In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. TiDB also provides mandatory configuration items such as `isolation-level` for you to more flexibly control the topological isolation level of data according to different scenarios. From b2df8999f7a5ca71089fd2bc622774318089ee1e Mon Sep 17 00:00:00 2001 From: ireneontheway <48651140+ireneontheway@users.noreply.github.com> Date: Mon, 14 Sep 2020 14:51:19 +0800 Subject: [PATCH 13/17] Update schedule-replicas-by-topology-labels.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- schedule-replicas-by-topology-labels.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index 0c87e0aa66274..7a0f083bc4947 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -130,6 +130,6 @@ In the case of the 5-replica configuration, if z3 fails or is isolated as a whol For example, a TiKV cluster is distributed across three data zones z1, z2, and z3. Each Region has three replicas as required, and PD distributes the three replicas of the same Region to these three data zones respectively. If a power outage occurs in z1 and cannot be recovered after a period of time, PD determines that the Region replicas on z1 are no longer available. However, because `isolation-level` is set to `zone`, PD needs to strictly guarantee that different replicas of the same Region will not be scheduled on the same data zone. Because both z2 and z3 already have replicas, PD will not perform any scheduling under the minimum isolation level restriction of `isolation-level`, even if there are only two replicas at this moment. -Similarly, when `isolation-level` is set to `rack`, the minimum isolation level is different racks in the same data center. Under this setting, if isolation can be guaranteed at the zone level, it will be guaranteed first. Only when the zone level isolation can not be completed, will the scheduling in the same zone and the same rack be considered to avoid. +Similarly, when `isolation-level` is set to `rack`, the minimum isolation level applies to different racks in the same data center. With this configuration, the isolation at the zone layer is guaranteed first if possible. When the isolation at the zone level cannot be guaranteed, PD tries to avoid scheduling different replicas to the same rack in the same zone. The scheduling works similarly when `isolation-level` is set to `host` where PD first guarantees the isolation level of rack, and then the level of host. In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`. TiDB also provides mandatory configuration items such as `isolation-level` for you to more flexibly control the topological isolation level of data according to different scenarios. From e03873a654c6a8513c92a026f16d46b868ad0d1e Mon Sep 17 00:00:00 2001 From: ireneontheway <48651140+ireneontheway@users.noreply.github.com> Date: Mon, 14 Sep 2020 15:29:19 +0800 Subject: [PATCH 14/17] Update schedule-replicas-by-topology-labels.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- schedule-replicas-by-topology-labels.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index 7a0f083bc4947..5f87d35d4693d 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -64,7 +64,7 @@ The `location-labels` configuration is an array of strings, and each item corres ### Configure `isolation-level` for PD -If `location-labels` has been configured, you can further enhance the topological isolation requirements on TiKV clusters by configuring `isolation-level` in the PD configuration file: +If `location-labels` has been configured, you can further enhance the topological isolation requirements on TiKV clusters by configuring `isolation-level` in the PD configuration file. Assume that you have made a three-layer cluster topology by configuring `location-labels` according to the instructions above: zone -> rack -> host, and have configured the `isolation-level` as follows: From 89c967b3517383b7dbbcd94134139c3dbe9cb60b Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 14 Sep 2020 15:36:44 +0800 Subject: [PATCH 15/17] Update schedule-replicas-by-topology-labels.md --- schedule-replicas-by-topology-labels.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index 5f87d35d4693d..52e85a103dc24 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -66,7 +66,7 @@ The `location-labels` configuration is an array of strings, and each item corres If `location-labels` has been configured, you can further enhance the topological isolation requirements on TiKV clusters by configuring `isolation-level` in the PD configuration file. -Assume that you have made a three-layer cluster topology by configuring `location-labels` according to the instructions above: zone -> rack -> host, and have configured the `isolation-level` as follows: +Assume that you have made a three-layer cluster topology by configuring `location-labels` according to the instructions above: zone -> rack -> host, you can configure the `isolation-level` to `zone` as follows: {{< copyable "" >}} From 0233719a98fbcf9479626a3ef58792bfcf8aef47 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 16 Sep 2020 15:54:37 +0800 Subject: [PATCH 16/17] Update schedule-replicas-by-topology-labels.md --- schedule-replicas-by-topology-labels.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index 52e85a103dc24..321eff95ff62d 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -14,11 +14,11 @@ To make this mechanism effective, you need to properly configure TiKV and PD so ### Configure `labels` for TiKV -You can use the command-line flag or set the TiKV configuration file to bind some attributes in the form of key-value pairs. These attributes are called `labels`. TiKV reports its `labels` to PD so users can identify the location of TiKV nodes. +You can use the command-line flag or set the TiKV configuration file to bind some attributes in the form of key-value pairs. These attributes are called `labels`. After TiKV is started, it reports its `labels` to PD so users can identify the location of TiKV nodes. -Assume that the topology has three layers: zone > rack > host, and you can use these labels (zone, rack, host) to set the TiKV location. +Assume that the topology has three layers: zone > rack > host, and you can use these labels (zone, rack, host) to set the TiKV location in one of the following methods: -+ If the TiKV cluster is not started, use the command-line flag: ++ Use the command-line flag: {{< copyable "" >}} @@ -26,7 +26,7 @@ Assume that the topology has three layers: zone > rack > host, and you can use t tikv-server --labels zone=,rack=,host= ``` -+ If the TiKV cluster is started, configure in the TiKV configuration file: ++ Configure in the TiKV configuration file: {{< copyable "" >}} @@ -39,7 +39,7 @@ Assume that the topology has three layers: zone > rack > host, and you can use t According to the description above, the label can be any key-value pair used to describe TiKV attributes. But PD cannot identify the location-related labels and the layer relationship of these labels. Therefore, you need to make the following configuration for PD to understand the TiKV node topology. -+ If the PD cluster is not started, configure `location-labels` in the PD configuration file: ++ If the PD cluster is not initialized, configure `location-labels` in the PD configuration file: {{< copyable "" >}} From dbd532133b09dbae13883a698ef08a13a73906ff Mon Sep 17 00:00:00 2001 From: ireneontheway Date: Thu, 17 Sep 2020 11:22:50 +0800 Subject: [PATCH 17/17] Update schedule-replicas-by-topology-labels.md --- schedule-replicas-by-topology-labels.md | 59 +++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index 321eff95ff62d..546bab38743aa 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -89,6 +89,65 @@ The `location-level` configuration is an array of strings, which needs to corres > > `isolation-level` is empty by default, which means there is no mandatory restriction on the isolation level. To set it, you need to configure `location-labels` for PD and ensure that the value of `isolation-level` is one of `location-labels` names. +### Configure a cluster using TiUP (recommended) + +When using TiUP to deploy a cluster, you can configure the TiKV location in the [initialization configuration file](/production-deployment-using-tiup.md#step-3-edit-the-initialization-configuration-file). TiUP will generate the corresponding TiKV and PD configuration files during deployment. + +In the following example, a two-layer topology of `zone/host` is defined. The TiKV nodes of the cluster are distributed among three zones, each zone with two hosts. In z1, two TiKV instances are deployed per host. In z2 and z3, one TiKV instance is deployed per host. In the following example, `tikv-n` represents the IP address of the `n`th TiKV node. + +``` +server_configs: + pd: + replication.location-labels: ["zone", "host"] + +tikv_servers: +# z1 + - host: tikv-1 + config: + server.labels: + zone: z1 + host: h1 + - host: tikv-2 + config: + server.labels: + zone: z1 + host: h1 + - host: tikv-3 + config: + server.labels: + zone: z1 + host: h2 + - host: tikv-4 + config: + server.labels: + zone: z1 + host: h2 +# z2 + - host: tikv-5 + config: + server.labels: + zone: z2 + host: h1 + - host: tikv-6 + config: + server.labels: + zone: z2 + host: h2 +# z3 + - host: tikv-7 + config: + server.labels: + zone: z3 + host: h1 + - host: tikv-8 + config: + server.labels: + zone: z3 + host: h2s +``` + +For details, see [Geo-distributed Deployment topology](/geo-distributed-deployment-topology.md). +
Configure a cluster using TiDB Ansible