diff --git a/TOC.md b/TOC.md index 49009599d880a..0e9d70fd0d942 100644 --- a/TOC.md +++ b/TOC.md @@ -58,7 +58,7 @@ - [Docker Deployment](/test-deployment-using-docker.md) + Geographic Redundancy - [Overview](/geo-redundancy-deployment.md) - - [Configure Location Awareness](/location-awareness.md) + - [Schedule Replicas by Topology Labels](/schedule-replicas-by-topology-labels.md) - [Data Migration with Ansible](https://pingcap.com/docs/tidb-data-migration/stable/deploy-a-dm-cluster-using-ansible/) + Configure - [Time Zone](/configure-time-zone.md) diff --git a/location-awareness.md b/location-awareness.md deleted file mode 100644 index c73d4837fb0fd..0000000000000 --- a/location-awareness.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -title: Cluster Topology Configuration -summary: Learn to configure cluster topology to maximize the capacity for disaster recovery. -aliases: ['/docs/v3.1/location-awareness/','/docs/v3.1/how-to/deploy/geographic-redundancy/location-awareness/'] ---- - -# Cluster Topology Configuration - -## Overview - -PD schedules according to the topology of the TiKV cluster to maximize the TiKV's capability for disaster recovery. - -Before you begin, see [Deploy TiDB Using TiDB Ansible (Recommended)](/online-deployment-using-ansible.md) and [Deploy TiDB Using Docker](/test-deployment-using-docker.md). - -## TiKV reports the topological information - -TiKV reports the topological information to PD according to the startup parameter or configuration of TiKV. - -Assuming that the topology has three structures: zone > rack > host, use lables to specify the following information: - -Startup parameter: - -``` -tikv-server --labels zone=,rack=,host= -``` - -Configuration: - -``` toml -[server] -labels = "zone=,rack=,host=" -``` - -## PD understands the TiKV topology - -PD gets the topology of TiKV cluster through the PD configuration. - -``` toml -[replication] -max-replicas = 3 -location-labels = ["zone", "rack", "host"] -``` - -`location-labels` needs to correspond to the TiKV `labels` name so that PD can understand that the `labels` represents the TiKV topology. - -> **Note:** -> -> You must configure `location-labels` for PD and `labels` for TiKV at the same time for `labels` to take effect. - -## PD schedules based on the TiKV topology - -PD makes optimal scheduling according to the topological information. You just need to care about what kind of topology can achieve the desired effect. - -If you use 3 replicas and hope that the TiDB cluster is always highly available even when a data zone goes down, you need at least 4 data zones. - -Assume that you have 4 data zones, each zone has 2 racks, and each rack has 2 hosts. You can start 2 TiKV instances on each host: - -``` -# zone=z1 -tikv-server --labels zone=z1,rack=r1,host=h1 -tikv-server --labels zone=z1,rack=r1,host=h2 -tikv-server --labels zone=z1,rack=r2,host=h1 -tikv-server --labels zone=z1,rack=r2,host=h2 - -# zone=z2 -tikv-server --labels zone=z2,rack=r1,host=h1 -tikv-server --labels zone=z2,rack=r1,host=h2 -tikv-server --labels zone=z2,rack=r2,host=h1 -tikv-server --labels zone=z2,rack=r2,host=h2 - -# zone=z3 -tikv-server --labels zone=z3,rack=r1,host=h1 -tikv-server --labels zone=z3,rack=r1,host=h2 -tikv-server --labels zone=z3,rack=r2,host=h1 -tikv-server --labels zone=z3,rack=r2,host=h2 - -# zone=z4 -tikv-server --labels zone=z4,rack=r1,host=h1 -tikv-server --labels zone=z4,rack=r1,host=h2 -tikv-server --labels zone=z4,rack=r2,host=h1 -tikv-server --labels zone=z4,rack=r2,host=h2 -``` - -In other words, 16 TiKV instances are distributed across 4 data zones, 8 racks and 16 machines. - -In this case, PD will schedule different replicas of each datum to different data zones. - -- If one of the data zones goes down, the high availability of the TiDB cluster is not affected. -- If the data zone cannot recover within a period of time, PD will remove the replica from this data zone. - -To sum up, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to reach a certain level of disaster recovery, deploy many machines in different sites according to the topology. The number of machines must be more than the number of `max-replicas`. diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md new file mode 100644 index 0000000000000..646f7c6bf6bd9 --- /dev/null +++ b/schedule-replicas-by-topology-labels.md @@ -0,0 +1,101 @@ +--- +title: Schedule Replicas by Topology Labels +summary: Learn how to schedule replicas by topology labels. +aliases: ['/docs/v3.1/location-awareness/','/docs/v3.1/how-to/deploy/geographic-redundancy/location-awareness/','/tidb/v3.1/location-awareness'] +--- + +# Schedule Replicas by Topology Labels + +To improve the high availability and disaster recovery capability of TiDB clusters, it is recommended that TiKV nodes are physically scattered as much as possible. For example, TiKV nodes can be distributed on different racks or even in different data centers. According to the topology information of TiKV, the PD scheduler automatically performs scheduling at the background to isolate each replica of a Region as much as possible, which maximizes the capability of disaster recovery. + +To make this mechanism effective, you need to properly configure TiKV and PD so that the topology information of the cluster, especially the TiKV location information, is reported to PD during deployment. Before you begin, see [Deploy TiDB Using TiDB Ansible](/online-deployment-using-ansible.md) first. + +## Configure `labels` based on the cluster topology + +### Configure `labels` for TiKV + +You can use the command-line flag or set the TiKV configuration file to bind some attributes in the form of key-value pairs. These attributes are called `labels`. After TiKV is started, it reports its `labels` to PD so users can identify the location of TiKV nodes. + +Assume that the topology has three layers: zone > rack > host, and you can use these labels (zone, rack, host) to set the TiKV location in one of the following methods: + ++ Use the command-line flag: + + {{< copyable "" >}} + + ``` + tikv-server --labels zone=,rack=,host= + ``` + ++ Configure in the TiKV configuration file: + + {{< copyable "" >}} + + ```toml + [server] + labels = "zone=,rack=,host=" + ``` + +### Configure `location-labels` for PD + +According to the description above, the label can be any key-value pair used to describe TiKV attributes. But PD cannot identify the location-related labels and the layer relationship of these labels. Therefore, you need to make the following configuration for PD to understand the TiKV node topology. + ++ If the PD cluster is not initialized, configure `location-labels` in the PD configuration file: + + {{< copyable "" >}} + + ```toml + [replication] + location-labels = ["zone", "rack", "host"] + ``` + ++ If the PD cluster is already initialized, use the pd-ctl tool to make online changes: + + {{< copyable "shell-regular" >}} + + ```bash + pd-ctl config set location-labels zone,rack,host + ``` + +The `location-labels` configuration is an array of strings, and each item corresponds to the key of TiKV `labels`. The sequence of each key represents the layer relationship of different labels. + +> **Note:** +> +> You must configure `location-labels` for PD and `labels` for TiKV at the same time for the configurations to take effect. Otherwise, PD does not perform scheduling according to the topology. + +### Configure a cluster using TiDB Ansible + +When using TiDB Ansible to deploy a cluster, you can directly configure the TiKV location in the `inventory.ini` file. `tidb-ansible` will generate the corresponding TiKV and PD configuration files during deployment. + +In the following example, a two-layer topology of `zone/host` is defined. The TiKV nodes of the cluster are distributed among three zones, each zone with two hosts. In z1, two TiKV instances are deployed per host. In z2 and z3, one TiKV instance is deployed per host. + +``` +[tikv_servers] +# z1 +tikv-1 labels="zone=z1,host=h1" +tikv-2 labels="zone=z1,host=h1" +tikv-3 labels="zone=z1,host=h2" +tikv-4 labels="zone=z1,host=h2" +# z2 +tikv-5 labels="zone=z2,host=h1" +tikv-6 labels="zone=z2,host=h2" +# z3 +tikv-7 labels="zone=z3,host=h1" +tikv-8 labels="zone=z3,host=h2" + +[pd_servers:vars] +location_labels = ["zone", "host"] +``` + +## PD schedules based on topology label + +PD schedules replicas according to the label layer to make sure that different replicas of the same data are scattered as much as possible. + +Take the topology in the previous section as an example. + +Assume that the number of cluster replicas is 3 (`max-replicas=3`). Because there are 3 zones in total, PD ensures that the 3 replicas of each Region are respectively placed in z1, z2, and z3. In this way, the TiDB cluster is still available when one data center fails. + +Then, assume that the number of cluster replicas is 5 (`max-replicas=5`). Because there are only 3 zones in total, PD cannot guarantee the isolation of each replica at the zone level. In this situation, the PD scheduler will ensure replica isolation at the host level. In other words, multiple replicas of a Region might be distributed in the same zone but not on the same host. + +In the case of the 5-replica configuration, if z3 fails or is isolated as a whole, and cannot be recovered after a period of time (controlled by `max-store-down-time`), PD will make up the 5 replicas through scheduling. At this time, only 3 hosts are available. This means that host-level isolation cannot be guaranteed and that multiple replicas might be scheduled to the same host. + +In summary, PD maximizes the disaster recovery of the cluster according to the current topology. Therefore, if you want to achieve a certain level of disaster recovery, deploy more machines on different sites according to the topology than the number of `max-replicas`.