From 59102c90b07fdd23a9be79327e9d8e85ae78b5b2 Mon Sep 17 00:00:00 2001 From: Ran Date: Tue, 7 Apr 2020 15:40:03 +0800 Subject: [PATCH 1/5] tiflash: add scale.md doc --- TOC.md | 1 + reference/tiflash/scale.md | 128 +++++++++++++++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 reference/tiflash/scale.md diff --git a/TOC.md b/TOC.md index 14682038d2c02..2eba1c283ea56 100644 --- a/TOC.md +++ b/TOC.md @@ -305,6 +305,7 @@ - [Overview](/reference/tiflash/overview.md) - [Deploy a TiFlash Cluster](/reference/tiflash/deploy.md) - [Use TiFlash](/reference/tiflash/use-tiflash.md) + - [Scale](/reference/tiflash/scale.md) + TiDB Binlog - [Overview](/reference/tidb-binlog/overview.md) - [Deploy](/reference/tidb-binlog/deploy.md) diff --git a/reference/tiflash/scale.md b/reference/tiflash/scale.md new file mode 100644 index 0000000000000..31afdd8d9aabd --- /dev/null +++ b/reference/tiflash/scale.md @@ -0,0 +1,128 @@ +--- +title: Scale the TiFlash Cluster +summary: Learn how to scale in and out nodes in the TiFlash cluster. +category: reference +--- + +# Scale the TiFlash Cluster + +This document describes how to scale in and out nodes in the TiFlash cluster. + +## Scale out a TiFlash node + +The following example shows how to scale out a TiFlash node if you deploy TiFlash on the `192.168.1.1` node. + +1. Edit the `inventory.ini` file, and add the TiFlash node information: + + Currently, this feature only supports IP but not domain name. + + {{< copyable "" >}} + + ```ini + [tiflash_servers] + 192.168.1.1 + ``` + +2. Edit the `hosts.ini` file, and add the node information: + + {{< copyable "" >}} + + ```ini + [servers] + 192.168.1.1 + [all:vars] + username = tidb + ntp_server = pool.ntp.org + ``` + +3. Initialize the new node: + + - Configure the SSH mutual trust and sudo rules on the Control Machine: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook -i hosts.ini create_users.yml -l 192.168.1.1 -u root -k + ``` + + - Install the NTP service on the target machine: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook -i hosts.ini deploy_ntp.yml -u tidb -b + ``` + + - Initialize the node on the target machine: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook bootstrap.yml -l 192.168.1.1 + ``` + +4. Deploy the new node: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook deploy.yml -l 192.168.1.1 + ``` + +5. Start the new node: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook start.yml -l 192.168.1.1 + ``` + +6. Update the configuration of Prometheus and restart: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook rolling_update_monitor.yml --tags=prometheus + ``` + +7. Access the monitoring platform using your browser, and view the status of the cluster and the new node. + +## Scale in a TiFlash node + +The following example shows how to scale in a TiFlash node if you deploy TiFlash on the `192.168.1.1` node. + +> **Note:** +> +> The offline process described in this section does not delete the data on the offline node. If you need to take the node online again, delete the data manually. + +1. Take the node to be scaled in offline. See [Take TiFlash nodes offline](/reference/tiflash/maintain.md#take-tiflash-nodes-offline) for details. + +2. Check whether the node has been offline successfully using Grafana or pd-ctl (the offline process takes some time). + +3. After the `store` corresponding to TiFlash disappears, or the `state_name` becomes `Tomestone`, execute the following command to shutdown the TiFlash process: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook stop.yml -l 192.168.1.1 + ``` + + If the node still has other services and you want to stop TiFlash only, use the following command: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook stop.yml -t tiflash -l 192.168.1.1 + ``` + +4. Edit the `inventory.ini` and `hosts.ini` files to remove the node information. + +5. Update the configuration of Prometheus and restart: + + {{< copyable "shell-regular" >}} + + ```shell + ansible-playbook rolling_update_monitor.yml --tags=prometheus + ``` + +6. Access the monitoring platform using your browser, and view the status of the cluster. \ No newline at end of file From 10ba29b17e623c00ba1b59bcd81d0b5a36d79a10 Mon Sep 17 00:00:00 2001 From: Ran Date: Wed, 8 Apr 2020 13:38:34 +0800 Subject: [PATCH 2/5] Apply suggestions from code review Co-Authored-By: TomShawn <41534398+TomShawn@users.noreply.github.com> --- reference/tiflash/scale.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/tiflash/scale.md b/reference/tiflash/scale.md index 31afdd8d9aabd..a564994bb8862 100644 --- a/reference/tiflash/scale.md +++ b/reference/tiflash/scale.md @@ -12,9 +12,9 @@ This document describes how to scale in and out nodes in the TiFlash cluster. The following example shows how to scale out a TiFlash node if you deploy TiFlash on the `192.168.1.1` node. -1. Edit the `inventory.ini` file, and add the TiFlash node information: +1. Edit the `inventory.ini` file to add the TiFlash node information: - Currently, this feature only supports IP but not domain name. + Currently, you can only add IP but not domain name. {{< copyable "" >}} @@ -23,7 +23,7 @@ The following example shows how to scale out a TiFlash node if you deploy TiFlas 192.168.1.1 ``` -2. Edit the `hosts.ini` file, and add the node information: +2. Edit the `hosts.ini` file to add the node information: {{< copyable "" >}} @@ -89,7 +89,7 @@ The following example shows how to scale out a TiFlash node if you deploy TiFlas ## Scale in a TiFlash node -The following example shows how to scale in a TiFlash node if you deploy TiFlash on the `192.168.1.1` node. +The following example shows how to scale in a TiFlash node if you stop the TiFlash service on the `192.168.1.1` node. > **Note:** > @@ -125,4 +125,4 @@ The following example shows how to scale in a TiFlash node if you deploy TiFlash ansible-playbook rolling_update_monitor.yml --tags=prometheus ``` -6. Access the monitoring platform using your browser, and view the status of the cluster. \ No newline at end of file +6. Access the monitoring platform using your browser, and view the status of the cluster. From 033ca0dfa9775b7c480be3c7323c8d7cd7fc3a48 Mon Sep 17 00:00:00 2001 From: Ran Date: Wed, 8 Apr 2020 14:27:23 +0800 Subject: [PATCH 3/5] Update TOC.md Co-Authored-By: TomShawn <41534398+TomShawn@users.noreply.github.com> --- TOC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TOC.md b/TOC.md index f720078ad9d83..6af0044b0968c 100644 --- a/TOC.md +++ b/TOC.md @@ -308,7 +308,7 @@ - [Deploy a TiFlash Cluster](/reference/tiflash/deploy.md) - [Use TiFlash](/reference/tiflash/use-tiflash.md) - [Monitor TiFlash](/reference/tiflash/monitor.md) - - [Scale](/reference/tiflash/scale.md) + - [Scale TiFlash](/reference/tiflash/scale.md) - [Configure TiFlash](/reference/tiflash/configuration.md) + TiDB Binlog - [Overview](/reference/tidb-binlog/overview.md) From b246f19bdd20382736866ac998af68a899b97a7a Mon Sep 17 00:00:00 2001 From: Ran Date: Wed, 8 Apr 2020 14:49:31 +0800 Subject: [PATCH 4/5] minor update --- reference/tiflash/scale.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/tiflash/scale.md b/reference/tiflash/scale.md index a564994bb8862..d1d9287a47602 100644 --- a/reference/tiflash/scale.md +++ b/reference/tiflash/scale.md @@ -77,7 +77,7 @@ The following example shows how to scale out a TiFlash node if you deploy TiFlas ansible-playbook start.yml -l 192.168.1.1 ``` -6. Update the configuration of Prometheus and restart: +6. Update the configuration of Prometheus and restart it: {{< copyable "shell-regular" >}} @@ -117,7 +117,7 @@ The following example shows how to scale in a TiFlash node if you stop the TiFla 4. Edit the `inventory.ini` and `hosts.ini` files to remove the node information. -5. Update the configuration of Prometheus and restart: +5. Update the configuration of Prometheus and restart it: {{< copyable "shell-regular" >}} From 5ed50ceb5bdfac611ff57896a3e71aff1ac1310b Mon Sep 17 00:00:00 2001 From: Ran Date: Wed, 8 Apr 2020 19:34:33 +0800 Subject: [PATCH 5/5] update anchor --- reference/tiflash/scale.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/tiflash/scale.md b/reference/tiflash/scale.md index d1d9287a47602..a3044c65e400f 100644 --- a/reference/tiflash/scale.md +++ b/reference/tiflash/scale.md @@ -95,7 +95,7 @@ The following example shows how to scale in a TiFlash node if you stop the TiFla > > The offline process described in this section does not delete the data on the offline node. If you need to take the node online again, delete the data manually. -1. Take the node to be scaled in offline. See [Take TiFlash nodes offline](/reference/tiflash/maintain.md#take-tiflash-nodes-offline) for details. +1. Take down the node to be scaled in. See [Take a TiFlash node down](/reference/tiflash/maintain.md#take-a-tiflash-node-down) for details. 2. Check whether the node has been offline successfully using Grafana or pd-ctl (the offline process takes some time).