From b46128216d559f88117f02db5927e34778d90ca3 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 23 Oct 2025 14:16:24 +0800 Subject: [PATCH 01/17] Add temp.md --- temp.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 temp.md diff --git a/temp.md b/temp.md new file mode 100644 index 0000000000000..af27ff4986a7b --- /dev/null +++ b/temp.md @@ -0,0 +1 @@ +This is a test file. \ No newline at end of file From e4e8bc4430bb9e9d0d3f8e8936f412c1c799e2a4 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 23 Oct 2025 14:16:28 +0800 Subject: [PATCH 02/17] Delete temp.md --- temp.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 temp.md diff --git a/temp.md b/temp.md deleted file mode 100644 index af27ff4986a7b..0000000000000 --- a/temp.md +++ /dev/null @@ -1 +0,0 @@ -This is a test file. \ No newline at end of file From 5ecad228701d77e30ded5d472b593d935d5b88f9 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 23 Oct 2025 09:32:06 +0000 Subject: [PATCH 03/17] Auto-sync: Update English docs from Chinese PR https://github.com/pingcap/docs-cn/pull/20703 Synced from: https://github.com/pingcap/docs-cn/pull/20703 Target PR: https://github.com/pingcap/docs/pull/21962 AI Provider: gemini Co-authored-by: github-actions[bot] --- follower-read.md | 87 +++++++++++++++++++++++++-------------- grafana-tidb-dashboard.md | 7 +++- system-variables.md | 2 +- 3 files changed, 64 insertions(+), 32 deletions(-) diff --git a/follower-read.md b/follower-read.md index 76156e996f61a..2b3fd861a8b39 100644 --- a/follower-read.md +++ b/follower-read.md @@ -5,69 +5,96 @@ summary: This document describes the use and implementation of Follower Read. # Follower Read -When a read hotspot appears in a Region, the Region leader can become a read bottleneck for the entire system. In this situation, enabling the Follower Read feature can significantly reduce the load of the leader, and improve the throughput of the whole system by balancing the load among multiple followers. This document introduces the use and implementation mechanism of Follower Read. +In TiDB, to ensure high availability and data security, TiKV stores multiple replicas for each Region, one of which is the leader and the others are followers. By default, all read and write requests are processed by the leader. The Follower Read feature allows reading data from the follower replicas of a Region while maintaining strong consistency, thereby distributing the read pressure on the leader and improving the overall read throughput of the cluster. -## Overview +When performing Follower Read, TiDB selects the appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to determine whether a replica is a local replica: when the `zone` label of TiDB is the same as that of the target TiKV, TiDB considers the replica as a local replica. For more information, see [Schedule replicas by topology labels](schedule-replicas-by-topology-labels.md). -The Follower Read feature refers to using any follower replica of a Region to serve a read request under the premise of strongly consistent reads. This feature improves the throughput of the TiDB cluster and reduces the load of the leader. It contains a series of load balancing mechanisms that offload TiKV read loads from the leader replica to the follower replica in a Region. TiKV's Follower Read implementation provides users with strongly consistent reads. +By allowing followers to participate in data reading, Follower Read can achieve the following goals: + +- Distribute read hotspots and reduce leader load. +- In multi-AZ or multi-datacenter deployments, prioritize reading local replicas to reduce cross-region traffic. + +## Applicable scenarios + +Follower Read is suitable for the following scenarios: + +- Businesses with large read request volume and obvious read hotspots. +- Scenarios where it is desirable to prioritize reading local replicas in multi-AZ deployments to save bandwidth. +- In a read-write separation architecture, it is desirable to further improve the overall read performance of the cluster. > **Note:** > -> To achieve strongly consistent reads, the follower node currently needs to request the current execution progress from the leader node (that is `ReadIndex`), which causes an additional network request overhead. Therefore, the main benefits of Follower Read are to isolate read requests from write requests in the cluster and to increase overall read throughput. +> To ensure the strong consistency of the read results, Follower Read needs to communicate with the leader to confirm the current commit progress (that is, execute the Raft `ReadIndex` operation) before reading, which introduces an additional network interaction. Therefore, Follower Read is most effective when there are a large number of read requests or read-write isolation is required; however, the performance improvement may not be significant for low-latency single queries. ## Usage -To enable TiDB's Follower Read feature, modify the value of the `tidb_replica_read` variable as follows: +To enable TiDB's Follower Read feature, set the value of the `tidb_replica_read` variable to the desired value: {{< copyable "sql" >}} ```sql -set [session | global] tidb_replica_read = ''; +set [session | global] tidb_replica_read = '<目标值>'; ``` Scope: SESSION | GLOBAL Default: leader -This variable is used to set the expected data read mode. +This variable is used to set the expected data read mode. From v8.5.4, this variable only takes effect on read-only SQL statements. -- When you set the value of `tidb_replica_read` to `leader` or an empty string, TiDB maintains its default behavior and sends all read operations to the leader replica to perform. -- When you set the value of `tidb_replica_read` to `follower`, TiDB selects a follower replica of the Region to perform read operations. If the Region has learner replicas, TiDB also considers them for reads with the same priority. If no available follower or learner replicas exist for the current Region, TiDB reads from the leader replica. -- When the value of `tidb_replica_read` is set to `leader-and-follower`, TiDB can select any replicas to perform read operations. In this mode, read requests are load balanced between the leader and follower. -- When the value of `tidb_replica_read` is set to `prefer-leader`, TiDB prefers to select the leader replica to perform read operations. If the leader replica is obviously slow in processing read operations (such as caused by disk or network performance jitter), TiDB will select other available follower replicas to perform read operations. -- When the value of `tidb_replica_read` is set to `closest-replicas`, TiDB prefers to select a replica in the same availability zone to perform read operations, which can be a leader or a follower. If there is no replica in the same availability zone, TiDB reads from the leader replica. -- When the value of `tidb_replica_read` is set to `closest-adaptive`: +In scenarios where you need to save cross-region traffic by reading local replicas, the following configurations are recommended: - - If the estimated result of a read request is greater than or equal to the value of [`tidb_adaptive_closest_read_threshold`](/system-variables.md#tidb_adaptive_closest_read_threshold-new-in-v630), TiDB prefers to select a replica in the same availability zone for read operations. To avoid unbalanced distribution of read traffic across availability zones, TiDB dynamically detects the distribution of availability zones for all online TiDB and TiKV nodes. In each availability zone, the number of TiDB nodes whose `closest-adaptive` configuration takes effect is limited, which is always the same as the number of TiDB nodes in the availability zone with the fewest TiDB nodes, and the other TiDB nodes automatically read from the leader replica. For example, if TiDB nodes are distributed across 3 availability zones (A, B, and C), where A and B each contains 3 TiDB nodes and C contains only 2 TiDB nodes, the number of TiDB nodes whose `closest-adaptive` configuration takes effect in each availability zone is 2, and the other TiDB node in each of the A and B availability zones automatically selects the leader replica for read operations. - - If the estimated result of a read request is less than the value of [`tidb_adaptive_closest_read_threshold`](/system-variables.md#tidb_adaptive_closest_read_threshold-new-in-v630), TiDB can only select the leader replica for read operations. +- The default value `leader` provides the best performance. +- `closest-adaptive` saves traffic as much as possible with minimal performance loss. +- `closest-replicas` can save network traffic to the greatest extent. -- When you set the value of `tidb_replica_read` to `learner`, TiDB reads data from the learner replica. If no learner replica is available for the current Region, TiDB reads from an available leader or follower replica. +If you are currently using other configurations, refer to the following table to modify them to the recommended configurations: - +| Configuration being used | Recommended configuration to modify to | +| ------------- | ------------- | +| `follower` | `closest-replicas` | +| `leader-and-follower` | `closest-replicas` | +| `prefer-leader` | `closest-adaptive` | +| `learner` | `closest-replicas` | -> **Note:** -> -> When you set `tidb_replica_read` to `closest-replicas` or `closest-adaptive`, to ensure that replicas are distributed across availability zones according to the specified configuration, you need to configure `location-labels` for PD and set the correct `labels` for TiDB and TiKV according to [Schedule replicas by topology labels](/schedule-replicas-by-topology-labels.md). TiDB depends on the `zone` label to match TiKV nodes in the same availability zone, so you need to make sure that the `zone` label is included in the `location-labels` of PD and `zone` is included in the configuration of each TiDB and TiKV node. If your cluster is deployed using TiDB Operator, refer to [High availability of data](https://docs.pingcap.com/tidb-in-kubernetes/stable/configure-a-tidb-cluster#high-availability-of-data). -> -> For TiDB v7.5.0 and earlier versions: -> -> - If you set `tidb_replica_read` to `follower` and no follower or learner replicas are available, TiDB returns an error. -> - If you set `tidb_replica_read` to `learner` and no learner replicas are available, TiDB returns an error. +If you want to use a more precise read replica selection policy, refer to the complete list of optional configurations: + +## Basic monitoring - +By observing the [**TiDB** > **KV Request** > **Read Req Traffic** panel (new in v8.5.4)](/grafana-tidb-dashboard.md#kv-request), you can determine whether you need to use Follower Read and view the effect of saving traffic after enabling Follower Read. ## Implementation mechanism Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers can do nothing to handle read requests but receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover. -To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy. +Follower Read includes a series of load balancing mechanisms that offload TiKV read load from Region leader replicas to follower replicas. To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader node. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy. ### Strongly consistent reads -When the follower node processes a read request, it first uses `ReadIndex` of the Raft protocol to interact with the leader of the Region, to obtain the latest commit index of the current Raft group. After the latest commit index of the leader is applied locally to the follower, the processing of a read request starts. +When the follower node processes a read request, it first uses `ReadIndex` of the Raft protocol to interact with the leader node of the Region to obtain the latest commit index (read index) of the current Raft group. After the latest commit index of the leader node is applied locally to the follower, the processing of a read request starts. + +![read-index-flow](/media/follower-read/read-index.png) ### Follower replica selection strategy -Because the Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level, TiDB adopts the round-robin strategy to select the follower replica. Currently, for the coprocessor requests, the granularity of the Follower Read load balancing policy is at the connection level. For a TiDB client connected to a specific Region, the selected follower is fixed, and is switched only when it fails or the scheduling policy is adjusted. +The Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level. TiDB selects a replica for the first time based on the configuration of `tidb_replica_read`. From the second retry, TiDB will prioritize ensuring successful reading. Therefore, when the selected follower node has an inaccessible fault or other errors, it will switch to the leader for service. + +#### `leader` + +- Select the leader replica for reading, regardless of replica location. + +#### `closest-replicas` + +- When the replica in the same AZ as TiDB is the leader node, Follower Read is not used. +- When the replica in the same AZ as TiDB is not the leader node, Follower Read is used. + +#### `closest-adaptive` + +- If the estimated return result is not large enough, use the `leader` policy and do not perform Follower Read. +- If the estimated return result is large enough, use the `closest-replicas` policy. + +### Follower Read performance overhead + +To ensure strong data consistency, Follower Read needs to perform a `ReadIndex` regardless of how much data is read, which inevitably consumes more TiKV CPU resources. Therefore, in small query (such as point query) scenarios, the performance loss of Follower Read is relatively more obvious. At the same time, because the traffic saved by local reading for small queries is limited, it is more recommended to use Follower Read in large query or batch reading scenarios. -However, for the non-coprocessor requests, such as a point query, the granularity of the Follower Read load balancing policy is at the transaction level. For a TiDB transaction on a specific Region, the selected follower is fixed, and is switched only when it fails or the scheduling policy is adjusted. If a transaction contains both point queries and coprocessor requests, the two types of requests are scheduled for reading separately according to the preceding scheduling policy. In this case, even if a coprocessor request and a point query are for the same Region, TiDB processes them as independent events. +When `tidb_replica_read` is `closest-adaptive`, TiDB does not use Follower Read for small queries, so the extra overhead of TiKV CPU compared to the `leader` policy is generally within +10% in various workloads. diff --git a/grafana-tidb-dashboard.md b/grafana-tidb-dashboard.md index 73d0026926d3e..15f065cd32a2b 100644 --- a/grafana-tidb-dashboard.md +++ b/grafana-tidb-dashboard.md @@ -123,9 +123,14 @@ The following metrics relate to requests sent to TiKV. Retry requests are counte - **local**: the number of requests per second that attempt a stale read in the local zone - Stale Read Req Traffic: - **cross-zone-in**: the incoming traffic of responses to requests that attempt a stale read in a remote zone - - **cross-zone-out**: the outgoing traffic of requests that attempt a stale read in a remote zone + - **cross-zone-out**: the outgoing traffic of responses to requests that attempt a stale read in a remote zone - **local-in**: the incoming traffic of responses to requests that attempt a stale read in the local zone - **local-out**: the outgoing traffic of requests that attempt a stale read in the local zone +- Read Req Traffic + - **leader-local**: Traffic generated by Leader Read processing read requests in the local zone + - **leader-cross-zone**: Traffic generated by Leader Read processing read requests in the remote zone + - **follower-local**: Traffic generated by Follower Read processing read requests in the local zone + - **follower-cross-zone**: Traffic generated by Follower Read processing read requests in the remote zone ### PD Client diff --git a/system-variables.md b/system-variables.md index 6ad5c710e3ace..4ece152f463e2 100644 --- a/system-variables.md +++ b/system-variables.md @@ -5375,7 +5375,7 @@ SHOW WARNINGS; - Type: Enumeration - Default value: `leader` - Possible values: `leader`, `follower`, `leader-and-follower`, `prefer-leader`, `closest-replicas`, `closest-adaptive`, and `learner`. The `learner` value is introduced in v6.6.0. -- This variable is used to control where TiDB reads data. +- This variable is used to control where TiDB reads data. From v8.5.4, this variable only takes effect on read-only SQL statements. - For more details about usage and implementation, see [Follower read](/follower-read.md). ### tidb_restricted_read_only New in v5.2.0 From 4264bc8cca7ea6857ef0b553d174b8019e55651f Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 24 Oct 2025 10:53:25 +0800 Subject: [PATCH 04/17] Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- follower-read.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/follower-read.md b/follower-read.md index 2b3fd861a8b39..726c1a17a11ea 100644 --- a/follower-read.md +++ b/follower-read.md @@ -24,7 +24,7 @@ Follower Read is suitable for the following scenarios: > **Note:** > -> To ensure the strong consistency of the read results, Follower Read needs to communicate with the leader to confirm the current commit progress (that is, execute the Raft `ReadIndex` operation) before reading, which introduces an additional network interaction. Therefore, Follower Read is most effective when there are a large number of read requests or read-write isolation is required; however, the performance improvement may not be significant for low-latency single queries. +> To ensure the strong consistency of the read results, Follower Read needs to communicate with the leader to confirm the current commit progress (by executing the Raft `ReadIndex` operation) before reading, which introduces an additional network interaction. Therefore, Follower Read is most effective when there are a large number of read requests or read-write isolation is required; however, the performance improvement may not be significant for low-latency single queries. ## Usage @@ -33,7 +33,7 @@ To enable TiDB's Follower Read feature, set the value of the `tidb_replica_read` {{< copyable "sql" >}} ```sql -set [session | global] tidb_replica_read = '<目标值>'; +set [session | global] tidb_replica_read = ''; ``` Scope: SESSION | GLOBAL @@ -65,7 +65,7 @@ By observing the [**TiDB** > **KV Request** > **Read Req Traffic** panel (new in ## Implementation mechanism -Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers can do nothing to handle read requests but receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover. +Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers could not handle read requests, and they only receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover. Follower Read includes a series of load balancing mechanisms that offload TiKV read load from Region leader replicas to follower replicas. To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader node. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy. @@ -77,7 +77,7 @@ When the follower node processes a read request, it first uses `ReadIndex` of th ### Follower replica selection strategy -The Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level. TiDB selects a replica for the first time based on the configuration of `tidb_replica_read`. From the second retry, TiDB will prioritize ensuring successful reading. Therefore, when the selected follower node has an inaccessible fault or other errors, it will switch to the leader for service. +The Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level. TiDB selects a replica for the first time based on the configuration of `tidb_replica_read`. From the second retry, TiDB will prioritize ensuring successful reading. Therefore, when the selected follower node is inaccessible or has other errors, it will switch to the leader for service. #### `leader` @@ -85,8 +85,8 @@ The Follower Read feature does not affect TiDB's Snapshot Isolation transaction #### `closest-replicas` -- When the replica in the same AZ as TiDB is the leader node, Follower Read is not used. -- When the replica in the same AZ as TiDB is not the leader node, Follower Read is used. +- When the replica in the same AZ as TiDB is the leader node, TiDB does not use Follower Read for it. +- When the replica in the same AZ as TiDB is a follower node, TiDB uses Follower Read for it. #### `closest-adaptive` @@ -95,6 +95,5 @@ The Follower Read feature does not affect TiDB's Snapshot Isolation transaction ### Follower Read performance overhead -To ensure strong data consistency, Follower Read needs to perform a `ReadIndex` regardless of how much data is read, which inevitably consumes more TiKV CPU resources. Therefore, in small query (such as point query) scenarios, the performance loss of Follower Read is relatively more obvious. At the same time, because the traffic saved by local reading for small queries is limited, it is more recommended to use Follower Read in large query or batch reading scenarios. - +To ensure strong data consistency, Follower Read needs to perform a `ReadIndex` regardless of how much data is read, which inevitably consumes more TiKV CPU resources. Therefore, in small query scenarios (such as point queries), the performance loss of Follower Read is relatively more obvious. At the same time, because the traffic saved by local reading for small queries is limited, Follower Read is more recommended for large query or batch reading scenarios. When `tidb_replica_read` is `closest-adaptive`, TiDB does not use Follower Read for small queries, so the extra overhead of TiKV CPU compared to the `leader` policy is generally within +10% in various workloads. From f0125d8ebae9b14ca49d7c87631e9cca3c701254 Mon Sep 17 00:00:00 2001 From: qiancai Date: Fri, 24 Oct 2025 11:01:43 +0800 Subject: [PATCH 05/17] add the missing content --- follower-read.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/follower-read.md b/follower-read.md index 726c1a17a11ea..98fa14f131c5d 100644 --- a/follower-read.md +++ b/follower-read.md @@ -59,6 +59,31 @@ If you are currently using other configurations, refer to the following table to If you want to use a more precise read replica selection policy, refer to the complete list of optional configurations: +- When you set the value of `tidb_replica_read` to `leader` or an empty string, TiDB maintains its default behavior and sends all read operations to the leader replica to perform. +- When you set the value of `tidb_replica_read` to `follower`, TiDB selects a follower replica of the Region to perform read operations. If the Region has learner replicas, TiDB also considers them for reads with the same priority. If no available follower or learner replicas exist for the current Region, TiDB reads from the leader replica. +- When the value of `tidb_replica_read` is set to `leader-and-follower`, TiDB can select any replicas to perform read operations. In this mode, read requests are load balanced between the leader and follower. +- When the value of `tidb_replica_read` is set to `prefer-leader`, TiDB prefers to select the leader replica to perform read operations. If the leader replica is obviously slow in processing read operations (such as caused by disk or network performance jitter), TiDB will select other available follower replicas to perform read operations. +- When the value of `tidb_replica_read` is set to `closest-replicas`, TiDB prefers to select a replica in the same availability zone to perform read operations, which can be a leader or a follower. If there is no replica in the same availability zone, TiDB reads from the leader replica. +- When the value of `tidb_replica_read` is set to `closest-adaptive`: + + - If the estimated result of a read request is greater than or equal to the value of [`tidb_adaptive_closest_read_threshold`](/system-variables.md#tidb_adaptive_closest_read_threshold-new-in-v630), TiDB prefers to select a replica in the same availability zone for read operations. To avoid unbalanced distribution of read traffic across availability zones, TiDB dynamically detects the distribution of availability zones for all online TiDB and TiKV nodes. In each availability zone, the number of TiDB nodes whose `closest-adaptive` configuration takes effect is limited, which is always the same as the number of TiDB nodes in the availability zone with the fewest TiDB nodes, and the other TiDB nodes automatically read from the leader replica. For example, if TiDB nodes are distributed across 3 availability zones (A, B, and C), where A and B each contains 3 TiDB nodes and C contains only 2 TiDB nodes, the number of TiDB nodes whose `closest-adaptive` configuration takes effect in each availability zone is 2, and the other TiDB node in each of the A and B availability zones automatically selects the leader replica for read operations. + - If the estimated result of a read request is less than the value of [`tidb_adaptive_closest_read_threshold`](/system-variables.md#tidb_adaptive_closest_read_threshold-new-in-v630), TiDB can only select the leader replica for read operations. + +- When you set the value of `tidb_replica_read` to `learner`, TiDB reads data from the learner replica. If no learner replica is available for the current Region, TiDB reads from an available leader or follower replica. + + + +> **Note:** +> +> When you set `tidb_replica_read` to `closest-replicas` or `closest-adaptive`, to ensure that replicas are distributed across availability zones according to the specified configuration, you need to configure `location-labels` for PD and set the correct `labels` for TiDB and TiKV according to [Schedule replicas by topology labels](/schedule-replicas-by-topology-labels.md). TiDB depends on the `zone` label to match TiKV nodes in the same availability zone, so you need to make sure that the `zone` label is included in the `location-labels` of PD and `zone` is included in the configuration of each TiDB and TiKV node. If your cluster is deployed using TiDB Operator, refer to [High availability of data](https://docs.pingcap.com/tidb-in-kubernetes/stable/configure-a-tidb-cluster#high-availability-of-data). +> +> For TiDB v7.5.0 and earlier versions: +> +> - If you set `tidb_replica_read` to `follower` and no follower or learner replicas are available, TiDB returns an error. +> - If you set `tidb_replica_read` to `learner` and no learner replicas are available, TiDB returns an error. + + + ## Basic monitoring By observing the [**TiDB** > **KV Request** > **Read Req Traffic** panel (new in v8.5.4)](/grafana-tidb-dashboard.md#kv-request), you can determine whether you need to use Follower Read and view the effect of saving traffic after enabling Follower Read. From fb3371792301e3d3e55d65ffe655b92d097dfa5a Mon Sep 17 00:00:00 2001 From: qiancai Date: Fri, 24 Oct 2025 11:55:49 +0800 Subject: [PATCH 06/17] refine descriptions --- follower-read.md | 61 ++++++++++++++++++++++++------------------------ 1 file changed, 31 insertions(+), 30 deletions(-) diff --git a/follower-read.md b/follower-read.md index 98fa14f131c5d..77bf5ba3a40f4 100644 --- a/follower-read.md +++ b/follower-read.md @@ -5,59 +5,59 @@ summary: This document describes the use and implementation of Follower Read. # Follower Read -In TiDB, to ensure high availability and data security, TiKV stores multiple replicas for each Region, one of which is the leader and the others are followers. By default, all read and write requests are processed by the leader. The Follower Read feature allows reading data from the follower replicas of a Region while maintaining strong consistency, thereby distributing the read pressure on the leader and improving the overall read throughput of the cluster. +In TiDB, to ensure high availability and data security, TiKV stores multiple replicas for each Region, one of which is the leader and the others are followers. By default, all read and write requests are processed by the leader. The Follower Read feature enables TiDB to read data from follower replicas of a Region while maintaining strong consistency, thereby reducing the read workload on the leader and improving the overall read throughput of the cluster. -When performing Follower Read, TiDB selects the appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to determine whether a replica is a local replica: when the `zone` label of TiDB is the same as that of the target TiKV, TiDB considers the replica as a local replica. For more information, see [Schedule replicas by topology labels](schedule-replicas-by-topology-labels.md). +When performing Follower Read, TiDB selects an appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to identify local replicas: if the `zone` label of a TiDB node is the same as that of the target TiKV node, TiDB considers the replica as a local replica. For more information, see [Schedule replicas by topology labels](schedule-replicas-by-topology-labels.md). -By allowing followers to participate in data reading, Follower Read can achieve the following goals: +By enabling followers to handle read requests, Follower Read helps to achieve the following goals: -- Distribute read hotspots and reduce leader load. -- In multi-AZ or multi-datacenter deployments, prioritize reading local replicas to reduce cross-region traffic. +- Distribute read hotspots and reduce leader workload. +- Prioritize local replica reads in multi-AZ or multi-datacenter deployments to minimize cross-AZ traffic. -## Applicable scenarios +## Usage scenarios Follower Read is suitable for the following scenarios: -- Businesses with large read request volume and obvious read hotspots. -- Scenarios where it is desirable to prioritize reading local replicas in multi-AZ deployments to save bandwidth. -- In a read-write separation architecture, it is desirable to further improve the overall read performance of the cluster. +- Applications with heavy read requests or significant read hotspots. +- Multi-AZ deployments where prioritizing to read from local replicas helps reduce cross-AZ bandwidth usage. +- Read-write separation architectures that expect further improvement in overall read performance. > **Note:** > -> To ensure the strong consistency of the read results, Follower Read needs to communicate with the leader to confirm the current commit progress (by executing the Raft `ReadIndex` operation) before reading, which introduces an additional network interaction. Therefore, Follower Read is most effective when there are a large number of read requests or read-write isolation is required; however, the performance improvement may not be significant for low-latency single queries. +> To ensure strong consistency of the read results, Follower Read needs to communicate with the leader to confirm the latest commit progress (by executing the Raft `ReadIndex` operation) before reading, which introduces an additional network interaction. Therefore, Follower Read is most effective when there are a large number of read requests or read-write isolation is required. However, for low-latency single queries, the performance improvement might not be significant. ## Usage -To enable TiDB's Follower Read feature, set the value of the `tidb_replica_read` variable to the desired value: +To enable TiDB's Follower Read feature, modify the value of the `tidb_replica_read` variable as follows: {{< copyable "sql" >}} ```sql -set [session | global] tidb_replica_read = ''; +set [session | global] tidb_replica_read = ''; ``` Scope: SESSION | GLOBAL Default: leader -This variable is used to set the expected data read mode. From v8.5.4, this variable only takes effect on read-only SQL statements. +This variable defines the expected data read mode. Starting from v8.5.4 and v9.0.0, this variable only takes effect on read-only SQL statements. -In scenarios where you need to save cross-region traffic by reading local replicas, the following configurations are recommended: +In scenarios where you need to reduce cross-AZ traffic by reading from local replicas, the following configurations are recommended: - The default value `leader` provides the best performance. -- `closest-adaptive` saves traffic as much as possible with minimal performance loss. -- `closest-replicas` can save network traffic to the greatest extent. +- `closest-adaptive` minimizes cross-AZ traffic with minimal performance impact. +- `closest-replicas` minimizes cross-AZ traffic as much as possible. -If you are currently using other configurations, refer to the following table to modify them to the recommended configurations: +If you are using other configurations, refer to the following table to modify them to the recommended configurations: -| Configuration being used | Recommended configuration to modify to | +| Current configuration | Recommended configuration | | ------------- | ------------- | | `follower` | `closest-replicas` | | `leader-and-follower` | `closest-replicas` | | `prefer-leader` | `closest-adaptive` | | `learner` | `closest-replicas` | -If you want to use a more precise read replica selection policy, refer to the complete list of optional configurations: +If you want to use a more precise read replica selection policy, refer to the full list of available configurations as follows: - When you set the value of `tidb_replica_read` to `leader` or an empty string, TiDB maintains its default behavior and sends all read operations to the leader replica to perform. - When you set the value of `tidb_replica_read` to `follower`, TiDB selects a follower replica of the Region to perform read operations. If the Region has learner replicas, TiDB also considers them for reads with the same priority. If no available follower or learner replicas exist for the current Region, TiDB reads from the leader replica. @@ -86,39 +86,40 @@ If you want to use a more precise read replica selection policy, refer to the co ## Basic monitoring -By observing the [**TiDB** > **KV Request** > **Read Req Traffic** panel (new in v8.5.4)](/grafana-tidb-dashboard.md#kv-request), you can determine whether you need to use Follower Read and view the effect of saving traffic after enabling Follower Read. +You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4 and v9.0.0)](/grafana-tidb-dashboard.md#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. ## Implementation mechanism Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers could not handle read requests, and they only receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover. -Follower Read includes a series of load balancing mechanisms that offload TiKV read load from Region leader replicas to follower replicas. To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader node. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy. +Follower Read includes a series of load balancing mechanisms that offload TiKV read loads from the leader replica to a follower replica in a Region. To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader node. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy. ### Strongly consistent reads -When the follower node processes a read request, it first uses `ReadIndex` of the Raft protocol to interact with the leader node of the Region to obtain the latest commit index (read index) of the current Raft group. After the latest commit index of the leader node is applied locally to the follower, the processing of a read request starts. +When the follower node processes a read request, it first uses `ReadIndex` of the Raft protocol to interact with the leader of the Region, to obtain the latest commit index of the current Raft group. After the latest commit index of the leader is applied locally to the follower, the processing of a read request starts. ![read-index-flow](/media/follower-read/read-index.png) ### Follower replica selection strategy -The Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level. TiDB selects a replica for the first time based on the configuration of `tidb_replica_read`. From the second retry, TiDB will prioritize ensuring successful reading. Therefore, when the selected follower node is inaccessible or has other errors, it will switch to the leader for service. +The Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level. TiDB selects a replica for the first read attempt based on the configuration of `tidb_replica_read`. From the second retry onward, TiDB prioritizes ensuring successful reads. Therefore, when the selected follower node becomes inaccessible or has other errors, TiDB switches to the leader for service. #### `leader` -- Select the leader replica for reading, regardless of replica location. +- Always selects the leader replica for reads, regardless of its location. #### `closest-replicas` -- When the replica in the same AZ as TiDB is the leader node, TiDB does not use Follower Read for it. -- When the replica in the same AZ as TiDB is a follower node, TiDB uses Follower Read for it. +- When the replica in the same AZ as TiDB is the leader node, TiDB does not perform Follower Read from it. +- When the replica in the same AZ as TiDB is a follower node, TiDB performs Follower Read from it. #### `closest-adaptive` -- If the estimated return result is not large enough, use the `leader` policy and do not perform Follower Read. -- If the estimated return result is large enough, use the `closest-replicas` policy. +- If the estimated result is not large enough, TiDB uses the `leader` policy and does not perform Follower Read. +- If the estimated result is large enough, TiDB uses the `closest-replicas` policy. ### Follower Read performance overhead -To ensure strong data consistency, Follower Read needs to perform a `ReadIndex` regardless of how much data is read, which inevitably consumes more TiKV CPU resources. Therefore, in small query scenarios (such as point queries), the performance loss of Follower Read is relatively more obvious. At the same time, because the traffic saved by local reading for small queries is limited, Follower Read is more recommended for large query or batch reading scenarios. -When `tidb_replica_read` is `closest-adaptive`, TiDB does not use Follower Read for small queries, so the extra overhead of TiKV CPU compared to the `leader` policy is generally within +10% in various workloads. +To ensure strong data consistency, Follower Read performs a `ReadIndex` operation regardless of how much data is read, which inevitably consumes additional TiKV CPU resources. Therefore, in small-query scenarios (such as point queries), the performance loss of Follower Read is relatively more obvious. Moreover, because the traffic reduced by local reads for small queries is limited, Follower Read is more recommended for large queries or batch reading scenarios. + +When `tidb_replica_read` is set to `closest-adaptive`, TiDB does not perform Follower Read for small queries. As a result, the extra TiKV CPU overhead compared with that of the `leader` policy is generally within +10% in various workloads. From 43ffb51e198426ad0523d8e75794ca34061c80b6 Mon Sep 17 00:00:00 2001 From: qiancai Date: Fri, 24 Oct 2025 14:03:44 +0800 Subject: [PATCH 07/17] add the image --- grafana-tidb-dashboard.md | 4 ++-- media/follower-read/read-index.png | Bin 0 -> 50514 bytes system-variables.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) create mode 100644 media/follower-read/read-index.png diff --git a/grafana-tidb-dashboard.md b/grafana-tidb-dashboard.md index 15f065cd32a2b..7479bcb4eeca2 100644 --- a/grafana-tidb-dashboard.md +++ b/grafana-tidb-dashboard.md @@ -128,9 +128,9 @@ The following metrics relate to requests sent to TiKV. Retry requests are counte - **local-out**: the outgoing traffic of requests that attempt a stale read in the local zone - Read Req Traffic - **leader-local**: Traffic generated by Leader Read processing read requests in the local zone - - **leader-cross-zone**: Traffic generated by Leader Read processing read requests in the remote zone + - **leader-cross-zone**: Traffic generated by Leader Read processing read requests in a remote zone - **follower-local**: Traffic generated by Follower Read processing read requests in the local zone - - **follower-cross-zone**: Traffic generated by Follower Read processing read requests in the remote zone + - **follower-cross-zone**: Traffic generated by Follower Read processing read requests in a remote zone ### PD Client diff --git a/media/follower-read/read-index.png b/media/follower-read/read-index.png new file mode 100644 index 0000000000000000000000000000000000000000..b20a7047f905a9af711f5b740cfc9ac7ec102c9e GIT binary patch literal 50514 zcmd43c{tSX`!+6(Y-JfnNg)x&7KQA_n=ry)WDBWe9g>}F6UitlTMH6ljGc^qNhuL! z&Aw#c_kHPizpD4=y*%H~@%*0Scl?gy`LAOdGxzJh@9VnG^SsV$!Z14ON9c~zQBY7E z(bQ1Yr=U29rJ$fpqB#U!iATtfQ&4bIXsV*HUKX{S0oeq~j%fPX!YHE6V z_+#Ry2_~P}UEfoT2eKm&XRoPJseYCTe0nppuO`1`9&fq|z3 zldf_9^Sw;OClq)R`c}oEe|}2RIDYztb5(w?b9F!Xmg_8MF6D_;ZPw~p+*zv2HZS)e zZ;ih&%eAP?QVVDM%BVc_V}IcXrF`9Krh4h1iW5}U3iTTo(l_4Hj9M$uTs*yLkR|rk zWa)-KZM2!CJ(8#+Gsens_Np>>1OLlA1#1g~N?!}T{qIwWei=w#K+%ql9b}zBePw>@ z>V4)>+*(qn;!VE{NxkVto*)gX4zMPi?|f-`<@R)Dgn1;Vf@6AuM?u6%xt`mNA`|;_ zBaupEo0oo_^Hn=e9OBo4cZ!QTO&jkqoV3UFy!phl@_hA^+0Z7H9!os-i8!VA3oh>& z=MVRU`oo@5teie^bu(n-V{iAFA!#KQbs}$CnaA8Tc|E(ZS=M8z?r84^4(C4Mj`WxQ zJ=^PZ4kI5Px|$vM7;!S3i2UR8m!NLWs#ONcCYJi%-FmYC_8&~dV2YI#xT*Hb5X%ex z6$v;>|CQ5kl1zQ)ip+1I^)hxAs;haP_u`5Z!Y>QzOZx9_5(b(Y-}y6qrl;Pf4bq1# zODQdPT3%FuAy3up5`)?B$P*$1%^U}+*Z887mYXjLN~|>q}q!QG`AET*i)`y<#%Eylm?om4peXH6(#9QRjaX?yl?TH&Pcyg zycP2@zL%St_yv4pK9`IUR9<$63*yTfgQ_A@AjxQteZ4U46uZ}wVA@7s|^e1)qi+%$%9P0*%V2lLYVDOpSuN)i%ZCsit-chqvEZ{#rtPm z+pP@sX~Da_5B=lanu7l)@3w23`9JvVqdT28Lz^;rikH9A!JcUTTDSH%@g38>`a$XZiPPL)B0R*Z;8vu>}ZrOd4F%m$WT4ja@qB2?2gVL z_mKLp4_sSOQVLZ*1)bN@-_A<_#pmXbTS6V3B$vp&d7gEu?(IuT;37i;d(eWB^q90fR= z1Kg4;G+^zSqqxpEv@PF=N%a=9DW-C7y|=T-!N~JdE|1|+%RBkFilq-t>-$sbFUeJY zJ7x9JqAMrmJhF=H+Rm`Qhu;i&d64C%=(uuH&aZ$R%)2|4|BTc(j%`ACFZAH%~a9odWS0jFwyd`p1hRO`K5F{OVNO+}6a)Vuq8l*!Iv0#m~Dm6W*JKAH+y&W7CCtoM5kb`QP1JNMin$I36`vmZif4rDHRqzSKH(L|zenMMpD z*-&nwqR^GRkv$MCXfWGt^RlRXzN}F#70deUxU8GNHG=a4BxhFz#R29;v!Q~Eg3fzK zW~$F}d9|-=OZVT1;@2gE^p|OsH8r>gi|Oh2naS>#V^mmu6z(?F_~Ck_LZ{#GX1jr| zWN#7b?xAiJ^~oC|X5=@XgMQM&_xdWnfTytk^%Qlw7(x50mWv$S>Rf*9i>VAVTGmyW z<8~k+v7pA-#}hK>t_j$r;V}(^)E};5z33=^@C_B$SU)rv*yf1&5o_UNw=-M#Wj2gI zF;6E!*0ArAD}K63deDu_!+&qhs2rRt4itmiYXN&}-<;GxRR7p_c)jq1aC$0yBx75S`v8cr))9jiluwrEpDB0J)2FrY+z{y zD8B2_d%D%CzboM_vjc{!;je#J!E)yf(>J!mHgZZ|hN&;cGX$Y&Hbrj5w8z}DS?L@k z-hrc`)?`f76ME~8UB6>hj?;lntYZrZ#%UBa%F*UTyekzD(z)aM!3 z@D)0+s$@1Sc#Jq4NPz2;DUm%+<#RXR-8)kF?B?@~x3OW7krp_W!`BHHwpaOop!gq5Dpd`_IRQIS3LXE_|oq zPm4D7`*_mx4G4g?n~C1z1!=y65-+{Jho#wKt@N((o2yC8WEGa3Gul4%0v-?Q|AkC_ z-CD5|)kIjw(nl`mqZVGTvI`oJqNQf8>Nkxo?ML3(k5ilNCttUWdTuggkAIAo_%qGh zcXgsMf;%tCV^f#j$*+9AZji1cveo2brF1dk#FPZZv}61GFcuDLM)~=Yk=f*MQHlN* zuR*^JwT`{dosA$!OhG+78j@t{T)B3f%t)=7{BB#HgRAorrvJ`LlVqXoOTTp&cfmc! zY{GidL~35mSY6v{;ql)kO!7KUy<8C-+t8lQ3%8(sQn~&zVBe*8Ke|S9s;$zO!{+nT z;U{_m`62TqttT%>H-LJC95LKI4`s)qB-7iU`1*Z@Vzi~Y-}`x=YZNYwmW&1)teEvI zY59zy+s%+nlWMPJedD3!?{?cJM}jJvA}iL4pNlh(CgkDK?BwXC%c-6V18I!1e%q@q z0pGR`r7Ae*obX$1nP0wYK9CXaZ@7xeaQK$v^x|c~NVE9I6R5Jpu&vv^5e*K^Hyn z=tz##>l~h_5$H{GqTx;W1ywjYHtB_k6~Y1AmE(Jgtv`esJFkU{+yr>-oL~8soC<(g z(z4}c>0db~x!*+NXYX-1-TF~EUpA9G?UVNN;)(rQ)N;XYcGdRu`uA|Lo-3|R)_;UD z6jiSGG)jtl%o@+Mrqa)7;rb9IAWghwC)Egycz^v8v}CN7$)s`U*B(ie^ezjb482~q zgY2FS{#?}eB@cG{tAUsLB^S${H+CcX=`)}2CVrEu;qu=+g$WiyirVGIZ810BU8Mh* z+lfbSpdWH7l)V+OX6lF&F%^ogxo9~MEtDlzv}A1s;{PDZ(05!iyX8xeBxg9M8+B2L`R{VlH}|+t!UByZIG3m zQ}r{{>$t<$Bmme!{Q9~IF`V1cjyHB~SJsuxc`QhIRqY?*Xh=P-53(<^m0X?VBe*fV zdeWL>E_YPwTkV@6Jbjr__{JGu-)G@79o^IkkBxCf9ettcBIK^*?$2^`Edg6&aSut) zoU693)LT5OmGzvTWhkDx%gJ})_xWsU5NAILP{jzh^ka>#xP9)sP= zaUerCBDPxg!l_Fv{tSl=JYxrroF=|UQ8%tVjF^uR>~k8SMbZ*M;*-4rAWZJg(8^r# zpcMC&>klPo$-PeJ=FCJ5pU$n`FE}FI_tnCC4ACZYUoYiiOy393xuS19Z>hOv%QDu~CUONjAoTr3 z#~I3UKD5Z95o@{*(=@SrT7Gs=pau98Z8kB&3~}DyDx{RCJ1gEkFr4f^F8a!?$1dHl zPfizu;9@XTD+{>!`Kh?*!zm+6zcJC(_5-d9V0S;rqt-s3`*MCuN|U{sm26}VH#A7Q zl+PDyoP5n}WmDxRjo|2b8exk0G#{z7-?d6bn16x1WTDr^&-cps#$&Zhl(S&x-vCe{ zMG~Dzwis8xAPs0swGSOKo1Jh=38a4O>Nf!CJXB)F7L6Sp29{deZxxy$oAPyFOXGVCAR3NJzlNEvL! zZ(f87{u97Q_E#j~(Q$VBn+wml1YfP?=LIrt@7Mm`dEH&JOqH6 zDU-RooyFQl=I(~?QvLG}UMe~8i&xf$a*gvn0Cgq)kD;RxT+TKx0ta`MF?@YL{24Bl zY~{Z&5<{g%plMEebXhr-4?w(flHs@#Y7Lxs$tWxQW?8*%oWK4q?Jq93e>#w#dr^t< zbL6d*1OH(M;IwM@7A9(fUvQOsud!k+1NMqU$|jR@zSKR=8HaTMj{Uvasvof9&TKPj z&sYNd!S5{T$hyc1+t&hR-if#7KbsJggf989cjD=-mFv>EiTG)7PKzaj_chvkqKs|hksGCHVKCObL`Oe?*4`K|Kdkr zd%=;qdyGG`@v!S~N99Q@dZo52U_)+2e^!a}O?3IdKJSD?pZ%T8^7Lv9m^Q^A>hSo# ze_XfOCkr82GW0I{FzcTij6|COn>BPkLCYkoky z?BSumm?#sLcB2~LeJB6f@eTuwAhM&>)#in3vkVANe}{I@>esA3#u?w-^PMx~a<5T8 z!yNn-fx!%NpV0zfffqjh_nidDPpbdYz^gx~efPH}gxIj21K#O#WivVRCptk=<&-Nw zf9%)l(yR_&yM7y5+!p13IcPYxhx{!~o5LVE?Co9Zau`bppGL6#=HhT?p^fg*;q9?F z%j2o$ktc7c9Qk!5EV8p_g=UM?s~EEH2cparWmX!si2&sL%B6b~S|qIP~g0 zz&vxM6N!Z!&J`_tM$fh_e5UxIaw7!C$~uE;Z_^D*!NNc}AS;fcCq3>bm>>W13Y|Ta zFD?VE_s)U;dA+8z$got9L(BGe7vC|igN1g3D$B853_z_M>PvsQUZUf*< z7|DD3;eALPGKLV^M(_3si5vAy8z!S5H$?Wve=b^(Rs#3Nv-Hn<12DFSs!jZ5V=Jl+ z^%mkBxi_aJ!5~%Ee4dl!J^t)ttf{@AnO$m3mh)g0`O}#f@0*3S4}lQBQCbc1Y0*ia z$yZhH767#?iWM>2R{Yj?kcNcWgzwyrolsXunn7dv9$aZ=L>Wfv#STdAjzfW8we}BTq;W6%GDfiUJ%9)| z#VQu@mCFr6QiFF^x{o_#Spki&61p{f3$B?`Du_4PptSX{nY-ypuLDl!Yv3C)28v`M z*rEQts2r28?(`vJMc*t6J?45!tM2T~6-xmKdw$oeHkek*_A9ricL}^_QaF9!Tiy4} ztO9h@meh~kmGbu-4a&0C4XmDuqkzQcLj|eBwdMe|s;v5qi8r*@7seATi}3c-?HNv@ zcB0iwA2^)mbBhOj$bIL+cUt)3 zeAhaR=H|;6$R|Df^7VGg=g2SmLE)PpjF1^WR=)W?oMZ0_m`Y<+X*@XGIHt ziS*eNSlA^oW1Xw_$aXIZR3DEgRncq#u-q4S5{Z^_Z&5ocd3zARIY&s6>-LjWT1%}5 z^=S(%C4U(IRG|CEHr_AeuetJ>K-5=#4>a& z)}2#6KRrGGD0M5-Z7kNT*{N*m%uIWR z0nR5j%5?X{QX741Ri$q5F@7}VZ9G%JK7&YRS7(oAylgU{QM_B`q>!t zX>7-vRGd43W{Sf>jZ~40Q#5lbP3=ui_svW6`cCVan^1+j+~ScGa@3-$KPuHZv2rkF z0KpVz;gfOFyFO?b#>V`-Nyvp=Bm!HO zI}ap+^L__ze^J;t@6z(jCUL52@~c!8*Z|}sddUF3W}l6D91ubrye3GQIR)=P;my7d zxRX^>_UE4EinoV2e%_g&uRDsP(n>rmSp5M4MpJnz@=%Ow%+W(@Me)=p4k%IMX(>T> z0cfsc`-e1NXW=iqJ_FmeHIb z*P9Zl*t1Beif@TKU*}+Al^`pWA(?tsrLKoL*eA~hNZ!2o1{QauNkVZhMbu>~? zUyb%+&6l9J_QQfSH==%30JE?pV01?b?+b}h~o!eZYi@fdHXWcw%57({EfoI$9ivgYfzbXJWM$gAd{H=_+R zEUmvBJFWqhfZb?M{}?)C!d)z$CLsl>!>urK3+sFvF*0HxG38{;D>)RZwV| z_pv+Rw&mwPB4w}UOg?MAD|mMh9M7bpu^{sYpg)Zy(%Szac*r7{4i!_mmfNY&u)GZq zHcZ7ggddMV%j)GQ;0|2Ye+m@Q4`FBdaxl}b2{G7Ob{tFl`sg)ODz2u7q3=pC$>vMq ze0U1<49xFW-X<=+;F3gOLnhLAiD<;aBhHBo`;nTfMSw>XXx2CfHJauY_6?$8hF0J2 zaoZ5F#ZQ#IQlGqLwd#GcxI*3(Joi$O6+x&0jjzG}XQ`vdWruR+0?Lfgd&>kzj6N-{ zp9ARb&e5b>eW!^9WIEMvFNJf(P(+YFSy7$tFd*(eZ%kx=-RCv(aNeH;utTR>R(P?H zR=i-CYJSr4dwo|MqEwIV>Efbd3?|qlz~4!Ao%EVyHrdCjT?<$lumTj&|Jr-03O(9yM(phEXM@o7bRx;RQN@)+yP?sx5D_`n%m?UGk9fs2*Ph)+pYjjpzrGQ{hVOdvdkP~ zj9q4uqO?#?HcedBrRDiot=-6z=}QHaZ1l*^2=0N6fDg|8!E9LniL{1i_1?&Ui78qT zC7B`V+>f+>M@cFkecIG=e2U?zU0Yfy>lhll1h8PK^DTD<&F;NbT%Kr!h%l&ZXK{#| zv{(gn$UNWEMaSj&!j?fZAl5Sw;|R)v4XK4j=$ARkjo$9v6S_)!qwk-;kA*#hTY6EU zVw#G?2o~1fezdiXEZ8Q6%$ehED6-M&SSWd9>b5z8FA>9PYBuMlm&e zzLGbh2*O$17Kf65g|7~!)?-n%#OU`Izu42?vg&bLFvo3lG>e*F&Z*zZDr}yf&(tRTh$}v9R1l9Q z{lzM=K!;S2?uaLTmp5D^T|F(ES2h78gZ8R)1b*Q?}E zvC6v?%p?`cEb7lRG>cR>u-60E_>cwdFQ`B#Kjj(tJ0HN#Z9#KihY+RgQ+~6^}}taG>&8 z%H68NA2Rw3{(5v68@5OhtVwPn0~}J~lhvG9*e+X%DLE#%Y`Jrg_w2`et4z?BApZK2 zASy#v5InCY8$TZcY6bk>sU|v-4>-SxdlfI!K_~h5Ss;0^ zpDAfzr)w4(ug2fUgp9oJ-Qpc8u8IcV0Jf=F!COgpY@+54gM-hJpdUYD1{g6W{wE_w z%IhS;Odr-Y8JRng0Z{SWu+{>%axj4gQAk1gmS%BB;ut>A!@U$TI+ti6+Rp7w{)~@#wi#tYCrmUU;tg!f8FS zn)QQ;8sc=i1M#KK_>*q@)TIZk9$_~@FWTVCM z676Y|V)3L{551!>(y1D+oO{O?l(}EPj2R22jC`nqc$u#HY2uIKdxw1Uofk=HrB8u0 zT8}oOR0Egdd7ZD)!@l#?_%MkDn1(9H(&kmYZY-xExoy!qTt8Dod9?7LEhi-{PmRwb zZkwxgFo}yGF3cI!FsY4}#vJDku$4SLsl39Z!E~VWE==a20rw+u$}F*|+X8r``F?HB zQ`)?mx><_np~`rgJWk$v_NSCV517Gn;{G^&6gyQLRP;AL6ZB6!i!z4FrY`DVFCS-C z#)xzO_C5Xie^^Zllp)^*i@X1EME(39V17FM|IYlxeH*3LqkhOFmga0?;i&dDWza3K zY2aewXq1!W4|K-4J!F6xmhFZViL+b{3hzE}o-qiN5|NC~^j)MKZkjM58U$a z{^7P`qS2v1A>t9b|9aV0l>${4_?Ij1=l)^%4!{g!hP;mabLT1WKhwzHZ)Xx3oDuTv z((!*CM?Dq7j*I<<2~Ukz^AqVQdRQ&Ecr-wLJw*|;4DFSqEE9HW`Do{dOMz6PH>tnM z!{DlcYuu>j8djPc4=84|%1#jeuh{O#^dz_88qrS6B;n?2F0r8NVRq3*h1bivQv040eG{Ydh2XYwwDUnv~t(%$MduM75n#k zfl_6|sJNMwnUV*Z4ntm-{uXN*3g8Jpi|ztl^U8V6?{+}&_ZSMI?g84weAaXE>-QdV zDGjJ4lmW*=eXIEO4_&clPIi#s6|Ry1I!~Qt?hZ%mo}2{&&BQEdEEEI9xAC&2pSv$0 z+#IwL;Ikvrz5jZ&pi4~3n7Y}ExHhPWzzn(Dub%Rdow`S z9d1^);}`^9F2`n>3E`cAuKerSX2niY3l%FpK##r-gy9#yY$ybcdsOJLFC8u;A~ZK1 zgQUw=iGIob=rE|npBDeX2r*$%nrfSUz(DGVM!SqWlk-l%n+mR?xoYc1+|IGhwpLBwy1Nti+IfUboi!LUs>0rGz4-AG?Jtnq^%^JX+T>u`R9*oW#P*gd2j%V~A$Z6kmEu{Z|k? zq5~E1Y5(eauX8VGidl0w<^_fGc#nZLtZVOfH0BzR-rqo}3rJmUi>}sGvjdgoJLRHV z3Ty2K+3qu)^9yLFrQMY;<=H`H0cdxn^{(p=r`186dOZ*Q3mVqLpj6=mI(ql#Km!o! z8+G+K2 z=reZ%lTqiSyDp|lVWW?{zdaN8`YW)ilmnZ}wKJ+Kf3IH}qR{tNwNx5Fq1HH_e#yTl zEj`O0Zq#4q?uxgABHLjMi1u?JMSI$H7ExhTJ_&_{2@`}CVTP{#MleSZj^ZpN?UidO zN~!Z}ByIZkzs&+XQm1@Ok8D-!^fYdID*n!+-`yzgj9-VLk-JJZ@a(Ti@&sj1o#;+` z;bh+^QXJ2c*Fh)r!Id&0T$Aa37|0cuhV0lVWuTB2K+qlFdfE?yg1V=OPrip8 z?|tQJ47{~&nu2bOF=SY?)WC}U7#{NNEtZWOsF?z}_tIhABz=^cDpKCn{;yK!%^jH3a5ru$g$jYC5j;fypN|X#R(+nM*w=np?@!8e zAPy*@5LgZI#FS;GH5G0@@NBIU=Y;Pmn?h8h&frH8%P%?-18blqC8;zx#8Ty;sTYsK zBs2heysML{#iT(4cuVjFo+Ne}Qecvoa6gPdWtgf0)Mck1*A-yN+%?pR0z`d&821nl z=Z}p5hWCKnQaqsqVa$4gmgOA_xVJ^%?T7*Dc8Z`ZW{}}K<2^V^rSXu;@Q5|d4FWi^v_4H6?neUPyL1-Hl?^j@Lgt$am^5 zi}71ey30@iyw;N2Tfj$Z5864ev#a;FJ990S99ojq7Il_*VRWESoIUZsM#BU6)gbzL zVrATKRN&U$DU2VaPJ@o! zxI@VVu##Fq_8;v;`R>Yd5a>7^DSr)a8~p=bGL`ni5lS*uN#}I~*8-VAMp9M#shPF3 zhbIy42^HOp29)^LM=5@yQ^nFn(CFX0q%!e||K%GhHscz@ithv=+5DdlJm7e@D1f?! zA3=BcyxcS~WwupI4A;pmrs57#Ru^b;4=iQ{KZPZ{zsqBrTee9ZIi)-Nz0Nq%A zc|NyDU|V{yUf|JxO@Vk)%N|bwJI(@P`P+SP+J6caLkt4^M9w>ls9vkklaBdyY=uz$ z#8s~}fJEm6FfExyZreGd6+ockki|dk`FzCHR$#ET1GtN<8Fi*~?(SS~G2=k#14VC8 zm%5nMPgsGE5yv9~BTX17mA^9s=>Q4tuAx=<93RZkankO>L%{s;zS+@9J^&47`EJZh zO~kwQfa#@q9&E@Mup5y(E}J!}k=_*LmQ;XRLLPta&_gGbH5B~FQt;@jd=d>Rftael8&SE+OYoSL425@ZvY=_@T{ z5dd>GWS5Z--H`8Wv=Zu(<&;e$eWz06W^nd=Brv0I+W9sWUg=yWp;}nweWw04zuYxS1$FYOXtZ(0BxS zz{kONIOOQ__RP5$$j-<$ePus>jXTaMy^V=T4JK_(GnWcx|{A+)Z|+2vl|H^L(#(UJ1PAwc=S?14mmvVMcbGf;|h? z$xkYShBq$14`DN{ujEI)_8b5Pj*W$aMxhQmB(A@$Q==nWQ8No*|Ur)A%a{1e)s7HDL ze9)li1ps?DU`$rPYK+_T#937U<7!`b3c#+V!-I%1{p$2f&#q~S6k$)J)eifae0FYWA9HLR?hEpacwRRRjAYpO%ow;13!unGjx%;Md@9IDl1@Lg_ ze%^`InTTi2f6~>mUdj#nrh<(xr?GD|uW3^*;35%9M@45(;4B7*_$1-mF5x;e(K=Th^g2?2k$dbfPjQy3~ z1BfE;?_vB`g4M@LMj~8!G?;nR^6=668yN3}sS|NzMmC$mUhr4At~@Iba4x;T&e8Ll zJ(VTy!ee4oNgG)~t` zE+DczGltzQv^Sgtw(R3d>vXckm#T#V+08)0$SxBb5}@iYJqN| znws#27>AtF4-AkmLEXle)uc7{LPv(H2Am&7000g6;3Ar(rVc-Dz7B%NM4L3#Q7u7C@U))5f)!hW;ibUScaKhPP-!zS6pEHN2L*qo@ zc?dRG%uDG%Pd^D%MMMfECDZ+2j7)NE)!ozQ`w%8WeW)@#Jd3 z+o`jGD6NchSTGop3e8vK&{-c@ZPxL2_hCL6rli)^0DAInXLR zkLJWvbXZxflwmOs+8=b6Cm>7yxu)|5DZ&93=ICYd+oKxA5B2V>Wi{I(ZnCIis7{*5 z|K=2%T+jl+Oe|PnsW0(Ozb^YI>#G)%GrE1O9Isf%plO!g_4HsyV0tJOyv_HR^-Z)E z+NIk%t{bqyxq<<~>y}na%~FC>!B2EI5zdV-2MkuTO9aG2RLz!Rl@{RFaXTw;{qx^q zlTG25$6C|01kk}~0YsX%Sk4M$p@n9y0Gk-$xk1>RlCVU8-7DVu9Xm;aLBEte4@lW# zm~Xm`M@xG3j&_m-X9gJqYEq!up6eGLD zdpD;Bp+x0YbBz7B+$Baen5j$8lHKl7Gm`x_`oHC}Z@5~+SP#;k zp-r#74LQipS^DD}d_S^8^h9J#zHF!oIyeW^3HvA6XCJx9+_}HT9jcSw$V6lT30gKN zNADaB>3GoV=XZW4=!tPy3^^U}t)>LSR%);QDC`v=f*d{S(!!N`b`XGd-23~?Z;zaH zZOMWVOQ{Vy#_QRt+sk&}g~|!?QIY^Yr95pF%mP$sbxpx7)V^Zh;BtXU&S@t};t)y9 zqLmt#-NvSpsv#DEnVLuTm!^)MDBIEWG(xb2DI`=sF}=+Z8g~+{lJ?rOVoW@tx4pKf(tCl?srxHmk)RIT3bbckjmyq=X=RIw3owu2wpQ*WxjOU= z5I^5T*T0&m`vHb+&*K7C382;b+_y6Q+>Nrs#@ERDr#S9JzuwX8yz92ZICJXJ4S+W^ z*L4hf|E36+9NdM)*dLd)j|OuRmqqE;K5#gn?`}?KT!yBQSpbnW<1e%@DDB?zoXXo473X=IOcM{o zxCA#isN>bpo;JeDBumR&8Ap?R^aHp0)jvxDX+e(rfEDKp&7rB;ns5yxatQ!>*Bn}oH z$TFp}N3(c<_Uh^OA;JAM+k zM~0Mjc#{697(>xj`k#n^1b!fa1j%<0-&01@dg19##pqLC1Kj|MpkUByA{~2 zu?7ApNq-8cGjERQ$^G2?Q9L-uqfTl6Y*8w+g5%S}r0V>t-@j!i?xWz<(p%>eg8$N? zz&0myLk)>0H&&;V<`U3*9O2O|XVL(~_U`BwES&oh0@~kY-hZFM|I$MJ|D_{QKjU+2 zCcD@a3<2S_Z9gsWP!|D^hGKKi{vuLY${jmUAIH150P&LQjov=%wK#O<%O$`Y zM$?H6AW=C2;P;>sz)z(CRoHEfrG(J$`EN!fN_)fI!7 zFDjPTD^PpX;OEgywF5X{+eaTf=N^AiYL50l71c_+Y;}PXjBczxIw%36?X! zf(6d)0%(|IxH90!-ZzNO&a|aRLUwJB-@-mi$pDcnOBLfyb_7ry63L;_{cE6gqtkxo zI39WX{rN{%fmQ>Pqi*{xj%c37=wQ4(=E3BQ2gJBR6XOP`%-KD=xmDW@YH5%J>YccG#m^lhYwa%dZn1~`kIl+1BbNnI<+-)#W zjFK-y>~OApq3U7tvh7YcI4&pf@gzSmUbPOK$-N-6UIlJSce71sy8fef?{$wXHzWJ}M&%jnsnaf5%!NuRBZOScHul)QA-mh$4I7D2GN4*M~#7q%y{#sR2l+ zg?_6*>Bm-NFWR%qDzx+mTO9P;0gzGpD6K*892hsa&0KC#>C^oxl+*O~PQv@)x}%n} zJX(;^JzO{}lB|nVNiVbjAXHuseAB(d?AHLEDOLa?bB^NH4B(QU9-^R`)miE5DL3^x z3Y}Q2&#Rlf5=70J2ZoB@uBrP$f^>hWY`6LJ;h0@N(5dKx`U{Y-0!^Iiydm5rTaqq4 zlc)pZ(KFwpXihPl1x z4|*@f)F9AWL2Z-obRJHMXlqq=vbo@JfPsx3^wfIGW zMa>?56TN(@mzQMEN92`XD4pPp(s<$rmw>wBN`($tM zE#W#RgapGJh1synhH;|ieV5S*ccT{go^k@2VJ z(w_BIXX+>j{g`>s1$vh+7ESQp-XQZgy!bdJ_WNSOfLxl&*rN*)K@^}*oM7gpJRI$< z7GT7y#+6u9r=v%o>(%lanl>R`cF_yA1-+8&sn}`|1^u*=YHwGE2ZYcCuQb}hWpvu; z_k7867y0teau^F&Pa^T$&t{#MdVr-%9fBia?}=CR*+&;1*F3qfN_gM?2cyCb=tl}Z zG2{x|KAlEbZkJF=-A1TE!kA-D>GJi^=zqOZSad!2sv=OU!}_D?r#O z7kH}r^nEH!hKf=XYzF$ddLH#JnhcUcDu_}>f|IS>Q;AW*+9VNQw9c>*>zdx`#~q3} z1xqm$pI*8EK^32T5pGWq&aPN)IFcqr;t}U{Nm-o^sSB^Slj{LFf8?9_Om|^=qYjY< z1Aa57f5qOicL*BlDQ?QY!10oNz~0PUges;vUC&c?z5`6?lV$Ie2}O~a=1 zO}`#bMxMY@Nb?&6JyPsD_C$8OEBqQkkx&hbk8@MilSz0Hbu3B%st^e^SSfWi89^ou zo?px=YSP1z6iYEuYI?hZI%pPD4)M?nf=@Fb&c zUbxMZO1x~!?FEVNuhiz0cfQ9;B}};pkK{e5g3)lHResrTOgE6)+MSvVqCKb!XZ30LVQs zdMCmk64ACJP+{UY_f-P9)ZA~aW8U#;vM8KSHg_}z5UBd_W_*H2kDUj&+LCr!Eyoi$ zNU2Hs-SPgr(8Y3CeF}T| zo{Z)cqi>$s{w{s_$+j!ZP+#G$c^FizHHxpY!mgbH#X6}ywi+6%!THNhB-@6Qix6yl z`d`(`;B2yYQwlW*l$u?WR~|HrfO$*=e*&gcnXd-ZDT6}^3-^=>ydsJO$}Sk>2ngP7 z=FN}?(>ysZAuDqDoBI!w@C9s&{F(k8klgmGV`)pcfHX;N_?ZRDltdwEA&^DI}2dJNL z32r=I47?6A3J64V;&(FAIJ!m0|2k2!rs_5173 zL4+F!Hfy%%lKWa1LCTC9C(+!m1V&9AvODS(w>*H%?BR(+&>tliD>a<%{7 zanf=@hh5`~;5bj`CmVsyv>BFAGc^2?NL~%4lNR{pE-z0(&1l>5m*k!dR#5hs*VH|Q z1r^E@+fJGx+tRdMr#{5A%8R1-DjlbW9#Hwf~_el=eN)1Oz^7mo_&MJYdwdBpG41O z<7~HUM8uP6qV}VOn$&>qC(&hR_UwuJm zOSs77;TWC^a2TAa;Sh1se5uk5ehIDv$H+RT5+A_5|J?p*do@ylseh`=+qxqq~4;NG2XR0*U&AJxQTXlezm3O`zm{ImV`JkJ9Fw(wxZB* z==&$lP2Oq?(|4XVTV!~78X)*Tf93lEpZ-qtB(#yg-Faki3h-9I#+Z&p{_(5(wtNY| zny`n8UP3Bk{U>=(CEZ{o%ugOHR|S3YhqN;mQQel#UoJKhC!;R%6Y8@?tM7_$4H=83U($2`wdU5rBKu z_NY|4m?k~XdxA5A;Ma|A5Omh&`+MCyV_*dmh}eeubTZ;Cq`S=ARh>j;W_{QLv;Zp(-Z)-_X0kGoV(46W1 zqrkYEmc2^u>Bnl3DqXas9=WpH%3_s-4H zt%PCID&SzK5SM{V$Hl1WI11Bom=7@)1Tqh2#U-v&Ou%!9I;$g36(s+Y;0GfhTiucW zPGWLbk_Y8}2-L?f4L8210 zL_{Tv6ap$qa*&J!kpiJmf|4l_5fCLSD#<{WSfW4?3Pf^71Vkh!34)4B1|cR225y zoJy&kz}?fg$*;OoYy2~Y+0ad0_qP2JIC^cqTgp&6Xv4KdvKc@|1#BF#ns@#;&Gygp z`wNW!Z<;L{j!d)>|NoO_+xYJ0)pjN7emE}i_ySA~6{`0^I_~0;kHQUhTk-!C{M(QIx@Vx=Au_(%D+a`q7H60a_vg4sKA~DAmBtnI!&CISz3J{BtP_5;xVy0(h;A z%v@AGbhW<&4X~l|OCoLOav>llgOxld($^&WFYmE^4ZtCD-!mu^;aU}GGnVGdU;QN$}6I1R4s!sD^t$+Omk?C0^FK|*i=zyrkF{S&jAfb^75E8_+C}nJ` z$5VWV)R(_#pQ#-eqGb~R9rm#G{6mmcwPzHe>-UA+iFO2VaIS!BS>mR;!=;BW_qMfMu5{YKz{w?*=$2T~toq<fI{U8pqG8HH_WCOTq z9p%UoKVtCUjsnI%Y&-Beft2%@FN@TdHxE5H+%a;}(=T36gKXb!Jg~^tp8dQ_w-)2k z%$LB-F0*R{-s!htDOepd2x@1)Qt}EiylNZ!8~DwA7jIbdClt)&z96*#6#H6LX)kV0 zus%Qw-@+?=IbQyU;7Pv69r?CxnZMpA>@vRSnR>`)<#Ucu5JT<_{I0@8owISZDTJ#7 z@)-u&N;s@#VC!3fvZMazZal~~&b|0RTYx$HOb|;K@^ax(;`7zi*SZ7Dr>QXRYm*_Y zGY3U&Lf3)LZ*MQ)HhmU_f+BvD= zH)p2aCr0IeXAc}ifJR~4#0QGpf%z-;_CUIj#Xtk3T`7kHRR?={1L*e1X;j#0$2TEc zO3>w6Jk!wq{0qvaYtE`yg5Mzc1E~r>1?zw6W8`yvG`>!F>CIIecUF%&mB99W z;ee}Z0I!{4ozCFk-i^7=F4p&-X>&~Ju&OJ_fV0l*;bY3E^|Y5@Zc#4eKai6_GH{t| z?Zv2(>R#$4%Uc&;#&V}On&oMnV}GfgqU%ANI1}$AB;PeUHYK(pD>9`W=hc}&{@bHW zX->}YS}R~rv)nvL!mll>e;zgoj$%2T$5hoRGmctb#FPbn1Wsr&P2A3xe@OD4&ZV}m&m_(xc@{g-_aYu`~UETMT`Ycf9!?duX29U<(SsxNGviPc?TpN zT8be|uusAqQ+glmsi z59~BLN&>i$KjtwYo`c`6_o_Ylysiar9WWpr^QyOvH#AS(=!%vn%m}&-Ym5c#KK02j z9KU&eSi@vn{GI?88f^a;!fJ$fV333?zS8Af$v3@;Ha9ih!Oh6s@^+8yj8$|mY5Ix{ z(?!HllKFMozFB)~M(D-mW0ayWdYuo;fQynvZ*#z35sN(Mfznwpq|JPgKy|KLgY9QU z3i;+hwD^r57K7kQoVA=TpDp-0iDmE6vr12ioow8EQa_3qZie097>N2*6LL-5RuLR~ zNd{p0HDg&TMr2e0~aeTvrm=i-fzwN)Bgg; z2YRZgP4h9k>x|()UzyEdUq0vE35RpJfNHV1y)a=7v|D2*zhi88VdAk+HZ`YD`?K7? z*m3CGz-Icb96G8fpPcf7f>akUDst-(`aWV#YqF{`mS;G^VmOjjUW52^2pne*lDEFG zC_Eyh8$WC9hmhGJwk+kdd~?^q!>Tkic(`9u@9MrZlNf#DCT~JecagfY@Y<@hjzZhJ zweTa!Vz!Pz;T9y0Q~yoPT=yo&=}CbB)Ib6Rqb@u1wrpE2oT{ zP3&Z2#}v9ip?Uxg1!pL5zSQ>N+|K!>8`QG(+_o&jL%rKmdtk9VX;}YvHk|Nx4T~JZUJzQ9j0XCKK4q2!o(c9q+{Rozz#R&b<+!^>)|d zE3=Fux4i5-G2xevV!^dGg5y2#yM}Sma{W)I-C2yZ1~1tEE(p~7sUt9Q#GjY>DZz`y zB)C&^FZS9On2wXi!Jb2BQ?q?XQpO`yLSr=^C3?-jtqN%Rf@^;IgZr}sRn(*`7>W&f z8@|*#`zB0cpm8HU_#_6bKQ_uwDq?F>FY^rOX3Rt(-ua$kP%w!a9WaK?@3(`Oh9PDj zt?EX6$}N@;!^>wBd>344RFr_BPlcQOi%21MT2WbxgNVuKpkA(}+5IVw{}b&Uefefs`|CD3oN{6NKFcQah#L;j&_e5KS=dut{u3eXhj=o%7|q) zpvkN&+xME2NoKpebKPt}FaxJDj#>C{s|;)h6%V(qF@g%>s2O7>T!~eK@2vk|Te4)Q zxWs?v&akM|;29(5%_iEm2E##zb0q9?p@d}d5*O*()J^G&yWnSwci*6<=;zhf(r-EW z?$|Pv25s|SFVexF$iQER9vk;?24rHLDk6=6D^PJW$>%h17e|ZMaopJ@_kug9w$tq0 z^Jzh?tU={pwCDPA{v-K@N_Q{;nKy?kkRX+l7S*kPP)KZLYl>+!hTPdqhEJ}^CHilp zyruypq0{MI)?3gQiP4|479IeHIkYE$(I=tN=hm*=8hy}!=nENbBPpDfrqDtI1<$out36kI#Qk5kGkZ(o&O|wPjmO0JIJf!(=&Ntb^A#vC6tJv+& z0BY>UN1_vSlq|>*!E*Q93F`j`bpHQstMLC_TLr~snj5!z^WTf!+~WJOQ-ZPb;z)3D9mJ^USc4BSrwqS?)c0(~i%=---e?4V$3Ji7fQ*19|IcQt-R za&R-%qyI?FdP|XS*Kj#t#CterPE-D(rv)<$xx+t1HpzFkeMx08m8Az;i+PEUU?Ow% z8P8_%zmOm^YYo@&R0Vr04IF>>mR0S2T4zYpjl9+I!)y4T+i_oZG8>BT)N+FxQQY&* z*Wnzj%vxF}yLtDg>3s)>hd)=cU%6iQTOt7K!(pKm3^lvdMiX3Yuy2-Mbc(UV%tzDr zG2D@{bpB<~FURX0-gio#=xf%~v?#kzf^C*t7Kpnd4Ko}YPN|6#iSH0UyvxBiY309b?8E2niUG{ z(vufuL6delc3bre(P=Z$*VRZs_)Y@8O3_CA^02BfDR!GZ2X~)2B$Ejn zMWWhAOla<)HW1Opn|Ij{%~gjHgFjNAf5Z(Q8v@siFli$nvBcEzDAQlXy-TSyd}+$W z*}rB(86)X{|!H^;dK z9kR#)MiH@Npjlk%#kV1t5RB2gra`3?{9)J>9Bv0giNRjy>v|Q3Y7V$&8d? zTA0#?&#$VO8al8^mYiEFGXC@VVsuo9NOa(0#Q1po)6?w^EDL5*N=tw<48E{@oe+!V z<8&PI|Lb!zmyT-adb^NC-f}$>2JPZMxo)U@klhU1f3>;i{Q#P~M-PJ{81N9}v%1q; zD6b9Jz)KP(`vNkDC4jxic`;eyi>di!SNU`yODVDG@JX~4T#iyS@b%~Ww!?N{Bgyc$ zQ4>Z9Cy-~+#zP8l37*QxZvI%Z(s88#CTz!isym#gs+8et^z2Hwc%?|Ve4v~&U=nm) zFChl*M7afmj-~;-DMai`K9N3XCo!z5iH# zbLZEgYhMPlT-%9C?!cB8^ZX9BZp)#hbV#`AyR6nCsb5qN402+<5`)P3y zA__eU&5gh?VrI)xjZ9y`nF~xx;=qOdX^FRn=3yQjetcJW-3<}xtxGiZPlu6*vh()= z$H&iBBBdFKw+_VwIYY~re3{vj*O6XNU))dz=2y;DP2K{G3v%4x=!I29h?n;+LtTq> zvysz#*f3d5{}43W&@ITweo;#Y9upy@j4Oz?hLolfHy7=|Sy$(oJMS{e0TFjHk<$be z011g;oca_|B-}ydGysVoYX&i#6EUb}z;)E&#V;s^6ob&QzaPtdt4SbT>?NK;gIkTl z#7f(Y73i3^fZYad)$Db@CIMW}M-_Fa@FXvocSOt%A$FYn)P*4JpcB?&>utW9Uj_gj z7fGyH2Sxh~>YoHPl2Pn}*q67SUcLsTR<#rM(5S!Ehkc|*hG z6cr?}54i?T91JnW&{Clqgom$CGdD?~?9_k_&LJ~OeMb&)KRn>2A#@UhGK{T|gm>@Y-$Gvj~v=I_RB8a7jIfn-# z?zI`sPYIyhJ+jGilcf^GRK9zXHN~(I&r-B6NxH;{{aHYwVQNS82=SU?7PY&G=LfNNh5tj>d^2z8e;FuFI#W+#9vDI4`$UHv-5@@oi57{{3Zn#XaDMNrcIw}?DH=g3JP}89 zUV=z-fR;b^hO+Ya@u3m2_f7ThSgjPHn@KdCzoJw~mtVhpC+vs>%X{p=2HA**9l388 zd+UZ4koE9=hI^LW=59m!;WC5>-@Dw`RiLXh25aElq`pR2{dbTYM?5 zH61j85Wu==(1#Ay6}~j?Xs}Y^>*#)It&KDMMi1-(iQ zi#r1g_Zy3m{-NvbO0%c00ZV8)RJ7HOnuf=wjVaC82Jer zyvf$Fqu0{zhZJnej#wA7*1{^Y64NT7( zbS7)no7eza`_eZ1dXnyfHX9k+=FB#|U(}bPlpcH7Z~b;7O+7Pba>%6%4H{Jq#ga;? z9Jy;9&wbzaJ`^W(7%)>Ldg!C>W;2X$*U~uL)}E$vkY;6rayqKNPKDkn;<3j^wzhW{ z>9#gSGItG}$baD6yTSLZD}&hiBp%8d4X8*u3MH%A1HEK@a@l02`=HsgQJ;$UA;K}N zh1_YIKK$n=RgGl=w8n}JOSm+BP?9M=I9OM)x~t zKQB_tgop)aWHuoio7AG#64ZQCDYcfT#f$tsy+ixy(!*mQ!L=~9_rYxIx46A4oY?j) z?qyQ3S%PM}z0>9q#gQdG8f@CsPI6dp} zaW;7)Ikg#nGg3w0KIA=2;Y*i}`IE_)O+S+eEtEa5Z#wfP91suMP8QRw08BYW{i0o} zVQt3Qt{W^Xrcynb_Z(+vu;lCv-$dVV(iB<1xOni2%&nk?NX~Wz;l{gaLLo0F$gTRC z43nOY-3Mr~#XY)NhSCeYBzu|EBC1|MVQvSFklGB-JTFeespDL-3DT01@6BBM7p_g> zf76J*<&EIg>1xguobr74;qjeA4WrJ=$A&N7CC*ET?H}dYe&iI#v+zV6m<=6_x{0Mw zoP9s8p1qlR3#fGkW2%cg23ON>ue0b#Q;cE zvxa4MmpYnRR}Dn*c=tXaUK5lsRl+F+KG(tjHaz;Q7=y)i3<96iRqk|;)33yJJ6@7u zg)(mKCjM!sggI_Bi`DQ7E~jc-#lU8OB@=UoK)((J+-ZfTaSq!i2WvU!75qN`7Y(ZV zff$i&p(lz?dv3lhq30NWzap{rY86CzeZFUVA0FMP$?;8p^eN+J?53)T!rklpPE$Aa z7@owBnNPO0cT;!_CQr5{lyG?4w>IrP++Mc*9}{0oU-Ou@*i=G95uumf#`r4n2{ByX zSm3)_%SL04xxI_$Q;s=o;xMTc>#{^8G{9^qJ@t%WH?N7?IPlH;SzhA#$r*;I%x@VE z7nAUt0~kZyV@vLLwQOq*Rr4*n`D zMFF3!tndP7V5&2CWAV1hpnpMS*4Zv?l_il*)Bg5X*WXJ^37)fA9ycH0+gY%rDd=oc z@NzyCo~ls^vYpORws*^{LR8sN=;&uxrp%X@l3+0H7#|#K$ERZJSwJ8!AB%WLA2nc9 z)g5hU%wbB4Tapte6HBj6uerjy4`ePF(iiz0wap+eh*#(PCxcQCTWxPks--NUoRQH8 zB8zLk(64yh>bX}mL47xs_*#YUVr4<;{*G74y6sYMAZ6qoQmYjb0nr_!l?kdCSmB!# zyB*SWbNJ+OIUH-=YCK)Ix+E7#mTqU-ik=^B>+k(;R!?*12#dGReGltcP91JFx*s^q z|LkW|!nGy=cbSN|qO}a=IBf8Q?3rhyyUTt~biJwi+9m$;wN#7A6J4vWx0TYjMp|Pg z%qN#bmj4=!aHeR^UV5uSs( zBKCGp3`KeDBnO^1+Clr1jDMP~T58XQL1@j?3~R_XnWJ*CSYp#eR*$@Pbjs7w+W2Df zn^EVB2SX+$J(O^fJfTP`M?6PS<>N2D_Ug&vu^?Dx{j3`$23?F+Q3|tUhZy`*gNe&z zo9^FNbzP0px>lZ7nkz8yLDn`BTlD?ptRo-k!>_>0@miKsmFJK!#uEzKGZIp`un3Yp zG&tZh3^4sxfcthfOA3Xt7oK#|C11-esw--6QTWhPY}By$@Dr_Grhc~l;PXc`zv+vi#GLOPeB0j^{i&AzO|=O+NY}npXd7~jqh?S zDTEhZ>(dY`nOYp?BA40m8LU1dBn`Yw?O~{_GZNHm0sLMmjpLnFx(BzE<(8RM_4|jb zYk=z;-7TQVB;R0h2d|5q4sYr5=TQ&W7R)BoFDQL$=$#BMB2UUbr8CsnmYI@xC*J6+ zVSjEFn0E(XvKx#udbt=n?i6!Tf%7vd_(>Hlc9ru4&$1P^yl^r-W?74J_x|HGtyDJ~ zrl_K3qqg-d7?7%#8ZR4YCobb6~8fbIIt`QQ3M$ z)qps`)gVw2cczQclJ;0}E$-K0?1pyrAl6WXv^eJ8*0eX#4mUVCwz7JRWEg&7aMX^e zG0qQuRIg8`c`V$-vSC*iXpj&7br)Ly-JMx?O5=)-*$c)LO^^FZ^QYwRo?S4#jOh6$ zVDjyR6Aco;S#<3?nMm&3?oqj7JXpMV_}Ku89S$n@7QDk=S%L#>sWr>OpVECGx}FXf z1y6o1&DTkzvmmvjGMe-4l3ADhcJr~ZWx3A}u*k0UMCJ-W4bQnhO|iN=2WUOW^z+lg zG9H%gtALzv+3a?!`nYv(vgVVG*bd<4CckE%$s|56 z$;iZS9Hhf=>p%5hb8qCj*T*`WMxe=@{rZ9KDoc`8dcH})I?al-!r0gexqB*{9!m8) zm<*&X90e)9x(X^=`s#Y&--z7FKdDMs$}+9uB$#!O?bV&SdcJ{D;rkpcb0|FDl!&SZyi1TvF zePb91Omb@gn%i6&g_4y;+Rl0)wTFSsla;;Bu3iG$Dgje_63waQ}P;^ zWHqRgom1mzEBXf)pl*-aVAV} zA`q-xsI?HPbRVmdQDQ9){IrB806v7ZaudMJN{zb&Fw;TjQPfq@y#mH}6fqbSHWnkG zkFfppWIw!#a0JAOcO{@La(dDDouW}WuywKlfc7M+a3cX!3*6UwX8ie4)%>4iWDBZB zC0J+iSan~Xs^DP&Vs8yQd~kUq4o)l){gAphBX8Et#dHPnf(>W|a~!qRL`Qea`AcEe zh`m~VfL9G@Ty9+{d#!jb9>9A6Mb_iicZJP4M)r`iYHaln_$2@gxVH+=XL<=NH#He$ z5`01Vdi-U25f{l+eqW#&Q|Gz?4O1JnlD_9*&gO-XZF>^T=NbnNjKqs~IQ?26WSP^f z&O>{acu-V!Z`wt30w=G~>03F#g)(QaeqeMaJI#RAkXmhXA?nGK0KTuw4@Ih0QZV)` zsw}jV<_hEYz=cyGhDHjMePv;*Fi&Cy(qe1ZfpcmTKzrnPm#z#MsJ7h_3uF?W{q~k`!P=t=e6U-BU zAi~6#kn0hwC1=TUK0h#BwRMGdnVuYVI6h5JZYJ(cI9V8^Zgf~QY5zq#{z z?#d=$)62uiS2p+r=FxuJ1efVpmO`K@hdzNW#{{P%PcDZG;}a~_rp$jk5cY{u;!@r` zZ4Xh7HK(H*S+$_Sw(Yp%>4Q>P!NVr_(_Z;U;t0M~PD`#AR}(};Di_0t2GTh;E`^@B zdlrM#&9BeE)FDvS(`NSD#J4c8OEfQM^z7Z}FN6T>)gk~KcFjBwPSs9>7Tl)!#G3*R z(Y1n~yFf;pNy1t^v6J5(=d1%_o# z6cO3(u7CW!k&Iz;@XXQA@Xf0L4EGFzWNxsWGR1Xaqg7IkrXA`(0oB>rc#5~`O6j+K zd@6UV10P|-bDQ#ivYnS&&ND0AGz&| zC8PE58(#Yi*Zj#%*Ac0Z4OVCiH0rXZ(#A1G*8{zK+bAk zTv=rZUv7afvFR>*h4+&W>P$9@Y7x6KC(<4_4s&EGk9_6Fr&l&W4;EI@9|9@24QS?O zM;}st{J6O+SuUMEbDv(Q;L(Oz)>Re@QylUc4Q=)8kql}C<#ypcqQMzN{t^Y+`2kGM zRjZyMUtr#t$f$96Ul)wr>miv|q*MjURCmX;^JVknEIuQHBrXz5s@*K>oOh(m$Qf#= zKn6#&X6pLU<_s_&sq$%fUrhF?qQl~5yBr=CMca!zfbZT*e3P`ju0FYBBzXCa&$FUV zEy4b%qlx&=%(LFE-0HTWAEHZ)B%s}#wb3?>qZ%b8MSugxBRixdYF9z&CGUxY5k_6} zW>3TII^7i{ZMz3PIdw}3kdI~vNFdiQHt5KnJ>HRN>Fp~2J{P=rNf49Xa<~nwbBDV= z-I5NlF?@&+gGQNIMmi%c7>9PcP)BSL(G`bE7eedpBq{%xbYdp!%|iE6Hoy!{>IaB% zM*T{U`dDRLUT1JL-Gy#?Y<+to&aqwxPy5Z6AOp{5x?e~<&v*VU|I8}cCJd8l6uEkk zLJovWjmR*C!WRstS7YepuPiM*H1G%kpJq|Kx9(U7su$eSRm1-VPrsSFf;^S0UD*Q{ zX5_GkpOT%V)p6#EDOJvxH&+9+qILl1uZNvr-6bIvq>M4Ss$*4Vq@VF9fX>`5$^kU8 z1iLaB=g4$ocx(je4>lxZkYzYaRkrlA3hF!Vc#BG6XH%q6?xG1W@o(u=L&np742lm! zIc$KBlM_g3LieGXq47FO++$Bb(@*|`zr28|G=Boy2Qg-(G7!k=meX#pmG!iVXtj_# zJP*_E!$qBYN}c$uv;R<{M`v^)uTT*2T9#T^u&x;UY1?2#PRCy}<35A`Vz_3eu;p^S8>AfV$l0C<9{iz*6!L@GUu}%b@0`5< z?cLIiVpkt9W+sgs=MfnpH-zs9}guQLOiyb4h}RGjv$~=$kU9Q)7g(v@BsL`}w2F%mMc%SXa5n z-Exk;gFM2!alBukH}}gxVJ5?w_OGTiVjyVXmO)dMlT^{No#1lq`^ovrmbq{He`n zpIb2>ETs69^{69GFGcH&VYbSkXdzu6={S|3q2~CtU6f@Yw|mTtcr}xtjC|Y9sc%hj zljCeSogaL=R^^F%<#%%HLN7XG{W9Y#2lwu`Nw+)qsgP|l65*V(DIRjq576XRQWZ1n zO2$1>{_ck0X)KB0j}m+D^oZ6Y6RO1y()iScb2(U|UO;BAEv+u*;UkiZ4%IRD4=(D0 zl;44@xh+v=&c1-Q_7zdEX@KuBn`1#0&h@S5z~FSe@MS zJ(AFlA=%68T?mTsL}@((ANah32rovdvgAaNvrBf?aRY|ca20)lXzXk9%#E9+IdOD) zUDo9N8j1V^~cd^WT z?5*MM4InocZod1DDw@OmK32s2q?;BqT9FsMvSaA9$0r%$Z#l=XGbw$0Grjlq`Uv%e zQ#S0}uODCJ=BlAIo>AQM(cz7x<9DmZnCAgnPN;^157VXEe-T^=%dN!WKq`=X#THHv@Z)BD0q&hz+hZT0%K%C=L6Yg)uuUr9&EUK64i zzaKjKZ{5i+KsnyN#ygjBa$fq~bx8jJ0vJ)>V2;H!)_12Xr+EYZ8fi!K`Cm9p~J-YM6*aD^Ze8m$|&=>ngeaL z$6t9KChl(GsX9avI_*Bs_tH;&L|D+!hca%p{^2QJFrJT!-1}asCN^hBdqQ4@jr0#H zqn47?;HZZ2wKCtp_KT_gwFLA|4rpn}& zdevnevE)s|BWAXD!^JRl?W^RzIJ)!Wssn{OI2yO2cQN}07xu+hSA8`xmAS@hXus== zJlf(__rRC18~Ky!v27n6yPQyxc@VOdYbBys@Kom7UiCtb0pzN{0kU`}C-n*VKT%`b zYh@v+IR%%EoEK$5xP?13jV?YN;ur!V3vAq}FL3=ZO>m`6uxUH_nSu`dj;d@%omJ&D4T(_wV&khq@ z{^Qks=g*C|4?psvmG;`tvERQhskm=_*ReylVnc{@ashqjM`0+HU0CJ!&j}s$Y~K+s zqI8e(%MKo& z8w-uBfNWTduZC6P?RLo4YxE#D_i3QY+SJ~rq+|{5@iBce4;^Vl<8$BD=wg}I+stbHH8!bhmyf1d z*#^?oNhkn7Je~nAtbg;U{~!m{FmqO=Bz$*hzd1b0O;Ucyxy-5sx7Ys=$f)K;uYP#Z zFYlYS)8@{^1dNz`(KePd+B|R&-K~1$ALPsxD=4brw8|1+9i`bFe2N9XSq}qg|Mvsg5G|6o z%6|QtQTs~Yya-x^CV#sKj~96bwR|K-goOf6@z^-Gla zZK=W@3jKz+EJn6m&=1^A^9um18_;HNpI-J{fj5lIjOdpK)9`fq^PZC2JaVx@9^MA_o=u5m;rQlAHvzT&x?pgUXR@2|A7j_sJQ#=vv-}a zH>N%nxok0lrx|TxV8K__jXGPi6mmbP`!nEi|Jw!}YM6hs@m+ey`f;NYiUG~YZEtzV z!#sbXRfakdzyD)|-}luRw`O=+jy|eC?`<6|R62l^j{hhWOIj&XsiP97v_kKfgVxr~ z2<1TKYVeGCnWM3VKq1po)ITZ`$%W_9bttLb(eo3T z{&DF(^(wQ*yw~`2Cm?Kgh*AcK%Gh;LHDw2IqF(-f@y2H=1VFia`+x!9RxqY?9R#0! z)xHRc1J;ubF!1Ed1NE$?kSU$AJwotM|LP}zWtFW#!T|PXk)=RQYrz~d?8w3sq z1j`{Pc>cC(+Y|Kto-k_NW70r;eWtxJ5NyDLVs&4-6nbxhme*4r^>IS1fyMKI03^p` z`m|*scfKr&UrFGPg(CZC7BH#Uo+M zT?fQ5gU>Hng^80*EQ3l;Xy&|*Scl3ekiv?8i<}j|-1sQ^sauxC(t93~X$t*X!1Gns zf(Y>jbWXaQ{x@qh=J{sX$!`eZ5xTVbikhSEr@};NHyH9+Aa+qcz>;pXIltG(6pFf6 z5xV=})X4pugcZQZW|kEb{Q;aUo(_HwRrf9eF3ZS|ixl9KV0s`~?6;}=-bN}o2sF;e z#RdKCAq{7X<_2i461uIimvT{Gp&L6s8yFDz zAMK}xk5FO7w3Ohj; zb#d_)Vs~oY#>OhqM<~jyvZjHM_0gB;av~ku?jndsTp-q^m&q9)A*53dc*?6iIB0() z@w!2Im>yx}$LHlUQI66);{BH+`WN&auz>(S<$HJ|=sDhV^vRnur#|1Y0X#MDH+cjz zAWd30m&rm>X4MQfU;t)CGcv zACL=HuOV8!Gx8iam2!i1n*lN(1w3T30}v=@C!~Y_m4n6o$a22EK|E8cdd~U-{;@%~D1;#}V4h0<}=uV8mo>%77hLGN*^LcJI z3Pp}+`NCoK0)-+H*Y{uC2B8QAr>M#~2u0rhuY@9rTVX=|$s^Z1AOh##_9p@t-U`aH z7Xmk>d6>4sJlp=Wn?p32^OyXnwgXO2fAqY-8NMn3qNo;zD>{h~f(YQdO)ff!PnaShl z$u^V@fsy4ql~k(~iQ{g1BGLX(b36CGE9)rnrP?x;*-C^vu80Z(Hyw8m1l~G##+JvC1U6l22WBBWUx^ z*+wqXc!P^nL8mU(@K>%E1m-e-lj|)Peak&$iD|k8)#7~()Ovq%zSDZ`AEY0m@;9USpTUt!jIO`~P-pr@ z&j+aA(b-yRo~l3|#+!=seB(bXy)6DhJziW_fdlUNv`7p*$I|tM!-*@SIwhtX5h6Xo z{KG3cf1sO<)~eoKsD8yU8!Hl_xN24y;r|~1SL7yVd%~am|8CN+zs>SES6QyF9G5@- z2b5tyFTH#mcwzr3Pd^81`xuO$Wxb$;BeP+i>Oo(45NWWf9+5(z_ll<9YJXdaMUO8m z@5X;VCH@8~QIv$a5V6%IcAd9oecIOS1>Ili9AML1I12@|OQYk|V2*E$uD*Z!;S@|D z5A>!j;QAkL$|Pd2hqoD!wcr6^RrAwD{vCSM<{cP(hel5IMRoMHRkIl{&VNB$I7tV$ zkH)-60#~*3bF~BE0D(mv^h>zMUnR98=lsvF+9(sfxMvGwGS7_=@pl39?MEe)6nvAs zHUHoOtcRRECoT%?<*FGwg7X<(gW*->0_wB zQH9KW>d5v6xm=pMEvVA?ZFKE^N*rXVEF$P&c$4(+p5%YST>qD0u5b}JH8^Mrj{L#a z$f7mhp|dLDtjY4!P>z`>o|L$^V#81{GW1vk!+3nwQ}xdRtJEsm+W?*Y6u(5@fT@9# zS^l@v?7x5`aB9+o0ITQxTyTpibQ5EKV-BE)!1kYkySPy##q=YhrtvY)h8ROu3i8^K zUOxq1JKc~dJ!V93<2t-JFC(94dXE?^KELv>UGJ`+0hsE9vNYQ65SX=vLQ+}*Tk-F% z1uB;+F8aYG73#yF1&{-xtRnF;MKd)YwclqheM^dF5lbpKYK$sg6kP_mXzS%q%-ctI zJPd$d=%oEz;l(O74UZ2h$hkooBo|<(QBSmII%X)f8?IW-= zUfZ_+FvVeJ3;7>M_e10PU_fzc1&J{AUtF~i`sd#aP~cZU1>c&ufClM5h5}x`0x$DI zAEtkKF%n)Bc;)h!yZxVGgDd{R1pmkX!xUm=Yt*=L@F$Nvt z=l^Td`D+kEea$ZUr-qnlL}1UR!Ls}%$p6P^!l4c_%}*u&wN3gK6#kFPV17Od2V8{j z)D@)a{pXjifr75~|8X#E zyL-=1oidgYQOy;JR+O>kwRCo~0oY^zwW~K0eqS>(`_Emy*$=4i1%kNc1S#vG&2X6HE2l{|7U34`OYP2(f3Q4n2; zfl|Y)%klSWn{OPl(jFX((3Q@30!efW*v;GRmLBKqO4my^+;sWNAp@SI2GsYX?y$Uv z2@Rnbiy>s;$x_g*uLU^1S7xhzU0DFtTU;RP#S-J~X-XG;H{3*DGl<4&9(ht%0dCYb zk!RKH1xNrL?!L7FTUQH+QCk2lwEw>OOW8oz z%keTJ|dEispJPq8Z#Yei{1r3^=5Wi@v!R5?t;MbX} zHh$UpkI5^~+;~C%>~cr?)l1sx5Pke+cH$)=f?a+sonS27?cE2+^xXq9iz>l!&<4CG z^96AO1I8bb%vBVWFi)Uib1&UfBXU1oRJmT35#pp=$QS_NtK3b}s*c9BmX^N{BSzTM z9o!G%X&X>z-x39Sn|t{(U~+l3g@--AGA{s3I!L}VuQXO5(SLal4>KuEKK+T!0m1#& zNIheVL`z7*TRw9Y1gR%LML=HdO5W^l7eCSEFbH&eOGHa~X9bRi;&?(Kjxk=iY@BY%yb}ntp0+gH9e479531ya#xXS3yzR_01uV*wkp#Ov^Y# zMN3NWoWyPPA0aGtMOP0Sy>O5r@R5v6?fG264$|bYK{?+Kn2*O1S#GN*A!vQBq$T^@ zbnF7y;gV`-tCtRRe{nxbt3ZzQ25zK#lp|{GUaRbF_f0!M)8{>1X7%rl@9vJzRL0JT zwp?ytvprZoqn~9nsqMwvWY*Q68zwp2Hy{O_OXC2n=m?b@zM)332u1ut@9`wCrEwqn zLK#)pf<$6Ihya<`(C+Pkz{$srZk~e=M9O}HC1b&dBXM@z`Osqw?{Y*9CQ!x3t$udb zOY--n)>d}=d(j@R{`yXGYM9v&YJAmNuq`_tBk}HwHoK-NC|tmen#YQ_+(vpj2B^?a z17X*lD!oXv&(2!Qs-C>I7_DDPn~6}+WiBpg*nnd%1m$-F>Vq%}n71hMg}U(8MMqE? z*~9k(S+^N*Z>Qg;_&tTVvsorU<8qXA1z9*j6#8y^Y$Srn!`vp(GFlz z0pxpk(DUo#3qLbhQn1_r^{a$t@@HxuoB=jwk@N^snbJFzfFfZNQB>oB8`={JY{TVC zv~uLTL9d|Uf&=(*))#quLRZJ^m9A(V@`#iI(CpL)i|_P+qWuye;P12)7AmFQcrg*i zT@F%|LK^t?kW1PQPSVMGOizGd=MCpt>U{<#=>^Y?YY@inj8#-b<$h$8s){1Tc+&Isl zNWpOjCBtShkN`+$QM;Nm*+}2s+O9*Db;f-aHBc!Q)mrfaB=AC2xz(Y8i~MJCg9Y=? z+d!Xkx4_x}>_ZcA5fD!(F7Awi0R_s>G1N=Ka$C?Pzsoob+>&;u&IA=r&1SGl#Y2mbERbf_G^1E%#-=0fRTNq z_I9u@m>=#x|l|@L#c7C2FZO#WK;~HVb73TMQ%f*MC4L2v4uo#xy6)7qZzb~ zM7l`rO^7KK!_MxaB#M+>gtnfKd~^HXv_f1saf@V`mF;gP6tI-j z;-ySmYU2vRW8Vjr)3z=PfiF(}4H)%K!>Wsc$bipI>RNL?&jX=!YhO^$TEQhGT=^N0 zy5Zd+Fy}elT&)!X(^%-C{DxDA(3P3}==Ju}VtSp{9ht%ouZ&N5D>q~;P_rf9m=EFc zju?fC#SnZpWbOm7K5j|0IsJ3x{xfuWLP=Y4rIu4!G*gvMRVdvIXO) z*bnOnW3`q4A;A?Lf`C11kS6jyC5T>x)r|l*_o)l7Pdoe^oY@$kJNXyw>?bgnza;#O z+(-)PGF(?nWff*if<2#BKKRaJA?ng5wNJ?MEfCERl@%8f^;LJzR9Vv>od$mH;Q@> z;7~u-(Rqm2j%d|YZ*PP{E3meKAxG{wp&3ptZ#~8^D%|t1$(fn9osONHR7L zqNhXYgk!d3cn(Gd_X;3i%dyh*%3me|v851T*2HCyhKP2BncnkvlrEhL07Z+BZkGJF z8G`NLL(GrW2|m14C{3?u)vMdL{psz~F9b52tn&l>nse_hYId@0#MX~qQ+51s4@-() zz^%G)EH+_RiN~J^CO>0qt-NP6wY#g2Bjv}p@~|e1RyL`GkOXJAiGc16dZfV+r5j!I zB@$%XtvZ04(nas59x2;+SG=xZt4H$J6zfiNzzVWpzpt{Wn1zO?*WhsVkl{0z4J@O;9twlGz~*_7%)n|jLI%+w@j z<^+iIz6sMLa8qoC+??^-YHV5F&?<2o8wjc!uw*sEG)dcKrf4m;Y;=ayMYJmlHRxBV z!+RvZH=NT5LlDo@x zUROTzRPIY4YU+i3zP4ob)b`HqctM5jh?AKW@W5`>e2fcAC#t=!1m8Pymp-LbKianM zs9)Mma25a#CIIR1>K_k_(krgYYV5->RDWJ|E^k1Bb0dwt^!;hJKoWU(a0XWAJ7E}o zbKB4ZPJx283RdL^k4&AAp0AFLGBuKRtph_vFd!=2~6k^#N3V!tIJQinC3&foWyRXKYexU$uGg!+a$r}tMq?;CE*Iq!bjyby3 zZcdm_y%+ZQ1}?x9!D7G0Sd=N7Qp@(cE4TJnQ&Z6{0x0K2u@WzQ&l(gUb26j4&YNRF z%~ zBtphIB7#{X5v+LepcGxQw;N3Ezvl59ph_~LyDp0`FIpEWzXbC=72gwHWd-o2FpCGY zG0W5l-rZvObs~j=!fV`YUf$Xa!$vgswNQ*Tivv*kpcTf{rw;Wc3!ZAh=v$;Q)J z5}eUD)jDPv%T?8=QcbFFxcR~gPcd9c6NKdfjBh_ENTsU%*=%G%^*WKhsMMO>gXNtp zH>1+mZ+fSpiWzJb*Q&Vb2zQ1Kx-*8=CIdw2?G3V;z8J=;YG-#9ky~+6QaN)GkBfU* zyqsD(NEKGwGWHKbGqxdAl{&*%G=Vmg%C`Xi~vifBCl z%Vmb@%@$1jGTa|gkd4p7^+Ve4OF`Lq1Dt&9S5wumBZ<;HWdLniwI-;2oq@hx$6)vZ ziV8pG`>^dfW!M&b0^nPJUkvmTs>c8L8}_~iO3C)DX8;rWp(G;$niiatCX-eW2tr(& zp{4}HwX$CSBHcS-P4T+YiZE8J8yx^F5tBoKYQ9eyemHxXKxa50{@=5wrh#foHNh7A0~5obx7~sq-pd3pft%o~s;F{L7s$^@T^;qkb0M}? z8-52=nNlY(@zO{H-^u&U5E1vmht|CwfNVn#=y6)exNusXxplW=SA}6+GM4uXt6K_g zyIm$Fw(7&87?86#JglKuCp^mbBbJ~9t`_qw$H5C_h-$@P3a=9cip>%7FT7cP*QY<~ zK-iSycEw5n^fUtbS`vzICDe6+V$h+1!>bcDn#oiZ9IfU#WTRSOEUE*7H`gX0F}r$V zzY_Ta3~?@+F?m%g1e%ih@pT~fl+-ga4k(8gPlM!Y_iHJ;pSRwJB?rnvVc z7^4FFUW1Kna$iitie;?Lg~A7y*q)FSW6nf23bSG%h(PilcI`3@`Fw5`9O$}*FO*SC zf&ep<1*Qj0l**DPz&>h4T|{qyQ0PnB=$@>532TWiM3u*8aXg@@^{}u2+41EM7=GnZ zc76y0wEx6;Btdfq2O?VYxu=t64M}f50I6cR*{frdD?|8RlsijxZh9?9A0pz` z!M%aFM!0uS`b&H{hod7!?)a502NbVn;QQ@;eOYt|0FhAbqTI|E5J~#d*6X4npNeP( zbjjYC&)~H>DE9&~XagifY+P9MoAdP4*kzv-WL9}?qHmc_GgU9w>rY zh%9$pyJ84Re0TNjDK60}4RSPc@Fz zyao7@K5}>+H1jF(OmldU!w(57DbNjxO?P~Zv_8K+-BdRy73DmBIG;1ik~ZotvBtaZjE{ys?#V2nFVsaM9R{Ys~e?G+LrsS<(!n|6;j z+3Pmn<2|&%FE`_ub7+l$ZpgSC3A5ZuNJr?kJpd>PxqUvQw!m=IlGh+msdE95}t%lzvYOql$EMRkdmik9`1 zj`47RR9U?`m<5q4ic@EI7-(^E_6&Y+pP=#hj$2-tJ=earM(AsdrT9~XpI5O^ZOZV?F7KxUGeg?SBP96j!3LPZ!)pK^0 zJ52~MBd-Vi!Lq}#Yb>u(Vm6HT9<6en1pU6_q$dkOl;F>BkS)&zc25bTa}-%+X8LJv zqBdSb4iF45h5?qtU2w`pv z4z%P5Lx9<-gH{c`Pfa|LzRVT5!yd2{%XkAb<*He5B9uiRYyYwn z_#*Ftb?qeleR)tbZ{WjP9;|csKLNM&lA1j}8-RxS(^uVA)SP##b(YJ2CT6{FeSML? zgeX+)h~-i4Yeg~8z{gX|KhN;AArd1O1=-{Efm&pBdCaOmge&m%GWUAuzcqf~UbsO< z0@ZgaKWDQLDp3BHKYTz+Di0iArqDil0FS)Jf=RMfq=`y?8S7#HOAx-tWcewww6W#{S zF!p4dfON_sGXT`})>gcl^o@%fVm#t$uhdel*6!EQ7yX1xD6o=NL zavD9==-xx2WX2tN(1MzndNLDmH*f298^iu{Zhfx~(MfA~bQSE0*=%kFRM2(Yxl$n4YMLJjLqmD(>K-s>!uojaK3taN4ThJ+%+?yS zBFUr3z*-|l!M(V6=HUd_+{Q&-v$_}K(y}e%lWrAf($dwkml1pMa8b*LwDdUM-bcI* zL2UXH>wR2_;q?e9ABh8;^%5YS!19=L(C1m0Mv3yd(X-pRBa|rJ@RNIdhJp-4yP%2r zXoV*4SJMpax|Ou`HRJQ5_oG<w<-|JGKa@bai zkVg#0q(9n*x_m%=MBw05&&CDQt-_zq9&qT zp?NU^cTDQi&d5m(8g>0rfrOeU-2`h|HVE7KCogr}#BPFTowM96@~A0rTy{3S7`Uim zCn6%+Zn>FEdouK@NP>g%ijDx^D<)FVP7R|LZIg%rkOZ;0+Fei7j0iWRS^8(6-4|FK zd3OK^sa0_NG^AZ;zq4-vgxyok)u0$mL0wbXRXcejW>kc(A?>tR4KT{;eR^W!pPSea zjLw;WYw#T9X_}WNMssV~R=YzAwIO5FRlf4o82IZ&`Xeps6pPcI0C~tgUH0C83^j4# zoH8*ww_I>!Y*1*Bl-&KpAft~Hmtx2g9M=u!0SEGZqJ@|8Fts0$ND=-c@{c|Uk^9pm z<9tX%r+kQ!%Zb<~zI*R_`Q&u0Xgkw2<)*2miJI`#FSBTC1g-F#x)3(@>e`|y1K+H5>{2@|#{?_P%mlRv*tmP3L z|I@!;nFFuL?4QGiqgcYW_~n=XPAtEK2;bl@KRhG$Fv3NEoBHuca8O{Mq#sglNew in v5.2.0 From 6c8d40b01e1941258b53bb46e2fffcc7a6c94795 Mon Sep 17 00:00:00 2001 From: qiancai Date: Fri, 24 Oct 2025 14:52:35 +0800 Subject: [PATCH 08/17] Fix relative link in follower-read documentation --- follower-read.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/follower-read.md b/follower-read.md index 77bf5ba3a40f4..62339bdfc739a 100644 --- a/follower-read.md +++ b/follower-read.md @@ -7,7 +7,7 @@ summary: This document describes the use and implementation of Follower Read. In TiDB, to ensure high availability and data security, TiKV stores multiple replicas for each Region, one of which is the leader and the others are followers. By default, all read and write requests are processed by the leader. The Follower Read feature enables TiDB to read data from follower replicas of a Region while maintaining strong consistency, thereby reducing the read workload on the leader and improving the overall read throughput of the cluster. -When performing Follower Read, TiDB selects an appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to identify local replicas: if the `zone` label of a TiDB node is the same as that of the target TiKV node, TiDB considers the replica as a local replica. For more information, see [Schedule replicas by topology labels](schedule-replicas-by-topology-labels.md). +When performing Follower Read, TiDB selects an appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to identify local replicas: if the `zone` label of a TiDB node is the same as that of the target TiKV node, TiDB considers the replica as a local replica. For more information, see [Schedule replicas by topology labels](/schedule-replicas-by-topology-labels.md). By enabling followers to handle read requests, Follower Read helps to achieve the following goals: From 72867e4120d93a3efc473e2a50b632c48a34e632 Mon Sep 17 00:00:00 2001 From: qiancai Date: Mon, 27 Oct 2025 14:25:23 +0800 Subject: [PATCH 09/17] Update follower-read.md --- follower-read.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/follower-read.md b/follower-read.md index 62339bdfc739a..9c623e52e2b86 100644 --- a/follower-read.md +++ b/follower-read.md @@ -7,7 +7,17 @@ summary: This document describes the use and implementation of Follower Read. In TiDB, to ensure high availability and data security, TiKV stores multiple replicas for each Region, one of which is the leader and the others are followers. By default, all read and write requests are processed by the leader. The Follower Read feature enables TiDB to read data from follower replicas of a Region while maintaining strong consistency, thereby reducing the read workload on the leader and improving the overall read throughput of the cluster. -When performing Follower Read, TiDB selects an appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to identify local replicas: if the `zone` label of a TiDB node is the same as that of the target TiKV node, TiDB considers the replica as a local replica. For more information, see [Schedule replicas by topology labels](/schedule-replicas-by-topology-labels.md). + + +When performing Follower Read, TiDB selects an appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to identify local replicas: if the `zone` label of a TiDB node is the same as that of the target TiKV node, TiDB considers the replica as a local replica. For more information, see [Schedule Replicas by Topology Labels](/schedule-replicas-by-topology-labels.md). + + + + + +When performing Follower Read, TiDB selects an appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to identify local replicas: if the `zone` label of a TiDB node is the same as that of the target TiKV node, TiDB considers the replica as a local replica. For more information, see [Schedule Replicas by Topology Labels](https://docs.pingcap.com/tidb/stable/schedule-replicas-by-topology-labels). + + By enabling followers to handle read requests, Follower Read helps to achieve the following goals: @@ -86,8 +96,22 @@ If you want to use a more precise read replica selection policy, refer to the fu ## Basic monitoring +> **Note** +> +> This section is only applicable to TiDB Self-Managed. + + + You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4 and v9.0.0)](/grafana-tidb-dashboard.md#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. + + + + +You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4 and v9.0.0)](https://docs.pingcap.com/tidb/stable/grafana-tidb-dashboard#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. + + + ## Implementation mechanism Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers could not handle read requests, and they only receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover. From 4f2e79e3b22d028aa30a81e6e6e467f1f5e6b6c0 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Tue, 28 Oct 2025 13:41:41 +0800 Subject: [PATCH 10/17] Update follower-read.md Co-authored-by: you06 --- follower-read.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/follower-read.md b/follower-read.md index 9c623e52e2b86..fef764901c028 100644 --- a/follower-read.md +++ b/follower-read.md @@ -15,7 +15,7 @@ When performing Follower Read, TiDB selects an appropriate replica based on the -When performing Follower Read, TiDB selects an appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to identify local replicas: if the `zone` label of a TiDB node is the same as that of the target TiKV node, TiDB considers the replica as a local replica. For more information, see [Schedule Replicas by Topology Labels](https://docs.pingcap.com/tidb/stable/schedule-replicas-by-topology-labels). +When performing Follower Read, TiDB selects an appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to identify local replicas: if the `zone` label of a TiDB node is the same as that of the target TiKV node, TiDB considers the replica as a local replica. The `zone` label is set automatically in TiDB Cloud. From b0bcaf4e94addf5c973ed5bb2da20c1c90c58a6c Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Tue, 28 Oct 2025 13:42:52 +0800 Subject: [PATCH 11/17] Update follower-read.md --- follower-read.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/follower-read.md b/follower-read.md index fef764901c028..c50ca101e8d52 100644 --- a/follower-read.md +++ b/follower-read.md @@ -114,7 +114,7 @@ You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New i ## Implementation mechanism -Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers could not handle read requests, and they only receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover. +Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers cannot handle read requests, and they only receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover. Follower Read includes a series of load balancing mechanisms that offload TiKV read loads from the leader replica to a follower replica in a Region. To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader node. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy. From 0ee2bb0a05ab330476c3b67fe4e9e26ac4d6aad2 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 30 Oct 2025 14:03:41 +0800 Subject: [PATCH 12/17] Apply suggestions from code review Co-authored-by: Lilian Lee --- follower-read.md | 16 ++++++++-------- grafana-tidb-dashboard.md | 8 ++++---- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/follower-read.md b/follower-read.md index c50ca101e8d52..a49bf4f995280 100644 --- a/follower-read.md +++ b/follower-read.md @@ -5,7 +5,7 @@ summary: This document describes the use and implementation of Follower Read. # Follower Read -In TiDB, to ensure high availability and data security, TiKV stores multiple replicas for each Region, one of which is the leader and the others are followers. By default, all read and write requests are processed by the leader. The Follower Read feature enables TiDB to read data from follower replicas of a Region while maintaining strong consistency, thereby reducing the read workload on the leader and improving the overall read throughput of the cluster. +In TiDB, to ensure high availability and data safety, TiKV stores multiple replicas for each Region, one of which is the leader and the others are followers. By default, all read and write requests are processed by the leader. The Follower Read feature enables TiDB to read data from follower replicas of a Region while maintaining strong consistency, thereby reducing the read workload on the leader and improving the overall read throughput of the cluster. @@ -19,9 +19,9 @@ When performing Follower Read, TiDB selects an appropriate replica based on the -By enabling followers to handle read requests, Follower Read helps to achieve the following goals: +By enabling followers to handle read requests, Follower Read achieves the following goals: -- Distribute read hotspots and reduce leader workload. +- Distribute read hotspots and reduce the leader workload. - Prioritize local replica reads in multi-AZ or multi-datacenter deployments to minimize cross-AZ traffic. ## Usage scenarios @@ -29,12 +29,12 @@ By enabling followers to handle read requests, Follower Read helps to achieve th Follower Read is suitable for the following scenarios: - Applications with heavy read requests or significant read hotspots. -- Multi-AZ deployments where prioritizing to read from local replicas helps reduce cross-AZ bandwidth usage. -- Read-write separation architectures that expect further improvement in overall read performance. +- Multi-AZ deployments where you want to prioritize reading from local replicas to reduce cross-AZ bandwidth usage. +- Read-write separation architectures that you want to further improve overall read performance. > **Note:** > -> To ensure strong consistency of the read results, Follower Read needs to communicate with the leader to confirm the latest commit progress (by executing the Raft `ReadIndex` operation) before reading, which introduces an additional network interaction. Therefore, Follower Read is most effective when there are a large number of read requests or read-write isolation is required. However, for low-latency single queries, the performance improvement might not be significant. +> To ensure strong consistency of the read results, Follower Read communicates with the leader before reading to confirm the latest commit progress (by executing the Raft `ReadIndex` operation). This introduces an additional network interaction. Therefore, Follower Read is most effective where a large number of read requests exist or read-write isolation is required. However, for low-latency single queries, the performance improvement might not be significant. ## Usage @@ -126,7 +126,7 @@ When the follower node processes a read request, it first uses `ReadIndex` of th ### Follower replica selection strategy -The Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level. TiDB selects a replica for the first read attempt based on the configuration of `tidb_replica_read`. From the second retry onward, TiDB prioritizes ensuring successful reads. Therefore, when the selected follower node becomes inaccessible or has other errors, TiDB switches to the leader for service. +The Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level. TiDB selects a replica based on the `tidb_replica_read` configuration for the first read attempt. From the second retry onward, TiDB prioritizes ensuring successful reads. Therefore, when the selected follower node becomes inaccessible or has other errors, TiDB switches to the leader for service. #### `leader` @@ -146,4 +146,4 @@ The Follower Read feature does not affect TiDB's Snapshot Isolation transaction To ensure strong data consistency, Follower Read performs a `ReadIndex` operation regardless of how much data is read, which inevitably consumes additional TiKV CPU resources. Therefore, in small-query scenarios (such as point queries), the performance loss of Follower Read is relatively more obvious. Moreover, because the traffic reduced by local reads for small queries is limited, Follower Read is more recommended for large queries or batch reading scenarios. -When `tidb_replica_read` is set to `closest-adaptive`, TiDB does not perform Follower Read for small queries. As a result, the extra TiKV CPU overhead compared with that of the `leader` policy is generally within +10% in various workloads. +When `tidb_replica_read` is set to `closest-adaptive`, TiDB does not perform Follower Read for small queries. As a result, under various workloads, the additional CPU overhead on TiKV is typically no more than 10% compared with the `leader` policy. diff --git a/grafana-tidb-dashboard.md b/grafana-tidb-dashboard.md index 7479bcb4eeca2..b7526e1d954ad 100644 --- a/grafana-tidb-dashboard.md +++ b/grafana-tidb-dashboard.md @@ -127,10 +127,10 @@ The following metrics relate to requests sent to TiKV. Retry requests are counte - **local-in**: the incoming traffic of responses to requests that attempt a stale read in the local zone - **local-out**: the outgoing traffic of requests that attempt a stale read in the local zone - Read Req Traffic - - **leader-local**: Traffic generated by Leader Read processing read requests in the local zone - - **leader-cross-zone**: Traffic generated by Leader Read processing read requests in a remote zone - - **follower-local**: Traffic generated by Follower Read processing read requests in the local zone - - **follower-cross-zone**: Traffic generated by Follower Read processing read requests in a remote zone + - **leader-local**: traffic generated by Leader Read processing read requests in the local zone + - **leader-cross-zone**: traffic generated by Leader Read processing read requests in a remote zone + - **follower-local**: traffic generated by Follower Read processing read requests in the local zone + - **follower-cross-zone**: traffic generated by Follower Read processing read requests in a remote zone ### PD Client From da0dc7be128fb29d02917bfb0ad7f27354cd5f4d Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 30 Oct 2025 14:13:13 +0800 Subject: [PATCH 13/17] Update follower-read.md Co-authored-by: Lilian Lee --- follower-read.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/follower-read.md b/follower-read.md index a49bf4f995280..06fc2e699bb1c 100644 --- a/follower-read.md +++ b/follower-read.md @@ -116,7 +116,7 @@ You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New i Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers cannot handle read requests, and they only receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover. -Follower Read includes a series of load balancing mechanisms that offload TiKV read loads from the leader replica to a follower replica in a Region. To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader node. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy. +Follower Read includes a set of load balancing mechanisms that offload TiKV read requests from the leader replica to a follower replica in a Region. To allow data reading from the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader node. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy. ### Strongly consistent reads From 139284dd3f1be000f0477263640cf508562304a9 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 30 Oct 2025 15:35:34 +0800 Subject: [PATCH 14/17] Update follower-read.md Co-authored-by: Lilian Lee --- follower-read.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/follower-read.md b/follower-read.md index 06fc2e699bb1c..d063f6d711f6e 100644 --- a/follower-read.md +++ b/follower-read.md @@ -54,9 +54,9 @@ This variable defines the expected data read mode. Starting from v8.5.4 and v9.0 In scenarios where you need to reduce cross-AZ traffic by reading from local replicas, the following configurations are recommended: -- The default value `leader` provides the best performance. -- `closest-adaptive` minimizes cross-AZ traffic with minimal performance impact. -- `closest-replicas` minimizes cross-AZ traffic as much as possible. +- `leader`: the default value, providing the best performance. +- `closest-adaptive`: minimizes cross-AZ traffic while keeping performance loss to a minimum. +- `closest-replicas`: maximizes cross-AZ traffic savings but might cause some performance degradation. If you are using other configurations, refer to the following table to modify them to the recommended configurations: From 10946db356dcb656cec58588ab3e670eb0b7fbd4 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Tue, 4 Nov 2025 15:39:13 +0800 Subject: [PATCH 15/17] update for v8.5.4 --- follower-read.md | 4 ++-- system-variables.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/follower-read.md b/follower-read.md index d063f6d711f6e..37dfe3f780e14 100644 --- a/follower-read.md +++ b/follower-read.md @@ -102,13 +102,13 @@ If you want to use a more precise read replica selection policy, refer to the fu -You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4 and v9.0.0)](/grafana-tidb-dashboard.md#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. +You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4)](/grafana-tidb-dashboard.md#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. -You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4 and v9.0.0)](https://docs.pingcap.com/tidb/stable/grafana-tidb-dashboard#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. +You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4)](https://docs.pingcap.com/tidb/stable/grafana-tidb-dashboard#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. diff --git a/system-variables.md b/system-variables.md index c7384c24eec9b..b955c78a27f6f 100644 --- a/system-variables.md +++ b/system-variables.md @@ -5375,7 +5375,7 @@ SHOW WARNINGS; - Type: Enumeration - Default value: `leader` - Possible values: `leader`, `follower`, `leader-and-follower`, `prefer-leader`, `closest-replicas`, `closest-adaptive`, and `learner`. The `learner` value is introduced in v6.6.0. -- This variable is used to control where TiDB reads data. Starting from v8.5.4 and v9.0.0, this variable only takes effect on read-only SQL statements. +- This variable is used to control where TiDB reads data. Starting from v8.5.4, this variable only takes effect on read-only SQL statements. - For more details about usage and implementation, see [Follower read](/follower-read.md). ### tidb_restricted_read_only New in v5.2.0 From 68d5bd4cb13b65b79b5f2d78f6a0da85f90c861d Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Tue, 4 Nov 2025 15:39:36 +0800 Subject: [PATCH 16/17] Apply suggestions from code review --- follower-read.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/follower-read.md b/follower-read.md index 37dfe3f780e14..fa7b0bb25f10a 100644 --- a/follower-read.md +++ b/follower-read.md @@ -50,7 +50,7 @@ Scope: SESSION | GLOBAL Default: leader -This variable defines the expected data read mode. Starting from v8.5.4 and v9.0.0, this variable only takes effect on read-only SQL statements. +This variable defines the expected data read mode. Starting from v8.5.4, this variable only takes effect on read-only SQL statements. In scenarios where you need to reduce cross-AZ traffic by reading from local replicas, the following configurations are recommended: From 7eb942cc0a82a93a795c7d06b95c535e1a2208c9 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Tue, 4 Nov 2025 16:23:39 +0800 Subject: [PATCH 17/17] set basic monitoring as TiDB specific --- follower-read.md | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/follower-read.md b/follower-read.md index fa7b0bb25f10a..672263f7880c3 100644 --- a/follower-read.md +++ b/follower-read.md @@ -94,21 +94,11 @@ If you want to use a more precise read replica selection policy, refer to the fu -## Basic monitoring - -> **Note** -> -> This section is only applicable to TiDB Self-Managed. - -You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4)](/grafana-tidb-dashboard.md#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. - - - - +## Basic monitoring -You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4)](https://docs.pingcap.com/tidb/stable/grafana-tidb-dashboard#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it. +You can check the [**TiDB** > **KV Request** > **Read Req Traffic** panel (New in v8.5.4)](/grafana-tidb-dashboard.md#kv-request) to determine whether to enable Follower Read and observe the traffic reduction effect after enabling it.