Activate Follower-Handle

[CabinfeverB]

Development Task

PD serves as the central node and provides services for the cluster. In the past, PD leaders have been carrying all the services and follower resources have not been utilized.

• Region API

Summary

An important function of PD is to provide routing information for data replicas. For example, GetRegion is a hot path in TiDB cluster. The pd provides the sync region mechanism. The pd leader synchronizes the region with the changed meta information to a follower. So we can handle the region requeset in PD follower server.

Additional knowledge

Sync region

The pd provides the sync region mechanism. The pd leader synchronizes the region with the changed meta information to a follower.
And the followers hold basicCluster which has region tree.

Follower forward mechanism

When enable-forwarding is enabled (the default value in TiDB is false), the pd periodically checks the connectivity and health status of the pd leader. If false, pd client will try to send request with pd-forwarded-hostto followers and followers will forward the request to the leader.

Region Cache in TiKV client-go

The region cache deletes a region item in either of the following situations:

1. GC triggered by TTL
2. When reloading region, insert new region item and delete old region
3. client-go will do ScanRegions for all regions caused by GC. It will ignore the region cache and request to PD.
   Therefore, if a region is unavailable in client-Go, the client does not delete the old region first and then find a new region

design

There may be a delay in synchronizing region data to followers. As a result, the followers data is not up-to-date. To reduce potential retries on the associated path, callers need to select followers or leaders according to the usage scenario.
Specifically, if the caller wants to obtain a new region first time, it can do so from the followers (we can assume that most of the region information in the followers is up to date). If the caller needs to update the region information, we think that this time must get the latest, then this time from the leader.
According to above additional knowledge, this transformation is very easy to do in client-go.

implement

Client-go Impl

1. In findRegionByKey, if region is nil, the region cache can get region with follower enabled optin. Otherwise, no option.

• client-go/pull/1079

2. We should check epoch in region cache.

• client-go/pull/1072

PD Client Implement

1. Only do BuildForwardContext, if we need to forward request to the leader. client: avoid to add redundant grpc metadata #7471
2. For the GetRegion interface, add FollowerHandleOption in parameters. client: add follower option #7465
3. Maintain follower health status in the client. If gRPC calling returns error, the error of response is not nil, the region of response is nil, remove this follower from candidate servers. And pd client should get region from the leader rather than return nil and error. client: Introduce ServiceClient #7489 syncer: add region syncer client status #7461
4. If FollowerHandle is true, pd client sends the requests to the Leader or followers one by one. Add follower-handle in gRPC metadata. client: Introduce ServiceClient #7489 vars: add pd_enable_follower_handle_region to support get region from pd follower pingcap/tidb#49231

Server Implement

1. If PD server is follower, check whether the gRPC context holds ForwardMetadataKey. If it holds, send it to the leader as before. If not, check the active-follower. If with it, check the status of Sync client first. If the stream is broken, an error is returned. Otherwise, try to get the region from the region tree and return . *: follower support to handle GetRegion and other region api #7432

metrics

#7619

[CabinfeverB]

Since we have no plans to support more APIs at this time, I close this issue