Implement the merging operation of Keyspace Group

[JmPotato]

Development Task

As we now support the split operation of Keyspace Group, it is also necessary to provide reverse merge operations, which can help us better manage the number of Keyspace Groups and do some rollback operations.

To implement the merging operation of Keyspace Group, it may contain the following process:

1. Put all merging groups into the merging state.
   a. Once the groups that need to be merged watch this change, it will stop serving any TSO request to the outside.
   b. Once the merging target groups watch this change, it will start the merging process described as follows.
2. The merging target server sends the merging gRPC requests to all group primaries which need to be merged.
3. Once a primary received the merging request, it will:
   a. Stop serving any TSO request to the outside if it has not been performed step 1.a yet.
   b. Return the current TSO as the gRPC response.
4. The merging target server keeps trying step 3 until all the primaries responded, and then:
   a. Set its current TSO to the biggest one among all the primary responses.
   b. Ask the PD leader to delete these groups from the meta and move the keyspaces.

[binshi-bing]

What if the primary of a keyspace group, which has already stopped the service and responded the PD leader, transferred to another server before 4.a. Without persistent info, will the new primary of the keyspace group still serve the requests before 4.a?

[binshi-bing]

To solve the above problem, can we have the following steps?

1. Once the default keyspace group watches this change, it deletes all non-default keyspace groups in etcd.
2. The keyspace group managers on tso servers watch this change and handle the DELETE events to stop serving these keyspace groups.
3. Once the default keyspace group watches all primaries of non-default keyspace groups are gone, it reads the timestamps saved in etcd of all non-default keyspace groups, reset own timestmp with the max(all non-default persisted timestamps) + a time window.

After synced offline with @JmPotato , we'll go with this improvement.

[binshi-bing]

We already have step 2 implemented. We also have part of 3 implemented and the other part has the similar logic and code.