TestSplitKeyspaceGroup "Condition never satisfied" randomly

[binshi-bing]

Flaky Test

Which jobs are failing

--- FAIL: TestSplitKeyspaceGroup (44.24s)
testutil.go:63:
Error Trace: /home/runner/work/pd/pd/pkg/utils/testutil/testutil.go:63
/home/runner/work/pd/pd/tests/pdctl/keyspace/keyspace_group_test.go:124
Error: Condition never satisfied
Test: TestSplitKeyspaceGroup

CI link

https://github.com/tikv/pd/actions/runs/5203268375/jobs/9385886991?pr=6565

Reason for failure (if possible)

unknown

Anything else

[lhy1024]

In https://github.com/tikv/pd/actions/runs/5218488382/jobs/9420912472?pr=6584, I added some log.

2023-06-09T07:14:02.5578661Z [2023/06/09 07:13:38.570 +00:00] [INFO] [tso_keyspace_group.go:691] ["alloc nodes for keyspace group"] [keyspace-group-id=0] [nodes="[{\"address\":\"http://127.0.0.1:33849\"},{\"address\":\"http://127.0.0.1:41395\"}]"]
2023-06-09T07:14:02.5578956Z [2023/06/09 07:13:38.573 +00:00] [INFO] [tso_keyspace_group.go:193] ["all keyspace groups have equal or more than default replica count, stop to alloc node"]
....
2023-06-09T07:14:02.6855813Z [2023/06/09 07:13:42.792 +00:00] [INFO] [keyspace_group_test.go:143] ["split output"] [output="Failed! [500] \"not enough replicas in the keyspace group\""]

[lhy1024]

This is because UpdateKeyspaceForGroup overrides the result of allocNodesToAllKeyspaceGroups

pd server	T1	T2	T3
A	GroupManager.Bootstrap creates default keyspace group and put it into GroupManager.groups	...	keyspaceManager.Bootstrap preallocate keyspace and updateKeyspaceForGroupLocked overwrite default keyspace group with nil member, which is from GroupManager.groups
B	...	allocNodesToAllKeyspaceGroups load default keyspace group and alloc node for it	...

So, we need to run allocNodesToAllKeyspaceGroups only on pd leader and updateKeyspaceForGroupLocked cannot modify other params