replica selector refactor

[crazycs520]

close #1144

What changed and how does it work?

Since the old version replicaSelector is hard to test and understand, This PR introduces a new replica selector implementation replicaSelectorV2, which is less code and easer to understand.

For rollback, I don't remove the old version replicaSelector, and add a new config EnableReplicaSelectorV2 whose default value is true, which means use replicaSelectorV2 by default, set it's value to false will use old version replicaSelector. After replicaSelectorV2 is stable, maybe we can consider remove old version replicaSelector and config EnableReplicaSelectorV2.

How to test?

• All the old tests are pass.

• I add many test case in replica_selector_test.go to make sure EnableReplicaSelectorV2 has the same behavior with old version replicaSelector.

  • TestReplicaReadAccessPathByCase
  • TestReplicaReadAccessPathByCase2
  • TestReplicaReadAccessPathByBasicCase
  • TestReplicaReadAccessPathByLeaderCase
  • TestReplicaReadAccessPathByFollowerCase
  • TestReplicaReadAccessPathByMixedAndPreferLeaderCase
  • TestReplicaReadAccessPathByStaleReadCase
  • TestReplicaReadAccessPathByTryIdleReplicaCase
  • TestReplicaReadAccessPathByFlashbackInProgressCase
  • TestReplicaReadAccessPathByProxyCase
  • TestReplicaReadAccessPathByLearnerCase

• I write a TestReplicaReadAccessPathByGenError test to generate various combinations of region errors to verify that EnableReplicaSelectorV2 has the same behavior with old version replicaSelector. From the following log, you can see this test generated a total 2200723 cases.

  [2024/03/05 10:27:52.562 +08:00] [INFO] [replica_selector_test.go:2511] ["TestReplicaReadAccessPathByGenError Finished"] [total-case=2200723] [valid-case=857630] [invalid-case=1343093]
  

• The test coverage of replicaSelectorV2 is more than 98%.

Advice when reviewing PR

If you have any questions about the logic about replicaSelector, you can try to modify the logic code, and then run the test to see whether the test results are as expected.

cd internal/locate
go test -run=TestReplicaReadAccessPathBy.*Case   # run some test cases, this is fast and less than 1s on my m1 mbp.
go test > test.log   # run all pkg test, this may cost 230s on my m1 mbp.

[zyguan]

Overall LGTM. We may need to do some basic benchmarks and stablility tests to make sure there is no obvious regression.

[MyonKeminta]

Ok. It's fine to me for this PR to keep the logic consistent, but actually I think it doesn't make sense to let this error be recorded as an rpcError, which marks the RPC's result undetermined:

			// (in getRPCContext)
			if err != nil {
				s.rpcError = err
				return nil, nil
			}

I think it would be good if the error handling can be refined in the future

[MyonKeminta]

Is it sure that the deadlineErrUsingConfTimeout is produced during the current attempt?

[crazycs520]

No. The logic is if have meet deadlineErrUsingConfTimeout error in this round, don't invalidate region to let up layer to remove kv_read_timeout then do fast retry.

[crazycs520]

This logic is same with the old version.

[MyonKeminta]

• Should a not-slow normal follower (NotSlow + NormalPeer -> 2) have the same priority as a slow leader in prefer-leader mode (PreferLeader -> 2) ?

• Is there an assumption that in cases that leader must be expected (such as the first attempt in prefer-leader mode and the second attempt in stale read mode) must be done with the ReplicaSelectLeaderStrategy instead of using the calculateScore? If so, I think there should be some comments to explain that this function doesn't handle those cases.

• If LabelMatch always has higher priority than PreferLeader, and PreferLeader always has higher priority than NormalPeer and NotSlow, is it possible that we use different bits for these factors, so that the more significant bit a factor is at, the more dominant status the factor is? For factors that has the same priority, we reserve more bits for them. For example, we can have such definition:

  const (
  	flagLabelMatches = 1 << 3
  	flagPreferLeader = 1 << 2
  	flagNormalPeer = 1
  	flagNotSlow = 1
  	
  	// So that we have definition of the final score:
  	// MSB                                                                               LSB
  	// [unused bits][1 bit: LabelMatches][1 bit: PreferLeader][2 bits: NormalPeer + NotSlow]
  )

  I'm not sure if this is possible to give the same result as your current code, but I think in this way, we can have less cases like:

  • Factor A < Factor C
  • Factor B < Factor C
  • Factor A + Factor B > Factor C

  which I'm afraid may make the priority calculation less comprehensible. Another benefit is that we can mostly know how a score comes from by reading its bits, which may help in diagnosing.

[crazycs520]

Great catch, done as your advice.

[MyonKeminta]

• Is there an assumption that in cases that leader must be expected (such as the first attempt in prefer-leader mode and the second attempt in stale read mode) must be done with the ReplicaSelectLeaderStrategy instead of using the calculateScore? If so, I think there should be some comments to explain that this function doesn't handle those cases.

I'm not very sure. If so, please consider adding some comments to note that to avoid misunderstanding of the scoring rule.

[MyonKeminta]

Mostly LGTM but I'm not sure if I've understood it totally... I need some more time

[MyonKeminta]

Is there any reason to create a new sender instead of using the prepared one?

[crazycs520]

I was worried that sharing the same sender for each test might have concurrency issues, so I create a new sender for each test.

[MyonKeminta]

🤔 I assume that the fields initialized in SetupTest should be done for each single test

// SetupTestSuite has a SetupTest method, which will run before each
// test in the suite.
type SetupTestSuite interface {
	SetupTest()
}

[MyonKeminta]

So is it that it won't trigger upper-layer's backoff in this case?

[crazycs520]

This is for #1207

Since I need to test replicaSelectorV2 has some beha behavior with old replicaSelector, so I just change the old replicaSelector behavior here. If I don't change this, I'll have to modify the replicaSelectorV2 code to accommodate this strange behavior.

[MyonKeminta]

• Is there an assumption that in cases that leader must be expected (such as the first attempt in prefer-leader mode and the second attempt in stale read mode) must be done with the ReplicaSelectLeaderStrategy instead of using the calculateScore? If so, I think there should be some comments to explain that this function doesn't handle those cases.

I'm not very sure. If so, please consider adding some comments to note that to avoid misunderstanding of the scoring rule.

[MyonKeminta]

Rest LGTM

[MyonKeminta]

LGTM

[you06]

rest LGTM