tests: unify frequency check time to make test stable

[HuSharp]

What problem does this PR solve?

Issue Number: Close #8555

What is changed and how does it work?

Unify changeFrequencyTimes and skipCampaignLeaderCheck

We have 2 failpoints, both designed to address frequently change(There are many resign leader in this test and slow disk can make frequently happened in high probability, we can unify them.

for #8555

We can find PD2 campaign times by log:

1. SetupSuite make PD2 as leader ----- 1

2. TestRetryOnLeaderChange

TestRetryOnLeaderChange is before TestTransferLeader, which have 3 times resign

Pd2->PD1
Pd1->PD2 -------- 2
Pd2->PD1 --------- during this resign etcd had stuck which make PD2 still can start campaign leader

This is the reason why the test is unstable, if it's not stuck, for the whole test, here the campaign times won't +1

2024-08-21T09:59:37.1252137Z [2024/08/21 09:59:21.828 +00:00] [INFO] [member.go:362] ["try to resign etcd leader to next pd-server"] [from=pd2] [to=]
2024-08-21T09:59:37.1253432Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:710] ["node state check job has been stopped"]
2024-08-21T09:59:37.1254169Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:686] ["metrics are reset"]
2024-08-21T09:59:37.1254945Z [2024/08/21 09:59:21.828 +00:00] [INFO] [server.go:127] ["region syncer has been stopped"]
2024-08-21T09:59:37.1255938Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:688] ["metrics collection job has been stopped"]
2024-08-21T09:59:37.1256731Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:520] ["sync store config job is stopped"]
2024-08-21T09:59:37.1257462Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:407] ["service check job is stopped"]
2024-08-21T09:59:37.1258166Z [2024/08/21 09:59:21.828 +00:00] [INFO] [labeler.go:70] ["region labeler GC stopped"]
2024-08-21T09:59:37.1259073Z [2024/08/21 09:59:21.828 +00:00] [INFO] [runner.go:131] ["stopping async task runner"] [name=heartbeat-async]
2024-08-21T09:59:37.1260105Z [2024/08/21 09:59:21.828 +00:00] [INFO] [runner.go:131] ["stopping async task runner"] [name=log-async]
2024-08-21T09:59:37.1261069Z [2024/08/21 09:59:21.828 +00:00] [INFO] [runner.go:131] ["stopping async task runner"] [name=misc-async]
2024-08-21T09:59:37.1261982Z [2024/08/21 09:59:21.828 +00:00] [INFO] [scheduling_controller.go:161] ["scheduling metrics are reset"]
2024-08-21T09:59:37.1263113Z [2024/08/21 09:59:21.828 +00:00] [INFO] [scheduling_controller.go:163] ["scheduling metrics collection job has been stopped"]
2024-08-21T09:59:37.1263986Z [2024/08/21 09:59:21.828 +00:00] [INFO] [coordinator.go:232] ["coordinator stops running"]
2024-08-21T09:59:37.1264715Z [2024/08/21 09:59:21.829 +00:00] [INFO] [coordinator.go:210] ["coordinator is stopping"]
2024-08-21T09:59:37.1265559Z [2024/08/21 09:59:21.829 +00:00] [INFO] [scheduling_controller.go:139] ["statistics background jobs has been stopped"]
2024-08-21T09:59:37.1266409Z [2024/08/21 09:59:21.829 +00:00] [INFO] [coordinator.go:213] ["coordinator has been stopped"]
2024-08-21T09:59:37.1267216Z [2024/08/21 09:59:21.829 +00:00] [INFO] [scheduling_controller.go:87] ["scheduling service is stopped"]
2024-08-21T09:59:37.1267975Z [2024/08/21 09:59:21.830 +00:00] [INFO] [cluster.go:769] ["raft cluster is stopped"]
2024-08-21T09:59:37.1268676Z [2024/08/21 09:59:21.830 +00:00] [INFO] [tso.go:436] ["reset the timestamp in memory"] []
2024-08-21T09:59:37.1270080Z [2024/08/21 09:59:21.831 +00:00] [INFO] [server.go:1481] ["leadership transfer starting"] [local-member-id=e298409efaeb2293] [current-leader-member-id=e298409efaeb2293] [transferee-member-id=319ee8b43d843ed7]
2024-08-21T09:59:37.1272407Z [2024/08/21 09:59:21.831 +00:00] [INFO] [raft] [zap_raft.go:77] ["e298409efaeb2293 [term 4] starts to transfer leadership to 319ee8b43d843ed7"]
2024-08-21T09:59:37.1275064Z [2024/08/21 09:59:21.831 +00:00] [INFO] [raft] [zap_raft.go:77] ["e298409efaeb2293 sends MsgTimeoutNow to 319ee8b43d843ed7 immediately as 319ee8b43d843ed7 already has up-to-date log"]
2024-08-21T09:59:37.1277696Z [2024/08/21 09:59:21.831 +00:00] [INFO] [raft] [zap_raft.go:77] ["319ee8b43d843ed7 [term 4] received MsgTimeoutNow from e298409efaeb2293 and starts an election to get leadership."]
2024-08-21T09:59:37.1279845Z [2024/08/21 09:59:21.831 +00:00] [INFO] [raft] [zap_raft.go:77] ["319ee8b43d843ed7 is starting a new election at term 4"]
2024-08-21T09:59:37.1281814Z [2024/08/21 09:59:21.831 +00:00] [INFO] [server.go:1690] ["start to campaign PD leader"] [campaign-leader-name=pd2]

3. start TestTransferLeader
   PD1 -> PD2 ------- 4 And then meet frequently error

tests: unify frequency check time to make test stable

Check List

Tests

• Unit test
• Integration test

Release note

None.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.58%. Comparing base (32829f1) to head (614e4e5).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8556      +/-   ##
==========================================
- Coverage   77.68%   77.58%   -0.11%     
==========================================
  Files         474      474              
  Lines       61877    61873       -4     
==========================================
- Hits        48069    48002      -67     
- Misses      10280    10336      +56     
- Partials     3528     3535       +7     

Flag	Coverage Δ
unittests	77.58% <100.00%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lhy1024, okJiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [lhy1024]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-08-22 02:44:11.940995655 +0000 UTC m=+406247.075445774: ☑️ agreed by lhy1024.
• 2024-08-22 03:48:03.03392074 +0000 UTC m=+410078.168370861: ☑️ agreed by okJiang.