Skip to content

*: fix flaky test TestPreparingProgress and TestRemovingProgress#9465

Merged
ti-chi-bot[bot] merged 9 commits intotikv:masterfrom
okJiang:fix-unstable
Jul 3, 2025
Merged

*: fix flaky test TestPreparingProgress and TestRemovingProgress#9465
ti-chi-bot[bot] merged 9 commits intotikv:masterfrom
okJiang:fix-unstable

Conversation

@okJiang
Copy link
Member

@okJiang okJiang commented Jul 1, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: Close #9461,

What is changed and how does it work?

time.Sleep(100 * time.Millisecond) // wait for the removing progress to be created

Here we wait for a checkStores interval to trigger the progress manager logic to get the removing status.

For TestPreparingProgress, let's take this as an example https://github.com/tikv/pd/actions/runs/16019981411/job/45194384196

    testutil.go:68: 
        	Error Trace:	/home/runner/work/pd/pd/pkg/utils/testutil/testutil.go:68
        	            				/home/runner/work/pd/pd/tests/server/api/api_test.go:1086
        	Error:      	Condition never satisfied
        	Test:       	TestPreparingProgress

This is because after checkStoresLoop, the store may have entered the preparing state, so it may fail. Therefore, we need to block checkStores to obtain the intermediate state.

Check List

Tests

  • Unit test
  • Integration test

Release note

None.

@okJiang
fix flaky test
56d2652 
Signed-off-by: okJiang <819421878@qq.com>
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jul 1, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has signed the dco. labels Jul 1, 2025
@okJiang okJiang marked this pull request as ready for review July 1, 2025 08:45
@ti-chi-bot ti-chi-bot bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jul 1, 2025
@okJiang
Copy link
Member Author

okJiang commented Jul 1, 2025

/retest

4 similar comments
@okJiang
Copy link
Member Author

okJiang commented Jul 1, 2025

/retest

@okJiang
Copy link
Member Author

okJiang commented Jul 1, 2025

/retest

@okJiang
Copy link
Member Author

okJiang commented Jul 1, 2025

/retest

@okJiang
Copy link
Member Author

okJiang commented Jul 1, 2025

/retest

okJiang added 2 commits July 1, 2025 18:29
@okJiang
mute TestSwitchBurst
23e0f3a 
Signed-off-by: okJiang <819421878@qq.com>
@okJiang
add failpoint
7d7582b 
Signed-off-by: okJiang <819421878@qq.com>
@ti-chi-bot ti-chi-bot bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 1, 2025
@okJiang
update
71e79bb 
Signed-off-by: okJiang <819421878@qq.com>
@okJiang
Copy link
Member Author

okJiang commented Jul 2, 2025

/retest

1 similar comment
@okJiang
Copy link
Member Author

okJiang commented Jul 2, 2025

/retest

@okJiang
add failpoint
4752d85 
Signed-off-by: okJiang <819421878@qq.com>
@ti-chi-bot ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 2, 2025
@okJiang
fix eventually
655f3a7 
Signed-off-by: okJiang <819421878@qq.com>
@codecov
Copy link

codecov bot commented Jul 2, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.19%. Comparing base (c546b9c) to head (4556365).
Report is 7 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #9465   +/-   ##
=======================================
  Coverage   76.18%   76.19%           
=======================================
  Files         481      481           
  Lines       75517    75541   +24     
=======================================
+ Hits        57534    57556   +22     
- Misses      14419    14420    +1     
- Partials     3564     3565    +1
Flag Coverage Δ
unittests 76.19% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

okJiang
okJiang commented Jul 2, 2025
View reviewed changes
server/cluster/cluster.go Outdated
if !readyToServe && (c.IsPrepared() || (c.IsServiceIndependent(constant.SchedulingServiceName) && c.isStorePrepared())) {
threshold = c.getThreshold(stores, store)
log.Debug("store preparing threshold", zap.Uint64("store-id", storeID),
log.Info("store preparing threshold", zap.Uint64("store-id", storeID),
Copy link
Member Author

@okJiang okJiang Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be revert before pr merge.

lhy1024
lhy1024 approved these changes Jul 2, 2025
View reviewed changes
tests/integrations/mcs/resourcemanager/resource_manager_test.go

// TestSwitchBurst is used to test https://github.com/tikv/pd/issues/6209
func (suite *resourceManagerClientTestSuite) TestSwitchBurst() {
suite.T().Skip("skip this test because it is not stable")
Copy link
Contributor

@lhy1024 lhy1024 Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when will we open it again?

Copy link
Member Author

@okJiang okJiang Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe after someone fixes it😂

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 2, 2025
@okJiang
trigger test
9c19bf8 
Signed-off-by: okJiang <819421878@qq.com>
@okJiang
Copy link
Member Author

okJiang commented Jul 2, 2025

/retest

@okJiang
revert log level
61bc51c 
Signed-off-by: okJiang <819421878@qq.com>
@okJiang
Copy link
Member Author

okJiang commented Jul 2, 2025

/retest

@rleungx
Copy link
Member

rleungx commented Jul 2, 2025

How about adding some descriptions for the reason?

@okJiang
Copy link
Member Author

okJiang commented Jul 3, 2025

How about adding some descriptions for the reason?

time.Sleep(100 * time.Millisecond) // wait for the removing progress to be created

Here we wait for a checkStores interval to trigger the progress manager logic to get the removing status.

For TestPreparingProgress, let's take this as an example https://github.com/tikv/pd/actions/runs/16019981411/job/45194384196

    testutil.go:68: 
        	Error Trace:	/home/runner/work/pd/pd/pkg/utils/testutil/testutil.go:68
        	            				/home/runner/work/pd/pd/tests/server/api/api_test.go:1086
        	Error:      	Condition never satisfied
        	Test:       	TestPreparingProgress

This is because after checkStoresLoop, the store may have entered the preparing state, so it may fail. Therefore, we need to block checkStores to obtain the intermediate state.

@ti-chi-bot ti-chi-bot bot added the lgtm label Jul 3, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jul 3, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lhy1024, rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 3, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jul 3, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-07-02 09:20:23.957867618 +0000 UTC m=+1473076.681046599: ☑️ agreed by lhy1024.
  • 2025-07-03 03:41:36.193766649 +0000 UTC m=+1539148.916945631: ☑️ agreed by rleungx.

@okJiang
Copy link
Member Author

okJiang commented Jul 3, 2025

/retest

1 similar comment
@okJiang
Copy link
Member Author

okJiang commented Jul 3, 2025

/retest

@purelind
Copy link
Contributor

purelind commented Jul 3, 2025

/test pull-build pull-build-next-gen

@ti-chi-bot ti-chi-bot bot merged commit 41ef10e into tikv:master Jul 3, 2025
30 of 48 checks passed
@okJiang okJiang deleted the fix-unstable branch July 3, 2025 07:11
This was referenced Mar 18, 2026
Open
Draft
Draft
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lhy1024 lhy1024 lhy1024 approved these changes

@rleungx rleungx rleungx approved these changes

Assignees

No one assigned

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Flaky test TestPreparingProgress

4 participants

@okJiang @rleungx @purelind @lhy1024