*: make unit test great

[HuSharp]

What problem does this PR solve?

Issue Number: Ref #7969

What is changed and how does it work?

Add back parallel by run a single test case in a isolated OS process

We use a wrong way to parallel. Some test cases are not side effect free, for example:

• the enable and disable of the failpoint
• modification of the configuration
• change of the global variable in a single process.
  etc...

If the test cases with side effect run parallelly with others, it could cause some test fail unexpectedly.
When they're in a single OS process, make them parallel is not a big deal.

usage

// run all tests
pd-ut

// show usage
pd-ut -h

// list all packages
pd-ut list

// list test cases of a single package
pd-ut list $package

// list test cases that match a pattern
pd-ut list $package 'r:$regex'

// run all tests
pd-ut run

// run test all cases of a single package
pd-ut run $package

// run test cases of a single package
pd-ut run $package $test

// run test cases that match a pattern
pd-ut run $package 'r:$regex'

// build all test package
pd-ut build

// build a test package
pd-ut build xxx

// write the junitfile
pd-ut run --junitfile xxx

// test with race flag
pd-ut run --race

use xprog to accelerate bin.test build

go test --exec=xprog, the Go compile toolchain would call your xprog program passing the test binary path as the argument

So you can do this trick, write you own xprog, and move the test binary to some place else, save it for later use.

print function time

when using ./pd-ut run --junitfile xxx, can get junitfile as XML file to get every function cost time.

	<testsuite tests="11" failures="0" time="1.789263" name="github.com/tikv/pd/pkg/btree">
		<properties>
			<property name="go.version" value="go1.21.0 linux/amd64"></property>
		</properties>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestAscendLessThanG" time="0.038767"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestAscendGreaterOrEqualG" time="0.033858"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDeleteMaxG" time="0.201940"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestBTreeSizeInfo" time="0.136755"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDescendGreaterThanG" time="0.013433"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDeleteMinG" time="0.262618"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestCloneConcurrentOperationsG" time="0.335037"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestBTreeG" time="0.348103"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDescendLessOrEqualG" time="0.018799"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDescendRangeG" time="0.302686"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestAscendRangeG" time="0.097267"></testcase>
	</testsuite>

Performance

Since we run parallel now, the CI runs faster than before, on my server computer (Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GH, 144 core 72 CPU):

reduce from 6min -> 1min

for make ut:

time make ut
building task finish, parallelism=144, count=641, takes=10.154869881s
run all tasks takes 50.559951196s
> make ut  327.15s user 48.61s system 615% cpu 1:01.03 total

for make basic-test

time make basic-test
CGO_ENABLED=1 go test -tags tso_function_test,deadlock -timeout 20m -race
> make basic-test 1176.46s user 169.41s system 473% cpu 5:44.15 total

Check List

Tests

• Unit test
• Manual test

Release note

introduce pd-ut to accelerate unit-test

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• nolouch
• rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[lhy1024]

Do we still need to call suite.Clean()?

[codecov]

Codecov Report

Merging #7952 (c235324) into master (b9240a0) will decrease coverage by 3.64%.
The diff coverage is n/a.

❗ Current head c235324 differs from pull request most recent head 9f93b0e. Consider uploading reports for the commit 9f93b0e to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7952      +/-   ##
==========================================
- Coverage   77.30%   73.66%   -3.64%     
==========================================
  Files         468      436      -32     
  Lines       60879    48525   -12354     
==========================================
- Hits        47061    35748   -11313     
+ Misses      10284     9719     -565     
+ Partials     3534     3058     -476     

Flag	Coverage Δ
unittests	73.66% <ø> (-3.64%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[nolouch]

I'd prefer to name this as dev, as we can do more scope with dev command , such as

./dev lint
./dev test pkg/util
./dev generator swagger
./dev bench heartbeat

[HuSharp]

maybe can unify all tools in another pr

[rleungx]

Do we have a timeout?

[HuSharp]

maybe need to reduce timeout for each test when enabled race
https://github.com/tikv/pd/pull/7952/files#diff-d1de7e6511458d33a769c0c501ec0ea45494dd9d2e830dc315e8fe623eb34b76R569-R574

[HuSharp]

/merge

[ti-chi-bot]

@HuSharp: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: c235324

[ti-chi-bot]

@HuSharp: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.