Run all integration tests in parallel

[NikitaSkrynnik]

Description

Each integration test is launched in its own namespace. If we launch several tests simultaneously they won't interfere with each other. It will reduce time of testing on CI and also it will improve coverage and quality of the project.

Results:

Suite	Old	New	Difference
Basic Suite	659.59s	192.82s	360.78 %
Feature Suite	972.89s	283.35s	343.35 %
Heal Suite	1646.46s	1667.61s	0 %
Ipsec Suite	199.73s	38.52s	518.51 %
Monolith Suite	242.04s	258.16s	0 %
Memory Suite	146.37s	37.42s	391.15 %
Observability Suite	97.84s	89.21s	0 %
Rvlan Suite	1017.88s	1018.52s	0 %

Solution

We can use worker pool pattern to launch tests in parallel. That is necessary for public clusters because they have limited resources.

We need to add some changes to gotestmd to generate test suites that can run in parallel. Also we need to add special flag parallel for each suite to have a possibility to launch them sequentially.

We can launch tests with

s.T().Run()

and add special flag

s.T().Parallel()

to launch several tests in parallel.

[NikitaSkrynnik]

It looks like testify package is not designed to run suite tests in parallel. Suite runs all its tests sequentially and uses only one instance of *testing.T for each test inside the suite.

Option 1

We can make a custom suite, which wraps an original suite. The custom suite uses Setup and Cleanup from the original suite. It can get all the tests from the original suite using reflect package and run each test with its own *testing.T instance. All *testing.T instances are stored in a map in the custom suite. Therefore we are able to run several tests in parallel, because each test has its own instance of *testing.T.

Option 2

We can get rid of testify package and create our own parallel version of this package.

[NikitaSkrynnik]

Basic Suite execution time measurement

We used worker pool pattern to lauch tests in parallel. Here are the results of running tests Locally and on CI with 1 and 5 workers

	Local	on CI
1 worker	606 s.	730 s.
5 workers	166 s.	330 s.

[NikitaSkrynnik]

Decomposition

1. Run tests in parallel using sub-tests pattern - 12h
2. Test on CI - 7h
3. Add generation on parallel tests in gotestmd - 8h
4. Test on public clusters and tune a number of tests that could be run in parallel - 12h
5. Add excluded tests list for each suite - 5h
6. Pass CI - 7h

Total: 51h

[NikitaSkrynnik]

Current behavior

Integration tests are launched sequentially in suites.

Suites are not designed to run tests in parallel. Here is the issue about the feature stretchr/testify#187.

Solution

How do we run tests in parallel

In order to launch tests in parallel we can use sub-test pattern using package testing. It will allow us to run tests in parallel with method:

t.Parallel()

Here is an example of how a parallel version of tests could look:

func TestAll(t *testing.T) {
	s := Suite{}
	s.SetT(t)
	s.SetupSuite()

	tests := []struct {
		Name string
		Test func(t *testing.T)
	}{
		{"Kernel2Ethernet2Kernel", s.Kernel2Ethernet2Kernel},
		{"Kernel2Kernel", s.Kernel2Kernel},
		{"Memif2Memif", s.Memif2Memif},
	}
	for _, test := range tests {
		t.Run(test.Name, test.Test)
	}
}

func (s *Suite) TestKernel2Ethernet2Kernel(t *testing.T) {
	t.Parallel()
        ...
}
func (s *Suite) TestKernel2Kernel(t *testing.T) {
	t.Parallel()
	...
}
func (s *Suite) TestMemif2Memif(t *testing.T) {
	t.Parallel()
        ...
}

We add one top-level test TestAll which creates sub-tests specified in suite. We can use s.SetupSuite() from base suite to do a setup for sub-tests. This method calls SetupSuite() methods from parent suites (prefetch.SetupSuite(), checkout.SetupSuite(), spire.SetupSuite()). All sub-tests are stored as base suite methods (for example, s.Kernel2Ethernet2Kernel). This is needed because base suite contains shell suite which is necessary for calling test's bash commands.

The way we lauch tests is different now.
Old way:

func TestRunBasicSuite(t *testing.T) {
	suite.Run(t, new(basic.Suite))
}

New way:

func TestRunBasicSuite(t *testing.T) {
	basic.TestAll(t)
}

How do we control a number of concurrent tests

We can control a number of concurrent tests with flag -parallel $COUNT for go test command. By default, $COUNT is equal to a number of cores on CPU.

If there are tests, which conflict with other tests or require a lot of resources (for example, nse-composition), we can run them sequentially simply removing line with t.Parallel() method.

Changes in gotestmd

We add special flag parallel for gotestmd which indicates that we want to generate tests that can be run in parallel. If this flag isn't specified gotestmd generates tests in suites (as it is now).

We pass parallel flag to suite struct before generating suites. Based on the value we decide whether we add top-level test TestAll to a file.

We also pass this flag to Test struct. If flag is true, we add t *testing.T argument to a sub-test function, call t.Parallel() method, pass t to shell.Runner and also call t.Cleanup instead of s.T().Cleanup.

Changes in shell.Suite

We add argument t *testing.T to shell.Runner method to be able to pass different t *testing.T.

Open questions

1. How do we track tests which should be run sequentially and separately from the rest of the tests.

My suggestion:

We add flag Parallel to every test in README.md in deployments-k8s. This requires an additional flag for Test struct in gotestmd to be able to decide whether to add t.Parallel() for the test or not. In this case Test struct will have two flags: one global Parallel flag to generate test as sub-test and second flag which tells if this particular sub-test should be parallel (with or without t.Parallel() method call).

2. How do we collect logs for each test if they are launched in parallel?

In sequential version of tests suite automatically calls BeforeTest() and AfterTest() methods to collect logs and describe pods for each tests. If we run sub-tests without suite, we need to call this methods manually before and after each test.

Conclusion

Pros:

1. No additional frameworks are required
2. Easy to control a number of parallel tests and which tests should be run sequentially
3. Doesn't require significant changes in gotestmd

Cons:

1. Two different ways of launching parallel and non-parallel versions
2. Additional flags parallel for some tests in README.md in deployments-k8s

[denis-tingaikin]

PR with parallel running in testify stretchr/testify#1109

[NikitaSkrynnik]

Decomposition

Sub Task	Estimation	Done
Rework exlude prefix test	2h	Done
Rework vl3 tests	3h	Done
Pass CI	5h
Rework log collection	7h	Done
Rework pod describe collection	3h
Review log collection code	3h
Test log collection on CI	0.5h
Excluded tests fuctionality	0.5h	Done
Add excluded tests for each suite	0.5h	Done
Test on public clusters and tune a number of tests that could be run in parallel	4h
Review	3h
Total	31h

[NikitaSkrynnik]

Results

Suite	Old	New	Difference
Basic Suite	659.59s	192.82s	360.78 %
Feature Suite	972.89s	283.35s	343.35 %
Heal Suite	1646.46s	1667.61s	0 %
Ipsec Suite	199.73s	38.52s	518.51 %
Monolith Suite	242.04s	258.16s	0 %
Memory Suite	146.37s	37.42s	391.15 %
Observability Suite	97.84s	89.21s	0 %
Rvlan Suite	1017.88s	1018.52s	0 %