Flaky Test: TestSecurity, runTestretryFetchCert

[zrggw]

#1432

Problem

The test runTestretryFetchCert in test TestSecurity is flaky and failed in Kmesh CI Workflow.

/retest

[hzxuzhonghu]

Can reproduce the flake with main branch in your local env?

[zrggw]

Here is my reproduction process

TestSecurity/TestCertRotate (in manager_test.go)

# git checkout main
# cd ./pkg/controller/security

1. Test when CPU resources are not limited:
# go test -race -run TestSecurity
time="2025-07-14T14:21:38+08:00" level=error msg="fetchCert for [identity] error: abnormal test" subsys=security
PASS
ok kmesh.net/kmesh/pkg/controller/security 5.552s

2. Then I created a new cgroup to limit CPU resources:
# cd /sys/fs/cgroup/
# mkdir test &&cd test
# echo 1301455 > cgroup.procs
# echo 20000 100000 | tee cpu.max

3. Then I ran the test again, a null pointer dereference issue occurred in the TestCertRotate function:
# go test -race -run TestSecurity
--- FAIL: TestSecurity (2.49s)
--- FAIL: TestSecurity/TestCertRotate (2.19s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1428909]

goroutine 51 [running]:
testing.tRunner.func1.2({0x16077c0, 0x22e5c60})
/usr/local/go/src/testing/testing.go:1734 +0x3eb
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1737 +0x696
panic({0x16077c0?, 0x22e5c60?})
/usr/local/go/src/runtime/panic.go:792 +0x132
kmesh.net/kmesh/pkg/controller/security.runTestCertRotate(0xc00018ca80)
/root/code/kmesh/pkg/controller/security/manager_test.go:127 +0x829
kmesh.net/kmesh/pkg/controller/security.TestSecurity.func2(0xc00018ca80)
/root/code/kmesh/pkg/controller/security/manager_test.go:43 +0x27
testing.tRunner(0xc00018ca80, 0x183a7a0)
/usr/local/go/src/testing/testing.go:1792 +0x226
created by testing.(*T).Run in goroutine 24
/usr/local/go/src/testing/testing.go:1851 +0x8f3
exit status 2
FAIL kmesh.net/kmesh/pkg/controller/security 2.700s

The specific line of code that threw the error is as follows:

// line 127
		if cert2 != nil && cert2.cert.CreatedTime != oldCert.CreatedTime {

Test TestSecurity/runTestretryFetchCert

1. Comment out the TestCertRotate part(line 42-44) in the TestSecurity function to test runTestretryFetchCert:
2. Remove CPU resource limits and test TestSecurity:
# echo max 100000 | tee cpu.max
# go test -race -run TestSecurity
time="2025-07-14T14:42:13+08:00" level=error msg="fetchCert for [identity] error: abnormal test" subsys=security
PASS
ok kmesh.net/kmesh/pkg/controller/security 3.449s

3. limit CPU resources:
# echo 20000 100000 | tee cpu.max

4. Run the test again, a nil value occurred:
# go test -race -run TestSecurity
time="2025-07-14T14:51:28+08:00" level=error msg="fetchCert for [identity] error: abnormal test" subsys=security
--- FAIL: TestSecurity (2.50s)
--- FAIL: TestSecurity/TestretryFetchCert (2.19s)
manager_test.go:169:
Error Trace: /root/code/kmesh/pkg/controller/security/manager_test.go:169
/root/code/kmesh/pkg/controller/security/manager_test.go:46
Error: Expected value not to be nil.
Test: TestSecurity/TestretryFetchCert
FAIL
exit status 1
FAIL kmesh.net/kmesh/pkg/controller/security 2.703s

The specific line of code that threw the error is as follows:

// line 169
	assert.NotNil(t, secretManager.GetCert(identity).cert)