Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: Travis: TestGetIdentity/Multiple_identities: test timed out after 10m0s #30873

Closed
lmb opened this issue Feb 20, 2024 · 1 comment · Fixed by #30885
Closed

CI: Travis: TestGetIdentity/Multiple_identities: test timed out after 10m0s #30873

lmb opened this issue Feb 20, 2024 · 1 comment · Fixed by #30885
Assignees
@bimmlerd
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! sig/agent Cilium agent related. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies.

Comments

@lmb
Copy link
Contributor

lmb commented Feb 20, 2024

CI failure

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓                                                   

┃   PANIC  package: github.com/cilium/cilium/pkg/k8s/identitybackend • TestGetIdentity/Multiple_identities   ┃                                                   

┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛                                                   

panic: test timed out after 10m0s                                                                                                                                

running tests:                                                                                                                                                   

	TestGetIdentity (10m0s)                                                                                                                                          

	TestGetIdentity/Multiple_identities (10m0s)                                                                                                                      

                                                                                                                                                                 

goroutine 30 [running]:                                                                                                                                          

testing.(*M).startAlarm.func1()                                                                                                                                  

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:2259 +0x320                                                                             

created by time.goFunc                                                                                                                                           

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/time/sleep.go:176 +0x38                                                                                    

                                                                                                                                                                 

goroutine 1 [chan receive, 10 minutes]:                                                                                                                          

testing.(*T).Run(0x4000003860, {0x1fcb0e2?, 0x163e9a4cdfb?}, 0x20f9a30)                                                                                          

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:1649 +0x350                                                                             

testing.runTests.func1(0x40003d4bd0?)                                                                                                                            

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:2054 +0x48                                                                              

testing.tRunner(0x4000003860, 0x4000647c28)                                                                                                                      

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:1595 +0xe8                                                                              

testing.runTests(0x40003c3720?, {0x36702a0, 0x2, 0x2}, {0x40?, 0x1dafee0?, 0x36a1260?})                                                                          

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:2052 +0x3b4                                                                             

testing.(*M).Run(0x40003c3720)                                                                                                                                   

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:1925 +0x538                                                                             

main.main()                                                                                                                                                      

	_testmain.go:83 +0x1c8                                                                                                                                           

                                                                                                                                                                 

goroutine 9 [chan receive, 10 minutes]:                                                                                                                          

testing.(*T).Run(0x400025c000, {0x1fd4598?, 0x0?}, 0x4000380530)                                                                                                 

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:1649 +0x350                                                                             

github.com/cilium/cilium/pkg/k8s/identitybackend.TestGetIdentity(0x400025c000?)                                                                                  

	/home/travis/gopath/src/github.com/cilium/cilium/pkg/k8s/identitybackend/identity_test.go:204 +0x940                                                             

testing.tRunner(0x400025c000, 0x20f9a30)                                                                                                                         

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:1595 +0xe8                                                                              

created by testing.(*T).Run in goroutine 1                                                                                                                       

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:1648 +0x33c                                                                             

                                                                                                                                                                 

goroutine 22 [semacquire, 10 minutes]:                                                                                                                           

sync.runtime_Semacquire(0x2318428?)                                                                                                                              

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/runtime/sema.go:62 +0x2c                                                                                   

sync.(*WaitGroup).Wait(0x40002be100)                                                                                                                             

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/sync/waitgroup.go:116 +0x74                                                                                

github.com/cilium/cilium/pkg/k8s/identitybackend.TestGetIdentity.func1(0x400066c000)                                                                             

	/home/travis/gopath/src/github.com/cilium/cilium/pkg/k8s/identitybackend/identity_test.go:241 +0x4cc                                                             

testing.tRunner(0x400066c000, 0x4000380530)                                                                                                                      

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:1595 +0xe8                                                                              

created by testing.(*T).Run in goroutine 9                                                                                                                       

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/testing/testing.go:1648 +0x33c                                                                             

                                                                                                                                                                 

goroutine 11 [select]:                                                                                                                                           

k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x40001029c0?, {0x230de60, 0x400058e090}, 0x1, 0x400010f920)                                                      

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:238 +0xf8                                                   

k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x4000150490?, 0x3b9aca00, 0x0, 0x60?, 0xffff64c40d58?)                                                            

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x80                                                   

k8s.io/apimachinery/pkg/util/wait.Until(...)                                                                                                                     

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161                                                         

k8s.io/client-go/tools/cache.(*controller).Run(0x4000471d60, 0x400010f920)                                                                                       

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x2d0                                                    

github.com/cilium/cilium/pkg/k8s/informer.(*privateRunner).Run(0x40003c9160, 0x400010f920)                                                                       

	/home/travis/gopath/src/github.com/cilium/cilium/pkg/k8s/informer/informer.go:46 +0xa8                                                                           

github.com/cilium/cilium/pkg/k8s/identitybackend.(*crdBackend).ListAndWatch(0x40003d5f20, {0x0?, 0x0?}, {0x23348c0?, 0x400054e760}, 0x400010f920)                

	/home/travis/gopath/src/github.com/cilium/cilium/pkg/k8s/identitybackend/identity.go:407 +0x540                                                                  

created by github.com/cilium/cilium/pkg/k8s/identitybackend.TestGetIdentity.func1 in goroutine 10                                                                

	/home/travis/gopath/src/github.com/cilium/cilium/pkg/k8s/identitybackend/identity_test.go:225 +0x2e4                                                             

                                                                                                                                                                 

goroutine 23 [sync.Cond.Wait, 10 minutes]:                                                                                                                       

sync.runtime_notifyListWait(0x400024df18, 0x0)                                                                                                                   

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/runtime/sema.go:527 +0x154                                                                                 

sync.(*Cond).Wait(0x400024df08)                                                                                                                                  

	/home/travis/.gimme/versions/go1.21.6.linux.arm64/src/sync/cond.go:70 +0xcc                                                                                      

k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0x400024def0, 0x40006520c0)                                                                                        

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x1dc                                                    

k8s.io/client-go/tools/cache.(*controller).processLoop(0x4000476640)                                                                                             

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x3c                                                     

k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x400064c270?)                                                                                              

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x40                                                   

k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x400025c4e0?, {0x230de60, 0x400064a660}, 0x1, 0x400005a420)                                                      

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0x90                                                   

k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x40002be120?, 0x3b9aca00, 0x0, 0xc0?, 0xffff64c40d58?)                                                            

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x80                                                   

k8s.io/apimachinery/pkg/util/wait.Until(...)                                                                                                                     

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161                                                         

k8s.io/client-go/tools/cache.(*controller).Run(0x4000476640, 0x400005a420)                                                                                       

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x2d0                                                    

github.com/cilium/cilium/pkg/k8s/informer.(*privateRunner).Run(0x4000662100, 0x400005a420)                                                                       

	/home/travis/gopath/src/github.com/cilium/cilium/pkg/k8s/informer/informer.go:46 +0xa8                                                                           

github.com/cilium/cilium/pkg/k8s/identitybackend.(*crdBackend).ListAndWatch(0x400064a510, {0x0?, 0x0?}, {0x23348c0?, 0x4000380670}, 0x400005a420)                

	/home/travis/gopath/src/github.com/cilium/cilium/pkg/k8s/identitybackend/identity.go:407 +0x540                                                                  

created by github.com/cilium/cilium/pkg/k8s/identitybackend.TestGetIdentity.func1 in goroutine 22                                                                

	/home/travis/gopath/src/github.com/cilium/cilium/pkg/k8s/identitybackend/identity_test.go:225 +0x2e4                                                             

                                                                                                                                                                 

goroutine 15 [select, 10 minutes]:                                                                                                                               

k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x36a1260?}, {0x231d658, 0x400037d1d0}, {0xffff64c40d58, 0x4000696420}, {0x23556a8?, 0x1f55f40}, 0x0, ...)

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:724 +0x120                                                     

k8s.io/client-go/tools/cache.(*Reflector).watch(0x4000682e00, {0x0?, 0x0?}, 0x400010f920, 0x0?)                                                                  

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:437 +0x3e0                                                     

k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0x4000682e00, 0x400010f920)                                                                               

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x3f4                                                     

k8s.io/client-go/tools/cache.(*Reflector).Run.func1()                                                                                                            

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x28                                                      

k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)                                                                                                       

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x40                                                   

k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000332b40?, {0x230de80, 0x4000399f90}, 0x1, 0x400010f920)                                                      

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0x90                                                   

k8s.io/client-go/tools/cache.(*Reflector).Run(0x4000682e00, 0x400010f920)                                                                                        

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x184                                                     

k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()                                                                                 

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x2c                                                       

k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()                                                                                                         

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x58                                                       

created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 11                                                                                      

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x7c                                                       

                                                                                                                                                                 

goroutine 33 [select, 10 minutes]:                                                                                                                               

k8s.io/client-go/tools/cache.(*Reflector).startResync(0x4000682e00, 0x400010f920, 0x400010fa40, 0x4000079708?)                                                   

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:369 +0xbc                                                      

created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 15                                                                                

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:356 +0x3dc                                                     

                                                                                                                                                                 

goroutine 26 [chan receive, 10 minutes]:                                                                                                                         

k8s.io/client-go/tools/cache.(*controller).Run.func1()                                                                                                           

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x2c                                                     

created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 23                                                                                        

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x94                                                     

                                                                                                                                                                 

goroutine 27 [select, 10 minutes]:                                                                                                                               

k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x36a1260?}, {0x231d658, 0x400037d218}, {0xffff64c40d58, 0x400024def0}, {0x23556a8?, 0x1f55f40}, 0x0, ...)

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:724 +0x120                                                     

k8s.io/client-go/tools/cache.(*Reflector).watch(0x4000191dc0, {0x0?, 0x0?}, 0x400005a420, 0x0?)                                                                  

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:437 +0x3e0                                                     

k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0x4000191dc0, 0x400005a420)                                                                               

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x3f4                                                     

k8s.io/client-go/tools/cache.(*Reflector).Run.func1()                                                                                                            

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x28                                                      

k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)                                                                                                       

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x40                                                   

k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x400065e040?, {0x230de80, 0x4000648230}, 0x1, 0x400005a420)                                                      

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0x90                                                   

k8s.io/client-go/tools/cache.(*Reflector).Run(0x4000191dc0, 0x400005a420)                                                                                        

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x184                                                     

k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()                                                                                 

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x2c                                                       

k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()                                                                                                         

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x58                                                       

created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 23                                                                                      

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x7c                                                       

                                                                                                                                                                 

goroutine 29 [select, 10 minutes]:                                                                                                                               

k8s.io/client-go/tools/cache.(*Reflector).startResync(0x4000191dc0, 0x400005a420, 0x400005a540, 0x4000673f08?)                                                   

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:369 +0xbc                                                      

created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 27                                                                                

	/home/travis/gopath/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/reflector.go:356 +0x3dc                                                     

FAIL	github.com/cilium/cilium/pkg/k8s/identitybackend	600.033s

log.txt
@lmb lmb added area/CI Continuous Integration testing issue or flake sig/policy Impacts whether traffic is allowed or denied based on user-defined policies. ci/flake This is a known failure that occurs in the tree. Please investigate me! sig/agent Cilium agent related. labels Feb 20, 2024
@bimmlerd bimmlerd self-assigned this Feb 21, 2024
@bimmlerd
Copy link
Member

It's a test bug. There's a race between when the call to Watch reaches the k8s fake client object tracker and the Create calls for the cilium identities. If the watch happens first, all is well. If it doesn't the addWaitGroup.Done() calls don't occur and we wait until the test times out. This is easily reproducible locally, I use while true; do go test ./pkg/k8s/identitybackend -run 'TestGetIdentity' -v -count=1 -timeout=1m -- -v=1 || break; done.

To test this hypothesis, I added a time.Sleep(time.Second) before identity creation, which is a crude fix for the race, but obviously not suited for CI robustness. Looking into how to ensure the watch happened deterministically now.

This was referenced Feb 21, 2024
Merged
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bimmlerd bimmlerd

Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! sig/agent Cilium agent related. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Address race condition in TestGetIdentity bimmlerd/cilium
2 participants
@lmb @bimmlerd