Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: Integration test: panic (timeout) for github.com/cilium/cilium/pkg/kvstore/store #24220

Closed
qmonnet opened this issue Mar 7, 2023 · 2 comments
Closed

CI: Integration test: panic (timeout) for github.com/cilium/cilium/pkg/kvstore/store #24220

qmonnet opened this issue Mar 7, 2023 · 2 comments
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

Comments

@qmonnet
Copy link
Member

qmonnet commented Mar 7, 2023

CI failure

Integration tests for package github.com/cilium/cilium/pkg/kvstore/store time out after 10 minutes, and panic.

Output:


┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓                                              
┃   PANIC  package: github.com/cilium/cilium/pkg/kvstore/store • Test   ┃                                              
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛      
panic: test timed out after 10m0s                                                                                      
running tests:                                                                                                         
	Test (10m0s)                                                                                                           
                                                                                                                       
goroutine 474 [running]:                                                                                               
testing.(*M).startAlarm.func1()                                                                                        
	/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:2241 +0x3b9                                                  
created by time.goFunc                                                                                                 
	/opt/hostedtoolcache/go/1.20.1/x64/src/time/sleep.go:176 +0x32                                                         
                                                                                                                       
goroutine 1 [chan receive, 10 minutes]:                                                                                
testing.(*T).Run(0xc000103ba0, {0x1077ace?, 0x56a3a5?}, 0x10e3a98)                                                     
	/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1630 +0x405                                                  
testing.runTests.func1(0x192e0c0?)                                                                                     
	/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:2036 +0x45                                                   
testing.tRunner(0xc000103ba0, 0xc0005bfc80)                                                                            
	/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1576 +0x10b                                                  
testing.runTests(0xc00032bcc0?, {0x1909f70, 0x1, 0x1}, {0xc000339b88?, 0x100c000339af0?, 0x192d580?})                  
	/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:2034 +0x489                                                  
testing.(*M).Run(0xc00032bcc0)                                                                                         
	/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1906 +0x63a                                                  
main.main()                                                                                                            
	_testmain.go:78 +0x1c5                                                                                                 
                                                                                                                       
goroutine 36 [chan receive, 9 minutes]:                                                                                
gopkg.in/check%2ev1.(*suiteRunner).runTest(...)                                                                        
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/check.go:813                                                  
gopkg.in/check%2ev1.(*suiteRunner).run(0xc000021200)                                                                   
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/check.go:618 +0x1bd                                           
gopkg.in/check%2ev1.Run({0xfb9b40?, 0x195cb40?}, 0x18cc1c0?)                                                           
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/run.go:92 +0x2a                                               
gopkg.in/check%2ev1.RunAll(0x18cc1c0?)                                                                                 
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/run.go:84 +0x85                                               
gopkg.in/check%2ev1.TestingT(0xc000103d40)                                                                             
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/run.go:72 +0x27c                                              
github.com/cilium/cilium/pkg/kvstore/store.Test(0x406d19?)                                                             
	/home/runner/work/cilium/cilium/pkg/kvstore/store/store_test.go:30 +0x19                                               
testing.tRunner(0xc000103d40, 0x10e3a98)                                                                               
	/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1576 +0x10b                                                  
created by testing.(*T).Run                                                                                            
	/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1629 +0x3ea                                                  
                                                                                                                       
goroutine 329 [select, 9 minutes]:                                                                                     
gopkg.in/check%2ev1.(*resultTracker)._loopRoutine(0xc000998090)                                                        
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/check.go:464 +0x8d                                            
created by gopkg.in/check%2ev1.(*resultTracker).start                                                                  
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/check.go:444 +0x56                                            
                                                                                                                       
goroutine 446 [select, 9 minutes]:                                                                                     
github.com/cilium/cilium/pkg/controller.(*Controller).runController(0xc0005e0ea0)                                      
	/home/runner/work/cilium/cilium/pkg/controller/controller.go:275 +0x1035                                               
created by github.com/cilium/cilium/pkg/controller.(*Manager).updateController                                         
	/home/runner/work/cilium/cilium/pkg/controller/manager.go:111 +0xb51                                                   
                                                                                                                       
goroutine 448 [chan receive, 9 minutes]:                                                                               
github.com/cilium/cilium/pkg/kvstore/store.(*SharedStore).watcher(0xc0003e69c0, 0xc00047c660?)                         
	/home/runner/work/cilium/cilium/pkg/kvstore/store/store.go:449 +0x14f                                                  
created by github.com/cilium/cilium/pkg/kvstore/store.(*SharedStore).listAndStartWatcher                               
	/home/runner/work/cilium/cilium/pkg/kvstore/store/store.go:435 +0xad                                                   
                                                                                                                       
goroutine 45 [chan receive, 9 minutes]:                                                                                
github.com/cilium/cilium/pkg/kvstore.connectEtcdClient.func4()                                                         
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:771 +0xd2                                                          
created by github.com/cilium/cilium/pkg/kvstore.connectEtcdClient                                                      
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:766 +0xa1e                                                         
                                                                                                                       
goroutine 397 [semacquire, 9 minutes]:                                                                                 
sync.runtime_Semacquire(0x1000100?)                                                                                    
	/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/sema.go:62 +0x27                                                        
sync.(*WaitGroup).Wait(0xc000222fc0?)                                                                                  
	/opt/hostedtoolcache/go/1.20.1/x64/src/sync/waitgroup.go:116 +0x4b                                                     
github.com/cilium/cilium/pkg/kvstore.(*Watcher).Stop.func1()                                                           
	/home/runner/work/cilium/cilium/pkg/kvstore/events.go:110 +0x128                                                       
sync.(*Once).doSlow(0xc0003b9130?, 0x0?)                                                                               
	/opt/hostedtoolcache/go/1.20.1/x64/src/sync/once.go:74 +0xc2                                                           
sync.(*Once).Do(...)                                                                                                   
	/opt/hostedtoolcache/go/1.20.1/x64/src/sync/once.go:65                                                                 
github.com/cilium/cilium/pkg/kvstore.(*Watcher).Stop(0x0?)                                                             
	/home/runner/work/cilium/cilium/pkg/kvstore/events.go:107 +0x45                                                        
github.com/cilium/cilium/pkg/kvstore/store.(*SharedStore).Release(0xc0003e69c0)                                        
	/home/runner/work/cilium/cilium/pkg/kvstore/store/store.go:241 +0xb6                                                   
github.com/cilium/cilium/pkg/kvstore/store.(*SharedStore).Close(0xc0003e69c0, {0x11ec060, 0xc0001ac008})               
	/home/runner/work/cilium/cilium/pkg/kvstore/store/store.go:251 +0x6c                                                   
github.com/cilium/cilium/pkg/kvstore/store.(*StoreSuite).TestStoreOperations(0x0?, 0x0?)                               
	/home/runner/work/cilium/cilium/pkg/kvstore/store/store_test.go:214 +0x832                                             
reflect.Value.call({0xfb9b40?, 0x195cb40?, 0x1613?}, {0x10778b2, 0x4}, {0xc0002f1f08, 0x1, 0x1?})                      
	/opt/hostedtoolcache/go/1.20.1/x64/src/reflect/value.go:586 +0xb07                                                     
reflect.Value.Call({0xfb9b40?, 0x195cb40?, 0xc000094570?}, {0xc000121f08?, 0x0?, 0xc000121e96?})                       
	/opt/hostedtoolcache/go/1.20.1/x64/src/reflect/value.go:370 +0xbc                                                      
gopkg.in/check%2ev1.(*suiteRunner).forkTest.func1(0xc0003b63c0)                                                        
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/check.go:775 +0x632                                           
gopkg.in/check%2ev1.(*suiteRunner).forkCall.func1()                                                                    
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/check.go:669 +0xa2                                            
created by gopkg.in/check%2ev1.(*suiteRunner).forkCall                                                                 
	/home/runner/work/cilium/cilium/vendor/gopkg.in/check.v1/check.go:666 +0x285                                           
                                                                                                                       
goroutine 177 [chan receive, 9 minutes]:                                                                               
github.com/cilium/cilium/pkg/kvstore.connectEtcdClient.func4()                                                         
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:771 +0xd2                                                          
created by github.com/cilium/cilium/pkg/kvstore.connectEtcdClient                                                      
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:766 +0xa1e                                                         
                                                                                                                       
goroutine 293 [chan receive, 9 minutes]:                                                                               
github.com/cilium/cilium/pkg/kvstore.connectEtcdClient.func4()                                                         
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:771 +0xd2                                                          
created by github.com/cilium/cilium/pkg/kvstore.connectEtcdClient                                                      
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:766 +0xa1e                                                         
                                                                                                                       
goroutine 113 [chan receive, 9 minutes]:                                                                               
github.com/cilium/cilium/pkg/kvstore.connectEtcdClient.func4()                                                         
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:771 +0xd2                                                          
created by github.com/cilium/cilium/pkg/kvstore.connectEtcdClient                                                      
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:766 +0xa1e                                                         
                                                                                                                       
goroutine 209 [chan receive, 9 minutes]:                                                                               
github.com/cilium/cilium/pkg/kvstore.connectEtcdClient.func4()                                                         
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:771 +0xd2                                                          
created by github.com/cilium/cilium/pkg/kvstore.connectEtcdClient                                                      
	/home/runner/work/cilium/cilium/pkg/kvstore/etcd.go:766 +0xa1e                                                         
                                                                                                                       
goroutine 465 [select, 9 minutes]:                                                                                     
net/http.(*http2ClientConn).RoundTrip(0xc000a10180, 0xc000832b00)                                                      
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:8231 +0x491                                               
net/http.(*http2Transport).RoundTripOpt(0xc000017320, 0xc000832b00, {0xc0?})                                           
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:7523 +0x1c5                                               
net/http.(*http2Transport).RoundTrip(...)                                                                              
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:7475                                                      
net/http.http2noDialH2RoundTripper.RoundTrip({0xc000168dc0?}, 0xc000832b00?)                                           
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:10060 +0x1b                                               
net/http.(*Transport).roundTrip(0xc000168dc0, 0xc000832b00)                                                            
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/transport.go:548 +0x3ca                                                
net/http.(*Transport).RoundTrip(0x5b924f?, 0x11e3260?)                                                                 
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/roundtrip.go:17 +0x19                                                  
net/http.send(0xc000832b00, {0x11e3260, 0xc000168dc0}, {0x8?, 0x1052800?, 0x0?})                                       
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/client.go:252 +0x5f7                                                   
net/http.(*Client).send(0xc0005bc3c0, 0xc000832b00, {0x7f7f74e59f18?, 0x100?, 0x0?})                                   
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/client.go:176 +0x9b                                                    
net/http.(*Client).do(0xc0005bc3c0, 0xc000832b00)                                                                      
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/client.go:716 +0x8fb                                                   
net/http.(*Client).Do(...)                                                                                             
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/client.go:582                                                          
github.com/hashicorp/consul/api.(*Client).doRequest(0xc0009842c0, 0xc00056a780?)                                       
	/home/runner/work/cilium/cilium/vendor/github.com/hashicorp/consul/api/api.go:1026 +0x6d                               
github.com/hashicorp/consul/api.(*KV).getInternal(0xc00007b558, {0xc0008c6df0?, 0x107ba0e?}, 0x7?, 0xc00007b300?)      
	/home/runner/work/cilium/cilium/vendor/github.com/hashicorp/consul/api/kv.go:135 +0x22a                                
github.com/hashicorp/consul/api.(*KV).List(0xc00007b590?, {0xc0008c6df0, 0xc}, 0x0?)                                   
	/home/runner/work/cilium/cilium/vendor/github.com/hashicorp/consul/api/kv.go:90 +0x13c                                 
github.com/cilium/cilium/pkg/kvstore.(*consulClient).Watch(0xc000148410, {0x11ec060?, 0xc0001ac000}, 0xc0008840a0)     
	/home/runner/work/cilium/cilium/pkg/kvstore/consul.go:320 +0x227                                                       
created by github.com/cilium/cilium/pkg/kvstore.(*consulClient).ListAndWatch                                           
	/home/runner/work/cilium/cilium/pkg/kvstore/consul.go:762 +0x1ea                                                       
                                                                                                                       
goroutine 443 [IO wait, 9 minutes]:                                                                                    
internal/poll.runtime_pollWait(0x7f7f4dabc948, 0x72)                                                                   
	/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/netpoll.go:306 +0x89                                                    
internal/poll.(*pollDesc).wait(0xc000578f00?, 0xc000b20000?, 0x0)                                                      
	/opt/hostedtoolcache/go/1.20.1/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32                                       
internal/poll.(*pollDesc).waitRead(...)                                                                                
	/opt/hostedtoolcache/go/1.20.1/x64/src/internal/poll/fd_poll_runtime.go:89                                             
internal/poll.(*FD).Read(0xc000578f00, {0xc000b20000, 0x800, 0x800})                                                   
	/opt/hostedtoolcache/go/1.20.1/x64/src/internal/poll/fd_unix.go:167 +0x299                                             
net.(*netFD).Read(0xc000578f00, {0xc000b20000?, 0xc000b2003f?, 0x36?})                                                 
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/fd_posix.go:55 +0x29                                                        
net.(*conn).Read(0xc0005f0268, {0xc000b20000?, 0x0?, 0xc0003af330?})                                                   
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/net.go:183 +0x45                                                            
crypto/tls.(*atLeastReader).Read(0xc000442c78, {0xc000b20000?, 0xc000442c78?, 0x0?})                                   
	/opt/hostedtoolcache/go/1.20.1/x64/src/crypto/tls/conn.go:788 +0x3d                                                    
bytes.(*Buffer).ReadFrom(0xc0003af410, {0x11df980, 0xc000442c78})                                                      
	/opt/hostedtoolcache/go/1.20.1/x64/src/bytes/buffer.go:202 +0x98                                                       
crypto/tls.(*Conn).readFromUntil(0xc0003af180, {0x11e31c0?, 0xc0005f0268}, 0x7c6?)                                     
	/opt/hostedtoolcache/go/1.20.1/x64/src/crypto/tls/conn.go:810 +0xe5                                                    
crypto/tls.(*Conn).readRecordOrCCS(0xc0003af180, 0x0)                                                                  
	/opt/hostedtoolcache/go/1.20.1/x64/src/crypto/tls/conn.go:617 +0x116                                                   
crypto/tls.(*Conn).readRecord(...)                                                                                     
	/opt/hostedtoolcache/go/1.20.1/x64/src/crypto/tls/conn.go:583                                                          
crypto/tls.(*Conn).Read(0xc0003af180, {0xc000b30000, 0x1000, 0x7cea80?})                                               
	/opt/hostedtoolcache/go/1.20.1/x64/src/crypto/tls/conn.go:1316 +0x16f                                                  
bufio.(*Reader).Read(0xc0004b95c0, {0xc0005be4a0, 0x9, 0x7dcfa5?})                                                     
	/opt/hostedtoolcache/go/1.20.1/x64/src/bufio/bufio.go:237 +0x1bb                                                       
io.ReadAtLeast({0x11df7c0, 0xc0004b95c0}, {0xc0005be4a0, 0x9, 0x9}, 0x9)                                               
	/opt/hostedtoolcache/go/1.20.1/x64/src/io/io.go:332 +0x9a                                                              
io.ReadFull(...)                                                                                                       
	/opt/hostedtoolcache/go/1.20.1/x64/src/io/io.go:351                                                                    
net/http.http2readFrameHeader({0xc0005be4a0?, 0x9?, 0xc00037b920?}, {0x11df7c0?, 0xc0004b95c0?})                       
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:1567 +0x6e                                                
net/http.(*http2Framer).ReadFrame(0xc0005be460)                                                                        
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:1831 +0x95                                                
net/http.(*http2clientConnReadLoop).run(0xc000189f98)                                                                  
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:9187 +0x12e                                               
net/http.(*http2ClientConn).readLoop(0xc000a10180)                                                                     
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:9082 +0x6f                                                
created by net/http.(*http2Transport).newClientConn                                                                    
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:7779 +0xc3c                                               
                                                                                                                       
goroutine 429 [sync.RWMutex.RLock, 7 minutes]:                                                                         
sync.runtime_SemacquireRWMutexR(0xc000a42ef8?, 0x4?, 0xc00022e800?)                                                    
	/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/sema.go:82 +0x26                                                        
sync.(*RWMutex).RLock(...)                                                                                             
	/opt/hostedtoolcache/go/1.20.1/x64/src/sync/rwmutex.go:71                                                              
github.com/cilium/cilium/pkg/kvstore/store.(*SharedStore).syncLocalKeys(0xc0003e69c0, {0x11ec028, 0xc0009961e0}, 0x17?)
	/home/runner/work/cilium/cilium/pkg/kvstore/store/store.go:294 +0x8f                                                   
github.com/cilium/cilium/pkg/kvstore/store.JoinSharedStore.func1({0x11ec028?, 0xc0009961e0?})                          
	/home/runner/work/cilium/cilium/pkg/kvstore/store/store.go:211 +0x3b                                                   
github.com/cilium/cilium/pkg/controller.(*Controller).runController(0xc0000b0360)                                      
	/home/runner/work/cilium/cilium/pkg/controller/controller.go:211 +0x1cf                                                
created by github.com/cilium/cilium/pkg/controller.(*Manager).updateController                                         
	/home/runner/work/cilium/cilium/pkg/controller/manager.go:111 +0xb51                                                   
                                                                                                                       
goroutine 384 [select, 9 minutes]:                                                                                     
net/http.(*http2clientStream).writeRequest(0xc000a10a80, 0xc000832b00)                                                 
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:8399 +0xa67                                               
net/http.(*http2clientStream).doRequest(0xc000546000?, 0xc0003b65a0?)                                                  
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:8261 +0x1e                                                
created by net/http.(*http2ClientConn).RoundTrip                                                                       
	/opt/hostedtoolcache/go/1.20.1/x64/src/net/http/h2_bundle.go:8190 +0x34a                                               
FAIL	github.com/cilium/cilium/pkg/kvstore/store	600.025s
@qmonnet qmonnet added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels Mar 7, 2023
@qmonnet qmonnet changed the title CI: Integration test: panic for github.com/cilium/cilium/pkg/kvstore/store CI: Integration test: panic (timeout) for github.com/cilium/cilium/pkg/kvstore/store Mar 7, 2023
@github-actions
Copy link

github-actions bot commented May 7, 2023

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label May 7, 2023
@github-actions
Copy link

This issue has not seen any activity since it was marked stale.
Closing.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@qmonnet