Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: RuntimeLB With Containers validates service recovery on restart #17629

Closed
maintainer-s-little-helper bot opened this issue Oct 18, 2021 · 8 comments
Closed

CI: RuntimeLB With Containers validates service recovery on restart #17629

maintainer-s-little-helper bot opened this issue Oct 18, 2021 · 8 comments
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

Comments

@maintainer-s-little-helper
Copy link

Test Name

RuntimeLB With Containers validates service recovery on restart

Failure Output

FAIL: Found 1 Cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 Cilium logs matching list of errors that must be investigated:
level=error
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
⚠️  Found "level=error" in logs 1 times
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
⚠️  Number of "Goroutine took lock for more than" in logs: 7
Top 2 errors/warnings:
endpoint regeneration failed
unable to queue endpoint build

Standard Error

Click to show. 
11:14:20 STEP: Running BeforeEach block for EntireTestsuite RuntimeLB
11:14:20 STEP: Running BeforeAll block for EntireTestsuite RuntimeLB With Containers
11:14:20 STEP: Creating containers for traffic test
11:14:26 STEP: Making HTTP request via the service before restart
11:14:26 STEP: Fetching service state before restart
11:14:27 STEP: Restarting Cilium
11:14:34 STEP: Checking that the service was restored correctly
11:14:34 STEP: Checking that BPF LB maps match the service
11:14:34 STEP: Making HTTP request via the service after restart
=== Test Finished at 2021-10-15T11:14:34Z====
11:14:34 STEP: Running JustAfterEach block for EntireTestsuite RuntimeLB
FAIL: Found 1 Cilium logs matching list of errors that must be investigated:
level=error
===================== TEST FAILED =====================
11:14:35 STEP: Running AfterFailed block for EntireTestsuite RuntimeLB
cmd: sudo cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                     
	 1          Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:fc8c   10.15.103.10    ready   
	 8          Disabled           Disabled          18205      container:id.httpd2           f00d::a0f:0:0:53bf   10.15.206.140   ready   
	                                                            container:id.service1                                                      
	 1221       Disabled           Disabled          1          reserved:host                                                      ready   
	 1477       Disabled           Disabled          34489      container:id.httpd1           f00d::a0f:0:0:81ad   10.15.126.107   ready   
	                                                            container:id.service1                                                      
	 2179       Disabled           Disabled          46749      container:id.client           f00d::a0f:0:0:9a03   10.15.152.14    ready   
	 
Stderr:
 	 

cmd: sudo cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend     Service Type   Backend                 
	 1    2.2.2.2:80   ClusterIP      1 => 10.15.126.107:80   
	                                  2 => 10.15.206.140:80   
	 
Stderr:
 	 

cmd: sudo cilium bpf lb list
Exitcode: 0 
Stdout:
 	 SERVICE ADDRESS   BACKEND ADDRESS
	 2.2.2.2:80        10.15.206.140:80 (1)                      
	                   10.15.126.107:80 (1)                      
	                   0.0.0.0:0 (1) [ClusterIP, non-routable]   
	 
Stderr:
 	 goroutine 1 [running]:
	 goroutine 1 runtime/debug.Stack()
	 goroutine 1 	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
	 goroutine 1 github.com/cilium/cilium/pkg/lock.(*internalRWMutex).Unlock(0xc0004c56f8)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/lock/lock_debug.go:58 +0x92
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).Open(0xc0004c5680)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:571 +0x95
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallback(0xc0004c5680, 0xc0006d3c48)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:632 +0x5b
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallbackIfExists(0xb, 0xc00087f7d0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:705 +0x3d
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.dumpSVC(0xc00087f7d0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:46 +0xb9
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.glob..func13(0x3718180, {0x20689be, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:115 +0x67
	 goroutine 1 github.com/spf13/cobra.(*Command).execute(0x3718180, {0x3cb80c8, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:856 +0x5f8
	 goroutine 1 github.com/spf13/cobra.(*Command).ExecuteC(0x3715200)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:960 +0x3ad
	 goroutine 1 github.com/spf13/cobra.(*Command).Execute(...)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:897
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.Execute()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/root.go:36 +0x25
	 goroutine 1 main.main()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/main.go:16 +0x17
	 

cmd: sudo cilium policy get
Exitcode: 0 
Stdout:
 	 []
	 Revision: 1
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
11:14:41 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|147e88a4_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip]]
11:14:43 STEP: Running AfterAll block for EntireTestsuite RuntimeLB With Containers

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//352/artifact/147e88a4_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip
https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//352/artifact/test_results_Cilium-PR-Runtime-net-next_352_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/352/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
@maintainer-s-little-helper maintainer-s-little-helper bot added the ci/flake This is a known failure that occurs in the tree. Please investigate me! label Oct 18, 2021
Merged
@maintainer-s-little-helper
Copy link
Author

PR #16525 hit this flake with 98.39% similarity:

Click to show.

Test Name

RuntimeLB With Containers validates service recovery on restart

Failure Output

FAIL: Found 1 Cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 Cilium logs matching list of errors that must be investigated:
level=error
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
⚠️  Found "level=error" in logs 1 times
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
⚠️  Number of "Goroutine took lock for more than" in logs: 7
Top 2 errors/warnings:
endpoint regeneration failed
unable to queue endpoint build

Standard Error

Click to show. 
19:39:34 STEP: Running BeforeEach block for EntireTestsuite RuntimeLB
19:39:34 STEP: Running BeforeAll block for EntireTestsuite RuntimeLB With Containers
19:39:34 STEP: Creating containers for traffic test
19:39:37 STEP: Making HTTP request via the service before restart
19:39:37 STEP: Fetching service state before restart
19:39:38 STEP: Restarting Cilium
19:39:45 STEP: Checking that the service was restored correctly
19:39:45 STEP: Checking that BPF LB maps match the service
19:39:46 STEP: Making HTTP request via the service after restart
=== Test Finished at 2021-10-20T19:39:46Z====
19:39:46 STEP: Running JustAfterEach block for EntireTestsuite RuntimeLB
FAIL: Found 1 Cilium logs matching list of errors that must be investigated:
level=error
===================== TEST FAILED =====================
19:39:46 STEP: Running AfterFailed block for EntireTestsuite RuntimeLB
cmd: sudo cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                     
	 700        Disabled           Disabled          1          reserved:host                                                      ready   
	 1152       Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:93d6   10.15.75.1      ready   
	 1875       Disabled           Disabled          15913      container:id.client           f00d::a0f:0:0:ecf    10.15.252.129   ready   
	 2144       Disabled           Disabled          52587      container:id.httpd1           f00d::a0f:0:0:3436   10.15.56.214    ready   
	                                                            container:id.service1                                                      
	 3039       Disabled           Disabled          8400       container:id.httpd2           f00d::a0f:0:0:688    10.15.72.13     ready   
	                                                            container:id.service1                                                      
	 
Stderr:
 	 

cmd: sudo cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend     Service Type   Backend                
	 1    2.2.2.2:80   ClusterIP      1 => 10.15.56.214:80   
	                                  2 => 10.15.72.13:80    
	 
Stderr:
 	 

cmd: sudo cilium bpf lb list
Exitcode: 0 
Stdout:
 	 SERVICE ADDRESS   BACKEND ADDRESS
	 2.2.2.2:80        10.15.56.214:80 (1)                       
	                   0.0.0.0:0 (1) [ClusterIP, non-routable]   
	                   10.15.72.13:80 (1)                        
	 
Stderr:
 	 goroutine 1 [running]:
	 goroutine 1 runtime/debug.Stack()
	 goroutine 1 	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
	 goroutine 1 github.com/cilium/cilium/pkg/lock.(*internalRWMutex).Unlock(0xc000275818)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/lock/lock_debug.go:58 +0x92
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).Open(0xc0002757a0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:571 +0x95
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallback(0xc0002757a0, 0xc00073bc48)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:632 +0x5b
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallbackIfExists(0xb, 0xc000240270)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:705 +0x3d
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.dumpSVC(0xc000240270)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:46 +0xb9
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.glob..func13(0x37232e0, {0x206da53, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:115 +0x67
	 goroutine 1 github.com/spf13/cobra.(*Command).execute(0x37232e0, {0x3cc3268, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
	 goroutine 1 github.com/spf13/cobra.(*Command).ExecuteC(0x3720360)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
	 goroutine 1 github.com/spf13/cobra.(*Command).Execute(...)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:902
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.Execute()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/root.go:36 +0x25
	 goroutine 1 main.main()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/main.go:16 +0x17
	 

cmd: sudo cilium policy get
Exitcode: 0 
Stdout:
 	 []
	 Revision: 1
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
19:39:52 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|6638eff1_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip]]
19:39:53 STEP: Running AfterAll block for EntireTestsuite RuntimeLB With Containers

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//391/artifact/6638eff1_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip
https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//391/artifact/e95b58fa_RuntimeConntrackInVethModeTest_Conntrack-related_configuration_options_for_endpoints.zip
https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//391/artifact/test_results_Cilium-PR-Runtime-net-next_391_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/391/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #17639 hit this flake with 98.39% similarity:

Click to show.

Test Name

RuntimeLB With Containers validates service recovery on restart

Failure Output

FAIL: Found 1 Cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 Cilium logs matching list of errors that must be investigated:
level=error
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
⚠️  Found "level=error" in logs 1 times
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
⚠️  Number of "Goroutine took lock for more than" in logs: 7
Top 5 errors/warnings:
endpoint regeneration failed
[caught ENVOY_SIGTERM
[StreamNetworkPolicies gRPC config stream closed: 13, 
[StreamListeners gRPC config stream closed: 13, 
unable to queue endpoint build

Standard Error

Click to show. 
15:32:53 STEP: Running BeforeEach block for EntireTestsuite RuntimeLB
15:32:53 STEP: Running BeforeAll block for EntireTestsuite RuntimeLB With Containers
15:32:53 STEP: Creating containers for traffic test
15:32:56 STEP: Making HTTP request via the service before restart
15:32:56 STEP: Fetching service state before restart
15:32:56 STEP: Restarting Cilium
15:33:04 STEP: Checking that the service was restored correctly
15:33:04 STEP: Checking that BPF LB maps match the service
15:33:04 STEP: Making HTTP request via the service after restart
=== Test Finished at 2021-10-25T15:33:04Z====
15:33:04 STEP: Running JustAfterEach block for EntireTestsuite RuntimeLB
FAIL: Found 1 Cilium logs matching list of errors that must be investigated:
level=error
===================== TEST FAILED =====================
15:33:04 STEP: Running AfterFailed block for EntireTestsuite RuntimeLB
cmd: sudo cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                     
	 946        Disabled           Disabled          2024       container:id.httpd1           f00d::a0f:0:0:3e7d   10.15.140.198   ready   
	                                                            container:id.service1                                                      
	 1515       Disabled           Disabled          1          reserved:host                                                      ready   
	 1761       Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:ba61   10.15.0.135     ready   
	 2822       Disabled           Disabled          11643      container:id.httpd2           f00d::a0f:0:0:9f7b   10.15.221.74    ready   
	                                                            container:id.service1                                                      
	 3734       Disabled           Disabled          23864      container:id.client           f00d::a0f:0:0:cb59   10.15.76.128    ready   
	 
Stderr:
 	 

cmd: sudo cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend     Service Type   Backend                 
	 1    2.2.2.2:80   ClusterIP      1 => 10.15.140.198:80   
	                                  2 => 10.15.221.74:80    
	 
Stderr:
 	 

cmd: sudo cilium bpf lb list
Exitcode: 0 
Stdout:
 	 SERVICE ADDRESS   BACKEND ADDRESS
	 2.2.2.2:80        0.0.0.0:0 (1) [ClusterIP, non-routable]   
	                   10.15.140.198:80 (1)                      
	                   10.15.221.74:80 (1)                       
	 
Stderr:
 	 goroutine 1 [running]:
	 goroutine 1 runtime/debug.Stack()
	 goroutine 1 	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
	 goroutine 1 github.com/cilium/cilium/pkg/lock.(*internalRWMutex).Unlock(0xc0001f0858)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/lock/lock_debug.go:58 +0x92
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).Open(0xc0001f07e0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:571 +0x95
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallback(0xc0001f07e0, 0xc00079dc48)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:632 +0x5b
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallbackIfExists(0x2, 0xc0003d5a40)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:705 +0x3d
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.dumpSVC(0xc0003d5a40)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:46 +0xb9
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.glob..func13(0x3723300, {0x206dd93, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:115 +0x67
	 goroutine 1 github.com/spf13/cobra.(*Command).execute(0x3723300, {0x3cc31e8, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
	 goroutine 1 github.com/spf13/cobra.(*Command).ExecuteC(0x3720380)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
	 goroutine 1 github.com/spf13/cobra.(*Command).Execute(...)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:902
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.Execute()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/root.go:36 +0x25
	 goroutine 1 main.main()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/main.go:16 +0x17
	 

cmd: sudo cilium policy get
Exitcode: 0 
Stdout:
 	 []
	 Revision: 1
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
15:33:12 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|796362b2_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip]]
15:33:13 STEP: Running AfterAll block for EntireTestsuite RuntimeLB With Containers

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//430/artifact/796362b2_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip
https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//430/artifact/test_results_Cilium-PR-Runtime-net-next_430_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/430/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #17640 hit this flake with 98.39% similarity:

Click to show.

Test Name

RuntimeLB With Containers validates service recovery on restart

Failure Output

FAIL: Found 1 Cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 Cilium logs matching list of errors that must be investigated:
level=error
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
⚠️  Found "level=error" in logs 1 times
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
⚠️  Number of "Goroutine took lock for more than" in logs: 7
Top 2 errors/warnings:
endpoint regeneration failed
unable to queue endpoint build

Standard Error

Click to show. 
10:05:12 STEP: Running BeforeEach block for EntireTestsuite RuntimeLB
10:05:12 STEP: Running BeforeAll block for EntireTestsuite RuntimeLB With Containers
10:05:12 STEP: Creating containers for traffic test
10:05:15 STEP: Making HTTP request via the service before restart
10:05:15 STEP: Fetching service state before restart
10:05:16 STEP: Restarting Cilium
10:05:23 STEP: Checking that the service was restored correctly
10:05:23 STEP: Checking that BPF LB maps match the service
10:05:23 STEP: Making HTTP request via the service after restart
=== Test Finished at 2021-10-26T10:05:23Z====
10:05:23 STEP: Running JustAfterEach block for EntireTestsuite RuntimeLB
FAIL: Found 1 Cilium logs matching list of errors that must be investigated:
level=error
===================== TEST FAILED =====================
10:05:24 STEP: Running AfterFailed block for EntireTestsuite RuntimeLB
cmd: sudo cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4           STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                    
	 1010       Disabled           Disabled          27843      container:id.httpd2           f00d::a0f:0:0:ca2c   10.15.89.60    ready   
	                                                            container:id.service1                                                     
	 1156       Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:341b   10.15.37.109   ready   
	 1767       Disabled           Disabled          55425      container:id.client           f00d::a0f:0:0:1b79   10.15.7.208    ready   
	 2803       Disabled           Disabled          1          reserved:host                                                     ready   
	 3298       Disabled           Disabled          5258       container:id.httpd1           f00d::a0f:0:0:86e4   10.15.92.41    ready   
	                                                            container:id.service1                                                     
	 
Stderr:
 	 

cmd: sudo cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend     Service Type   Backend               
	 1    2.2.2.2:80   ClusterIP      1 => 10.15.92.41:80   
	                                  2 => 10.15.89.60:80   
	 
Stderr:
 	 

cmd: sudo cilium bpf lb list
Exitcode: 0 
Stdout:
 	 SERVICE ADDRESS   BACKEND ADDRESS
	 2.2.2.2:80        10.15.92.41:80 (1)                        
	                   0.0.0.0:0 (1) [ClusterIP, non-routable]   
	                   10.15.89.60:80 (1)                        
	 
Stderr:
 	 goroutine 1 [running]:
	 goroutine 1 runtime/debug.Stack()
	 goroutine 1 	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
	 goroutine 1 github.com/cilium/cilium/pkg/lock.(*internalRWMutex).Unlock(0xc00064c078)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/lock/lock_debug.go:58 +0x92
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).Open(0xc00064c000)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:571 +0x95
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallback(0xc00064c000, 0xc000827c48)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:632 +0x5b
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallbackIfExists(0xb, 0xc000565110)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:705 +0x3d
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.dumpSVC(0xc000565110)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:46 +0xb9
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.glob..func13(0x37232e0, {0x206ddcb, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:115 +0x67
	 goroutine 1 github.com/spf13/cobra.(*Command).execute(0x37232e0, {0x3cc31c8, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
	 goroutine 1 github.com/spf13/cobra.(*Command).ExecuteC(0x3720360)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
	 goroutine 1 github.com/spf13/cobra.(*Command).Execute(...)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:902
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.Execute()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/root.go:36 +0x25
	 goroutine 1 main.main()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/main.go:16 +0x17
	 

cmd: sudo cilium policy get
Exitcode: 0 
Stdout:
 	 []
	 Revision: 1
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
10:05:31 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|785e493f_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip]]
10:05:32 STEP: Running AfterAll block for EntireTestsuite RuntimeLB With Containers

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//439/artifact/785e493f_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip
https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//439/artifact/test_results_Cilium-PR-Runtime-net-next_439_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/439/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #17718 hit this flake with 98.39% similarity:

Click to show.

Test Name

RuntimeLB With Containers validates service recovery on restart

Failure Output

FAIL: Found 1 Cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 Cilium logs matching list of errors that must be investigated:
level=error
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
⚠️  Found "level=error" in logs 1 times
Number of "context deadline exceeded" in logs: 1
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
⚠️  Number of "Goroutine took lock for more than" in logs: 7
Top 2 errors/warnings:
endpoint regeneration failed
unable to queue endpoint build

Standard Error

Click to show. 
17:42:24 STEP: Running BeforeEach block for EntireTestsuite RuntimeLB
17:42:24 STEP: Running BeforeAll block for EntireTestsuite RuntimeLB With Containers
17:42:24 STEP: Creating containers for traffic test
17:42:27 STEP: Making HTTP request via the service before restart
17:42:27 STEP: Fetching service state before restart
17:42:28 STEP: Restarting Cilium
17:42:35 STEP: Checking that the service was restored correctly
17:42:35 STEP: Checking that BPF LB maps match the service
17:42:35 STEP: Making HTTP request via the service after restart
=== Test Finished at 2021-10-28T17:42:36Z====
17:42:36 STEP: Running JustAfterEach block for EntireTestsuite RuntimeLB
FAIL: Found 1 Cilium logs matching list of errors that must be investigated:
level=error
===================== TEST FAILED =====================
17:42:36 STEP: Running AfterFailed block for EntireTestsuite RuntimeLB
cmd: sudo cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                     
	 375        Disabled           Disabled          45513      container:id.httpd1           f00d::a0f:0:0:d6be   10.15.21.225    ready   
	                                                            container:id.service1                                                      
	 608        Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:47ce   10.15.101.10    ready   
	 1713       Disabled           Disabled          33417      container:id.httpd2           f00d::a0f:0:0:6acd   10.15.8.201     ready   
	                                                            container:id.service1                                                      
	 1789       Disabled           Disabled          13004      container:id.client           f00d::a0f:0:0:d375   10.15.167.228   ready   
	 2036       Disabled           Disabled          1          reserved:host                                                      ready   
	 
Stderr:
 	 

cmd: sudo cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend     Service Type   Backend                
	 1    2.2.2.2:80   ClusterIP      1 => 10.15.21.225:80   
	                                  2 => 10.15.8.201:80    
	 
Stderr:
 	 

cmd: sudo cilium bpf lb list
Exitcode: 0 
Stdout:
 	 SERVICE ADDRESS   BACKEND ADDRESS
	 2.2.2.2:80        10.15.21.225:80 (1)                       
	                   0.0.0.0:0 (1) [ClusterIP, non-routable]   
	                   10.15.8.201:80 (1)                        
	 
Stderr:
 	 goroutine 1 [running]:
	 goroutine 1 runtime/debug.Stack()
	 goroutine 1 	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
	 goroutine 1 github.com/cilium/cilium/pkg/lock.(*internalRWMutex).Unlock(0xc0000a7ed8)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/lock/lock_debug.go:58 +0x92
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).Open(0xc0000a7e60)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:571 +0x95
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallback(0xc0000a7e60, 0xc0006ebc48)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:632 +0x5b
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallbackIfExists(0xb, 0xc00030dcb0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:705 +0x3d
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.dumpSVC(0xc00030dcb0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:46 +0xb9
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.glob..func13(0x37babe0, {0x20ce2d0, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:115 +0x67
	 goroutine 1 github.com/spf13/cobra.(*Command).execute(0x37babe0, {0x3d5b070, 0x0, 0x0})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
	 goroutine 1 github.com/spf13/cobra.(*Command).ExecuteC(0x37b7c60)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
	 goroutine 1 github.com/spf13/cobra.(*Command).Execute(...)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:902
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.Execute()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/root.go:36 +0x25
	 goroutine 1 main.main()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/main.go:16 +0x17
	 

cmd: sudo cilium policy get
Exitcode: 0 
Stdout:
 	 []
	 Revision: 1
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
17:42:43 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|dcdca93c_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip]]
17:42:44 STEP: Running AfterAll block for EntireTestsuite RuntimeLB With Containers

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//475/artifact/dcdca93c_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip
https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//475/artifact/test_results_Cilium-PR-Runtime-net-next_475_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/475/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@aanm aanm added the area/CI Continuous Integration testing issue or flake label Jan 6, 2022
@maintainer-s-little-helper
Copy link
Author

PR #18415 hit this flake with 98.39% similarity:

Click to show.

Test Name

RuntimeLB With Containers validates service recovery on restart

Failure Output

FAIL: Found 1 Cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 Cilium logs matching list of errors that must be investigated:
level=error
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
⚠️  Found "level=error" in logs 1 times
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
⚠️  Number of "Goroutine took lock for more than" in logs: 6
Top 2 errors/warnings:
endpoint regeneration failed
unable to queue endpoint build

Standard Error

Click to show. 
11:31:54 STEP: Running BeforeEach block for EntireTestsuite RuntimeLB
11:31:54 STEP: Running BeforeAll block for EntireTestsuite RuntimeLB With Containers
11:31:54 STEP: Creating containers for traffic test
11:31:58 STEP: Making HTTP request via the service before restart
11:31:58 STEP: Fetching service state before restart
11:31:58 STEP: Restarting Cilium
11:32:04 STEP: Checking that the service was restored correctly
11:32:04 STEP: Checking that BPF LB maps match the service
11:32:05 STEP: Making HTTP request via the service after restart
=== Test Finished at 2022-01-10T11:32:05Z====
11:32:05 STEP: Running JustAfterEach block for EntireTestsuite RuntimeLB
FAIL: Found 1 Cilium logs matching list of errors that must be investigated:
level=error
===================== TEST FAILED =====================
11:32:05 STEP: Running AfterFailed block for EntireTestsuite RuntimeLB
cmd: sudo cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4           STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                    
	 327        Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:682e   10.15.80.46    ready   
	 891        Disabled           Disabled          15413      container:id.httpd1           f00d::a0f:0:0:4053   10.15.174.98   ready   
	                                                            container:id.service1                                                     
	 1488       Disabled           Disabled          61914      container:id.httpd2           f00d::a0f:0:0:ee94   10.15.184.62   ready   
	                                                            container:id.service1                                                     
	 2322       Disabled           Disabled          1          reserved:host                                                     ready   
	 2324       Disabled           Disabled          19224      container:id.client           f00d::a0f:0:0:9bd4   10.15.83.142   ready   
	 
Stderr:
 	 

cmd: sudo cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend     Service Type   Backend                
	 1    2.2.2.2:80   ClusterIP      1 => 10.15.174.98:80   
	                                  2 => 10.15.184.62:80   
	 
Stderr:
 	 

cmd: sudo cilium bpf lb list
Exitcode: 0 
Stdout:
 	 SERVICE ADDRESS   BACKEND ADDRESS
	 2.2.2.2:80        10.15.174.98:80 (1)                       
	                   0.0.0.0:0 (1) [ClusterIP, non-routable]   
	                   10.15.184.62:80 (1)                       
	 
Stderr:
 	 goroutine 1 [running]:
	 goroutine 1 runtime/debug.Stack(0xc06f242d35df5a4d, 0x1a60973, 0x3719ec0)
	 goroutine 1 	/usr/local/go/src/runtime/debug/stack.go:24 +0x9f
	 goroutine 1 github.com/cilium/cilium/pkg/lock.(*internalRWMutex).Unlock(0xc000470bb8)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/lock/lock_debug.go:68 +0xc5
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).Open(0xc000470b40, 0x0, 0x0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:581 +0x9f
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallback(0xc000470b40, 0xc0007d3c18, 0x0, 0x0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:642 +0x65
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).DumpWithCallbackIfExists(0xc000470b40, 0xc0007d3c18, 0x2a, 0x203000)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:715 +0x65
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.dumpSVC(0xc0005727e0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:57 +0xc5
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.glob..func13(0x36eca60, 0x3c74310, 0x0, 0x0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_lb_list.go:126 +0x15d
	 goroutine 1 github.com/spf13/cobra.(*Command).execute(0x36eca60, 0x3c74310, 0x0, 0x0, 0x36eca60, 0x3c74310)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
	 goroutine 1 github.com/spf13/cobra.(*Command).ExecuteC(0x36e95e0, 0x371b180, 0x0, 0xc000068778)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:958 +0x375
	 goroutine 1 github.com/spf13/cobra.(*Command).Execute(...)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:895
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.Execute()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/root.go:47 +0x31
	 goroutine 1 main.main()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/main.go:22 +0x25
	 

cmd: sudo cilium policy get
Exitcode: 0 
Stdout:
 	 []
	 Revision: 1
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
11:32:12 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|d3fcd2ff_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip]]
11:32:13 STEP: Running AfterAll block for EntireTestsuite RuntimeLB With Containers

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//954/artifact/d3fcd2ff_RuntimeLB_With_Containers_validates_service_recovery_on_restart.zip
https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//954/artifact/test_results_Cilium-PR-Runtime-net-next_954_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/954/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@github-actions
Copy link

github-actions bot commented Jul 9, 2022

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. and removed stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. labels Jul 9, 2022
@github-actions
Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Sep 14, 2022
@github-actions
Copy link

This issue has not seen any activity since it was marked stale.
Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@aanm