Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master #24379

Closed
maintainer-s-little-helper bot opened this issue Mar 15, 2023 · 1 comment
Closed

CI: K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master #24379

maintainer-s-little-helper bot opened this issue Mar 15, 2023 · 1 comment
Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me!

Comments

@maintainer-s-little-helper
Copy link

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: migrate-svc restart count values do not match

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
migrate-svc restart count values do not match
Expected
    <int>: 0
to be identical to
    <int>: 13
/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/k8s/updates.go:346

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
UpdateIdentities: Skipping Delete of a non-existing identity
Cilium pods: [cilium-6ld9g cilium-wgdlz]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::l7-policy 
Endpoint Policy Enforcement:
Pod                        Ingress   Egress
migrate-svc-server-h8nrx   false     false
migrate-svc-server-jwn9f   false     false
app1-755788dd65-6vxvq      false     false
app1-755788dd65-srrw5      false     false
app2-6dfff7bfbd-vhbkw      false     false
app3-68f6b87dd5-5qqvj      false     false
migrate-svc-client-4p7f9   false     false
migrate-svc-client-fzqwv   false     false
migrate-svc-client-cbd9t   false     false
migrate-svc-client-n6m9f   false     false
migrate-svc-client-tj5x7   false     false
migrate-svc-server-qrxc8   false     false
coredns-758664cbbf-zjzt7   false     false
Cilium agent 'cilium-6ld9g': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 55 Failed 0
Cilium agent 'cilium-wgdlz': Status: Ok  Health:  Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 50 Failed 0

Standard Error

Click to show. 
06:19:10 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
06:19:11 STEP: Ensuring the namespace kube-system exists
06:19:11 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
06:19:11 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
06:19:11 STEP: Waiting for pods to be terminated
06:19:26 STEP: Deleting Cilium and CoreDNS
06:19:27 STEP: Waiting for pods to be terminated
06:19:27 STEP: Cleaning Cilium state (201fc9400fc69e476eeca411bb1bcff65a7526e8)
06:19:27 STEP: Cleaning up Cilium components
06:19:28 STEP: Waiting for Cilium to become ready
06:19:42 STEP: Cleaning Cilium state (v1.12)
06:19:42 STEP: Cleaning up Cilium components
06:19:46 STEP: Waiting for Cilium to become ready
06:20:22 STEP: Deploying Cilium 1.12
06:20:22 STEP: Waiting for Cilium to become ready
06:21:06 STEP: Validating Cilium Installation
06:21:06 STEP: Performing Cilium controllers preflight check
06:21:06 STEP: Performing Cilium health check
06:21:06 STEP: Performing Cilium status preflight check
06:21:06 STEP: Checking whether host EP regenerated
06:21:14 STEP: Performing Cilium service preflight check
06:21:14 STEP: Performing K8s service preflight check
06:21:20 STEP: Waiting for cilium-operator to be ready
06:21:20 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
06:21:20 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
06:21:20 STEP: Cilium "1.12" is installed and running
06:21:20 STEP: Restarting DNS Pods
06:21:21 STEP: Waiting for kube-dns to be ready
06:21:21 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns")
06:21:24 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns") => <nil>
06:21:24 STEP: Running kube-dns preflight check
06:21:32 STEP: Performing K8s service preflight check
06:21:32 STEP: Creating some endpoints and L7 policy
06:21:32 STEP: WaitforPods(namespace="default", filter="-l zgroup=testapp")
06:21:43 STEP: WaitforPods(namespace="default", filter="-l zgroup=testapp") => <nil>
06:21:51 STEP: Creating service and clients for migration
06:21:51 STEP: WaitforPods(namespace="default", filter="-l app=migrate-svc-server")
06:21:53 STEP: WaitforPods(namespace="default", filter="-l app=migrate-svc-server") => <nil>
06:21:53 STEP: WaitforPods(namespace="default", filter="-l app=migrate-svc-client")
06:21:55 STEP: WaitforPods(namespace="default", filter="-l app=migrate-svc-client") => <nil>
06:21:55 STEP: Validate that endpoints are ready before making any connection
06:21:58 STEP: Waiting for kube-dns to be ready
06:21:58 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns")
06:21:58 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns") => <nil>
06:21:58 STEP: Running kube-dns preflight check
06:22:05 STEP: Performing K8s service preflight check
06:22:06 STEP: Making L7 requests between endpoints
06:22:06 STEP: No interrupts in migrated svc flows
06:22:06 STEP: Install Cilium pre-flight check DaemonSet
06:22:06 STEP: Waiting for all cilium pre-flight pods to be ready
06:22:06 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-pre-flight-check")
06:22:17 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-pre-flight-check") => <nil>
06:22:17 STEP: Removing Cilium pre-flight check DaemonSet
06:22:17 STEP: Waiting for Cilium to become ready
06:22:17 STEP: Upgrading Cilium to 1.12.90
06:22:18 STEP: Validating pods have the right image version upgraded
06:22:22 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium")
06:22:58 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium") => <nil>
06:22:58 STEP: Checking that installed image is "201fc9400fc69e476eeca411bb1bcff65a7526e8"
06:22:58 STEP: Waiting for Cilium to become ready
06:22:58 STEP: Validating Cilium Installation
06:22:58 STEP: Performing Cilium controllers preflight check
06:22:58 STEP: Performing Cilium status preflight check
06:22:58 STEP: Performing Cilium health check
06:22:58 STEP: Checking whether host EP regenerated
06:23:06 STEP: Performing Cilium service preflight check
06:23:06 STEP: Performing K8s service preflight check
06:23:12 STEP: Waiting for cilium-operator to be ready
06:23:12 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
06:23:12 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
06:23:12 STEP: Validate that endpoints are ready before making any connection
06:23:15 STEP: Waiting for kube-dns to be ready
06:23:15 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns")
06:23:15 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns") => <nil>
06:23:15 STEP: Running kube-dns preflight check
06:23:22 STEP: Performing K8s service preflight check
06:23:22 STEP: Making L7 requests between endpoints
06:23:22 STEP: No interrupts in migrated svc flows
06:23:26 STEP: Check whether svc flows are not interrupted upon cilium-agent restart
06:23:38 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium")
06:24:11 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium") => <nil>
06:24:13 STEP: No interrupts in migrated svc flows
FAIL: migrate-svc restart count values do not match
Expected
    <int>: 0
to be identical to
    <int>: 13
=== Test Finished at 2023-03-15T06:24:13Z====
06:24:13 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
06:24:14 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS             RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-585bb89877-wmvsm           0/1     Running            0          24m     10.0.1.46       k8s2   <none>           <none>
	 cilium-monitoring   prometheus-8885c5888-xtdzz         1/1     Running            0          24m     10.0.1.227      k8s2   <none>           <none>
	 default             app1-755788dd65-6vxvq              2/2     Running            0          2m47s   10.0.0.193      k8s1   <none>           <none>
	 default             app1-755788dd65-srrw5              2/2     Running            0          2m47s   10.0.0.81       k8s1   <none>           <none>
	 default             app2-6dfff7bfbd-vhbkw              1/1     Running            0          2m47s   10.0.0.207      k8s1   <none>           <none>
	 default             app3-68f6b87dd5-5qqvj              1/1     Running            0          2m47s   10.0.0.150      k8s1   <none>           <none>
	 default             migrate-svc-client-4p7f9           1/1     Running            2          2m26s   10.0.0.123      k8s1   <none>           <none>
	 default             migrate-svc-client-cbd9t           1/1     Running            2          2m26s   10.0.1.210      k8s2   <none>           <none>
	 default             migrate-svc-client-fzqwv           0/1     CrashLoopBackOff   2          2m26s   10.0.1.169      k8s2   <none>           <none>
	 default             migrate-svc-client-n6m9f           1/1     Running            2          2m26s   10.0.0.225      k8s1   <none>           <none>
	 default             migrate-svc-client-tj5x7           1/1     Running            2          2m26s   10.0.1.213      k8s2   <none>           <none>
	 default             migrate-svc-server-h8nrx           1/1     Running            1          2m28s   10.0.1.137      k8s2   <none>           <none>
	 default             migrate-svc-server-jwn9f           1/1     Running            1          2m28s   10.0.0.50       k8s1   <none>           <none>
	 default             migrate-svc-server-qrxc8           1/1     Running            1          2m28s   10.0.1.96       k8s2   <none>           <none>
	 kube-system         cilium-6ld9g                       1/1     Running            0          42s     192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-5d64d97457-gcth4   1/1     Running            0          2m1s    192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-5d64d97457-hpqfz   1/1     Running            0          2m1s    192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-wgdlz                       1/1     Running            0          52s     192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-758664cbbf-zjzt7           1/1     Running            0          2m59s   10.0.1.216      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running            0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running            0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running            0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-gltbj                   1/1     Running            0          24m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-hjrjb                   1/1     Running            0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running            0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-jwpn2                 1/1     Running            0          24m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-vjj99                 1/1     Running            0          24m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-cbqwt               1/1     Running            0          24m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-wz85r               1/1     Running            0          24m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-6ld9g cilium-wgdlz]
cmd: kubectl exec -n kube-system cilium-6ld9g -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                            IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                 
	 291        Disabled           Disabled          29356      k8s:id=app3                                            fd02::48   10.0.0.150   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                               
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                        
	                                                            k8s:io.kubernetes.pod.namespace=default                                                
	                                                            k8s:zgroup=testapp                                                                     
	 655        Disabled           Disabled          41268      k8s:app=migrate-svc-client                             fd02::56   10.0.0.225   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                               
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                        
	                                                            k8s:io.kubernetes.pod.namespace=default                                                
	                                                            k8s:zgroup=migrate-svc                                                                 
	 843        Disabled           Disabled          41268      k8s:app=migrate-svc-client                             fd02::22   10.0.0.123   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                               
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                        
	                                                            k8s:io.kubernetes.pod.namespace=default                                                
	                                                            k8s:zgroup=migrate-svc                                                                 
	 1198       Disabled           Disabled          4          reserved:health                                        fd02::d7   10.0.0.234   ready   
	 2506       Disabled           Disabled          29302      k8s:appSecond=true                                     fd02::3c   10.0.0.207   ready   
	                                                            k8s:id=app2                                                                            
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                               
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app2-account                                   
	                                                            k8s:io.kubernetes.pod.namespace=default                                                
	                                                            k8s:zgroup=testapp                                                                     
	 2536       Enabled            Disabled          26939      k8s:id=app1                                            fd02::4e   10.0.0.193   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                               
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                   
	                                                            k8s:io.kubernetes.pod.namespace=default                                                
	                                                            k8s:zgroup=testapp                                                                     
	 3226       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                     ready   
	                                                            k8s:node-role.kubernetes.io/master                                                     
	                                                            reserved:host                                                                          
	 3687       Enabled            Disabled          26939      k8s:id=app1                                            fd02::71   10.0.0.81    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                               
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                   
	                                                            k8s:io.kubernetes.pod.namespace=default                                                
	                                                            k8s:zgroup=testapp                                                                     
	 3877       Disabled           Disabled          39160      k8s:app=migrate-svc-server                             fd02::a7   10.0.0.50    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                               
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                        
	                                                            k8s:io.kubernetes.pod.namespace=default                                                
	                                                            k8s:zgroup=migrate-svc                                                                 
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-wgdlz -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                             
	 665        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                 ready   
	                                                            reserved:host                                                                      
	 1014       Disabled           Disabled          41268      k8s:app=migrate-svc-client                        fd02::108   10.0.1.210   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:zgroup=migrate-svc                                                             
	 1415       Disabled           Disabled          41268      k8s:app=migrate-svc-client                        fd02::194   10.0.1.213   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:zgroup=migrate-svc                                                             
	 2515       Disabled           Disabled          39160      k8s:app=migrate-svc-server                        fd02::195   10.0.1.137   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:zgroup=migrate-svc                                                             
	 2727       Disabled           Disabled          39160      k8s:app=migrate-svc-server                        fd02::1f0   10.0.1.96    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:zgroup=migrate-svc                                                             
	 2837       Disabled           Disabled          52726      k8s:io.cilium.k8s.policy.cluster=default          fd02::1a1   10.0.1.216   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                    
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                        
	                                                            k8s:k8s-app=kube-dns                                                               
	 2870       Disabled           Disabled          41268      k8s:app=migrate-svc-client                        fd02::198   10.0.1.169   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:zgroup=migrate-svc                                                             
	 3186       Disabled           Disabled          4          reserved:health                                   fd02::124   10.0.1.138   ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
06:24:26 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
06:24:31 STEP: Cleaning up Cilium components
06:24:38 STEP: Waiting for Cilium to become ready
06:24:58 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|0f13fdd2_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
06:24:59 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
06:24:59 STEP: Cleaning up Cilium components

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19//486/artifact/0f13fdd2_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19//486/artifact/test_results_Cilium-PR-K8s-1.16-kernel-4.19_486_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/486/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
@maintainer-s-little-helper maintainer-s-little-helper bot added the ci/flake This is a known failure that occurs in the tree. Please investigate me! label Mar 15, 2023
Merged
@aanm
Copy link
Member

aanm commented Mar 15, 2023

The PR #24317 didn't contain the fix from #24336, closing

@aanm aanm closed this as completed Mar 15, 2023
This was referenced Mar 15, 2023
Merged
Merged
@julianwiedmann julianwiedmann mentioned this issue Mar 17, 2023
Merged
@aspsk aspsk mentioned this issue Jun 8, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me!
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@aanm