Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: K8sFQDNTest Validate that multiple specs are working correctly #17071

Closed
maintainer-s-little-helper bot opened this issue Aug 6, 2021 · 6 comments
Closed

CI: K8sFQDNTest Validate that multiple specs are working correctly #17071

maintainer-s-little-helper bot opened this issue Aug 6, 2021 · 6 comments
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

Comments

@maintainer-s-little-helper
Copy link

Test Name

K8sFQDNTest Validate that multiple specs are working correctly

Failure Output

FAIL: Cannot install fqdn proxy policy

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Cannot install fqdn proxy policy
Expected
    <*errors.errorString | 0xc000a4f7f0>: {
        s: "Cannot retrieve cilium pod cilium-5r82l policy revision: cannot get the revision ",
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/k8sT/fqdn.go:257

Standard Output

Click to show. 
Cilium pods: [cilium-5r82l cilium-cbb2s]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
app2-58757b7dd5-h6bdx                   
app3-5d69599cdd-xxlqs                   
coredns-755cd654d4-qfgk2                
grafana-5747bcc8f9-7qff7                
prometheus-655fb888d7-2dg8p             
app1-6bf9bf9bd5-ccjfd                   
app1-6bf9bf9bd5-clfh6                   
Cilium agent 'cilium-5r82l': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 18 Failed 0
Cilium agent 'cilium-cbb2s': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 46 Failed 0

Standard Error

Click to show. 
FAIL: Cannot install fqdn proxy policy
Expected
    <*errors.errorString | 0xc000a4f7f0>: {
        s: "Cannot retrieve cilium pod cilium-5r82l policy revision: cannot get the revision ",
    }
to be nil
=== Test Finished at 2021-08-05T13:39:45Z====
===================== TEST FAILED =====================
13:39:45 STEP: Running AfterFailed block for EntireTestsuite K8sFQDNTest
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS             RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-7qff7           0/1     Running            0          62s     10.0.0.89       k8s1   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-2dg8p        1/1     Running            0          62s     10.0.0.8        k8s1   <none>           <none>
	 default             app1-6bf9bf9bd5-ccjfd              2/2     Running            0          27s     10.0.0.164      k8s1   <none>           <none>
	 default             app1-6bf9bf9bd5-clfh6              2/2     Running            0          27s     10.0.0.229      k8s1   <none>           <none>
	 default             app2-58757b7dd5-h6bdx              1/1     Running            0          27s     10.0.0.143      k8s1   <none>           <none>
	 default             app3-5d69599cdd-xxlqs              1/1     Running            0          27s     10.0.0.222      k8s1   <none>           <none>
	 kube-system         cilium-5r82l                       1/1     Running            0          59s     192.168.36.12   k8s2   <none>           <none>
	 kube-system         cilium-cbb2s                       1/1     Running            0          59s     192.168.36.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-6f57b5cdd5-c4rsb   0/1     Error              0          59s     192.168.36.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-6f57b5cdd5-d4s8b   1/1     Running            0          59s     192.168.36.12   k8s2   <none>           <none>
	 kube-system         coredns-755cd654d4-qfgk2           1/1     Running            0          40s     10.0.0.228      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running            0          7m44s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running            0          7m44s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       0/1     CrashLoopBackOff   2          7m44s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-ck2cq                   1/1     Running            0          4m32s   192.168.36.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-kh8v2                   1/1     Running            0          5m44s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                0/1     Error              2          7m44s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-hd5jq                 1/1     Running            0          65s     192.168.36.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-jmpkz                 1/1     Running            0          65s     192.168.36.11   k8s1   <none>           <none>
	 kube-system         registry-adder-p7skm               1/1     Running            0          4m29s   192.168.36.12   k8s2   <none>           <none>
	 kube-system         registry-adder-tnrqk               1/1     Running            0          4m29s   192.168.36.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-5r82l cilium-cbb2s]
cmd: kubectl exec -n kube-system cilium-5r82l -c cilium-agent -- cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend             Service Type   Backend                   
	 1    10.96.0.1:443        ClusterIP      1 => 192.168.36.11:6443   
	 2    10.96.0.10:53        ClusterIP      1 => 10.0.0.228:53        
	 3    10.96.0.10:9153      ClusterIP      1 => 10.0.0.228:9153      
	 4    10.109.55.232:3000   ClusterIP                                
	 5    10.102.127.24:9090   ClusterIP      1 => 10.0.0.8:9090        
	 6    10.104.198.195:69    ClusterIP      1 => 10.0.0.229:69        
	                                          2 => 10.0.0.164:69        
	 7    10.104.198.195:80    ClusterIP      1 => 10.0.0.229:80        
	                                          2 => 10.0.0.164:80        
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-5r82l -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6        IPv4        STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                        
	 1787       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                            ready   
	                                                            reserved:host                                                 
	 3762       Disabled           Disabled          4          reserved:health               fd02::1dd   10.0.1.64   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-cbb2s -c cilium-agent -- cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend             Service Type   Backend                   
	 1    10.96.0.1:443        ClusterIP      1 => 192.168.36.11:6443   
	 2    10.96.0.10:53        ClusterIP      1 => 10.0.0.228:53        
	 3    10.96.0.10:9153      ClusterIP      1 => 10.0.0.228:9153      
	 4    10.109.55.232:3000   ClusterIP                                
	 5    10.102.127.24:9090   ClusterIP      1 => 10.0.0.8:9090        
	 6    10.104.198.195:80    ClusterIP      1 => 10.0.0.229:80        
	                                          2 => 10.0.0.164:80        
	 7    10.104.198.195:69    ClusterIP      1 => 10.0.0.229:69        
	                                          2 => 10.0.0.164:69        
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-cbb2s -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 516        Disabled           Disabled          9943       k8s:id=app1                                                                        fd02::92   10.0.0.164   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                             
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                            
	                                                            k8s:zgroup=testapp                                                                                                 
	 623        Disabled           Disabled          33328      k8s:appSecond=true                                                                 fd02::bb   10.0.0.143   ready   
	                                                            k8s:id=app2                                                                                                        
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                             
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app2-account                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                            
	                                                            k8s:zgroup=testapp                                                                                                 
	 927        Disabled           Disabled          9943       k8s:id=app1                                                                        fd02::2c   10.0.0.229   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                             
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                            
	                                                            k8s:zgroup=testapp                                                                                                 
	 929        Disabled           Disabled          16808      k8s:id=app3                                                                        fd02::c7   10.0.0.222   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                             
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                            
	                                                            k8s:zgroup=testapp                                                                                                 
	 1777       Disabled           Disabled          4          reserved:health                                                                    fd02::a5   10.0.0.224   ready   
	 2922       Disabled           Disabled          28022      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system         fd02::2b   10.0.0.228   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                        
	                                                            k8s:k8s-app=kube-dns                                                                                               
	 3405       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node-role.kubernetes.io/master                                                                                 
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 3491       Disabled           Disabled          1250       k8s:app=prometheus                                                                 fd02::f7   10.0.0.8     ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 3895       Disabled           Disabled          42831      k8s:app=grafana                                                                    fd02::26   10.0.0.89    ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
13:40:06 STEP: Running AfterEach for block EntireTestsuite K8sFQDNTest
13:40:07 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|8ca96124_K8sFQDNTest_Validate_that_multiple_specs_are_working_correctly.zip]]
13:40:08 STEP: Running AfterAll block for EntireTestsuite K8sFQDNTest
13:40:08 STEP: Removing Cilium installation using generated helm manifest

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/1172/artifact/8ca96124_K8sFQDNTest_Validate_that_multiple_specs_are_working_correctly.zip/8ca96124_K8sFQDNTest_Validate_that_multiple_specs_are_working_correctly.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/1172/artifact/test_results_Cilium-PR-K8s-1.21-kernel-4.9_1172_BDD-Test-PR.zip/test_results_Cilium-PR-K8s-1.21-kernel-4.9_1172_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/1172/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
@maintainer-s-little-helper maintainer-s-little-helper bot added the ci/flake This is a known failure that occurs in the tree. Please investigate me! label Aug 6, 2021
@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Aug 6, 2021
Merged
@aanm
Copy link
Member

aanm commented Aug 6, 2021 •
edited

@nbusseneau it looks like kube-apiserver was crashloopbackoff which caused the test to fail. This was the first test executed after provisioning, the following tests worked / passed successfully [1] . Can we have some sort of sanity check before starting the tests? I thought we did already 🤔

[1] https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/1172/execution/node/168/log/?consoleFull

@nbusseneau
Copy link
Member

Well, we do in the BeforeAll:

cilium/test/k8sT/fqdn.go

Line 65 in f77996c

DeployCiliumAndDNS(kubectl, ciliumFilename)

Which runs preflight checks:

cilium/test/k8sT/assertionHelpers.go

Lines 198 to 200 in f77996c

err := vm.CiliumPreFlightCheck()
ExpectWithOffset(1, err).Should(BeNil(), "cilium pre-flight checks failed")
ExpectCiliumOperatorReady(vm)

And it seems these were OK here:

15:38:53  13:38:53 STEP: Running BeforeAll block for EntireTestsuite K8sFQDNTest
15:38:53  13:38:53 STEP: Ensuring the namespace kube-system exists
15:38:53  13:38:53 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
15:38:54  13:38:53 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
15:38:54  13:38:54 STEP: Installing Cilium
15:38:54  13:38:54 STEP: Waiting for Cilium to become ready
15:39:15  13:39:13 STEP: Restarting unmanaged pods coredns-755cd654d4-ntgtm in namespace kube-system
15:39:15  13:39:13 STEP: Validating if Kubernetes DNS is deployed
15:39:15  13:39:13 STEP: Checking if deployment is ready
15:39:15  13:39:13 STEP: Kubernetes DNS is not ready: only 0 of 1 replicas are available
15:39:15  13:39:13 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
15:39:15  13:39:13 STEP: Waiting for Kubernetes DNS to become operational
15:39:15  13:39:13 STEP: Checking if deployment is ready
15:39:15  13:39:13 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:15  13:39:14 STEP: Checking if deployment is ready
15:39:15  13:39:14 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:16  13:39:15 STEP: Checking if deployment is ready
15:39:16  13:39:15 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:17  13:39:16 STEP: Checking if deployment is ready
15:39:17  13:39:16 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:18  13:39:17 STEP: Checking if deployment is ready
15:39:18  13:39:17 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:18  13:39:18 STEP: Checking if deployment is ready
15:39:18  13:39:18 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:19  13:39:19 STEP: Checking if deployment is ready
15:39:19  13:39:19 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:20  13:39:20 STEP: Checking if deployment is ready
15:39:20  13:39:20 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:21  13:39:21 STEP: Checking if deployment is ready
15:39:22  13:39:21 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:39:23  13:39:22 STEP: Checking if deployment is ready
15:39:23  13:39:22 STEP: Checking if kube-dns service is plumbed correctly
15:39:23  13:39:22 STEP: Checking if pods have identity
15:39:23  13:39:22 STEP: Checking if DNS can resolve
15:39:23  13:39:23 STEP: Validating Cilium Installation
15:39:23  13:39:23 STEP: Performing Cilium controllers preflight check
15:39:23  13:39:23 STEP: Performing Cilium health check
15:39:23  13:39:23 STEP: Performing Cilium status preflight check
15:39:25  13:39:25 STEP: Performing Cilium service preflight check
15:39:25  13:39:25 STEP: Performing K8s service preflight check
15:39:26  13:39:26 STEP: Waiting for cilium-operator to be ready
15:39:26  13:39:26 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
15:39:26  13:39:26 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
15:39:26  13:39:26 STEP: Applying demo manifest
15:39:27  13:39:26 STEP: WaitforPods(namespace="default", filter="-l zgroup=testapp")
15:39:35  13:39:34 STEP: WaitforPods(namespace="default", filter="-l zgroup=testapp") => <nil>

So I'm not sure what's going on either 🤔

@nbusseneau
Copy link
Member

After digging a bit, it seems the various *PreFlightCheck() only check Cilium, and not Kubernetes components.

@qmonnet qmonnet mentioned this issue Oct 27, 2021
Merged
@nbusseneau nbusseneau mentioned this issue Dec 13, 2021
Merged
@aanm aanm added the area/CI Continuous Integration testing issue or flake label Jan 6, 2022
@github-actions
Copy link

github-actions bot commented Jul 9, 2022

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. and removed stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. labels Jul 9, 2022
@github-actions
Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Sep 14, 2022
@github-actions
Copy link

This issue has not seen any activity since it was marked stale.
Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nbusseneau @aanm