Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: GKE: K8sServicesTest Checks service across nodes Supports IPv4 fragments #16004

Closed
pchaigno opened this issue May 4, 2021 · 4 comments
Closed

CI: GKE: K8sServicesTest Checks service across nodes Supports IPv4 fragments #16004

pchaigno opened this issue May 4, 2021 · 4 comments
Assignees
@qmonnet
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! pinned These issues are not marked stale by our issue bot. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
CI Force

Comments

@pchaigno
Copy link
Member

pchaigno commented May 4, 2021

In #15959, we enabled this test in GKE by disabling endpoint routes (incompatible with fragment tracking, see #15958). But apparently, it's still not working 100% and is flaky: https://datastudio.google.com/s/hluBxpjzw24.

https://jenkins.cilium.io/job/Cilium-PR-K8s-GKE/5355/testReport/Suite-k8s-1/17/K8sServicesTest_Checks_service_across_nodes_Supports_IPv4_fragments/
f36dde5a_K8sServicesTest_Checks_service_across_nodes_Supports_IPv4_fragments.zip

Stacktrace

/home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Failed to account for IPv4 fragments to 10.128.15.227 (in)
Expected
    <[]int | len:2, cap:2>: [0, 0]
To satisfy at least one of these matchers: [%!s(*matchers.EqualMatcher=&{[0 4]}) %!s(*matchers.EqualMatcher=&{[4 0]})]
/home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/k8sT/Services.go:2927

Standard Output

Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 2
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
Unable to serve pprof API
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 2 errors/warnings:
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Unable to update ipcache map entry on pod add
Cilium pods: [cilium-76skg cilium-p68j5]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                                                         Ingress   Egress
testds-rlw7w                                                          
l7-default-backend-5b76b455d-wfmhl                                    
stackdriver-metadata-agent-cluster-level-6cb7db7c5d-hn279             
app2-5cc5d58844-lqw92                                                 
testds-fjprc                                                          
event-exporter-gke-666b7ffbf7-7hr76                                   
kube-dns-autoscaler-5c78d65cd9-9l424                                  
kube-dns-c9488f9fb-6vmhf                                              
kube-dns-c9488f9fb-qtqdf                                              
metrics-server-v0.3.6-547dc87f5f-t2sbp                                
app1-5798c5fb6b-h62f5                                                 
app3-6c7856c5b5-jxj5z                                                 
testclient-8c9l6                                                      
app1-5798c5fb6b-ngrjq                                                 
test-k8s2-5b756fd6c5-c6kc2                                            
testclient-kcxbp                                                      
Cilium agent 'cilium-76skg': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 46 Failed 0
Cilium agent 'cilium-p68j5': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 76 Failed 0

Standard Error

Click to show 
07:11:55 STEP: Installing Cilium
07:12:02 STEP: Waiting for Cilium to become ready
07:12:43 STEP: Validating if Kubernetes DNS is deployed
07:12:43 STEP: Checking if deployment is ready
07:12:43 STEP: Checking if kube-dns service is plumbed correctly
07:12:43 STEP: Checking if DNS can resolve
07:12:43 STEP: Checking if pods have identity
07:12:45 STEP: Kubernetes DNS is up and operational
07:12:45 STEP: Validating Cilium Installation
07:12:45 STEP: Performing Cilium controllers preflight check
07:12:45 STEP: Performing Cilium status preflight check
07:12:45 STEP: Performing Cilium health check
07:12:50 STEP: Performing Cilium service preflight check
07:12:50 STEP: Performing K8s service preflight check
07:12:50 STEP: Waiting for cilium-operator to be ready
07:12:50 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
07:12:50 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
07:12:50 STEP: WaitforPods(namespace="kube-system", filter="")
07:12:50 STEP: WaitforPods(namespace="kube-system", filter="") => <nil>
07:12:59 STEP: Sending a fragmented packet from testclient-kcxbp to endpoint 10.123.240.2:10069
07:13:14 STEP: Sending a fragmented packet from testclient-kcxbp to endpoint 10.128.15.227:30638
FAIL: Failed to account for IPv4 fragments to 10.128.15.227 (in)
Expected
    <[]int | len:2, cap:2>: [0, 0]
To satisfy at least one of these matchers: [%!s(*matchers.EqualMatcher=&{[0 4]}) %!s(*matchers.EqualMatcher=&{[4 0]})]
=== Test Finished at 2021-05-01T07:13:19Z====
07:13:19 STEP: Running JustAfterEach block for EntireTestsuite K8sServicesTest
===================== TEST FAILED =====================
07:13:20 STEP: Running AfterFailed block for EntireTestsuite K8sServicesTest
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE                                                         NAME                                                        READY   STATUS    RESTARTS   AGE    IP              NODE                                          NOMINATED NODE   READINESS GATES
	 202105010632k8sclicliidentityclitestingtestciliumbpfmetricslist   app1-5798c5fb6b-h62f5                                       2/2     Running   0          40m    10.120.1.75     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 202105010632k8sclicliidentityclitestingtestciliumbpfmetricslist   app1-5798c5fb6b-ngrjq                                       2/2     Running   0          40m    10.120.1.79     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 202105010632k8sclicliidentityclitestingtestciliumbpfmetricslist   app2-5cc5d58844-lqw92                                       1/1     Running   0          40m    10.120.1.6      gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 202105010632k8sclicliidentityclitestingtestciliumbpfmetricslist   app3-6c7856c5b5-jxj5z                                       1/1     Running   0          40m    10.120.1.152    gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 cilium-monitoring                                                 grafana-7fd557d749-pgj4n                                    0/1     Running   1          54m    10.120.2.50     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 cilium-monitoring                                                 prometheus-d87f8f984-fqvft                                  1/1     Running   1          53m    10.120.1.69     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 default                                                           test-k8s2-5b756fd6c5-c6kc2                                  2/2     Running   0          11m    10.120.2.49     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 default                                                           testclient-8c9l6                                            1/1     Running   0          11m    10.120.2.29     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 default                                                           testclient-kcxbp                                            1/1     Running   0          11m    10.120.1.57     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 default                                                           testds-fjprc                                                2/2     Running   0          5m9s   10.120.2.91     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 default                                                           testds-rlw7w                                                2/2     Running   0          10m    10.120.1.90     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       cilium-76skg                                                1/1     Running   0          82s    10.128.15.226   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       cilium-node-init-x9fmk                                      1/1     Running   0          82s    10.128.15.227   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       cilium-node-init-zz2cm                                      1/1     Running   0          82s    10.128.15.226   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       cilium-operator-7f79bfd9b7-qsw7z                            1/1     Running   0          82s    10.128.15.227   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       cilium-operator-7f79bfd9b7-xhrvg                            1/1     Running   0          82s    10.128.15.226   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       cilium-p68j5                                                1/1     Running   0          82s    10.128.15.227   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       event-exporter-gke-666b7ffbf7-7hr76                         2/2     Running   0          43m    10.120.1.88     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       fluentbit-gke-2r4hx                                         2/2     Running   0          57m    10.128.15.226   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       fluentbit-gke-8p54s                                         2/2     Running   0          57m    10.128.15.227   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       gke-metrics-agent-22b7p                                     1/1     Running   0          57m    10.128.15.226   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       gke-metrics-agent-m6gjj                                     1/1     Running   0          57m    10.128.15.227   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       kube-dns-autoscaler-5c78d65cd9-9l424                        1/1     Running   0          43m    10.120.1.157    gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       kube-dns-c9488f9fb-6vmhf                                    4/4     Running   0          25m    10.120.1.245    gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       kube-dns-c9488f9fb-qtqdf                                    4/4     Running   0          25m    10.120.2.253    gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       kube-proxy-gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w      1/1     Running   0          57m    10.128.15.227   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       kube-proxy-gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r      1/1     Running   0          57m    10.128.15.226   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       l7-default-backend-5b76b455d-wfmhl                          1/1     Running   0          43m    10.120.2.84     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       log-gatherer-6t2x5                                          1/1     Running   0          54m    10.128.15.226   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       log-gatherer-w2jmz                                          1/1     Running   0          54m    10.128.15.227   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       metrics-server-v0.3.6-547dc87f5f-t2sbp                      2/2     Running   0          43m    10.120.1.35     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       prometheus-to-sd-6lmc5                                      1/1     Running   0          57m    10.128.15.226   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-np2r   <none>           <none>
	 kube-system                                                       prometheus-to-sd-smm8m                                      1/1     Running   0          57m    10.128.15.227   gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 kube-system                                                       stackdriver-metadata-agent-cluster-level-6cb7db7c5d-hn279   2/2     Running   0          43m    10.120.1.72     gke-cilium-ci-14-cilium-ci-14-1fe71fe9-jd1w   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-76skg cilium-p68j5]
cmd: kubectl exec -n kube-system cilium-76skg -- cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend                Service Type   Backend                   
	 1    10.123.251.167:80       ClusterIP      1 => 10.120.2.84:8080     
	 2    10.128.15.226:30197     NodePort       1 => 10.120.2.84:8080     
	 3    0.0.0.0:30197           NodePort       1 => 10.120.2.84:8080     
	 4    10.123.254.173:3000     ClusterIP                                
	 5    10.123.253.89:9090      ClusterIP      1 => 10.120.1.69:9090     
	 6    10.123.249.6:443        ClusterIP      1 => 10.120.1.35:443      
	 7    10.123.240.1:443        ClusterIP      1 => 35.227.169.143:443   
	 8    10.123.240.10:53        ClusterIP      1 => 10.120.1.245:53      
	                                             2 => 10.120.2.253:53      
	 9    10.123.251.103:80       ClusterIP      1 => 10.120.1.79:80       
	                                             2 => 10.120.1.75:80       
	 10   10.123.251.103:69       ClusterIP      1 => 10.120.1.79:69       
	                                             2 => 10.120.1.75:69       
	 11   10.123.250.236:80       ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 12   10.123.250.236:69       ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 13   10.123.240.2:10069      ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 14   10.123.240.2:10080      ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 15   10.128.15.226:31292     NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 16   0.0.0.0:31292           NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 17   10.128.15.226:30638     NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 18   0.0.0.0:30638           NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 19   10.123.240.196:10069    ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 20   10.123.240.196:10080    ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 21   10.128.15.226:30389     NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 22   0.0.0.0:30389           NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 23   10.128.15.226:32497     NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 24   0.0.0.0:32497           NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 25   10.123.249.183:10080    ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 26   10.123.249.183:10069    ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 27   0.0.0.0:31349           NodePort       1 => 10.120.2.91:69       
	 28   0.0.0.0:31349/i         NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 29   10.128.15.226:31349     NodePort       1 => 10.120.2.91:69       
	 30   10.128.15.226:31349/i   NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 31   10.128.15.226:30305     NodePort       1 => 10.120.2.91:80       
	 32   10.128.15.226:30305/i   NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 33   0.0.0.0:30305           NodePort       1 => 10.120.2.91:80       
	 34   0.0.0.0:30305/i         NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 35   10.123.253.73:10069     ClusterIP      1 => 10.120.2.49:69       
	 36   10.123.253.73:10080     ClusterIP      1 => 10.120.2.49:80       
	 37   10.128.15.226:30727     NodePort       1 => 10.120.2.49:69       
	 38   10.128.15.226:30727/i   NodePort       1 => 10.120.2.49:69       
	 39   0.0.0.0:30727           NodePort       1 => 10.120.2.49:69       
	 40   0.0.0.0:30727/i         NodePort       1 => 10.120.2.49:69       
	 41   10.128.15.226:31628     NodePort       1 => 10.120.2.49:80       
	 42   10.128.15.226:31628/i   NodePort       1 => 10.120.2.49:80       
	 43   0.0.0.0:31628           NodePort       1 => 10.120.2.49:80       
	 44   0.0.0.0:31628/i         NodePort       1 => 10.120.2.49:80       
	 45   10.123.253.195:10069    ClusterIP      1 => 10.120.2.49:69       
	 46   10.123.253.195:10080    ClusterIP      1 => 10.120.2.49:80       
	 47   10.128.15.226:32426     NodePort       1 => 10.120.2.49:69       
	 48   0.0.0.0:32426           NodePort       1 => 10.120.2.49:69       
	 49   10.128.15.226:32553     NodePort       1 => 10.120.2.49:80       
	 50   0.0.0.0:32553           NodePort       1 => 10.120.2.49:80       
	 51   10.123.250.40:80        ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 52   10.128.15.226:30996     NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 53   0.0.0.0:30996           NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 54   10.123.252.25:80        ClusterIP      1 => 10.120.2.49:80       
	 55   10.128.15.226:30696     NodePort       1 => 10.120.2.49:80       
	 56   10.128.15.226:30696/i   NodePort       1 => 10.120.2.49:80       
	 57   0.0.0.0:30696           NodePort       1 => 10.120.2.49:80       
	 58   0.0.0.0:30696/i         NodePort       1 => 10.120.2.49:80       
	 59   10.123.248.192:20080    ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 60   10.123.248.192:20069    ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 61   192.0.2.233:20080       ExternalIPs    1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 62   192.0.2.233:20069       ExternalIPs    1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 63   10.128.15.226:31571     NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 64   0.0.0.0:31571           NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 65   10.128.15.226:32408     NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 66   0.0.0.0:32408           NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 67   10.128.15.226:8080      HostPort       1 => 10.120.2.49:80       
	 68   0.0.0.0:8080            HostPort       1 => 10.120.2.49:80       
	 69   10.128.15.226:6969      HostPort       1 => 10.120.2.49:69       
	 70   0.0.0.0:6969            HostPort       1 => 10.120.2.49:69       
	 71   34.82.185.180:80        LoadBalancer   1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 72   104.199.121.157:80      LoadBalancer   1 => 10.120.2.49:80       
	 73   104.199.121.157:80/i    LoadBalancer   1 => 10.120.2.49:80       
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-76skg -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6   IPv4           STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                             
	 216        Disabled           Disabled          29133      k8s:io.cilium.k8s.policy.cluster=default                    10.120.2.91    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:zgroup=testDS                                                                  
	 228        Disabled           Disabled          25170      k8s:io.cilium.k8s.policy.cluster=default                    10.120.2.49    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:zgroup=test-k8s2                                                               
	 1063       Disabled           Disabled          30729      k8s:io.cilium.k8s.policy.cluster=default                    10.120.2.29    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:zgroup=testDSClient                                                            
	 2101       Disabled           Disabled          4          reserved:health                                             10.120.2.119   ready   
	 3239       Disabled           Disabled          4713       k8s:io.cilium.k8s.policy.cluster=default                    10.120.2.84    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                        
	                                                            k8s:k8s-app=glbc                                                                   
	                                                            k8s:name=glbc                                                                      
	 3338       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                 ready   
	                                                            k8s:cloud.google.com/gke-nodepool=cilium-ci-14                                     
	                                                            k8s:cloud.google.com/gke-os-distribution=cos                                       
	                                                            k8s:cloud.google.com/machine-family=n1                                             
	                                                            k8s:node.kubernetes.io/instance-type=n1-standard-4                                 
	                                                            k8s:topology.kubernetes.io/region=us-west1                                         
	                                                            k8s:topology.kubernetes.io/zone=us-west1-c                                         
	                                                            reserved:host                                                                      
	 3541       Disabled           Disabled          10639      k8s:io.cilium.k8s.policy.cluster=default                    10.120.2.253   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=kube-dns                                   
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                        
	                                                            k8s:k8s-app=kube-dns                                                               
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-p68j5 -- cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend                Service Type   Backend                   
	 1    10.123.251.167:80       ClusterIP      1 => 10.120.2.84:8080     
	 2    10.128.15.227:30197     NodePort       1 => 10.120.2.84:8080     
	 3    0.0.0.0:30197           NodePort       1 => 10.120.2.84:8080     
	 4    10.123.240.10:53        ClusterIP      1 => 10.120.1.245:53      
	                                             2 => 10.120.2.253:53      
	 5    10.123.249.6:443        ClusterIP      1 => 10.120.1.35:443      
	 6    10.123.254.173:3000     ClusterIP                                
	 7    10.123.240.1:443        ClusterIP      1 => 35.227.169.143:443   
	 8    10.123.253.89:9090      ClusterIP      1 => 10.120.1.69:9090     
	 9    10.123.251.103:80       ClusterIP      1 => 10.120.1.79:80       
	                                             2 => 10.120.1.75:80       
	 10   10.123.251.103:69       ClusterIP      1 => 10.120.1.79:69       
	                                             2 => 10.120.1.75:69       
	 11   10.123.250.236:80       ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 12   10.123.250.236:69       ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 13   10.123.240.2:10080      ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 14   10.123.240.2:10069      ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 15   10.128.15.227:31292     NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 16   0.0.0.0:31292           NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 17   10.128.15.227:30638     NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 18   0.0.0.0:30638           NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 19   10.123.240.196:10080    ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 20   10.123.240.196:10069    ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 21   10.128.15.227:30389     NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 22   0.0.0.0:30389           NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 23   10.128.15.227:32497     NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 24   0.0.0.0:32497           NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 25   10.123.249.183:10069    ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 26   10.123.249.183:10080    ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 27   10.128.15.227:31349     NodePort       1 => 10.120.1.90:69       
	 28   10.128.15.227:31349/i   NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 29   0.0.0.0:31349           NodePort       1 => 10.120.1.90:69       
	 30   0.0.0.0:31349/i         NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 31   10.128.15.227:30305     NodePort       1 => 10.120.1.90:80       
	 32   10.128.15.227:30305/i   NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 33   0.0.0.0:30305           NodePort       1 => 10.120.1.90:80       
	 34   0.0.0.0:30305/i         NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 35   10.123.253.73:10069     ClusterIP      1 => 10.120.2.49:69       
	 36   10.123.253.73:10080     ClusterIP      1 => 10.120.2.49:80       
	 37   10.128.15.227:31628     NodePort                                 
	 38   10.128.15.227:31628/i   NodePort       1 => 10.120.2.49:80       
	 39   0.0.0.0:31628           NodePort                                 
	 40   0.0.0.0:31628/i         NodePort       1 => 10.120.2.49:80       
	 41   10.128.15.227:30727     NodePort                                 
	 42   10.128.15.227:30727/i   NodePort       1 => 10.120.2.49:69       
	 43   0.0.0.0:30727           NodePort                                 
	 44   0.0.0.0:30727/i         NodePort       1 => 10.120.2.49:69       
	 45   10.123.253.195:10080    ClusterIP      1 => 10.120.2.49:80       
	 46   10.123.253.195:10069    ClusterIP      1 => 10.120.2.49:69       
	 47   10.128.15.227:32553     NodePort       1 => 10.120.2.49:80       
	 48   0.0.0.0:32553           NodePort       1 => 10.120.2.49:80       
	 49   10.128.15.227:32426     NodePort       1 => 10.120.2.49:69       
	 50   0.0.0.0:32426           NodePort       1 => 10.120.2.49:69       
	 51   10.123.250.40:80        ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 52   10.128.15.227:30996     NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 53   0.0.0.0:30996           NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 54   10.123.252.25:80        ClusterIP      1 => 10.120.2.49:80       
	 55   10.128.15.227:30696     NodePort                                 
	 56   10.128.15.227:30696/i   NodePort       1 => 10.120.2.49:80       
	 57   0.0.0.0:30696           NodePort                                 
	 58   0.0.0.0:30696/i         NodePort       1 => 10.120.2.49:80       
	 59   10.123.248.192:20080    ClusterIP      1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 60   10.123.248.192:20069    ClusterIP      1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 61   192.0.2.233:20080       ExternalIPs    1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 62   192.0.2.233:20069       ExternalIPs    1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 63   10.128.15.227:32408     NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 64   0.0.0.0:32408           NodePort       1 => 10.120.2.91:69       
	                                             2 => 10.120.1.90:69       
	 65   10.128.15.227:31571     NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 66   0.0.0.0:31571           NodePort       1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 67   34.82.185.180:80        LoadBalancer   1 => 10.120.2.91:80       
	                                             2 => 10.120.1.90:80       
	 68   104.199.121.157:80      LoadBalancer                             
	 69   104.199.121.157:80/i    LoadBalancer   1 => 10.120.2.49:80       
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-p68j5 -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                                       IPv6   IPv4           STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                                          
	 300        Disabled           Disabled          30729      k8s:io.cilium.k8s.policy.cluster=default                                                                 10.120.1.57    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                                 
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                                         
	                                                            k8s:zgroup=testDSClient                                                                                                         
	 606        Disabled           Disabled          61947      k8s:io.cilium.k8s.policy.cluster=default                                                                 10.120.1.157   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=kube-dns-autoscaler                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                                     
	                                                            k8s:k8s-app=kube-dns-autoscaler                                                                                                 
	 647        Disabled           Disabled          58997      k8s:appSecond=true                                                                                       10.120.1.6     ready   
	                                                            k8s:id=app2                                                                                                                     
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                        
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app2-account                                                                            
	                                                            k8s:io.kubernetes.pod.namespace=202105010632k8sclicliidentityclitestingtestciliumbpfmetricslist                                 
	                                                            k8s:zgroup=testapp                                                                                                              
	 963        Disabled           Disabled          34863      k8s:app=stackdriver-metadata-agent                                                                       10.120.1.72    ready   
	                                                            k8s:cluster-level=true                                                                                                          
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                        
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=metadata-agent                                                                          
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                                     
	 967        Disabled           Disabled          16113      k8s:io.cilium.k8s.policy.cluster=default                                                                 10.120.1.35    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=metrics-server                                                                          
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                                     
	                                                            k8s:k8s-app=metrics-server                                                                                                      
	                                                            k8s:version=v0.3.6                                                                                                              
	 1103       Disabled           Disabled          21547      k8s:id=app3                                                                                              10.120.1.152   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                        
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                                 
	                                                            k8s:io.kubernetes.pod.namespace=202105010632k8sclicliidentityclitestingtestciliumbpfmetricslist                                 
	                                                            k8s:zgroup=testapp                                                                                                              
	 1565       Disabled           Disabled          40888      k8s:id=app1                                                                                              10.120.1.75    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                        
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                                            
	                                                            k8s:io.kubernetes.pod.namespace=202105010632k8sclicliidentityclitestingtestciliumbpfmetricslist                                 
	                                                            k8s:zgroup=testapp                                                                                                              
	 1991       Disabled           Disabled          14184      k8s:io.cilium.k8s.policy.cluster=default                                                                 10.120.1.88    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=event-exporter-sa                                                                       
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                                     
	                                                            k8s:k8s-app=event-exporter                                                                                                      
	                                                            k8s:version=v0.3.4                                                                                                              
	 2037       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                              ready   
	                                                            k8s:cloud.google.com/gke-nodepool=cilium-ci-14                                                                                  
	                                                            k8s:cloud.google.com/gke-os-distribution=cos                                                                                    
	                                                            k8s:cloud.google.com/machine-family=n1                                                                                          
	                                                            k8s:node.kubernetes.io/instance-type=n1-standard-4                                                                              
	                                                            k8s:topology.kubernetes.io/region=us-west1                                                                                      
	                                                            k8s:topology.kubernetes.io/zone=us-west1-c                                                                                      
	                                                            reserved:host                                                                                                                   
	 2075       Disabled           Disabled          29133      k8s:io.cilium.k8s.policy.cluster=default                                                                 10.120.1.90    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                                 
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                                         
	                                                            k8s:zgroup=testDS                                                                                                               
	 2471       Disabled           Disabled          10639      k8s:io.cilium.k8s.policy.cluster=default                                                                 10.120.1.245   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=kube-dns                                                                                
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                                     
	                                                            k8s:k8s-app=kube-dns                                                                                                            
	 3475       Disabled           Disabled          40888      k8s:id=app1                                                                                              10.120.1.79    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                        
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                                            
	                                                            k8s:io.kubernetes.pod.namespace=202105010632k8sclicliidentityclitestingtestciliumbpfmetricslist                                 
	                                                            k8s:zgroup=testapp                                                                                                              
	 4033       Disabled           Disabled          4          reserved:health                                                                                          10.120.1.36    ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
07:13:59 STEP: Running AfterEach for block EntireTestsuite K8sServicesTest
07:13:59 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|f36dde5a_K8sServicesTest_Checks_service_across_nodes_Supports_IPv4_fragments.zip]]
07:14:00 STEP: Running AfterAll block for EntireTestsuite K8sServicesTest Checks service across nodes
@pchaigno pchaigno added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels May 4, 2021
@pchaigno pchaigno mentioned this issue May 4, 2021
Merged
@jrajahalme jrajahalme mentioned this issue May 4, 2021
Merged
@pchaigno
Copy link
Member Author

pchaigno commented May 7, 2021

This is happening quite a bit: https://datastudio.google.com/s/n37ZusenpaE. I'll quarantine the test.
/cc @qmonnet

@pchaigno pchaigno added this to To Do: disabled tests (1.10) in CI Force May 7, 2021
@pchaigno pchaigno mentioned this issue May 7, 2021
Merged
@nbusseneau nbusseneau mentioned this issue May 7, 2021
Merged
@pchaigno pchaigno mentioned this issue May 10, 2021
Merged
@ungureanuvladvictor
Copy link
Member

Hit this in https://jenkins.cilium.io/job/Cilium-PR-K8s-GKE/5395/ via #16043. 😢

@tklauser tklauser mentioned this issue May 10, 2021
Merged
@pchaigno pchaigno mentioned this issue May 11, 2021
Merged
@christarazi christarazi mentioned this issue May 11, 2021
Merged
@tklauser tklauser mentioned this issue May 19, 2021
Merged
@stale
Copy link

stale bot commented Jul 11, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jul 11, 2021
@pchaigno pchaigno removed the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jul 12, 2021
@brb brb added the sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. label Feb 17, 2022
@qmonnet qmonnet added the pinned These issues are not marked stale by our issue bot. label Feb 17, 2022
@pchaigno
Copy link
Member Author

Our GKE CI job was dropped.

CI Force automation moved this from To Do: disabled tests (1.10) to Fixed / Done Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@qmonnet qmonnet

Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! pinned These issues are not marked stale by our issue bot. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
No open projects
CI Force
  
Fixed / Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@brb @pchaigno @ungureanuvladvictor @qmonnet