Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: K8sBandwidthTest Checks Bandwidth Rate-Limiting Checks Pod to Pod bandwidth, vxlan tunneling: Rate mismatch #13056

Closed
joestringer opened this issue Sep 2, 2020 · 1 comment
Closed

CI: K8sBandwidthTest Checks Bandwidth Rate-Limiting Checks Pod to Pod bandwidth, vxlan tunneling: Rate mismatch #13056

joestringer opened this issue Sep 2, 2020 · 1 comment
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me!

Comments

@joestringer
Copy link
Member

Recently added test flaking out:

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.12-net-next/359/testReport/Suite-k8s-1/12/K8sBandwidthTest_Checks_Bandwidth_Rate_Limiting_Checks_Pod_to_Pod_bandwidth__vxlan_tunneling/
4b8d4b1b_K8sBandwidthTest_Checks_Bandwidth_Rate-Limiting_Checks_Pod_to_Pod_bandwidth,_vxlan_tunneling.zip

Stacktrace

/home/jenkins/workspace/Cilium-PR-K8s-1.12-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:514
Rate mismatch
Expected
    <*errors.errorString | 0xc0012a4970>: {
        s: "Expected result 19 (19.29) is not in the range of [20, 30]",
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.12-net-next/src/github.com/cilium/cilium/test/k8sT/Bandwidth.go:142

Standard Output

Number of "context deadline exceeded" in logs: 0
⚠️  Number of "level=error" in logs: 8
⚠️  Number of "level=warning" in logs: 24
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Mutation detector is enabled, this will result in memory leakage.
Unable to enqueue endpoint policy visibility event
Unable to enqueue endpoint policy bandwidth event
Hubble server will be exposing its API insecurely on this address
Cilium pods: [cilium-2xzzm cilium-qmx4l]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                                   Ingress   Egress
grafana-559c9666fd-7b4zv                        
prometheus-7b5bd859df-rpzlf                     
netperf-10-6479786bf5-6k7cj                     
netperf-25-87c675dcf-bb665                      
netperf-client-pod-764477dcc7-wb82r             
coredns-5544f57546-xzh4g                        
Cilium agent 'cilium-2xzzm': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 27 Failed 0
Cilium agent 'cilium-qmx4l': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 27 Failed 0

Standard Error

21:07:38 STEP: Running BeforeAll block for EntireTestsuite K8sBandwidthTest
21:07:38 STEP: Ensuring the namespace kube-system exists
21:07:39 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
21:07:39 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
21:07:39 STEP: Installing Cilium
21:07:41 STEP: Waiting for Cilium to become ready
21:07:41 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are scheduled
21:07:46 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:07:51 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:07:56 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:08:01 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
21:08:06 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
21:08:12 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
21:08:16 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
21:08:21 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
21:08:26 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
21:08:31 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
21:08:36 STEP: Number of ready Cilium pods: 2
21:08:36 STEP: Validating if Kubernetes DNS is deployed
21:08:36 STEP: Checking if deployment is ready
21:08:37 STEP: Checking if kube-dns service is plumbed correctly
21:08:37 STEP: Checking if pods have identity
21:08:37 STEP: Checking if DNS can resolve
21:08:38 STEP: Kubernetes DNS is up and operational
21:08:38 STEP: Validating Cilium Installation
21:08:38 STEP: Performing Cilium health check
21:08:38 STEP: Performing Cilium controllers preflight check
21:08:38 STEP: Performing Cilium status preflight check
21:08:40 STEP: Performing Cilium service preflight check
21:08:40 STEP: Performing K8s service preflight check
21:08:41 STEP: Waiting for cilium-operator to be ready
21:08:41 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
21:08:41 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
21:08:41 STEP: Running BeforeAll block for EntireTestsuite K8sBandwidthTest Checks Bandwidth Rate-Limiting
21:08:41 STEP: WaitforPods(namespace="default", filter="-l run=netperf-10")
21:08:46 STEP: WaitforPods(namespace="default", filter="-l run=netperf-10") => <nil>
21:08:46 STEP: WaitforPods(namespace="default", filter="-l run=netperf-25")
21:08:47 STEP: WaitforPods(namespace="default", filter="-l run=netperf-25") => <nil>
21:08:47 STEP: WaitforPods(namespace="default", filter="-l run=netperf-client-pod")
21:08:47 STEP: WaitforPods(namespace="default", filter="-l run=netperf-client-pod") => <nil>
21:08:47 STEP: WaitforPods(namespace="default", filter="-l run=netperf-client-host")
21:08:47 STEP: WaitforPods(namespace="default", filter="-l run=netperf-client-host") => <nil>
21:08:47 STEP: Installing Cilium
21:08:50 STEP: Waiting for Cilium to become ready
21:08:50 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:08:55 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:09:00 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:09:05 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:09:10 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:09:15 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:09:20 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
21:09:25 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
21:09:30 STEP: Number of ready Cilium pods: 2
21:09:30 STEP: Validating if Kubernetes DNS is deployed
21:09:30 STEP: Checking if deployment is ready
21:09:30 STEP: Checking if kube-dns service is plumbed correctly
21:09:30 STEP: Checking if pods have identity
21:09:30 STEP: Checking if DNS can resolve
21:09:31 STEP: Kubernetes DNS is up and operational
21:09:31 STEP: Validating Cilium Installation
21:09:31 STEP: Performing Cilium status preflight check
21:09:31 STEP: Performing Cilium controllers preflight check
21:09:31 STEP: Performing Cilium health check
21:09:34 STEP: Performing Cilium service preflight check
21:09:34 STEP: Performing K8s service preflight check
21:09:35 STEP: Waiting for cilium-operator to be ready
21:09:35 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
21:09:35 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
21:09:35 STEP: Running 1 netperf session from netperf-client-pod-764477dcc7-wb82r pod to pod with IP 10.0.1.25 (expected rate: 10)
21:09:46 STEP: Session test completed, netperf result raw: 9.29
21:09:46 STEP: Running 1 netperf session from netperf-client-pod-764477dcc7-wb82r pod to pod with IP 10.0.1.88 (expected rate: 25)
21:09:57 STEP: Session test completed, netperf result raw: 20.85
21:09:57 STEP: Running 1 netperf session from netperf-client-host-6bc6664445-z4h92 pod to pod with IP 10.0.1.25 (expected rate: 10)
21:10:09 STEP: Session test completed, netperf result raw: 9.3
21:10:09 STEP: Running 1 netperf session from netperf-client-host-6bc6664445-z4h92 pod to pod with IP 10.0.1.88 (expected rate: 25)
21:10:20 STEP: Session test completed, netperf result raw: 19.29
FAIL: Rate mismatch
Expected
    <*errors.errorString | 0xc0012a4970>: {
        s: "Expected result 19 (19.29) is not in the range of [20, 30]",
    }
to be nil
=== Test Finished at 2020-09-02T21:10:20Z====
21:10:20 STEP: Running JustAfterEach block for EntireTestsuite K8sBandwidthTest
===================== TEST FAILED =====================
21:10:20 STEP: Running AfterFailed block for EntireTestsuite K8sBandwidthTest
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                   READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE
	 cilium-monitoring   grafana-559c9666fd-7b4zv               0/1     Running   0          27m     10.0.0.191      k8s2   <none>
	 cilium-monitoring   prometheus-7b5bd859df-rpzlf            1/1     Running   0          27m     10.0.0.74       k8s2   <none>
	 default             netperf-10-6479786bf5-6k7cj            1/1     Running   0          102s    10.0.1.25       k8s1   <none>
	 default             netperf-25-87c675dcf-bb665             1/1     Running   0          102s    10.0.1.88       k8s1   <none>
	 default             netperf-client-host-6bc6664445-z4h92   1/1     Running   0          102s    192.168.36.12   k8s2   <none>
	 default             netperf-client-pod-764477dcc7-wb82r    1/1     Running   0          102s    10.0.0.114      k8s2   <none>
	 kube-system         cilium-2xzzm                           1/1     Running   0          93s     192.168.36.12   k8s2   <none>
	 kube-system         cilium-operator-84c6d45d97-7lwhr       1/1     Running   0          93s     192.168.36.13   k8s3   <none>
	 kube-system         cilium-qmx4l                           1/1     Running   0          93s     192.168.36.11   k8s1   <none>
	 kube-system         coredns-5544f57546-xzh4g               1/1     Running   0          9m33s   10.0.0.176      k8s2   <none>
	 kube-system         etcd-k8s1                              1/1     Running   0          29m     192.168.36.11   k8s1   <none>
	 kube-system         kube-apiserver-k8s1                    1/1     Running   0          29m     192.168.36.11   k8s1   <none>
	 kube-system         kube-controller-manager-k8s1           1/1     Running   0          29m     192.168.36.11   k8s1   <none>
	 kube-system         kube-scheduler-k8s1                    1/1     Running   0          29m     192.168.36.11   k8s1   <none>
	 kube-system         log-gatherer-846m9                     1/1     Running   0          27m     192.168.36.11   k8s1   <none>
	 kube-system         log-gatherer-bkxzz                     1/1     Running   0          27m     192.168.36.12   k8s2   <none>
	 kube-system         log-gatherer-dvnjl                     1/1     Running   0          27m     192.168.36.13   k8s3   <none>
	 kube-system         registry-adder-4cp2s                   1/1     Running   0          28m     192.168.36.12   k8s2   <none>
	 kube-system         registry-adder-cfm5n                   1/1     Running   0          28m     192.168.36.13   k8s3   <none>
	 kube-system         registry-adder-ff5fl                   1/1     Running   0          28m     192.168.36.11   k8s1   <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-2xzzm cilium-qmx4l]
cmd: kubectl exec -n kube-system cilium-2xzzm -- cilium bpf bandwidth list
Exitcode: 0 
Stdout:
 	 
Stderr:
 	 No entries found.
	 

cmd: kubectl exec -n kube-system cilium-2xzzm -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                            
	 1212       Disabled           Disabled          4          reserved:health                                   fd00::11   10.0.0.196   ready   
	 1556       Disabled           Disabled          1733       k8s:io.cilium.k8s.policy.cluster=default          fd00::8a   10.0.0.114   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                   
	                                                            k8s:io.kubernetes.pod.namespace=default                                           
	                                                            k8s:run=netperf-client-pod                                                        
	 3303       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                ready   
	                                                            reserved:host                                                                     
	 3551       Disabled           Disabled          34200      k8s:io.cilium.k8s.policy.cluster=default          fd00::cf   10.0.0.176   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                   
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                       
	                                                            k8s:k8s-app=kube-dns                                                              
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-qmx4l -- cilium bpf bandwidth list
Exitcode: 0 
Stdout:
 	 IDENTITY   EGRESS BANDWIDTH (BitsPerSec)
	 103        25M
	 1125       10M
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-qmx4l -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                             
	 6          Disabled           Disabled          4          reserved:health                                   fd00::112   10.0.1.127   ready   
	 103        Disabled           Disabled          20969      k8s:io.cilium.k8s.policy.cluster=default          fd00::159   10.0.1.88    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:run=netperf-25                                                                 
	 173        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                 ready   
	                                                            k8s:node-role.kubernetes.io/master                                                 
	                                                            reserved:host                                                                      
	 1125       Disabled           Disabled          7433       k8s:io.cilium.k8s.policy.cluster=default          fd00::143   10.0.1.25    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                    
	                                                            k8s:io.kubernetes.pod.namespace=default                                            
	                                                            k8s:run=netperf-10                                                                 
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
21:10:46 STEP: Running AfterEach for block EntireTestsuite K8sBandwidthTest
21:10:46 STEP: Running AfterEach for block EntireTestsuite
@joestringer joestringer added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels Sep 2, 2020
@joestringer joestringer mentioned this issue Sep 2, 2020
Merged
@sayboras sayboras mentioned this issue Sep 3, 2020
Merged
2 tasks
@borkmann
Copy link
Member

borkmann commented Sep 4, 2020

dup with #13062

@pchaigno pchaigno closed this as completed Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me!
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@borkmann @joestringer @pchaigno