Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v1.12] CI: K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining #22555

Closed
maintainer-s-little-helper bot opened this issue Dec 5, 2022 · 18 comments
Closed

[v1.12] CI: K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining #22555

maintainer-s-little-helper bot opened this issue Dec 5, 2022 · 18 comments
Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

Comments

@maintainer-s-little-helper
Copy link

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 1 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 k8s-app=cilium logs matching list of errors that must be investigated:
level=error
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "level=error" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
⚠️  Number of "level=warning" in logs: 19
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Key allocation attempt failed
Waiting for k8s node information
Unable to get node resource
Cannot create CEP
Cilium pods: [cilium-5m7vv cilium-pf4mr]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-external-1111-56548587dc-xfslk                        false     false
prometheus-655fb888d7-56h99                                  false     false
echo-a-d576c5f8b-ndh98                                       false     false
pod-to-b-multi-node-hostport-57fc8854f5-gpkp6                false     false
pod-to-c-multi-node-proxy-ingress-policy-74749c8794-gbt2x    false     false
grafana-5747bcc8f9-nfqnl                                     false     false
echo-c-58cc9b67d9-d42ln                                      false     false
pod-to-b-multi-node-nodeport-54446bdbb9-6zxf2                false     false
pod-to-c-intra-node-proxy-ingress-policy-5f95cf647c-nkxsd    false     false
pod-to-external-fqdn-allow-google-cnp-5ff4986c89-vlzx7       false     false
coredns-69b675786c-kr88g                                     false     false
pod-to-a-allowed-cnp-7b6d5ff99f-2k797                        false     false
pod-to-a-denied-cnp-6887b57579-9w86c                         false     false
pod-to-b-intra-node-nodeport-569d7c647-65vjh                 false     false
pod-to-b-multi-node-headless-64b6cbdd49-lw4l7                false     false
pod-to-b-intra-node-hostport-7d656d7bb9-lfc26                false     false
pod-to-b-multi-node-clusterip-fdf45bbbc-8pjks                false     false
pod-to-c-intra-node-proxy-to-proxy-policy-79584b8d68-pfgk8   false     false
pod-to-c-multi-node-proxy-to-proxy-policy-9f68b7595-c8sk5    false     false
echo-b-787dc99778-qr6fk                                      false     false
pod-to-a-57695cc7ff-4x2ks                                    false     false
pod-to-a-intra-node-proxy-egress-policy-74f796f479-bzz47     false     false
pod-to-a-multi-node-proxy-egress-policy-5b5f7d94d8-4l484     false     false
Cilium agent 'cilium-5m7vv': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0
Cilium agent 'cilium-pf4mr': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0

Standard Error

Click to show. 
13:30:32 STEP: Running BeforeAll block for EntireTestsuite
13:30:32 STEP: Starting tests: command line parameters: {Reprovision:false HoldEnvironment:false PassCLIEnvironment:true SSHConfig: ShowCommands:false TestScope: SkipLogGathering:false CiliumImage:quay.io/cilium/cilium-ci CiliumTag:39bbeff4891dc0ad8d1f38d35c68aa7d880da160 CiliumOperatorImage:quay.io/cilium/operator CiliumOperatorTag:39bbeff4891dc0ad8d1f38d35c68aa7d880da160 CiliumOperatorSuffix:-ci HubbleRelayImage:quay.io/cilium/hubble-relay-ci HubbleRelayTag:39bbeff4891dc0ad8d1f38d35c68aa7d880da160 ProvisionK8s:true Timeout:2h50m0s Kubeconfig:/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/vagrant-kubeconfig KubectlPath:/tmp/kubectl RegistryCredentials: Multinode:true RunQuarantined:false Help:false} environment variables: [JENKINS_HOME=/var/jenkins_home ghprbSourceBranch=yutaro/v3-backend-map-downgrade ghprbTriggerAuthorEmail=yhayakawa3720@gmail.com VM_MEMORY=8192 MAIL=/var/mail/root SSH_CLIENT=54.148.123.155 47914 22 ghprbPullAuthorEmail=yhayakawa3720@gmail.com USER=root PROJ_PATH=src/github.com/cilium/cilium RUN_CHANGES_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/2238/display/redirect?page=changes ghprbPullDescription=GitHub pull request #22416 of commit 39bbeff4891dc0ad8d1f38d35c68aa7d880da160, no merge conflicts. NETNEXT=0 ghprbActualCommit=39bbeff4891dc0ad8d1f38d35c68aa7d880da160 SHLVL=1 CILIUM_TAG=39bbeff4891dc0ad8d1f38d35c68aa7d880da160 NODE_LABELS=baremetal ginkgo nightly node-renewing-rhino vagrant HUDSON_URL=https://jenkins.cilium.io/ GIT_COMMIT=e95bbc004b0d0f738e6dfa3707eef73b779f20b0 OLDPWD=/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9 GINKGO_TIMEOUT=170m HOME=/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9 ghprbTriggerAuthorLoginMention=@YutaroHayakawa BUILD_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/2238/ ghprbPullAuthorLoginMention=@YutaroHayakawa HUDSON_COOKIE=a6c4aa23-6aab-4e6d-98a7-eb6b610931d9 JENKINS_SERVER_COOKIE=durable-9750e7264fcae5fc51abf70bead51bfc ghprbGhRepository=cilium/cilium DOCKER_TAG=39bbeff4891dc0ad8d1f38d35c68aa7d880da160 JobKernelVersion=49 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus KERNEL=49 CONTAINER_RUNTIME=docker WORKSPACE=/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9 ghprbPullLongDescription=This PR implements the downgrading path for v3 backend maps introduced in #21797 \r\n\r\n```release-note\r\nbpf: Implement downgrading path from v3 to v2 backend map\r\n```\r\n K8S_NODES=2 TESTDIR=/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test LOGNAME=root NODE_NAME=node-renewing-rhino ghprbCredentialsId=ciliumbot _=/usr/bin/java HUBBLE_RELAY_IMAGE=quay.io/cilium/hubble-relay-ci STAGE_NAME=BDD-Test-PR GIT_BRANCH=origin/pr/22416/merge EXECUTOR_NUMBER=0 ghprbTriggerAuthorLogin=YutaroHayakawa TERM=xterm XDG_SESSION_ID=4 HOST_FIREWALL=0 CILIUM_OPERATOR_TAG=39bbeff4891dc0ad8d1f38d35c68aa7d880da160 BUILD_DISPLAY_NAME=bpf: Implement downgrading path from v3 to v2 backend map  https://github.com/cilium/cilium/pull/22416  #2238 ghprbPullAuthorLogin=YutaroHayakawa HUDSON_HOME=/var/jenkins_home ghprbTriggerAuthor=Yutaro Hayakawa JOB_BASE_NAME=Cilium-PR-K8s-1.21-kernel-4.9 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/go/bin:/root/go/bin sha1=origin/pr/22416/merge KUBECONFIG=/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/vagrant-kubeconfig FOCUS=K8s BUILD_ID=2238 XDG_RUNTIME_DIR=/run/user/0 BUILD_TAG=jenkins-Cilium-PR-K8s-1.21-kernel-4.9-2238 RUN_QUARANTINED=false CILIUM_IMAGE=quay.io/cilium/cilium-ci JENKINS_URL=https://jenkins.cilium.io/ LANG=C.UTF-8 ghprbCommentBody=/test-backport-1.12 JOB_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/ ghprbPullTitle=bpf: Implement downgrading path from v3 to v2 backend map GIT_URL=https://github.com/cilium/cilium ghprbPullLink=https://github.com/cilium/cilium/pull/22416 BUILD_NUMBER=2238 JENKINS_NODE_COOKIE=dc1c1466-7840-4c07-9e7e-94aed39fbe7e SHELL=/bin/bash GOPATH=/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9 RUN_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/2238/display/redirect IMAGE_REGISTRY=quay.io/cilium ghprbAuthorRepoGitUrl=https://github.com/YutaroHayakawa/cilium.git FAILFAST=false HUDSON_SERVER_COOKIE=693c250bfb7e85bf ghprbTargetBranch=v1.12 JOB_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/display/redirect K8S_VERSION=1.21 JOB_NAME=Cilium-PR-K8s-1.21-kernel-4.9 PWD=/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test SSH_CONNECTION=54.148.123.155 47914 139.178.82.81 22 ghprbPullId=22416 CILIUM_OPERATOR_IMAGE=quay.io/cilium/operator HUBBLE_RELAY_TAG=39bbeff4891dc0ad8d1f38d35c68aa7d880da160 JobK8sVersion=1.21 VM_CPUS=3 CILIUM_OPERATOR_SUFFIX=-ci]
13:30:33 STEP: Ensuring the namespace kube-system exists
13:30:33 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
13:30:35 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
13:30:35 STEP: Preparing cluster
13:30:35 STEP: Labelling nodes
13:30:35 STEP: Cleaning up Cilium components
13:30:35 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
13:30:36 STEP: Ensuring the namespace kube-system exists
13:30:36 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
13:30:36 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
13:30:36 STEP: Installing Cilium
13:30:37 STEP: Waiting for Cilium to become ready
13:30:56 STEP: Restarting unmanaged pods coredns-69b675786c-qtjfs in namespace kube-system
13:30:56 STEP: Validating if Kubernetes DNS is deployed
13:30:56 STEP: Checking if deployment is ready
13:30:56 STEP: Kubernetes DNS is not ready: only 0 of 1 replicas are available
13:30:56 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
13:30:56 STEP: Waiting for Kubernetes DNS to become operational
13:30:56 STEP: Checking if deployment is ready
13:30:56 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
13:30:57 STEP: Checking if deployment is ready
13:30:57 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
13:30:58 STEP: Checking if deployment is ready
13:30:58 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
13:30:59 STEP: Checking if deployment is ready
13:30:59 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
13:31:00 STEP: Checking if deployment is ready
13:31:00 STEP: Checking if kube-dns service is plumbed correctly
13:31:00 STEP: Checking if pods have identity
13:31:00 STEP: Checking if DNS can resolve
13:31:01 STEP: Validating Cilium Installation
13:31:01 STEP: Performing Cilium controllers preflight check
13:31:01 STEP: Performing Cilium health check
13:31:01 STEP: Performing Cilium status preflight check
13:31:01 STEP: Checking whether host EP regenerated
13:31:01 STEP: Performing Cilium service preflight check
13:31:01 STEP: Performing K8s service preflight check
13:31:03 STEP: Waiting for cilium-operator to be ready
13:31:03 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
13:31:03 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
13:31:03 STEP: Making sure all endpoints are in ready state
13:31:06 STEP: WaitforPods(namespace="default", filter="")
13:31:51 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2022-12-05T13:31:51Z====
13:31:51 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 1 k8s-app=cilium logs matching list of errors that must be investigated:
level=error
===================== TEST FAILED =====================
13:31:52 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-nfqnl                                     1/1     Running   0          78s     10.0.1.252      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-56h99                                  1/1     Running   0          78s     10.0.1.197      k8s2   <none>           <none>
	 default             echo-a-d576c5f8b-ndh98                                       1/1     Running   0          48s     10.0.1.10       k8s2   <none>           <none>
	 default             echo-b-787dc99778-qr6fk                                      1/1     Running   0          48s     10.0.1.56       k8s2   <none>           <none>
	 default             echo-b-host-7f97669598-vpcxm                                 1/1     Running   0          48s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-58cc9b67d9-d42ln                                      1/1     Running   0          48s     10.0.1.171      k8s2   <none>           <none>
	 default             echo-c-host-6ff9f5c9fc-mkd95                                 1/1     Running   0          48s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6fd884bcf7-85wwp              1/1     Running   0          46s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-79f7df47b9-882k4               1/1     Running   0          46s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-57695cc7ff-4x2ks                                    1/1     Running   0          48s     10.0.1.19       k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-7b6d5ff99f-2k797                        1/1     Running   0          48s     10.0.1.164      k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-6887b57579-9w86c                         1/1     Running   0          48s     10.0.0.141      k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-74f796f479-bzz47     2/2     Running   0          48s     10.0.1.233      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5b5f7d94d8-4l484     2/2     Running   0          47s     10.0.0.168      k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-7d656d7bb9-lfc26                1/1     Running   0          45s     10.0.1.118      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-569d7c647-65vjh                 1/1     Running   0          45s     10.0.1.248      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-fdf45bbbc-8pjks                1/1     Running   0          46s     10.0.0.234      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-64b6cbdd49-lw4l7                1/1     Running   0          46s     10.0.0.50       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-57fc8854f5-gpkp6                1/1     Running   0          46s     10.0.0.1        k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-54446bdbb9-6zxf2                1/1     Running   0          45s     10.0.0.241      k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-5f95cf647c-nkxsd    2/2     Running   0          47s     10.0.1.86       k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-79584b8d68-pfgk8   2/2     Running   0          47s     10.0.1.187      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-74749c8794-gbt2x    2/2     Running   0          47s     10.0.0.113      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-9f68b7595-c8sk5    2/2     Running   0          47s     10.0.0.22       k8s1   <none>           <none>
	 default             pod-to-external-1111-56548587dc-xfslk                        1/1     Running   0          48s     10.0.1.220      k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-5ff4986c89-vlzx7       1/1     Running   0          48s     10.0.0.69       k8s1   <none>           <none>
	 kube-system         cilium-5m7vv                                                 1/1     Running   0          77s     192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-7c9869bc65-d4z7w                             1/1     Running   0          77s     192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-7c9869bc65-wtwpj                             1/1     Running   0          77s     192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-pf4mr                                                 1/1     Running   0          77s     192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-69b675786c-kr88g                                     1/1     Running   0          58s     10.0.0.239      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          5m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   0          5m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   2          5m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-vlpst                                             1/1     Running   0          118s    192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-w9nw4                                             1/1     Running   0          5m14s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   2          5m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-dvmgx                                           1/1     Running   0          81s     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-kcpbq                                           1/1     Running   0          81s     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-dsmnw                                         1/1     Running   0          116s    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-f8gs5                                         1/1     Running   0          116s    192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-5m7vv cilium-pf4mr]
cmd: kubectl exec -n kube-system cilium-5m7vv -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                              
	 172        Enabled            Disabled          7808       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::1bb   10.0.1.171   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=echo-c                                                                                                     
	 380        Disabled           Disabled          13971      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::166   10.0.1.248   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                                               
	 534        Disabled           Disabled          11985      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::1fc   10.0.1.86    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                                                   
	 1099       Disabled           Disabled          4          reserved:health                                                                    fd02::184   10.0.1.73    ready   
	 1114       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                                  ready   
	                                                            reserved:host                                                                                                       
	 1146       Disabled           Enabled           28693      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::110   10.0.1.233   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                                                    
	 1365       Disabled           Disabled          48648      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::117   10.0.1.19    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=pod-to-a                                                                                                   
	 1478       Disabled           Disabled          61071      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::116   10.0.1.118   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                                               
	 1908       Disabled           Enabled           46674      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::153   10.0.1.164   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=pod-to-a-allowed-cnp                                                                                       
	 2284       Disabled           Disabled          28438      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::1e0   10.0.1.56    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=echo-b                                                                                                     
	 2402       Disabled           Disabled          12528      k8s:app=grafana                                                                    fd02::173   10.0.1.252   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                    
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                   
	 2648       Disabled           Enabled           16013      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::148   10.0.1.187   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                                                  
	 2814       Disabled           Disabled          59057      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::1bc   10.0.1.10    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=echo-a                                                                                                     
	 3632       Disabled           Disabled          53588      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default             fd02::108   10.0.1.220   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                             
	                                                            k8s:name=pod-to-external-1111                                                                                       
	 3651       Disabled           Disabled          18708      k8s:app=prometheus                                                                 fd02::11e   10.0.1.197   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                    
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                              
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-pf4mr -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                       
	 207        Disabled           Disabled          4          reserved:health                                                              fd02::20   10.0.0.14    ready   
	 585        Disabled           Disabled          19350      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::57   10.0.0.239   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                         
	 1480       Disabled           Enabled           12515      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::d1   10.0.0.69    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                                               
	 2020       Disabled           Disabled          15935      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::11   10.0.0.50    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-b-multi-node-headless                                                                        
	 2256       Disabled           Disabled          8744       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::be   10.0.0.113   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                                            
	 2443       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                           ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                    
	                                                            k8s:node-role.kubernetes.io/master                                                                           
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                  
	                                                            reserved:host                                                                                                
	 2649       Disabled           Disabled          23270      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::1    10.0.0.234   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                                       
	 2686       Disabled           Disabled          24621      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::e0   10.0.0.1     ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                                        
	 2766       Disabled           Enabled           3965       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::f3   10.0.0.22    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                                           
	 3054       Disabled           Enabled           17542      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::82   10.0.0.168   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                                             
	 3514       Disabled           Enabled           57492      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::fe   10.0.0.141   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-a-denied-cnp                                                                                 
	 3820       Disabled           Disabled          14221      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::da   10.0.0.241   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                                        
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
13:32:05 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
13:32:49 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|1b2940c7_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9//2238/artifact/1b2940c7_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9//2238/artifact/test_results_Cilium-PR-K8s-1.21-kernel-4.9_2238_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/2238/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
@maintainer-s-little-helper maintainer-s-little-helper bot added the ci/flake This is a known failure that occurs in the tree. Please investigate me! label Dec 5, 2022
@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Dec 5, 2022
Merged
@gandro
Copy link
Member

gandro commented Dec 7, 2022

This test has been removed on master. Only happening on v1.12.

@gandro gandro changed the title CI: K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining [v1.12] CI: K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining Dec 7, 2022
@bimmlerd bimmlerd mentioned this issue Dec 8, 2022
Merged
14 tasks
@maintainer-s-little-helper
Copy link
Author

PR #23465 hit this flake with 95.99% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 1 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 k8s-app=cilium logs matching list of errors that must be investigated:
2023-01-30T18:47:37.785491491Z level=error msg="Cannot create CEP" containerID= controller="sync-to-k8s-ciliumendpoint (7)" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=7 error="etcdserver: request timed out" identity=16683 ipv4= ipv6= k8sPodName=/ subsys=endpointsynchronizer
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
Network status error received, restarting client connections
⚠️  Found "2023-01-30T18:47:37.785491491Z level=error msg=\"Cannot create CEP\" containerID= controller=\"sync-to-k8s-ciliumendpoint (7)\" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=7 error=\"etcdserver: request timed out\" identity=16683 ipv4= ipv6= k8sPodName=/ subsys=endpointsynchronizer" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
⚠️  Number of "level=warning" in logs: 12
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Key allocation attempt failed
Unable to get node resource
Waiting for k8s node information
Cannot create CEP
Cilium pods: [cilium-bbpt7 cilium-mmxfw]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
echo-c-688dcdfd44-ltpcg                                      false     false
pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-dzw49       false     false
coredns-5b7c49d4d8-pj4w2                                     false     false
pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-kzp85   false     false
prometheus-d87f8f984-ccrhv                                   false     false
pod-to-a-57c7db48fd-xhms5                                    false     false
pod-to-a-allowed-cnp-548f548798-zjmnq                        false     false
pod-to-a-denied-cnp-64b85f46b8-vq2tx                         false     false
pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-q7v4x     false     false
pod-to-b-multi-node-nodeport-658b99d8-n2csn                  false     false
pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-nd6ls   false     false
grafana-7fd557d749-7pfc5                                     false     false
echo-a-6954f488f5-fkt4d                                      false     false
pod-to-a-multi-node-proxy-egress-policy-5466dc57db-mmsdj     false     false
pod-to-b-intra-node-nodeport-7b59546c64-cktfj                false     false
pod-to-b-multi-node-headless-595bf6dfb-t84b5                 false     false
pod-to-external-1111-694c4c87dd-ljbpk                        false     false
echo-b-6c864d75d7-mfchh                                      false     false
pod-to-b-intra-node-hostport-5c58cb8758-899xp                false     false
pod-to-b-multi-node-clusterip-6c74fdb4b8-nm984               false     false
pod-to-b-multi-node-hostport-869554778-gtzlb                 false     false
pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-d28rk    false     false
pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-2btrp    false     false
Cilium agent 'cilium-bbpt7': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0
Cilium agent 'cilium-mmxfw': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0

Standard Error

Click to show. 
18:46:50 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
18:46:50 STEP: Ensuring the namespace kube-system exists
18:46:50 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
18:46:50 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
18:46:50 STEP: Installing Cilium
18:46:51 STEP: Waiting for Cilium to become ready
18:48:31 STEP: Restarting unmanaged pods coredns-5b7c49d4d8-4dhgn in namespace kube-system
18:48:41 STEP: Validating if Kubernetes DNS is deployed
18:48:41 STEP: Checking if deployment is ready
18:48:41 STEP: Kubernetes DNS is not ready: only 0 of 1 replicas are available
18:48:41 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
18:48:41 STEP: Waiting for Kubernetes DNS to become operational
18:48:41 STEP: Checking if deployment is ready
18:48:41 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:48:42 STEP: Checking if deployment is ready
18:48:42 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:48:43 STEP: Checking if deployment is ready
18:48:43 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:48:44 STEP: Checking if deployment is ready
18:48:44 STEP: Checking if kube-dns service is plumbed correctly
18:48:44 STEP: Checking if pods have identity
18:48:44 STEP: Checking if DNS can resolve
18:48:48 STEP: Validating Cilium Installation
18:48:48 STEP: Performing Cilium controllers preflight check
18:48:48 STEP: Performing Cilium status preflight check
18:48:48 STEP: Performing Cilium health check
18:48:48 STEP: Checking whether host EP regenerated
18:48:55 STEP: Performing Cilium service preflight check
18:48:55 STEP: Performing K8s service preflight check
18:49:01 STEP: Waiting for cilium-operator to be ready
18:49:01 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
18:49:02 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
18:49:02 STEP: Making sure all endpoints are in ready state
18:49:05 STEP: WaitforPods(namespace="default", filter="")
18:49:28 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-01-30T18:49:28Z====
18:49:28 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 1 k8s-app=cilium logs matching list of errors that must be investigated:
2023-01-30T18:47:37.785491491Z level=error msg="Cannot create CEP" containerID= controller="sync-to-k8s-ciliumendpoint (7)" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=7 error="etcdserver: request timed out" identity=16683 ipv4= ipv6= k8sPodName=/ subsys=endpointsynchronizer
===================== TEST FAILED =====================
18:49:29 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-7fd557d749-7pfc5                                     1/1     Running   0          2m44s   10.0.1.174      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-d87f8f984-ccrhv                                   1/1     Running   0          2m44s   10.0.0.100      k8s1   <none>           <none>
	 default             echo-a-6954f488f5-fkt4d                                      1/1     Running   0          30s     10.0.1.186      k8s2   <none>           <none>
	 default             echo-b-6c864d75d7-mfchh                                      1/1     Running   0          30s     10.0.1.39       k8s2   <none>           <none>
	 default             echo-b-host-75d477c875-fhffh                                 1/1     Running   0          30s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-688dcdfd44-ltpcg                                      1/1     Running   0          30s     10.0.1.68       k8s2   <none>           <none>
	 default             echo-c-host-5575b5cb-khm9l                                   1/1     Running   0          30s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6868bbd4f8-7lbf6              1/1     Running   0          27s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-69c9465576-hhjkr               1/1     Running   0          27s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-57c7db48fd-xhms5                                    1/1     Running   0          29s     10.0.1.244      k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-548f548798-zjmnq                        1/1     Running   0          29s     10.0.0.246      k8s1   <none>           <none>
	 default             pod-to-a-denied-cnp-64b85f46b8-vq2tx                         1/1     Running   0          29s     10.0.0.157      k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-q7v4x     2/2     Running   0          29s     10.0.1.171      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5466dc57db-mmsdj     2/2     Running   0          28s     10.0.0.68       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-5c58cb8758-899xp                1/1     Running   0          26s     10.0.1.253      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-7b59546c64-cktfj                1/1     Running   0          26s     10.0.1.135      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-6c74fdb4b8-nm984               1/1     Running   0          27s     10.0.0.242      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-595bf6dfb-t84b5                 1/1     Running   0          27s     10.0.0.218      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-869554778-gtzlb                 1/1     Running   0          27s     10.0.0.80       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-658b99d8-n2csn                  1/1     Running   0          26s     10.0.0.109      k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-d28rk    2/2     Running   0          28s     10.0.1.82       k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-nd6ls   2/2     Running   0          28s     10.0.1.182      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-2btrp    2/2     Running   0          28s     10.0.0.238      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-kzp85   2/2     Running   0          28s     10.0.0.76       k8s1   <none>           <none>
	 default             pod-to-external-1111-694c4c87dd-ljbpk                        1/1     Running   0          29s     10.0.1.243      k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-dzw49       1/1     Running   0          29s     10.0.1.136      k8s2   <none>           <none>
	 kube-system         cilium-bbpt7                                                 1/1     Running   0          2m43s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-mmxfw                                                 1/1     Running   0          2m43s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-cdcd7ddfd-cln2t                              1/1     Running   0          2m43s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-cdcd7ddfd-jxswx                              1/1     Running   1          2m43s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-5b7c49d4d8-pj4w2                                     1/1     Running   0          53s     10.0.1.242      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          7m5s    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   0          7m5s    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   3          7m5s    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-hhgb6                                             1/1     Running   0          3m23s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-vb2nq                                             1/1     Running   0          6m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   3          7m4s    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-49jkt                                           1/1     Running   0          2m47s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-whb98                                           1/1     Running   0          2m47s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-4bhzx                                         1/1     Running   0          3m21s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-vqjzl                                         1/1     Running   0          3m21s   192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-bbpt7 cilium-mmxfw]
cmd: kubectl exec -n kube-system cilium-bbpt7 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                              IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                   
	 62         Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                       ready   
	                                                            k8s:node-role.kubernetes.io/master                                                       
	                                                            reserved:host                                                                            
	 468        Disabled           Enabled           3457       k8s:io.cilium.k8s.policy.cluster=default                 fd02::4a   10.0.0.246   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-a-allowed-cnp                                                            
	 1067       Disabled           Enabled           8224       k8s:io.cilium.k8s.policy.cluster=default                 fd02::9c   10.0.0.157   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-a-denied-cnp                                                             
	 1251       Disabled           Enabled           13759      k8s:io.cilium.k8s.policy.cluster=default                 fd02::3    10.0.0.76    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                       
	 1359       Disabled           Disabled          2469       k8s:io.cilium.k8s.policy.cluster=default                 fd02::86   10.0.0.109   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                    
	 1746       Disabled           Disabled          59874      k8s:app=prometheus                                       fd02::9b   10.0.0.100   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                   
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                        
	 2122       Disabled           Enabled           2603       k8s:io.cilium.k8s.policy.cluster=default                 fd02::ce   10.0.0.68    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                         
	 2225       Disabled           Disabled          15352      k8s:io.cilium.k8s.policy.cluster=default                 fd02::f4   10.0.0.80    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                    
	 3062       Disabled           Disabled          23808      k8s:io.cilium.k8s.policy.cluster=default                 fd02::67   10.0.0.218   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-b-multi-node-headless                                                    
	 3253       Disabled           Disabled          24052      k8s:io.cilium.k8s.policy.cluster=default                 fd02::30   10.0.0.238   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                        
	 3545       Disabled           Disabled          4          reserved:health                                          fd02::5    10.0.0.128   ready   
	 4030       Disabled           Disabled          2431       k8s:io.cilium.k8s.policy.cluster=default                 fd02::34   10.0.0.242   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                  
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-mmxfw -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 7          Disabled           Disabled          16683      k8s:app=grafana                                      fd02::12f   10.0.1.174   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                              
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                     
	 99         Enabled            Disabled          33591      k8s:io.cilium.k8s.policy.cluster=default             fd02::158   10.0.1.68    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-c                                                                       
	 219        Disabled           Enabled           53668      k8s:io.cilium.k8s.policy.cluster=default             fd02::127   10.0.1.182   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                    
	 321        Disabled           Disabled          23961      k8s:io.cilium.k8s.policy.cluster=default             fd02::1f9   10.0.1.186   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-a                                                                       
	 340        Disabled           Disabled          22366      k8s:io.cilium.k8s.policy.cluster=default             fd02::143   10.0.1.253   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                 
	 667        Disabled           Enabled           56584      k8s:io.cilium.k8s.policy.cluster=default             fd02::195   10.0.1.136   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                        
	 735        Disabled           Disabled          30194      k8s:io.cilium.k8s.policy.cluster=default             fd02::173   10.0.1.135   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                 
	 740        Disabled           Disabled          13090      k8s:io.cilium.k8s.policy.cluster=default             fd02::18c   10.0.1.243   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-1111                                                         
	 1560       Disabled           Enabled           39360      k8s:io.cilium.k8s.policy.cluster=default             fd02::13f   10.0.1.171   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                      
	 1804       Disabled           Disabled          26568      k8s:io.cilium.k8s.policy.cluster=default             fd02::14b   10.0.1.242   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                       
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                           
	                                                            k8s:k8s-app=kube-dns                                                                  
	 2244       Disabled           Disabled          47696      k8s:io.cilium.k8s.policy.cluster=default             fd02::1ba   10.0.1.82    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                     
	 2727       Disabled           Disabled          26339      k8s:io.cilium.k8s.policy.cluster=default             fd02::198   10.0.1.39    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-b                                                                       
	 2930       Disabled           Disabled          4          reserved:health                                      fd02::14d   10.0.1.92    ready   
	 3364       Disabled           Disabled          14460      k8s:io.cilium.k8s.policy.cluster=default             fd02::1c0   10.0.1.244   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a                                                                     
	 3560       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                    ready   
	                                                            reserved:host                                                                         
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
18:49:46 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
18:50:33 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|7365cf55_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1059/artifact/7365cf55_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1059/artifact/test_results_Cilium-PR-K8s-1.17-kernel-4.9_1059_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9/1059/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
12 tasks
@maintainer-s-little-helper
Copy link
Author

PR #24386 hit this flake with 96.01% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 1 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 1 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-15T15:05:38.069919826Z level=error msg="Cannot create CEP" containerID= controller="sync-to-k8s-ciliumendpoint (2863)" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2863 error="etcdserver: request timed out" identity=14666 ipv4= ipv6= k8sPodName=/ subsys=endpointsynchronizer
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
Network status error received, restarting client connections
⚠️  Found "2023-03-15T15:05:38.069919826Z level=error msg=\"Cannot create CEP\" containerID= controller=\"sync-to-k8s-ciliumendpoint (2863)\" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2863 error=\"etcdserver: request timed out\" identity=14666 ipv4= ipv6= k8sPodName=/ subsys=endpointsynchronizer" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
⚠️  Number of "level=warning" in logs: 13
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Key allocation attempt failed
Unable to get node resource
Waiting for k8s node information
Cannot create CEP
Cilium pods: [cilium-fttwc cilium-nsjqw]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
echo-c-688dcdfd44-mdfmr                                      false     false
pod-to-b-intra-node-hostport-5c58cb8758-pmdhm                false     false
pod-to-b-multi-node-headless-595bf6dfb-jjk2v                 false     false
pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-szcc2    false     false
pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-6z6js   false     false
grafana-7fd557d749-c2pkx                                     false     false
echo-a-6954f488f5-qf2b4                                      false     false
echo-b-6c864d75d7-66qvm                                      false     false
pod-to-external-1111-694c4c87dd-mrnjr                        false     false
coredns-5b7c49d4d8-kqxcm                                     false     false
pod-to-a-57c7db48fd-6kbnp                                    false     false
pod-to-a-multi-node-proxy-egress-policy-5466dc57db-dtk88     false     false
pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-hx985   false     false
pod-to-b-multi-node-nodeport-658b99d8-vvxvz                  false     false
pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-59kcz       false     false
prometheus-d87f8f984-88spz                                   false     false
pod-to-b-multi-node-clusterip-6c74fdb4b8-c45mf               false     false
pod-to-b-multi-node-hostport-869554778-j7n7b                 false     false
pod-to-b-intra-node-nodeport-7b59546c64-fn886                false     false
pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-pxn8x    false     false
pod-to-a-allowed-cnp-548f548798-mfz2d                        false     false
pod-to-a-denied-cnp-64b85f46b8-d5rbh                         false     false
pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-trjlz     false     false
Cilium agent 'cilium-fttwc': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0
Cilium agent 'cilium-nsjqw': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0

Standard Error

Click to show. 
15:04:52 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
15:04:52 STEP: Ensuring the namespace kube-system exists
15:04:52 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
15:04:52 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
15:04:52 STEP: Installing Cilium
15:04:53 STEP: Waiting for Cilium to become ready
15:05:23 STEP: Restarting unmanaged pods coredns-5b7c49d4d8-9fqk6 in namespace kube-system
15:05:43 STEP: Validating if Kubernetes DNS is deployed
15:05:43 STEP: Checking if deployment is ready
15:05:43 STEP: Kubernetes DNS is not ready: only 0 of 1 replicas are available
15:05:43 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
15:05:43 STEP: Waiting for Kubernetes DNS to become operational
15:05:43 STEP: Checking if deployment is ready
15:05:43 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:44 STEP: Checking if deployment is ready
15:05:44 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:45 STEP: Checking if deployment is ready
15:05:45 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:46 STEP: Checking if deployment is ready
15:05:46 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:47 STEP: Checking if deployment is ready
15:05:47 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:48 STEP: Checking if deployment is ready
15:05:48 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:49 STEP: Checking if deployment is ready
15:05:49 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:50 STEP: Checking if deployment is ready
15:05:50 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:51 STEP: Checking if deployment is ready
15:05:51 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:52 STEP: Checking if deployment is ready
15:05:52 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:53 STEP: Checking if deployment is ready
15:05:54 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:54 STEP: Checking if deployment is ready
15:05:54 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:55 STEP: Checking if deployment is ready
15:05:55 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:56 STEP: Checking if deployment is ready
15:05:56 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:57 STEP: Checking if deployment is ready
15:05:57 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:58 STEP: Checking if deployment is ready
15:05:58 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:05:59 STEP: Checking if deployment is ready
15:05:59 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:00 STEP: Checking if deployment is ready
15:06:00 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:01 STEP: Checking if deployment is ready
15:06:01 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:02 STEP: Checking if deployment is ready
15:06:02 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:03 STEP: Checking if deployment is ready
15:06:03 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:04 STEP: Checking if deployment is ready
15:06:04 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:05 STEP: Checking if deployment is ready
15:06:05 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:06 STEP: Checking if deployment is ready
15:06:06 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:07 STEP: Checking if deployment is ready
15:06:07 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:08 STEP: Checking if deployment is ready
15:06:08 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:09 STEP: Checking if deployment is ready
15:06:09 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:10 STEP: Checking if deployment is ready
15:06:10 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:11 STEP: Checking if deployment is ready
15:06:11 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:12 STEP: Checking if deployment is ready
15:06:12 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:13 STEP: Checking if deployment is ready
15:06:13 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:14 STEP: Checking if deployment is ready
15:06:14 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:15 STEP: Checking if deployment is ready
15:06:15 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:16 STEP: Checking if deployment is ready
15:06:16 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:17 STEP: Checking if deployment is ready
15:06:17 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:18 STEP: Checking if deployment is ready
15:06:22 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:22 STEP: Checking if deployment is ready
15:06:22 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:22 STEP: Checking if deployment is ready
15:06:22 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:23 STEP: Checking if deployment is ready
15:06:23 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:24 STEP: Checking if deployment is ready
15:06:24 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:25 STEP: Checking if deployment is ready
15:06:25 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:26 STEP: Checking if deployment is ready
15:06:26 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:27 STEP: Checking if deployment is ready
15:06:27 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:28 STEP: Checking if deployment is ready
15:06:28 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:29 STEP: Checking if deployment is ready
15:06:29 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:30 STEP: Checking if deployment is ready
15:06:30 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:31 STEP: Checking if deployment is ready
15:06:31 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:32 STEP: Checking if deployment is ready
15:06:32 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:33 STEP: Checking if deployment is ready
15:06:33 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:34 STEP: Checking if deployment is ready
15:06:34 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
15:06:35 STEP: Checking if deployment is ready
15:06:35 STEP: Checking if kube-dns service is plumbed correctly
15:06:35 STEP: Checking if pods have identity
15:06:35 STEP: Checking if DNS can resolve
15:06:41 STEP: Validating Cilium Installation
15:06:41 STEP: Performing Cilium controllers preflight check
15:06:41 STEP: Performing Cilium health check
15:06:41 STEP: Checking whether host EP regenerated
15:06:41 STEP: Performing Cilium status preflight check
15:06:48 STEP: Performing Cilium service preflight check
15:06:48 STEP: Performing K8s service preflight check
15:06:54 STEP: Waiting for cilium-operator to be ready
15:06:55 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
15:06:55 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
15:06:55 STEP: Making sure all endpoints are in ready state
15:06:58 STEP: WaitforPods(namespace="default", filter="")
15:07:22 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-03-15T15:07:22Z====
15:07:22 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 1 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-15T15:05:38.069919826Z level=error msg="Cannot create CEP" containerID= controller="sync-to-k8s-ciliumendpoint (2863)" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2863 error="etcdserver: request timed out" identity=14666 ipv4= ipv6= k8sPodName=/ subsys=endpointsynchronizer
===================== TEST FAILED =====================
15:07:22 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-7fd557d749-c2pkx                                     1/1     Running   0          2m36s   10.0.1.121      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-d87f8f984-88spz                                   1/1     Running   0          2m36s   10.0.1.169      k8s1   <none>           <none>
	 default             echo-a-6954f488f5-qf2b4                                      1/1     Running   0          31s     10.0.0.28       k8s2   <none>           <none>
	 default             echo-b-6c864d75d7-66qvm                                      1/1     Running   0          31s     10.0.0.33       k8s2   <none>           <none>
	 default             echo-b-host-75d477c875-nmhfv                                 1/1     Running   0          31s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-688dcdfd44-mdfmr                                      1/1     Running   0          31s     10.0.0.159      k8s2   <none>           <none>
	 default             echo-c-host-5575b5cb-b4tz2                                   1/1     Running   0          31s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6868bbd4f8-mbmgb              1/1     Running   0          28s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-69c9465576-7mdqt               1/1     Running   0          28s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-57c7db48fd-6kbnp                                    1/1     Running   0          31s     10.0.0.149      k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-548f548798-mfz2d                        1/1     Running   0          30s     10.0.0.167      k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-64b85f46b8-d5rbh                         1/1     Running   0          30s     10.0.0.6        k8s2   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-trjlz     2/2     Running   0          30s     10.0.0.168      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5466dc57db-dtk88     2/2     Running   0          30s     10.0.1.84       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-5c58cb8758-pmdhm                1/1     Running   0          28s     10.0.0.136      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-7b59546c64-fn886                1/1     Running   0          27s     10.0.0.19       k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-6c74fdb4b8-c45mf               1/1     Running   0          29s     10.0.1.233      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-595bf6dfb-jjk2v                 1/1     Running   0          28s     10.0.1.93       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-869554778-j7n7b                 1/1     Running   0          28s     10.0.1.225      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-658b99d8-vvxvz                  1/1     Running   0          27s     10.0.1.63       k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-pxn8x    2/2     Running   0          29s     10.0.0.21       k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-hx985   2/2     Running   0          29s     10.0.0.211      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-szcc2    2/2     Running   0          29s     10.0.1.200      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-6z6js   2/2     Running   0          29s     10.0.1.215      k8s1   <none>           <none>
	 default             pod-to-external-1111-694c4c87dd-mrnjr                        1/1     Running   0          31s     10.0.0.104      k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-59kcz       1/1     Running   0          30s     10.0.1.162      k8s1   <none>           <none>
	 kube-system         cilium-fttwc                                                 1/1     Running   0          2m35s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-nsjqw                                                 1/1     Running   0          2m35s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-6d49c55d79-4hnpv                             1/1     Running   1          2m35s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-6d49c55d79-xfpgl                             1/1     Running   0          2m35s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-5b7c49d4d8-kqxcm                                     1/1     Running   0          2m5s    10.0.0.109      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          6m45s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   0          6m39s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   3          6m39s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-jdz9j                                             1/1     Running   0          4m40s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-nxcrm                                             1/1     Running   0          3m18s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   3          6m39s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-ntjjg                                           1/1     Running   0          2m39s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-rxwmg                                           1/1     Running   0          2m39s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-29688                                         1/1     Running   0          3m16s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-gmjsx                                         1/1     Running   0          3m16s   192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-fttwc cilium-nsjqw]
cmd: kubectl exec -n kube-system cilium-fttwc -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                              IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                    
	 653        Disabled           Enabled           15213      k8s:io.cilium.k8s.policy.cluster=default                 fd02::110   10.0.1.215   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                   
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                        
	 882        Disabled           Disabled          4          reserved:health                                          fd02::171   10.0.1.220   ready   
	 1159       Disabled           Disabled          1370       k8s:io.cilium.k8s.policy.cluster=default                 fd02::162   10.0.1.233   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                   
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                    
	 1167       Disabled           Disabled          57870      k8s:app=prometheus                                       fd02::1c8   10.0.1.169   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                         
	 1280       Disabled           Disabled          30882      k8s:io.cilium.k8s.policy.cluster=default                 fd02::19d   10.0.1.63    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                   
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                     
	 1300       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                        ready   
	                                                            k8s:node-role.kubernetes.io/master                                                        
	                                                            reserved:host                                                                             
	 1808       Disabled           Disabled          56220      k8s:app=grafana                                          fd02::1c9   10.0.1.121   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                         
	 2106       Disabled           Enabled           45939      k8s:io.cilium.k8s.policy.cluster=default                 fd02::176   10.0.1.162   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                   
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                            
	 2241       Disabled           Disabled          347        k8s:io.cilium.k8s.policy.cluster=default                 fd02::18d   10.0.1.200   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                   
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                         
	 2276       Disabled           Enabled           6410       k8s:io.cilium.k8s.policy.cluster=default                 fd02::1ca   10.0.1.84    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                   
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                          
	 3420       Disabled           Disabled          65436      k8s:io.cilium.k8s.policy.cluster=default                 fd02::18c   10.0.1.93    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                   
	                                                            k8s:name=pod-to-b-multi-node-headless                                                     
	 3870       Disabled           Disabled          13244      k8s:io.cilium.k8s.policy.cluster=default                 fd02::1b9   10.0.1.225   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                   
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                     
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-nsjqw -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 124        Disabled           Disabled          32889      k8s:io.cilium.k8s.policy.cluster=default             fd02::fc   10.0.0.21    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 337        Disabled           Disabled          751        k8s:io.cilium.k8s.policy.cluster=default             fd02::d4   10.0.0.104   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 841        Disabled           Enabled           52226      k8s:io.cilium.k8s.policy.cluster=default             fd02::75   10.0.0.6     ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-denied-cnp                                                         
	 912        Disabled           Enabled           23706      k8s:io.cilium.k8s.policy.cluster=default             fd02::e0   10.0.0.167   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-allowed-cnp                                                        
	 929        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 1059       Disabled           Disabled          4          reserved:health                                      fd02::3c   10.0.0.171   ready   
	 1393       Disabled           Disabled          3465       k8s:io.cilium.k8s.policy.cluster=default             fd02::56   10.0.0.19    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 1634       Disabled           Disabled          33640      k8s:io.cilium.k8s.policy.cluster=default             fd02::59   10.0.0.33    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 1672       Disabled           Disabled          20996      k8s:io.cilium.k8s.policy.cluster=default             fd02::d1   10.0.0.28    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 1958       Disabled           Enabled           20563      k8s:io.cilium.k8s.policy.cluster=default             fd02::fd   10.0.0.211   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 2361       Disabled           Disabled          33708      k8s:io.cilium.k8s.policy.cluster=default             fd02::63   10.0.0.149   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a                                                                    
	 2863       Disabled           Disabled          14666      k8s:io.cilium.k8s.policy.cluster=default             fd02::d6   10.0.0.109   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                          
	                                                            k8s:k8s-app=kube-dns                                                                 
	 2967       Disabled           Disabled          26337      k8s:io.cilium.k8s.policy.cluster=default             fd02::c5   10.0.0.136   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 3459       Enabled            Disabled          54445      k8s:io.cilium.k8s.policy.cluster=default             fd02::a9   10.0.0.159   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 3819       Disabled           Enabled           41589      k8s:io.cilium.k8s.policy.cluster=default             fd02::34   10.0.0.168   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
15:07:38 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
15:08:24 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|81796ea2_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1211/artifact/81796ea2_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1211/artifact/test_results_Cilium-PR-K8s-1.17-kernel-4.9_1211_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9/1211/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
1 task
@maintainer-s-little-helper
Copy link
Author

PR #24293 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-15T20:58:42.119545073Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.21 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-15T20:58:30.118846891Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.179 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-03-15T20:58:42.119545073Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.21 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-03-15T20:58:30.118846891Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.179 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 469
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
+ true
100:\tfrom all lookup local
Waiting for k8s node information
Command execution failed
+ '[' false = true ']'
Cilium pods: [cilium-fprpt cilium-x8k4k]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
echo-b-5c5d7c49b5-9txfr                                                
pod-to-c-intra-node-proxy-to-proxy-policy-77b98c977d-knm44             
pod-to-c-multi-node-proxy-to-proxy-policy-78647869d5-2w9rv             
pod-to-external-fqdn-allow-google-cnp-7d5cc887cd-xkqr5                 
coredns-86c74c674b-s59gf                                               
pod-to-a-intra-node-proxy-egress-policy-65f4654f99-tk6hz               
pod-to-b-intra-node-hostport-d568f6bd4-9k9j8                           
pod-to-b-multi-node-clusterip-86dbd49cdb-8xw54                         
pod-to-c-multi-node-proxy-ingress-policy-56889f84cd-ldw9s              
pod-to-external-1111-f466d786b-kn24x                                   
echo-a-78667c8fdf-sllcz                                                
echo-c-78dc7fc59d-n7lc2                                                
pod-to-a-69cc5b49b8-7lgvt                                              
pod-to-b-multi-node-hostport-6d54854b6d-bdlw8                          
pod-to-b-multi-node-nodeport-8697db657d-qx8c8                          
pod-to-c-intra-node-proxy-ingress-policy-fccbcc9bf-5ftfv               
pod-to-a-allowed-cnp-86b8c7bf44-w7w5t                                  
pod-to-a-denied-cnp-5f9b6b964b-wxsvx                                   
pod-to-a-multi-node-proxy-egress-policy-69b8569968-kp2c4               
pod-to-b-intra-node-nodeport-79d9fc898b-qr429                          
pod-to-b-multi-node-headless-7bcf8566b-866nh                           
Cilium agent 'cilium-fprpt': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 70 Failed 0
Cilium agent 'cilium-x8k4k': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0

Standard Error

Click to show. 
20:58:08 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
20:58:08 STEP: Ensuring the namespace kube-system exists
20:58:08 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
20:58:08 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
20:58:09 STEP: Installing Cilium
20:58:09 STEP: Waiting for Cilium to become ready
20:59:16 STEP: Restarting unmanaged pods coredns-86c74c674b-pgwfj in namespace kube-system
20:59:23 STEP: Validating if Kubernetes DNS is deployed
20:59:23 STEP: Checking if deployment is ready
20:59:23 STEP: Checking if kube-dns service is plumbed correctly
20:59:23 STEP: Checking if pods have identity
20:59:23 STEP: Checking if DNS can resolve
20:59:27 STEP: Kubernetes DNS is up and operational
20:59:27 STEP: Validating Cilium Installation
20:59:27 STEP: Performing Cilium controllers preflight check
20:59:27 STEP: Performing Cilium status preflight check
20:59:27 STEP: Performing Cilium health check
20:59:27 STEP: Checking whether host EP regenerated
20:59:35 STEP: Performing Cilium service preflight check
20:59:35 STEP: Performing K8s service preflight check
20:59:41 STEP: Waiting for cilium-operator to be ready
20:59:41 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
20:59:41 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
20:59:41 STEP: Making sure all endpoints are in ready state
20:59:44 STEP: WaitforPods(namespace="default", filter="")
21:00:02 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-03-15T21:00:02Z====
21:00:02 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-15T20:58:42.119545073Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.21 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-15T20:58:30.118846891Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.179 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
21:00:03 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE    IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-54dbdc987-r7v4f                                      0/1     Running   0          89m    10.0.1.100      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-6ff848df8b-dcm8l                                  1/1     Running   0          89m    10.0.1.189      k8s2   <none>           <none>
	 default             echo-a-78667c8fdf-sllcz                                      1/1     Running   0          24s    10.0.0.19       k8s2   <none>           <none>
	 default             echo-b-5c5d7c49b5-9txfr                                      1/1     Running   0          24s    10.0.0.244      k8s2   <none>           <none>
	 default             echo-b-host-7df9796db6-2jc45                                 1/1     Running   0          24s    192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-78dc7fc59d-n7lc2                                      1/1     Running   0          24s    10.0.0.101      k8s2   <none>           <none>
	 default             echo-c-host-7779f885fc-nj66d                                 1/1     Running   0          24s    192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6744fdbf6-pkznn               1/1     Running   0          21s    192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-87b4cc8cd-nhn59                1/1     Running   0          21s    192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-69cc5b49b8-7lgvt                                    1/1     Running   0          24s    10.0.1.130      k8s1   <none>           <none>
	 default             pod-to-a-allowed-cnp-86b8c7bf44-w7w5t                        1/1     Running   0          24s    10.0.0.241      k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-5f9b6b964b-wxsvx                         1/1     Running   0          24s    10.0.1.176      k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-65f4654f99-tk6hz     2/2     Running   0          23s    10.0.0.96       k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-69b8569968-kp2c4     2/2     Running   0          23s    10.0.1.220      k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-d568f6bd4-9k9j8                 1/1     Running   0          21s    10.0.0.196      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-79d9fc898b-qr429                1/1     Running   0          20s    10.0.0.52       k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-86dbd49cdb-8xw54               1/1     Running   0          22s    10.0.1.94       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-7bcf8566b-866nh                 1/1     Running   0          21s    10.0.1.199      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-6d54854b6d-bdlw8                1/1     Running   0          21s    10.0.1.27       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-8697db657d-qx8c8                1/1     Running   0          21s    10.0.1.78       k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-fccbcc9bf-5ftfv     2/2     Running   0          23s    10.0.0.133      k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-77b98c977d-knm44   2/2     Running   0          23s    10.0.0.149      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-56889f84cd-ldw9s    2/2     Running   0          23s    10.0.1.9        k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-78647869d5-2w9rv   2/2     Running   0          22s    10.0.1.114      k8s1   <none>           <none>
	 default             pod-to-external-1111-f466d786b-kn24x                         1/1     Running   0          24s    10.0.0.67       k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7d5cc887cd-xkqr5       1/1     Running   0          23s    10.0.1.115      k8s1   <none>           <none>
	 kube-system         cilium-fprpt                                                 1/1     Running   1          119s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-8687f945fd-69rwv                             1/1     Running   0          119s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-8687f945fd-bbl7f                             1/1     Running   0          119s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-x8k4k                                                 1/1     Running   1          119s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-86c74c674b-s59gf                                     1/1     Running   0          52s    10.0.0.87       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          98m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   3          98m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   4          98m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-2ckdn                                             1/1     Running   0          98m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-q6zrh                                             1/1     Running   0          90m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   2          98m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-kjdfx                                           1/1     Running   0          89m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-lnzxx                                           1/1     Running   0          89m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-vnbxk                                         1/1     Running   0          90m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-vt4j6                                         1/1     Running   0          90m    192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-fprpt cilium-x8k4k]
cmd: kubectl exec -n kube-system cilium-fprpt -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 63         Disabled           Enabled           7169       k8s:io.cilium.k8s.policy.cluster=default             fd02::19   10.0.0.241   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-allowed-cnp                                                        
	 247        Disabled           Enabled           33539      k8s:io.cilium.k8s.policy.cluster=default             fd02::1e   10.0.0.96    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 935        Disabled           Disabled          52777      k8s:io.cilium.k8s.policy.cluster=default             fd02::e1   10.0.0.196   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 1092       Disabled           Disabled          3845       k8s:io.cilium.k8s.policy.cluster=default             fd02::ae   10.0.0.244   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 1726       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 2045       Enabled            Disabled          48084      k8s:io.cilium.k8s.policy.cluster=default             fd02::4d   10.0.0.101   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 2244       Disabled           Enabled           11179      k8s:io.cilium.k8s.policy.cluster=default             fd02::b1   10.0.0.149   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 2279       Disabled           Disabled          4          reserved:health                                      fd02::93   10.0.0.174   ready   
	 2402       Disabled           Disabled          55708      k8s:io.cilium.k8s.policy.cluster=default             fd02::a5   10.0.0.87    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                          
	                                                            k8s:k8s-app=kube-dns                                                                 
	 3082       Disabled           Disabled          62666      k8s:io.cilium.k8s.policy.cluster=default             fd02::f1   10.0.0.19    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 3690       Disabled           Disabled          51322      k8s:io.cilium.k8s.policy.cluster=default             fd02::36   10.0.0.67    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 3952       Disabled           Disabled          4943       k8s:io.cilium.k8s.policy.cluster=default             fd02::ac   10.0.0.133   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 4083       Disabled           Disabled          42569      k8s:io.cilium.k8s.policy.cluster=default             fd02::1c   10.0.0.52    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-x8k4k -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 403        Disabled           Disabled          44622      k8s:io.cilium.k8s.policy.cluster=default             fd02::1cf   10.0.1.130   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a                                                                     
	 678        Disabled           Enabled           52641      k8s:io.cilium.k8s.policy.cluster=default             fd02::14e   10.0.1.115   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                        
	 1396       Disabled           Enabled           6735       k8s:io.cilium.k8s.policy.cluster=default             fd02::1bf   10.0.1.114   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                    
	 1722       Disabled           Disabled          37984      k8s:io.cilium.k8s.policy.cluster=default             fd02::16f   10.0.1.199   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-headless                                                 
	 1754       Disabled           Enabled           8694       k8s:io.cilium.k8s.policy.cluster=default             fd02::16e   10.0.1.220   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                      
	 1838       Disabled           Disabled          13178      k8s:io.cilium.k8s.policy.cluster=default             fd02::1b7   10.0.1.9     ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                     
	 2346       Disabled           Disabled          4          reserved:health                                      fd02::14c   10.0.1.229   ready   
	 2421       Disabled           Enabled           8377       k8s:io.cilium.k8s.policy.cluster=default             fd02::1d9   10.0.1.176   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-denied-cnp                                                          
	 2514       Disabled           Disabled          31127      k8s:io.cilium.k8s.policy.cluster=default             fd02::148   10.0.1.78    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                 
	 2690       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                    ready   
	                                                            k8s:node-role.kubernetes.io/master                                                    
	                                                            reserved:host                                                                         
	 3569       Disabled           Disabled          2755       k8s:io.cilium.k8s.policy.cluster=default             fd02::15b   10.0.1.94    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                
	 3734       Disabled           Disabled          3247       k8s:io.cilium.k8s.policy.cluster=default             fd02::167   10.0.1.27    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                 
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
21:00:19 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
21:01:04 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|447ee200_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9//2568/artifact/447ee200_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9//2568/artifact/test_results_Cilium-PR-K8s-1.18-kernel-4.9_2568_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9/2568/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #24604 hit this flake with 94.08% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-28T14:28:56.637623037Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.22 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-28T14:28:55.884087750Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.222 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-03-28T14:28:56.637623037Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.22 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-03-28T14:28:55.884087750Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.222 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 4
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 474
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
100:\tfrom all lookup local
Waiting for k8s node information
+ true
+ '[' false = true ']'
Cilium pods: [cilium-f44c8 cilium-m9r4b]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-b-multi-node-headless-64b6cbdd49-7864q                          
pod-to-c-intra-node-proxy-ingress-policy-5f95cf647c-b6b4k              
pod-to-c-multi-node-proxy-ingress-policy-74749c8794-77bqv              
echo-b-787dc99778-bwbqw                                                
pod-to-a-57695cc7ff-gvrhc                                              
pod-to-b-intra-node-hostport-7d656d7bb9-9596f                          
pod-to-c-multi-node-proxy-to-proxy-policy-9f68b7595-254j9              
pod-to-external-fqdn-allow-google-cnp-5ff4986c89-lvlw4                 
echo-a-d576c5f8b-494sl                                                 
pod-to-b-multi-node-clusterip-fdf45bbbc-lhrxn                          
pod-to-b-multi-node-hostport-57fc8854f5-5kbx6                          
pod-to-c-intra-node-proxy-to-proxy-policy-79584b8d68-dvkn2             
coredns-69b675786c-79rw5                                               
echo-c-58cc9b67d9-4hmsl                                                
pod-to-a-allowed-cnp-7b6d5ff99f-pp8p6                                  
pod-to-a-multi-node-proxy-egress-policy-5b5f7d94d8-mzfp4               
pod-to-b-multi-node-nodeport-54446bdbb9-r8clf                          
pod-to-external-1111-56548587dc-7rvvh                                  
pod-to-a-denied-cnp-6887b57579-4mng9                                   
pod-to-a-intra-node-proxy-egress-policy-74f796f479-cz694               
pod-to-b-intra-node-nodeport-569d7c647-69nqv                           
Cilium agent 'cilium-f44c8': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 70 Failed 0
Cilium agent 'cilium-m9r4b': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0

Standard Error

Click to show. 
14:28:23 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
14:28:23 STEP: Ensuring the namespace kube-system exists
14:28:23 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
14:28:23 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
14:28:23 STEP: Installing Cilium
14:28:23 STEP: Waiting for Cilium to become ready
14:29:54 STEP: Restarting unmanaged pods coredns-69b675786c-hsglp in namespace kube-system
14:30:12 STEP: Validating if Kubernetes DNS is deployed
14:30:12 STEP: Checking if deployment is ready
14:30:12 STEP: Checking if kube-dns service is plumbed correctly
14:30:12 STEP: Checking if pods have identity
14:30:12 STEP: Checking if DNS can resolve
14:30:16 STEP: Kubernetes DNS is up and operational
14:30:16 STEP: Validating Cilium Installation
14:30:16 STEP: Performing Cilium status preflight check
14:30:16 STEP: Performing Cilium health check
14:30:16 STEP: Checking whether host EP regenerated
14:30:16 STEP: Performing Cilium controllers preflight check
14:30:23 STEP: Performing Cilium service preflight check
14:30:23 STEP: Performing K8s service preflight check
14:30:29 STEP: Waiting for cilium-operator to be ready
14:30:29 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
14:30:30 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
14:30:30 STEP: Making sure all endpoints are in ready state
14:30:33 STEP: WaitforPods(namespace="default", filter="")
14:30:55 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-03-28T14:30:55Z====
14:30:55 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-28T14:28:56.637623037Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.22 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-28T14:28:55.884087750Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.222 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
14:30:56 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-mv9hp                                     0/1     Running   0          57m     10.0.1.41       k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-5j9z9                                  1/1     Running   0          57m     10.0.1.9        k8s2   <none>           <none>
	 default             echo-a-d576c5f8b-494sl                                       1/1     Running   0          28s     10.0.1.146      k8s2   <none>           <none>
	 default             echo-b-787dc99778-bwbqw                                      1/1     Running   0          28s     10.0.1.47       k8s2   <none>           <none>
	 default             echo-b-host-675cd8cfff-6z2h8                                 1/1     Running   0          28s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-58cc9b67d9-4hmsl                                      1/1     Running   0          28s     10.0.1.105      k8s2   <none>           <none>
	 default             echo-c-host-79c7cc6568-g6qpx                                 1/1     Running   0          28s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6fd884bcf7-jjdq2              1/1     Running   0          26s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-79f7df47b9-gr75r               1/1     Running   0          25s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-57695cc7ff-gvrhc                                    1/1     Running   0          28s     10.0.1.199      k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-7b6d5ff99f-pp8p6                        1/1     Running   0          28s     10.0.1.147      k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-6887b57579-4mng9                         1/1     Running   0          28s     10.0.0.79       k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-74f796f479-cz694     2/2     Running   0          27s     10.0.1.249      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5b5f7d94d8-mzfp4     2/2     Running   0          27s     10.0.0.175      k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-7d656d7bb9-9596f                1/1     Running   0          25s     10.0.1.239      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-569d7c647-69nqv                 1/1     Running   0          25s     10.0.1.220      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-fdf45bbbc-lhrxn                1/1     Running   0          26s     10.0.0.204      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-64b6cbdd49-7864q                1/1     Running   0          26s     10.0.0.118      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-57fc8854f5-5kbx6                1/1     Running   0          25s     10.0.0.238      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-54446bdbb9-r8clf                1/1     Running   0          25s     10.0.0.176      k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-5f95cf647c-b6b4k    2/2     Running   0          27s     10.0.1.240      k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-79584b8d68-dvkn2   2/2     Running   0          26s     10.0.1.21       k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-74749c8794-77bqv    2/2     Running   0          27s     10.0.0.140      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-9f68b7595-254j9    2/2     Running   0          26s     10.0.0.180      k8s1   <none>           <none>
	 default             pod-to-external-1111-56548587dc-7rvvh                        1/1     Running   0          28s     10.0.1.160      k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-5ff4986c89-lvlw4       1/1     Running   0          28s     10.0.0.85       k8s1   <none>           <none>
	 kube-system         cilium-f44c8                                                 1/1     Running   1          2m38s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-m9r4b                                                 1/1     Running   1          2m38s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-57585868fc-45zcn                             1/1     Running   0          2m38s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-57585868fc-bl4jc                             1/1     Running   0          2m38s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-69b675786c-79rw5                                     1/1     Running   0          67s     10.0.0.20       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   1          66m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   1          66m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   4          66m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-hkjzh                                             1/1     Running   0          58m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-t6jnc                                             1/1     Running   0          66m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   3          66m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-pmvbv                                           1/1     Running   0          57m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-tvjk4                                           1/1     Running   0          57m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-gs8wz                                         1/1     Running   0          58m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-sfx4p                                         1/1     Running   0          58m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-f44c8 cilium-m9r4b]
cmd: kubectl exec -n kube-system cilium-f44c8 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                              IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                    
	 31         Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                        ready   
	                                                            reserved:host                                                                                             
	 374        Disabled           Disabled          47681      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::1e6   10.0.1.160   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=pod-to-external-1111                                                                             
	 413        Disabled           Enabled           3588       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::147   10.0.1.147   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=pod-to-a-allowed-cnp                                                                             
	 488        Disabled           Disabled          4          reserved:health                                                          fd02::1ef   10.0.1.155   ready   
	 533        Disabled           Disabled          21504      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::12a   10.0.1.239   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                                     
	 544        Disabled           Disabled          21211      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::188   10.0.1.220   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                                     
	 1919       Disabled           Enabled           41171      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::149   10.0.1.21    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                                        
	 2161       Disabled           Disabled          27307      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::14b   10.0.1.47    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=echo-b                                                                                           
	 2574       Enabled            Disabled          13724      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::197   10.0.1.105   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=echo-c                                                                                           
	 2707       Disabled           Disabled          16187      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::1e9   10.0.1.240   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                                         
	 3233       Disabled           Disabled          23291      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::1aa   10.0.1.199   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=pod-to-a                                                                                         
	 3369       Disabled           Enabled           41738      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::1c2   10.0.1.249   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                                          
	 3735       Disabled           Disabled          46582      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::1bb   10.0.1.146   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                   
	                                                            k8s:name=echo-a                                                                                           
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-m9r4b -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                       
	 289        Disabled           Disabled          507        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::e    10.0.0.176   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                                        
	 347        Disabled           Disabled          43016      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::cb   10.0.0.238   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                                        
	 583        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                           ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                    
	                                                            k8s:node-role.kubernetes.io/master                                                                           
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                  
	                                                            reserved:host                                                                                                
	 602        Disabled           Enabled           9881       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::6d   10.0.0.180   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                                           
	 1058       Disabled           Enabled           9407       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::90   10.0.0.79    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-a-denied-cnp                                                                                 
	 1785       Disabled           Disabled          31231      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::91   10.0.0.204   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                                       
	 2308       Disabled           Enabled           7377       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::b6   10.0.0.175   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                                             
	 2420       Disabled           Enabled           51420      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::cf   10.0.0.85    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                                               
	 2537       Disabled           Disabled          13528      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::e9   10.0.0.118   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-b-multi-node-headless                                                                        
	 2890       Disabled           Disabled          36010      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::43   10.0.0.140   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                              
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                      
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                                            
	 3350       Disabled           Disabled          33637      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::67   10.0.0.20    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                         
	 3624       Disabled           Disabled          4          reserved:health                                                              fd02::d4   10.0.0.194   ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
14:31:12 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
14:31:58 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|bbc50ac3_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9//2503/artifact/bbc50ac3_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9//2503/artifact/test_results_Cilium-PR-K8s-1.21-kernel-4.9_2503_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/2503/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
5 tasks
@maintainer-s-little-helper
Copy link
Author

PR #24604 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-28T13:54:53.607287363Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.196 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-28T13:54:54.110475368Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.127 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-03-28T13:54:53.607287363Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.196 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-03-28T13:54:54.110475368Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.127 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 461
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
+ true
100:\tfrom all lookup local
+ '[' false = true ']'
++ awk '{print $2}'
Cilium pods: [cilium-2njmw cilium-lqs5s]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-b-multi-node-clusterip-7b5854d46c-ztqdc                         
pod-to-b-multi-node-nodeport-84fdc88d9f-q7zm7                          
pod-to-a-multi-node-proxy-egress-policy-5564db6bb7-hzqgg               
pod-to-b-intra-node-nodeport-5c8cd69ff5-tmlc5                          
pod-to-a-allowed-cnp-5ff69578c5-swkqr                                  
pod-to-a-denied-cnp-64d7765ddf-8tbwb                                   
pod-to-b-intra-node-hostport-54d55874b5-k6tqc                          
echo-a-56f9849b6b-5g4kz                                                
echo-b-5898f8c6c4-thn2b                                                
pod-to-b-multi-node-hostport-5cbc88fbcc-zt7g4                          
pod-to-c-intra-node-proxy-ingress-policy-8467c7ccf4-qqnzr              
pod-to-c-multi-node-proxy-ingress-policy-9c4bb99d5-n58lw               
pod-to-external-fqdn-allow-google-cnp-6d8b6d59fc-4mjv2                 
echo-c-6b4f857f7-8tvcj                                                 
pod-to-a-9dc6d768c-ngczq                                               
pod-to-c-intra-node-proxy-to-proxy-policy-7497c557df-xrn7x             
pod-to-c-multi-node-proxy-to-proxy-policy-6fff5cc8c4-2hkv6             
pod-to-external-1111-6b96d6cc7-jcxbz                                   
coredns-7c74c644b-6g2tx                                                
pod-to-a-intra-node-proxy-egress-policy-55bbdc8944-nch7j               
pod-to-b-multi-node-headless-77b698d8f5-n9lml                          
Cilium agent 'cilium-2njmw': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 58 Failed 0
Cilium agent 'cilium-lqs5s': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0

Standard Error

Click to show. 
13:54:30 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
13:54:30 STEP: Ensuring the namespace kube-system exists
13:54:30 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
13:54:30 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
13:54:30 STEP: Installing Cilium
13:54:31 STEP: Waiting for Cilium to become ready
13:55:17 STEP: Restarting unmanaged pods coredns-7c74c644b-ztrs7 in namespace kube-system
13:55:31 STEP: Validating if Kubernetes DNS is deployed
13:55:31 STEP: Checking if deployment is ready
13:55:31 STEP: Checking if kube-dns service is plumbed correctly
13:55:31 STEP: Checking if pods have identity
13:55:31 STEP: Checking if DNS can resolve
13:55:35 STEP: Kubernetes DNS is up and operational
13:55:35 STEP: Validating Cilium Installation
13:55:35 STEP: Performing Cilium controllers preflight check
13:55:35 STEP: Performing Cilium health check
13:55:35 STEP: Checking whether host EP regenerated
13:55:35 STEP: Performing Cilium status preflight check
13:55:42 STEP: Performing Cilium service preflight check
13:55:42 STEP: Performing K8s service preflight check
13:55:48 STEP: Waiting for cilium-operator to be ready
13:55:48 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
13:55:48 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
13:55:49 STEP: Making sure all endpoints are in ready state
13:55:52 STEP: WaitforPods(namespace="default", filter="")
13:56:34 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-03-28T13:56:34Z====
13:56:34 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-28T13:54:53.607287363Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.196 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-28T13:54:54.110475368Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.127 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
13:56:35 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE    IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-d69c97b9b-w2vh4                                      0/1     Running   0          26m    10.0.0.71       k8s1   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-r5mgl                                  1/1     Running   0          26m    10.0.0.213      k8s1   <none>           <none>
	 default             echo-a-56f9849b6b-5g4kz                                      1/1     Running   0          49s    10.0.0.94       k8s2   <none>           <none>
	 default             echo-b-5898f8c6c4-thn2b                                      1/1     Running   0          49s    10.0.0.174      k8s2   <none>           <none>
	 default             echo-b-host-7b949c7b9f-lkkx5                                 1/1     Running   0          49s    192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-6b4f857f7-8tvcj                                       1/1     Running   0          49s    10.0.0.190      k8s2   <none>           <none>
	 default             echo-c-host-6ccff4d6bb-k4vng                                 1/1     Running   0          49s    192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6c4db976b5-4xzfk              1/1     Running   0          46s    192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-68d54bb4b7-swrhl               1/1     Running   0          46s    192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-9dc6d768c-ngczq                                     1/1     Running   0          48s    10.0.0.74       k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-5ff69578c5-swkqr                        1/1     Running   0          48s    10.0.0.56       k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-64d7765ddf-8tbwb                         1/1     Running   0          48s    10.0.0.70       k8s2   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-55bbdc8944-nch7j     2/2     Running   0          48s    10.0.0.11       k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5564db6bb7-hzqgg     2/2     Running   0          47s    10.0.1.217      k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-54d55874b5-k6tqc                1/1     Running   0          45s    10.0.0.105      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-5c8cd69ff5-tmlc5                1/1     Running   0          45s    10.0.0.197      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-7b5854d46c-ztqdc               1/1     Running   0          46s    10.0.1.66       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-77b698d8f5-n9lml                1/1     Running   0          46s    10.0.1.117      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-5cbc88fbcc-zt7g4                1/1     Running   0          46s    10.0.1.244      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-84fdc88d9f-q7zm7                1/1     Running   0          45s    10.0.1.99       k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-8467c7ccf4-qqnzr    2/2     Running   0          47s    10.0.0.156      k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-7497c557df-xrn7x   2/2     Running   0          47s    10.0.0.180      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-9c4bb99d5-n58lw     2/2     Running   0          47s    10.0.1.108      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-6fff5cc8c4-2hkv6   2/2     Running   0          47s    10.0.1.197      k8s1   <none>           <none>
	 default             pod-to-external-1111-6b96d6cc7-jcxbz                         1/1     Running   0          48s    10.0.0.60       k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-6d8b6d59fc-4mjv2       1/1     Running   0          48s    10.0.0.57       k8s2   <none>           <none>
	 kube-system         cilium-2njmw                                                 1/1     Running   1          2m9s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-lqs5s                                                 1/1     Running   1          2m9s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-6587f7f9fd-7vtqt                             1/1     Running   0          2m9s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-6587f7f9fd-cxbwb                             1/1     Running   0          2m9s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-7c74c644b-6g2tx                                      1/1     Running   0          83s    10.0.1.45       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   1          34m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   1          34m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   2          34m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-ppzz2                                             1/1     Running   0          34m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-vjwdr                                             1/1     Running   0          27m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   2          34m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-68lmc                                           1/1     Running   0          26m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-79grb                                           1/1     Running   0          26m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-qflmr                                         1/1     Running   0          27m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-xhx7k                                         1/1     Running   0          27m    192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-2njmw cilium-lqs5s]
cmd: kubectl exec -n kube-system cilium-2njmw -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 195        Disabled           Disabled          57246      k8s:io.cilium.k8s.policy.cluster=default             fd02::108   10.0.1.117   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-headless                                                 
	 468        Disabled           Disabled          33865      k8s:io.cilium.k8s.policy.cluster=default             fd02::12f   10.0.1.244   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                 
	 497        Disabled           Disabled          24028      k8s:io.cilium.k8s.policy.cluster=default             fd02::1db   10.0.1.99    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                 
	 583        Disabled           Disabled          59753      k8s:io.cilium.k8s.policy.cluster=default             fd02::107   10.0.1.66    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                
	 1367       Disabled           Disabled          4          reserved:health                                      fd02::10e   10.0.1.98    ready   
	 1427       Disabled           Disabled          14251      k8s:io.cilium.k8s.policy.cluster=default             fd02::106   10.0.1.45    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                       
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                           
	                                                            k8s:k8s-app=kube-dns                                                                  
	 1602       Disabled           Disabled          13261      k8s:io.cilium.k8s.policy.cluster=default             fd02::1eb   10.0.1.108   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                     
	 2024       Disabled           Enabled           37040      k8s:io.cilium.k8s.policy.cluster=default             fd02::13f   10.0.1.197   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                    
	 2368       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                    ready   
	                                                            k8s:node-role.kubernetes.io/master                                                    
	                                                            reserved:host                                                                         
	 3443       Disabled           Enabled           49488      k8s:io.cilium.k8s.policy.cluster=default             fd02::189   10.0.1.217   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                      
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-lqs5s -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 104        Disabled           Disabled          16016      k8s:io.cilium.k8s.policy.cluster=default             fd02::39   10.0.0.94    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 142        Disabled           Enabled           55820      k8s:io.cilium.k8s.policy.cluster=default             fd02::9a   10.0.0.11    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 776        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 1052       Disabled           Enabled           29555      k8s:io.cilium.k8s.policy.cluster=default             fd02::b1   10.0.0.180   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 1217       Enabled            Disabled          40376      k8s:io.cilium.k8s.policy.cluster=default             fd02::3a   10.0.0.190   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 1419       Disabled           Disabled          12743      k8s:io.cilium.k8s.policy.cluster=default             fd02::e    10.0.0.174   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 1428       Disabled           Disabled          22371      k8s:io.cilium.k8s.policy.cluster=default             fd02::d7   10.0.0.60    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 1704       Disabled           Disabled          10935      k8s:io.cilium.k8s.policy.cluster=default             fd02::b4   10.0.0.197   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 1800       Disabled           Enabled           53491      k8s:io.cilium.k8s.policy.cluster=default             fd02::75   10.0.0.56    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-allowed-cnp                                                        
	 2320       Disabled           Enabled           1365       k8s:io.cilium.k8s.policy.cluster=default             fd02::e6   10.0.0.57    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                       
	 2393       Disabled           Disabled          7682       k8s:io.cilium.k8s.policy.cluster=default             fd02::6c   10.0.0.105   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 2469       Disabled           Disabled          16558      k8s:io.cilium.k8s.policy.cluster=default             fd02::f8   10.0.0.74    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a                                                                    
	 2975       Disabled           Disabled          43345      k8s:io.cilium.k8s.policy.cluster=default             fd02::a2   10.0.0.156   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 3099       Disabled           Disabled          4          reserved:health                                      fd02::b6   10.0.0.248   ready   
	 3814       Disabled           Enabled           23028      k8s:io.cilium.k8s.policy.cluster=default             fd02::6e   10.0.0.70    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-denied-cnp                                                         
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
13:56:51 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
13:57:37 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|1d8b7055_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9//2733/artifact/1d8b7055_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9//2733/artifact/test_results_Cilium-PR-K8s-1.19-kernel-4.9_2733_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9/2733/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper
Copy link
Author

PR #24604 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-28T18:41:20.406977962Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.162 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-28T18:41:18.459337100Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.14 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-03-28T18:41:20.406977962Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.162 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-03-28T18:41:18.459337100Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.14 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 4
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 469
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
100:\tfrom all lookup local
+ true
+ '[' false = true ']'
10:\tfrom all fwmark 0xa00/0xf00 lookup 2005
Cilium pods: [cilium-7lzft cilium-pprpl]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
echo-a-6954f488f5-njpwp                                                
pod-to-a-57c7db48fd-bz6ph                                              
pod-to-external-1111-694c4c87dd-mjr6p                                  
pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-2kgth                 
coredns-5b7c49d4d8-5gvvm                                               
pod-to-a-allowed-cnp-548f548798-266xt                                  
pod-to-a-denied-cnp-64b85f46b8-szrnz                                   
pod-to-a-multi-node-proxy-egress-policy-5466dc57db-jqpf6               
pod-to-b-intra-node-hostport-5c58cb8758-nvfkb                          
pod-to-b-multi-node-hostport-869554778-9m8fn                           
pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-z6s5n             
echo-b-6c864d75d7-wvjc9                                                
echo-c-688dcdfd44-2h8gc                                                
pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-ks54j               
pod-to-b-multi-node-nodeport-658b99d8-lx5c8                            
pod-to-b-intra-node-nodeport-7b59546c64-gj8q2                          
pod-to-b-multi-node-clusterip-6c74fdb4b8-dsxw6                         
pod-to-b-multi-node-headless-595bf6dfb-9x2v8                           
pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-qwwsj              
pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-hrsd6              
pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-sq89g             
Cilium agent 'cilium-7lzft': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 70 Failed 0
Cilium agent 'cilium-pprpl': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0

Standard Error

Click to show. 
18:40:57 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
18:40:57 STEP: Ensuring the namespace kube-system exists
18:40:57 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
18:40:57 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
18:40:57 STEP: Installing Cilium
18:40:58 STEP: Waiting for Cilium to become ready
18:42:14 STEP: Restarting unmanaged pods coredns-5b7c49d4d8-2t9l7 in namespace kube-system
18:42:28 STEP: Validating if Kubernetes DNS is deployed
18:42:28 STEP: Checking if deployment is ready
18:42:28 STEP: Checking if pods have identity
18:42:28 STEP: Checking if kube-dns service is plumbed correctly
18:42:28 STEP: Checking if DNS can resolve
18:42:31 STEP: Kubernetes DNS is up and operational
18:42:31 STEP: Validating Cilium Installation
18:42:31 STEP: Performing Cilium controllers preflight check
18:42:31 STEP: Performing Cilium health check
18:42:31 STEP: Performing Cilium status preflight check
18:42:31 STEP: Checking whether host EP regenerated
18:42:39 STEP: Performing Cilium service preflight check
18:42:39 STEP: Performing K8s service preflight check
18:42:45 STEP: Waiting for cilium-operator to be ready
18:42:45 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
18:42:45 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
18:42:45 STEP: Making sure all endpoints are in ready state
18:42:48 STEP: WaitforPods(namespace="default", filter="")
18:43:06 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-03-28T18:43:06Z====
18:43:06 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-28T18:41:20.406977962Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.162 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-28T18:41:18.459337100Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.14 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
18:43:07 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-7fd557d749-xl8bw                                     0/1     Running   0          76m     10.0.1.153      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-d87f8f984-mf4kn                                   1/1     Running   0          76m     10.0.1.152      k8s2   <none>           <none>
	 default             echo-a-6954f488f5-njpwp                                      1/1     Running   0          24s     10.0.1.123      k8s2   <none>           <none>
	 default             echo-b-6c864d75d7-wvjc9                                      1/1     Running   0          24s     10.0.1.6        k8s2   <none>           <none>
	 default             echo-b-host-fdb854bd-7g2jm                                   1/1     Running   0          24s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-688dcdfd44-2h8gc                                      1/1     Running   0          24s     10.0.1.231      k8s2   <none>           <none>
	 default             echo-c-host-5d6b79658c-mf7dt                                 1/1     Running   0          24s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6868bbd4f8-k9pp8              1/1     Running   0          22s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-69c9465576-jkgjx               1/1     Running   0          21s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-57c7db48fd-bz6ph                                    1/1     Running   0          24s     10.0.0.114      k8s1   <none>           <none>
	 default             pod-to-a-allowed-cnp-548f548798-266xt                        1/1     Running   0          24s     10.0.1.43       k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-64b85f46b8-szrnz                         1/1     Running   0          24s     10.0.0.85       k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-ks54j     2/2     Running   0          23s     10.0.1.129      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5466dc57db-jqpf6     2/2     Running   0          23s     10.0.0.195      k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-5c58cb8758-nvfkb                1/1     Running   0          21s     10.0.1.37       k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-7b59546c64-gj8q2                1/1     Running   0          20s     10.0.1.115      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-6c74fdb4b8-dsxw6               1/1     Running   0          22s     10.0.0.33       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-595bf6dfb-9x2v8                 1/1     Running   0          22s     10.0.0.213      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-869554778-9m8fn                 1/1     Running   0          21s     10.0.0.118      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-658b99d8-lx5c8                  1/1     Running   0          21s     10.0.0.23       k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-qwwsj    2/2     Running   0          23s     10.0.1.19       k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-z6s5n   2/2     Running   0          23s     10.0.1.104      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-hrsd6    2/2     Running   0          23s     10.0.0.205      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-sq89g   2/2     Running   0          22s     10.0.0.143      k8s1   <none>           <none>
	 default             pod-to-external-1111-694c4c87dd-mjr6p                        1/1     Running   0          24s     10.0.1.28       k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-2kgth       1/1     Running   0          23s     10.0.0.45       k8s1   <none>           <none>
	 kube-system         cilium-7lzft                                                 1/1     Running   1          2m14s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-6cd9c6c987-7hk6f                             1/1     Running   0          2m14s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-6cd9c6c987-vcblb                             1/1     Running   0          2m14s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-pprpl                                                 1/1     Running   1          2m14s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-5b7c49d4d8-5gvvm                                     1/1     Running   0          58s     10.0.1.79       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          85m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   1          85m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   3          85m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-fnmj8                                             1/1     Running   0          85m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-z46d6                                             1/1     Running   0          77m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   3          85m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-xjc7v                                           1/1     Running   0          76m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-zspz9                                           1/1     Running   0          76m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-9ntpz                                         1/1     Running   0          77m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-bdsxm                                         1/1     Running   0          77m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-7lzft cilium-pprpl]
cmd: kubectl exec -n kube-system cilium-7lzft -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 113        Disabled           Disabled          9245       k8s:io.cilium.k8s.policy.cluster=default             fd02::16e   10.0.1.28    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-1111                                                         
	 401        Disabled           Enabled           8003       k8s:io.cilium.k8s.policy.cluster=default             fd02::1b2   10.0.1.129   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                      
	 927        Disabled           Disabled          32528      k8s:io.cilium.k8s.policy.cluster=default             fd02::1fc   10.0.1.123   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-a                                                                       
	 1244       Disabled           Disabled          28484      k8s:io.cilium.k8s.policy.cluster=default             fd02::104   10.0.1.19    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                     
	 1648       Disabled           Enabled           10837      k8s:io.cilium.k8s.policy.cluster=default             fd02::1b0   10.0.1.104   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                    
	 1681       Disabled           Enabled           18834      k8s:io.cilium.k8s.policy.cluster=default             fd02::1c3   10.0.1.43    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-allowed-cnp                                                         
	 1899       Disabled           Disabled          4          reserved:health                                      fd02::18e   10.0.1.110   ready   
	 1916       Disabled           Disabled          30475      k8s:io.cilium.k8s.policy.cluster=default             fd02::181   10.0.1.37    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                 
	 2026       Disabled           Disabled          14732      k8s:io.cilium.k8s.policy.cluster=default             fd02::1a4   10.0.1.79    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                       
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                           
	                                                            k8s:k8s-app=kube-dns                                                                  
	 2219       Disabled           Disabled          25309      k8s:io.cilium.k8s.policy.cluster=default             fd02::11b   10.0.1.6     ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-b                                                                       
	 2764       Disabled           Disabled          45618      k8s:io.cilium.k8s.policy.cluster=default             fd02::113   10.0.1.115   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                 
	 3011       Enabled            Disabled          23707      k8s:io.cilium.k8s.policy.cluster=default             fd02::157   10.0.1.231   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-c                                                                       
	 3342       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                    ready   
	                                                            reserved:host                                                                         
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-pprpl -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 11         Disabled           Disabled          37005      k8s:io.cilium.k8s.policy.cluster=default             fd02::eb   10.0.0.118   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                
	 53         Disabled           Enabled           32923      k8s:io.cilium.k8s.policy.cluster=default             fd02::48   10.0.0.195   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                     
	 178        Disabled           Disabled          33341      k8s:io.cilium.k8s.policy.cluster=default             fd02::23   10.0.0.33    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                               
	 289        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                   ready   
	                                                            k8s:node-role.kubernetes.io/master                                                   
	                                                            reserved:host                                                                        
	 326        Disabled           Disabled          20219      k8s:io.cilium.k8s.policy.cluster=default             fd02::8b   10.0.0.114   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a                                                                    
	 849        Disabled           Enabled           4306       k8s:io.cilium.k8s.policy.cluster=default             fd02::f8   10.0.0.45    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                       
	 921        Disabled           Disabled          47935      k8s:io.cilium.k8s.policy.cluster=default             fd02::7a   10.0.0.213   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-headless                                                
	 1018       Disabled           Disabled          10427      k8s:io.cilium.k8s.policy.cluster=default             fd02::31   10.0.0.205   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                    
	 1021       Disabled           Enabled           29414      k8s:io.cilium.k8s.policy.cluster=default             fd02::fb   10.0.0.85    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-denied-cnp                                                         
	 1292       Disabled           Enabled           25121      k8s:io.cilium.k8s.policy.cluster=default             fd02::47   10.0.0.143   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                   
	 1726       Disabled           Disabled          4          reserved:health                                      fd02::3b   10.0.0.174   ready   
	 1879       Disabled           Disabled          9410       k8s:io.cilium.k8s.policy.cluster=default             fd02::b3   10.0.0.23    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
18:43:23 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
18:44:10 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|24390afe_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1226/artifact/24390afe_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1226/artifact/9a418557_K8sChaosTest_Restart_with_long_lived_connections_TCP_connection_is_not_dropped_when_cilium_restarts.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1226/artifact/ad64b64d_K8sChaosTest_Restart_with_long_lived_connections_L3-L4_policies_still_work_while_Cilium_is_restarted.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1226/artifact/bad62a0e_K8sChaosTest_Connectivity_demo_application_Endpoint_can_still_connect_while_Cilium_is_not_running.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1226/artifact/test_results_Cilium-PR-K8s-1.17-kernel-4.9_1226_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9/1226/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper
Copy link
Author

PR #24604 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-29T10:41:41.201609275Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.108 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-29T10:41:41.807730804Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.25 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-03-29T10:41:41.201609275Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.108 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-03-29T10:41:41.807730804Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.25 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 466
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
100:\tfrom all lookup local
+ true
+ '[' false = true ']'
++ awk '{print $2}'
10:\tfrom all fwmark 0xa00/0xf00 lookup 2005
Cilium pods: [cilium-8vq7h cilium-jtd95]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-f9ssb             
pod-to-external-1111-694c4c87dd-26thb                                  
pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-84q7m                 
coredns-5b7c49d4d8-mz6hz                                               
echo-c-688dcdfd44-q95d8                                                
pod-to-a-allowed-cnp-548f548798-hj8gp                                  
pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-46nxt              
pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-7zn8t              
pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-9zcgt             
echo-b-6c864d75d7-x8hh7                                                
pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-lnv5d               
pod-to-b-intra-node-nodeport-7b59546c64-qlblh                          
pod-to-b-multi-node-headless-595bf6dfb-4k7lq                           
pod-to-a-multi-node-proxy-egress-policy-5466dc57db-r5cm6               
pod-to-b-multi-node-hostport-869554778-q9w7t                           
pod-to-b-multi-node-nodeport-658b99d8-t4vjg                            
pod-to-b-multi-node-clusterip-6c74fdb4b8-fx4cs                         
echo-a-6954f488f5-7jkg2                                                
pod-to-a-57c7db48fd-np9zf                                              
pod-to-a-denied-cnp-64b85f46b8-6ct7j                                   
pod-to-b-intra-node-hostport-5c58cb8758-p4fgh                          
Cilium agent 'cilium-8vq7h': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0
Cilium agent 'cilium-jtd95': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 70 Failed 0

Standard Error

Click to show. 
10:41:18 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
10:41:18 STEP: Ensuring the namespace kube-system exists
10:41:19 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
10:41:19 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
10:41:19 STEP: Installing Cilium
10:41:19 STEP: Waiting for Cilium to become ready
10:42:10 STEP: Restarting unmanaged pods coredns-5b7c49d4d8-bj4lc in namespace kube-system
10:42:28 STEP: Validating if Kubernetes DNS is deployed
10:42:28 STEP: Checking if deployment is ready
10:42:28 STEP: Checking if kube-dns service is plumbed correctly
10:42:28 STEP: Checking if pods have identity
10:42:28 STEP: Checking if DNS can resolve
10:42:31 STEP: Kubernetes DNS is up and operational
10:42:31 STEP: Validating Cilium Installation
10:42:31 STEP: Performing Cilium controllers preflight check
10:42:31 STEP: Performing Cilium status preflight check
10:42:31 STEP: Performing Cilium health check
10:42:31 STEP: Checking whether host EP regenerated
10:42:39 STEP: Performing Cilium service preflight check
10:42:39 STEP: Performing K8s service preflight check
10:42:45 STEP: Waiting for cilium-operator to be ready
10:42:45 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
10:42:45 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
10:42:45 STEP: Making sure all endpoints are in ready state
10:42:48 STEP: WaitforPods(namespace="default", filter="")
10:43:09 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-03-29T10:43:09Z====
10:43:09 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-03-29T10:41:41.201609275Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.108 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-03-29T10:41:41.807730804Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.25 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
10:43:10 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE    IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-7fd557d749-l44xd                                     0/1     Running   0          88m    10.0.1.120      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-d87f8f984-4r4bg                                   1/1     Running   0          88m    10.0.1.151      k8s2   <none>           <none>
	 default             echo-a-6954f488f5-7jkg2                                      1/1     Running   0          27s    10.0.0.26       k8s2   <none>           <none>
	 default             echo-b-6c864d75d7-x8hh7                                      1/1     Running   0          27s    10.0.0.208      k8s2   <none>           <none>
	 default             echo-b-host-fdb854bd-p9jdn                                   1/1     Running   0          27s    192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-688dcdfd44-q95d8                                      1/1     Running   0          27s    10.0.0.12       k8s2   <none>           <none>
	 default             echo-c-host-5d6b79658c-6nmnt                                 1/1     Running   0          27s    192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6868bbd4f8-9msjg              1/1     Running   0          24s    192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-69c9465576-qp8hx               1/1     Running   0          24s    192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-57c7db48fd-np9zf                                    1/1     Running   0          27s    10.0.1.227      k8s1   <none>           <none>
	 default             pod-to-a-allowed-cnp-548f548798-hj8gp                        1/1     Running   0          27s    10.0.1.11       k8s1   <none>           <none>
	 default             pod-to-a-denied-cnp-64b85f46b8-6ct7j                         1/1     Running   0          27s    10.0.1.238      k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-lnv5d     2/2     Running   0          26s    10.0.0.151      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5466dc57db-r5cm6     2/2     Running   0          26s    10.0.1.86       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-5c58cb8758-p4fgh                1/1     Running   0          24s    10.0.0.102      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-7b59546c64-qlblh                1/1     Running   0          24s    10.0.0.17       k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-6c74fdb4b8-fx4cs               1/1     Running   0          25s    10.0.1.171      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-595bf6dfb-4k7lq                 1/1     Running   0          25s    10.0.1.3        k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-869554778-q9w7t                 1/1     Running   0          24s    10.0.1.228      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-658b99d8-t4vjg                  1/1     Running   0          24s    10.0.1.10       k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-46nxt    2/2     Running   0          26s    10.0.0.54       k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-9zcgt   2/2     Running   0          25s    10.0.0.172      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-7zn8t    2/2     Running   0          25s    10.0.1.7        k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-f9ssb   2/2     Running   0          25s    10.0.1.87       k8s1   <none>           <none>
	 default             pod-to-external-1111-694c4c87dd-26thb                        1/1     Running   0          27s    10.0.0.16       k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-84q7m       1/1     Running   0          26s    10.0.0.234      k8s2   <none>           <none>
	 kube-system         cilium-8vq7h                                                 1/1     Running   1          116s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-jtd95                                                 1/1     Running   1          116s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-6cd9c6c987-2lvbl                             1/1     Running   0          116s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-6cd9c6c987-rsnbh                             1/1     Running   0          116s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-5b7c49d4d8-mz6hz                                     1/1     Running   0          65s    10.0.0.128      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   1          96m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   1          96m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   4          96m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-5tvdl                                             1/1     Running   0          96m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-grdh2                                             1/1     Running   0          89m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   4          96m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-swsf2                                           1/1     Running   0          88m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-zf2kf                                           1/1     Running   0          88m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-2rbxc                                         1/1     Running   0          89m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-zcgxs                                         1/1     Running   0          89m    192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-8vq7h cilium-jtd95]
cmd: kubectl exec -n kube-system cilium-8vq7h -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 229        Disabled           Disabled          8663       k8s:io.cilium.k8s.policy.cluster=default             fd02::169   10.0.1.171   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                
	 482        Disabled           Enabled           2123       k8s:io.cilium.k8s.policy.cluster=default             fd02::14d   10.0.1.11    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-allowed-cnp                                                         
	 584        Disabled           Enabled           8064       k8s:io.cilium.k8s.policy.cluster=default             fd02::109   10.0.1.86    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                      
	 982        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                    ready   
	                                                            k8s:node-role.kubernetes.io/master                                                    
	                                                            reserved:host                                                                         
	 996        Disabled           Disabled          4          reserved:health                                      fd02::16f   10.0.1.97    ready   
	 1252       Disabled           Enabled           830        k8s:io.cilium.k8s.policy.cluster=default             fd02::1ac   10.0.1.238   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-denied-cnp                                                          
	 1625       Disabled           Disabled          19869      k8s:io.cilium.k8s.policy.cluster=default             fd02::18b   10.0.1.228   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                 
	 1656       Disabled           Disabled          38510      k8s:io.cilium.k8s.policy.cluster=default             fd02::165   10.0.1.227   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a                                                                     
	 1744       Disabled           Enabled           4575       k8s:io.cilium.k8s.policy.cluster=default             fd02::155   10.0.1.87    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                    
	 1887       Disabled           Disabled          7308       k8s:io.cilium.k8s.policy.cluster=default             fd02::112   10.0.1.3     ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-headless                                                 
	 2025       Disabled           Disabled          4846       k8s:io.cilium.k8s.policy.cluster=default             fd02::1e1   10.0.1.7     ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                     
	 2034       Disabled           Disabled          21004      k8s:io.cilium.k8s.policy.cluster=default             fd02::17c   10.0.1.10    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                 
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-jtd95 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 396        Disabled           Disabled          14450      k8s:io.cilium.k8s.policy.cluster=default             fd02::e5   10.0.0.16    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 793        Enabled            Disabled          24842      k8s:io.cilium.k8s.policy.cluster=default             fd02::5b   10.0.0.12    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 981        Disabled           Disabled          23269      k8s:io.cilium.k8s.policy.cluster=default             fd02::10   10.0.0.17    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 1011       Disabled           Disabled          1647       k8s:io.cilium.k8s.policy.cluster=default             fd02::95   10.0.0.54    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 1057       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 1303       Disabled           Disabled          30941      k8s:io.cilium.k8s.policy.cluster=default             fd02::22   10.0.0.26    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 1444       Disabled           Enabled           45419      k8s:io.cilium.k8s.policy.cluster=default             fd02::6d   10.0.0.234   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                       
	 1777       Disabled           Disabled          5108       k8s:io.cilium.k8s.policy.cluster=default             fd02::56   10.0.0.102   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 1897       Disabled           Enabled           15038      k8s:io.cilium.k8s.policy.cluster=default             fd02::84   10.0.0.172   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 2241       Disabled           Enabled           32183      k8s:io.cilium.k8s.policy.cluster=default             fd02::e1   10.0.0.151   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 2860       Disabled           Disabled          57719      k8s:io.cilium.k8s.policy.cluster=default             fd02::fc   10.0.0.208   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 2997       Disabled           Disabled          4          reserved:health                                      fd02::60   10.0.0.158   ready   
	 3248       Disabled           Disabled          5889       k8s:io.cilium.k8s.policy.cluster=default             fd02::31   10.0.0.128   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                          
	                                                            k8s:k8s-app=kube-dns                                                                 
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
10:43:27 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
10:44:12 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|88400b1a_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1230/artifact/88400b1a_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1230/artifact/test_results_Cilium-PR-K8s-1.17-kernel-4.9_1230_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9/1230/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper
Copy link
Author

PR #24710 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-04T14:35:42.850799817Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.127 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-04T14:35:42.786213623Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.142 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-04-04T14:35:42.850799817Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.127 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-04-04T14:35:42.786213623Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.142 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 4
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 472
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
100:\tfrom all lookup local
+ true
Waiting for k8s node information
+ '[' false = true ']'
Cilium pods: [cilium-9jl99 cilium-qdfvn]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-c-intra-node-proxy-ingress-policy-fccbcc9bf-mpml7               
pod-to-c-intra-node-proxy-to-proxy-policy-77b98c977d-kc7r7             
pod-to-external-fqdn-allow-google-cnp-7d5cc887cd-gr4f5                 
echo-b-5c5d7c49b5-zrbwm                                                
pod-to-a-denied-cnp-5f9b6b964b-lhgmd                                   
pod-to-a-intra-node-proxy-egress-policy-65f4654f99-r2scn               
pod-to-b-multi-node-hostport-6d54854b6d-sf6n8                          
pod-to-b-multi-node-clusterip-86dbd49cdb-sh929                         
pod-to-b-multi-node-headless-7bcf8566b-lk7pk                           
coredns-86c74c674b-zlrlx                                               
pod-to-b-intra-node-nodeport-79d9fc898b-k22b2                          
pod-to-b-multi-node-nodeport-8697db657d-gtvs2                          
pod-to-c-multi-node-proxy-ingress-policy-56889f84cd-5p9l2              
echo-a-78667c8fdf-fcvrz                                                
pod-to-a-69cc5b49b8-qfgvq                                              
pod-to-a-allowed-cnp-86b8c7bf44-6g8q8                                  
pod-to-a-multi-node-proxy-egress-policy-69b8569968-mddjx               
echo-c-78dc7fc59d-fxmhn                                                
pod-to-b-intra-node-hostport-d568f6bd4-ghxvl                           
pod-to-c-multi-node-proxy-to-proxy-policy-78647869d5-z8pkw             
pod-to-external-1111-f466d786b-dzttf                                   
Cilium agent 'cilium-9jl99': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0
Cilium agent 'cilium-qdfvn': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 58 Failed 0

Standard Error

Click to show. 
14:35:09 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
14:35:09 STEP: Ensuring the namespace kube-system exists
14:35:09 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
14:35:09 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
14:35:09 STEP: Installing Cilium
14:35:10 STEP: Waiting for Cilium to become ready
14:36:31 STEP: Restarting unmanaged pods coredns-86c74c674b-dsmcr in namespace kube-system
14:36:38 STEP: Validating if Kubernetes DNS is deployed
14:36:38 STEP: Checking if deployment is ready
14:36:38 STEP: Kubernetes DNS is not ready: only 0 of 1 replicas are available
14:36:38 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
14:36:39 STEP: Waiting for Kubernetes DNS to become operational
14:36:39 STEP: Checking if deployment is ready
14:36:39 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:40 STEP: Checking if deployment is ready
14:36:40 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:41 STEP: Checking if deployment is ready
14:36:41 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:42 STEP: Checking if deployment is ready
14:36:42 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:43 STEP: Checking if deployment is ready
14:36:43 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:44 STEP: Checking if deployment is ready
14:36:44 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:45 STEP: Checking if deployment is ready
14:36:45 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:46 STEP: Checking if deployment is ready
14:36:46 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:47 STEP: Checking if deployment is ready
14:36:47 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:48 STEP: Checking if deployment is ready
14:36:48 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:49 STEP: Checking if deployment is ready
14:36:49 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
14:36:50 STEP: Checking if deployment is ready
14:36:50 STEP: Checking if kube-dns service is plumbed correctly
14:36:50 STEP: Checking if DNS can resolve
14:36:50 STEP: Checking if pods have identity
14:36:53 STEP: Validating Cilium Installation
14:36:53 STEP: Performing Cilium controllers preflight check
14:36:53 STEP: Performing Cilium health check
14:36:53 STEP: Checking whether host EP regenerated
14:36:53 STEP: Performing Cilium status preflight check
14:37:01 STEP: Performing Cilium service preflight check
14:37:01 STEP: Performing K8s service preflight check
14:37:07 STEP: Waiting for cilium-operator to be ready
14:37:07 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
14:37:07 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
14:37:07 STEP: Making sure all endpoints are in ready state
14:37:10 STEP: WaitforPods(namespace="default", filter="")
14:37:38 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-04-04T14:37:38Z====
14:37:38 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-04T14:35:42.850799817Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.127 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-04T14:35:42.786213623Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.142 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
14:37:39 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-54dbdc987-sgdsv                                      0/1     Running   0          38m     10.0.0.121      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-6ff848df8b-8kmvl                                  1/1     Running   0          38m     10.0.0.202      k8s1   <none>           <none>
	 default             echo-a-78667c8fdf-fcvrz                                      1/1     Running   0          34s     10.0.0.15       k8s2   <none>           <none>
	 default             echo-b-5c5d7c49b5-zrbwm                                      1/1     Running   0          34s     10.0.0.219      k8s2   <none>           <none>
	 default             echo-b-host-7df9796db6-gzwkn                                 1/1     Running   0          34s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-78dc7fc59d-fxmhn                                      1/1     Running   0          34s     10.0.0.112      k8s2   <none>           <none>
	 default             echo-c-host-7779f885fc-jz6mw                                 1/1     Running   0          34s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6744fdbf6-kjbbz               1/1     Running   0          32s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-87b4cc8cd-vl92n                1/1     Running   0          31s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-69cc5b49b8-qfgvq                                    1/1     Running   0          34s     10.0.0.54       k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-86b8c7bf44-6g8q8                        1/1     Running   0          34s     10.0.0.101      k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-5f9b6b964b-lhgmd                         1/1     Running   0          34s     10.0.0.16       k8s2   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-65f4654f99-r2scn     2/2     Running   0          33s     10.0.0.77       k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-69b8569968-mddjx     2/2     Running   0          33s     10.0.1.29       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-d568f6bd4-ghxvl                 1/1     Running   0          31s     10.0.0.139      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-79d9fc898b-k22b2                1/1     Running   0          31s     10.0.0.167      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-86dbd49cdb-sh929               1/1     Running   0          32s     10.0.1.193      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-7bcf8566b-lk7pk                 1/1     Running   0          32s     10.0.1.248      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-6d54854b6d-sf6n8                1/1     Running   0          31s     10.0.1.92       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-8697db657d-gtvs2                1/1     Running   0          31s     10.0.1.171      k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-fccbcc9bf-mpml7     2/2     Running   0          33s     10.0.0.244      k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-77b98c977d-kc7r7   2/2     Running   0          32s     10.0.0.143      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-56889f84cd-5p9l2    2/2     Running   0          33s     10.0.1.162      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-78647869d5-z8pkw   2/2     Running   0          32s     10.0.1.105      k8s1   <none>           <none>
	 default             pod-to-external-1111-f466d786b-dzttf                         1/1     Running   0          34s     10.0.0.187      k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7d5cc887cd-gr4f5       1/1     Running   0          34s     10.0.1.46       k8s1   <none>           <none>
	 kube-system         cilium-9jl99                                                 1/1     Running   1          2m34s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-d5c77c59c-cmb74                              1/1     Running   0          2m34s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-d5c77c59c-zvfch                              1/1     Running   0          2m34s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-qdfvn                                                 1/1     Running   1          2m34s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-86c74c674b-zlrlx                                     1/1     Running   0          65s     10.0.0.179      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          47m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   2          47m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   3          47m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-wn8xt                                             1/1     Running   0          39m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-zvgc9                                             1/1     Running   0          47m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   2          47m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-98l8n                                           1/1     Running   0          38m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-lp8qm                                           1/1     Running   0          38m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-7kpz4                                         1/1     Running   0          39m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-r9qfn                                         1/1     Running   0          39m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-9jl99 cilium-qdfvn]
cmd: kubectl exec -n kube-system cilium-9jl99 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 32         Disabled           Enabled           1391       k8s:io.cilium.k8s.policy.cluster=default             fd02::f1   10.0.0.77    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 62         Enabled            Disabled          26602      k8s:io.cilium.k8s.policy.cluster=default             fd02::c0   10.0.0.112   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 501        Disabled           Disabled          4155       k8s:io.cilium.k8s.policy.cluster=default             fd02::bb   10.0.0.244   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 605        Disabled           Disabled          12593      k8s:io.cilium.k8s.policy.cluster=default             fd02::45   10.0.0.179   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                          
	                                                            k8s:k8s-app=kube-dns                                                                 
	 632        Disabled           Enabled           5762       k8s:io.cilium.k8s.policy.cluster=default             fd02::ae   10.0.0.16    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-denied-cnp                                                         
	 719        Disabled           Disabled          49536      k8s:io.cilium.k8s.policy.cluster=default             fd02::da   10.0.0.167   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 1175       Disabled           Enabled           44525      k8s:io.cilium.k8s.policy.cluster=default             fd02::76   10.0.0.143   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 1179       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 1598       Disabled           Disabled          27548      k8s:io.cilium.k8s.policy.cluster=default             fd02::44   10.0.0.187   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 2683       Disabled           Disabled          40117      k8s:io.cilium.k8s.policy.cluster=default             fd02::61   10.0.0.15    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 2748       Disabled           Disabled          1617       k8s:io.cilium.k8s.policy.cluster=default             fd02::54   10.0.0.139   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 2969       Disabled           Disabled          1203       k8s:io.cilium.k8s.policy.cluster=default             fd02::11   10.0.0.219   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 3021       Disabled           Enabled           2139       k8s:io.cilium.k8s.policy.cluster=default             fd02::af   10.0.0.101   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-allowed-cnp                                                        
	 3837       Disabled           Disabled          31827      k8s:io.cilium.k8s.policy.cluster=default             fd02::28   10.0.0.54    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a                                                                    
	 4085       Disabled           Disabled          4          reserved:health                                      fd02::d3   10.0.0.172   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-qdfvn -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 55         Disabled           Disabled          20874      k8s:io.cilium.k8s.policy.cluster=default             fd02::14a   10.0.1.193   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                
	 350        Disabled           Enabled           30180      k8s:io.cilium.k8s.policy.cluster=default             fd02::156   10.0.1.105   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                    
	 372        Disabled           Disabled          4          reserved:health                                      fd02::162   10.0.1.120   ready   
	 588        Disabled           Disabled          28359      k8s:io.cilium.k8s.policy.cluster=default             fd02::153   10.0.1.248   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-headless                                                 
	 686        Disabled           Enabled           12375      k8s:io.cilium.k8s.policy.cluster=default             fd02::11a   10.0.1.46    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                        
	 1004       Disabled           Disabled          11375      k8s:io.cilium.k8s.policy.cluster=default             fd02::1c7   10.0.1.92    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                 
	 2328       Disabled           Disabled          2026       k8s:io.cilium.k8s.policy.cluster=default             fd02::1c9   10.0.1.171   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                 
	 2340       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                    ready   
	                                                            k8s:node-role.kubernetes.io/master                                                    
	                                                            reserved:host                                                                         
	 2560       Disabled           Enabled           40318      k8s:io.cilium.k8s.policy.cluster=default             fd02::1d6   10.0.1.29    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                      
	 2978       Disabled           Disabled          6626       k8s:io.cilium.k8s.policy.cluster=default             fd02::1f5   10.0.1.162   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                     
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
14:37:56 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
14:38:40 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|df75be47_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9//2595/artifact/df75be47_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9//2595/artifact/test_results_Cilium-PR-K8s-1.18-kernel-4.9_2595_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9/2595/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Apr 4, 2023
Merged
1 task
@maintainer-s-little-helper
Copy link
Author

PR #24498 hit this flake with 93.21% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-08T01:48:11.719206881Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.6 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-08T01:48:12.741346913Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.235 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-04-08T01:48:11.719206881Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.6 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-04-08T01:48:12.741346913Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.235 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 468
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
+ true
100:\tfrom all lookup local
+ '[' false = true ']'
Key allocation attempt failed
Cilium pods: [cilium-626q8 cilium-6d2s7]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
echo-a-56f9849b6b-crrr4                                                
echo-c-6b4f857f7-fcccb                                                 
pod-to-a-allowed-cnp-5ff69578c5-f4ffj                                  
pod-to-a-multi-node-proxy-egress-policy-5564db6bb7-wfh5c               
pod-to-b-intra-node-nodeport-5c8cd69ff5-8wvj8                          
pod-to-b-multi-node-clusterip-7b5854d46c-z5c44                         
pod-to-c-intra-node-proxy-ingress-policy-8467c7ccf4-cmh5l              
coredns-7c74c644b-lm5bh                                                
pod-to-b-multi-node-hostport-5cbc88fbcc-6km4b                          
pod-to-c-multi-node-proxy-ingress-policy-9c4bb99d5-mc9v9               
echo-b-5898f8c6c4-xqrll                                                
pod-to-a-9dc6d768c-dwf6r                                               
pod-to-a-denied-cnp-64d7765ddf-47cn2                                   
pod-to-a-intra-node-proxy-egress-policy-55bbdc8944-8rzqf               
pod-to-c-intra-node-proxy-to-proxy-policy-7497c557df-fclgb             
pod-to-b-intra-node-hostport-54d55874b5-fz9q2                          
pod-to-b-multi-node-headless-77b698d8f5-stkkb                          
pod-to-b-multi-node-nodeport-84fdc88d9f-d9jvj                          
pod-to-c-multi-node-proxy-to-proxy-policy-6fff5cc8c4-kvkfv             
pod-to-external-1111-6b96d6cc7-df6bh                                   
pod-to-external-fqdn-allow-google-cnp-6d8b6d59fc-gd2qx                 
Cilium agent 'cilium-626q8': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0
Cilium agent 'cilium-6d2s7': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 70 Failed 0

Standard Error

Click to show. 
01:47:50 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
01:47:50 STEP: Ensuring the namespace kube-system exists
01:47:50 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
01:47:50 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
01:47:50 STEP: Installing Cilium
01:47:51 STEP: Waiting for Cilium to become ready
01:48:27 STEP: Restarting unmanaged pods coredns-7c74c644b-s9b5m in namespace kube-system
01:48:33 STEP: Validating if Kubernetes DNS is deployed
01:48:33 STEP: Checking if deployment is ready
01:48:33 STEP: Checking if kube-dns service is plumbed correctly
01:48:33 STEP: Checking if pods have identity
01:48:33 STEP: Checking if DNS can resolve
01:48:37 STEP: Kubernetes DNS is up and operational
01:48:37 STEP: Validating Cilium Installation
01:48:37 STEP: Performing Cilium controllers preflight check
01:48:37 STEP: Checking whether host EP regenerated
01:48:37 STEP: Performing Cilium health check
01:48:37 STEP: Performing Cilium status preflight check
01:48:45 STEP: Performing Cilium service preflight check
01:48:45 STEP: Performing K8s service preflight check
01:48:51 STEP: Waiting for cilium-operator to be ready
01:48:51 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
01:48:51 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
01:48:51 STEP: Making sure all endpoints are in ready state
01:48:55 STEP: WaitforPods(namespace="default", filter="")
01:49:15 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-04-08T01:49:15Z====
01:49:15 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-08T01:48:11.719206881Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.6 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-08T01:48:12.741346913Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.235 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
01:49:15 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE   IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-d69c97b9b-h6sp2                                      0/1     Running   0          86m   10.0.1.59       k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-lhxjh                                  1/1     Running   0          86m   10.0.1.122      k8s2   <none>           <none>
	 default             echo-a-56f9849b6b-crrr4                                      1/1     Running   0          27s   10.0.0.95       k8s2   <none>           <none>
	 default             echo-b-5898f8c6c4-xqrll                                      1/1     Running   0          27s   10.0.0.159      k8s2   <none>           <none>
	 default             echo-b-host-7b949c7b9f-2d64k                                 1/1     Running   0          27s   192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-6b4f857f7-fcccb                                       1/1     Running   0          27s   10.0.0.221      k8s2   <none>           <none>
	 default             echo-c-host-6ccff4d6bb-mbtrn                                 1/1     Running   0          27s   192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6c4db976b5-7sd4p              1/1     Running   0          24s   192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-68d54bb4b7-gxbpq               1/1     Running   0          24s   192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-9dc6d768c-dwf6r                                     1/1     Running   0          27s   10.0.1.122      k8s1   <none>           <none>
	 default             pod-to-a-allowed-cnp-5ff69578c5-f4ffj                        1/1     Running   0          26s   10.0.0.20       k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-64d7765ddf-47cn2                         1/1     Running   0          26s   10.0.1.185      k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-55bbdc8944-8rzqf     2/2     Running   0          26s   10.0.0.149      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5564db6bb7-wfh5c     2/2     Running   0          26s   10.0.1.33       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-54d55874b5-fz9q2                1/1     Running   0          23s   10.0.0.49       k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-5c8cd69ff5-8wvj8                1/1     Running   0          23s   10.0.0.135      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-7b5854d46c-z5c44               1/1     Running   0          25s   10.0.1.63       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-77b698d8f5-stkkb                1/1     Running   0          24s   10.0.1.74       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-5cbc88fbcc-6km4b                1/1     Running   0          24s   10.0.1.129      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-84fdc88d9f-d9jvj                1/1     Running   0          23s   10.0.1.128      k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-8467c7ccf4-cmh5l    2/2     Running   0          26s   10.0.0.179      k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-7497c557df-fclgb   2/2     Running   0          25s   10.0.0.59       k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-9c4bb99d5-mc9v9     2/2     Running   0          25s   10.0.1.212      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-6fff5cc8c4-kvkfv   2/2     Running   0          25s   10.0.1.90       k8s1   <none>           <none>
	 default             pod-to-external-1111-6b96d6cc7-df6bh                         1/1     Running   0          27s   10.0.0.55       k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-6d8b6d59fc-gd2qx       1/1     Running   0          26s   10.0.1.27       k8s1   <none>           <none>
	 kube-system         cilium-626q8                                                 1/1     Running   1          90s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-6d2s7                                                 1/1     Running   1          90s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-64fd479568-ljw78                             1/1     Running   0          90s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-64fd479568-lsdzf                             1/1     Running   0          90s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-7c74c644b-lm5bh                                      1/1     Running   0          54s   10.0.0.122      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   2          95m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   2          95m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   3          95m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-ssw7c                                             1/1     Running   0          87m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-zdvq9                                             1/1     Running   0          95m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   3          95m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-4srz8                                           1/1     Running   0          86m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-mf6b8                                           1/1     Running   0          86m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-4btjq                                         1/1     Running   0          87m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-6mqgz                                         1/1     Running   0          87m   192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-626q8 cilium-6d2s7]
cmd: kubectl exec -n kube-system cilium-626q8 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 64         Disabled           Disabled          50208      k8s:io.cilium.k8s.policy.cluster=default             fd02::195   10.0.1.74    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-headless                                                 
	 205        Disabled           Disabled          4          reserved:health                                      fd02::1a8   10.0.1.4     ready   
	 373        Disabled           Enabled           43054      k8s:io.cilium.k8s.policy.cluster=default             fd02::1bf   10.0.1.33    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                      
	 488        Disabled           Enabled           28206      k8s:io.cilium.k8s.policy.cluster=default             fd02::111   10.0.1.27    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                        
	 589        Disabled           Disabled          54044      k8s:io.cilium.k8s.policy.cluster=default             fd02::15f   10.0.1.212   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                     
	 740        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                    ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                             
	                                                            k8s:node-role.kubernetes.io/master                                                    
	                                                            reserved:host                                                                         
	 881        Disabled           Disabled          60784      k8s:io.cilium.k8s.policy.cluster=default             fd02::151   10.0.1.129   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                 
	 1495       Disabled           Disabled          32288      k8s:io.cilium.k8s.policy.cluster=default             fd02::14c   10.0.1.122   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a                                                                     
	 1921       Disabled           Disabled          10043      k8s:io.cilium.k8s.policy.cluster=default             fd02::10d   10.0.1.63    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                
	 1940       Disabled           Enabled           63799      k8s:io.cilium.k8s.policy.cluster=default             fd02::1c9   10.0.1.185   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-denied-cnp                                                          
	 2380       Disabled           Enabled           3800       k8s:io.cilium.k8s.policy.cluster=default             fd02::1dc   10.0.1.90    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                    
	 2556       Disabled           Disabled          2168       k8s:io.cilium.k8s.policy.cluster=default             fd02::1e3   10.0.1.128   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                 
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-6d2s7 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 975        Enabled            Disabled          38988      k8s:io.cilium.k8s.policy.cluster=default             fd02::85   10.0.0.221   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 1013       Disabled           Enabled           14351      k8s:io.cilium.k8s.policy.cluster=default             fd02::6    10.0.0.20    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-allowed-cnp                                                        
	 1121       Disabled           Enabled           58567      k8s:io.cilium.k8s.policy.cluster=default             fd02::a6   10.0.0.149   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 1230       Disabled           Disabled          56122      k8s:io.cilium.k8s.policy.cluster=default             fd02::4f   10.0.0.122   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                          
	                                                            k8s:k8s-app=kube-dns                                                                 
	 1260       Disabled           Disabled          28292      k8s:io.cilium.k8s.policy.cluster=default             fd02::37   10.0.0.55    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 1431       Disabled           Disabled          9892       k8s:io.cilium.k8s.policy.cluster=default             fd02::d7   10.0.0.159   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 1637       Disabled           Disabled          14460      k8s:io.cilium.k8s.policy.cluster=default             fd02::65   10.0.0.135   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 2161       Disabled           Disabled          44856      k8s:io.cilium.k8s.policy.cluster=default             fd02::2e   10.0.0.95    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 2220       Disabled           Disabled          22232      k8s:io.cilium.k8s.policy.cluster=default             fd02::d8   10.0.0.49    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 2642       Disabled           Enabled           43291      k8s:io.cilium.k8s.policy.cluster=default             fd02::2d   10.0.0.59    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 2998       Disabled           Disabled          4          reserved:health                                      fd02::66   10.0.0.131   ready   
	 3259       Disabled           Disabled          16912      k8s:io.cilium.k8s.policy.cluster=default             fd02::e8   10.0.0.179   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 3619       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
01:49:32 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
01:50:14 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|994aa682_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.9//1776/artifact/994aa682_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.9//1776/artifact/test_results_Cilium-PR-K8s-1.20-kernel-4.9_1776_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.9/1776/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Apr 8, 2023
Merged
1 task
@maintainer-s-little-helper
Copy link
Author

PR #24820 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-12T14:55:30.627917703Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.183 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-12T14:55:27.471369048Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.176 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-04-12T14:55:30.627917703Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.183 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-04-12T14:55:27.471369048Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.176 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 4
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 469
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
100:\tfrom all lookup local
+ true
+ '[' false = true ']'
Waiting for k8s node information
Cilium pods: [cilium-44rzd cilium-wf497]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-a-allowed-cnp-86b8c7bf44-xmscm                                  
pod-to-b-intra-node-nodeport-79d9fc898b-wjbzw                          
pod-to-b-multi-node-headless-7bcf8566b-gjwvc                           
pod-to-external-1111-f466d786b-7kh8m                                   
echo-a-78667c8fdf-fsnv7                                                
pod-to-a-69cc5b49b8-gl5hq                                              
pod-to-c-multi-node-proxy-ingress-policy-56889f84cd-rrnvb              
echo-b-5c5d7c49b5-5w4xs                                                
pod-to-b-intra-node-hostport-d568f6bd4-c59s8                           
pod-to-b-multi-node-hostport-6d54854b6d-8nj9s                          
pod-to-c-intra-node-proxy-ingress-policy-fccbcc9bf-lb7c9               
pod-to-c-intra-node-proxy-to-proxy-policy-77b98c977d-q9x78             
pod-to-c-multi-node-proxy-to-proxy-policy-78647869d5-ckvbj             
coredns-86c74c674b-7m2ml                                               
pod-to-a-denied-cnp-5f9b6b964b-t66cd                                   
pod-to-a-intra-node-proxy-egress-policy-65f4654f99-d7t95               
pod-to-a-multi-node-proxy-egress-policy-69b8569968-d9g2x               
pod-to-b-multi-node-clusterip-86dbd49cdb-254cb                         
pod-to-b-multi-node-nodeport-8697db657d-chpmt                          
pod-to-external-fqdn-allow-google-cnp-7d5cc887cd-dwlfl                 
echo-c-78dc7fc59d-cf8gz                                                
Cilium agent 'cilium-44rzd': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0
Cilium agent 'cilium-wf497': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 70 Failed 0

Standard Error

Click to show. 
14:55:04 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
14:55:04 STEP: Ensuring the namespace kube-system exists
14:55:04 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
14:55:04 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
14:55:04 STEP: Installing Cilium
14:55:05 STEP: Waiting for Cilium to become ready
14:56:30 STEP: Restarting unmanaged pods coredns-86c74c674b-6v5xx in namespace kube-system
14:56:48 STEP: Validating if Kubernetes DNS is deployed
14:56:48 STEP: Checking if deployment is ready
14:56:48 STEP: Checking if kube-dns service is plumbed correctly
14:56:48 STEP: Checking if pods have identity
14:56:48 STEP: Checking if DNS can resolve
14:56:51 STEP: Kubernetes DNS is up and operational
14:56:51 STEP: Validating Cilium Installation
14:56:51 STEP: Performing Cilium controllers preflight check
14:56:51 STEP: Performing Cilium status preflight check
14:56:51 STEP: Performing Cilium health check
14:56:51 STEP: Checking whether host EP regenerated
14:56:59 STEP: Performing Cilium service preflight check
14:56:59 STEP: Performing K8s service preflight check
14:57:05 STEP: Waiting for cilium-operator to be ready
14:57:05 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
14:57:05 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
14:57:05 STEP: Making sure all endpoints are in ready state
14:57:08 STEP: WaitforPods(namespace="default", filter="")
14:57:29 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-04-12T14:57:29Z====
14:57:29 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-12T14:55:30.627917703Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.183 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-12T14:55:27.471369048Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.176 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
14:57:30 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-54dbdc987-whll4                                      0/1     Running   0          55m     10.0.1.184      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-6ff848df8b-9kd68                                  1/1     Running   0          55m     10.0.1.30       k8s2   <none>           <none>
	 default             echo-a-78667c8fdf-fsnv7                                      1/1     Running   0          27s     10.0.0.23       k8s2   <none>           <none>
	 default             echo-b-5c5d7c49b5-5w4xs                                      1/1     Running   0          27s     10.0.0.154      k8s2   <none>           <none>
	 default             echo-b-host-7df9796db6-dgf2k                                 1/1     Running   0          27s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-78dc7fc59d-cf8gz                                      1/1     Running   0          27s     10.0.0.105      k8s2   <none>           <none>
	 default             echo-c-host-7779f885fc-nzfbn                                 1/1     Running   0          27s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6744fdbf6-zx5ph               1/1     Running   0          25s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-87b4cc8cd-nlc76                1/1     Running   0          24s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-69cc5b49b8-gl5hq                                    1/1     Running   0          27s     10.0.1.166      k8s1   <none>           <none>
	 default             pod-to-a-allowed-cnp-86b8c7bf44-xmscm                        1/1     Running   0          27s     10.0.0.185      k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-5f9b6b964b-t66cd                         1/1     Running   0          27s     10.0.1.29       k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-65f4654f99-d7t95     2/2     Running   0          26s     10.0.0.234      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-69b8569968-d9g2x     2/2     Running   0          26s     10.0.1.201      k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-d568f6bd4-c59s8                 1/1     Running   0          24s     10.0.0.143      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-79d9fc898b-wjbzw                1/1     Running   0          24s     10.0.0.106      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-86dbd49cdb-254cb               1/1     Running   0          25s     10.0.1.65       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-7bcf8566b-gjwvc                 1/1     Running   0          25s     10.0.1.156      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-6d54854b6d-8nj9s                1/1     Running   0          24s     10.0.1.238      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-8697db657d-chpmt                1/1     Running   0          24s     10.0.1.112      k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-fccbcc9bf-lb7c9     2/2     Running   0          26s     10.0.0.39       k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-77b98c977d-q9x78   2/2     Running   0          26s     10.0.0.71       k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-56889f84cd-rrnvb    2/2     Running   0          26s     10.0.1.122      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-78647869d5-ckvbj   2/2     Running   0          25s     10.0.1.149      k8s1   <none>           <none>
	 default             pod-to-external-1111-f466d786b-7kh8m                         1/1     Running   0          27s     10.0.0.8        k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7d5cc887cd-dwlfl       1/1     Running   0          26s     10.0.1.62       k8s1   <none>           <none>
	 kube-system         cilium-44rzd                                                 1/1     Running   1          2m30s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-84df5b667d-6zvcz                             1/1     Running   0          2m30s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-84df5b667d-g5t4x                             1/1     Running   0          2m30s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-wf497                                                 1/1     Running   1          2m30s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-86c74c674b-7m2ml                                     1/1     Running   0          65s     10.0.0.118      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          62m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   0          62m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   2          62m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-6twcw                                             1/1     Running   0          56m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-dglq4                                             1/1     Running   0          62m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   3          62m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-5dclk                                           1/1     Running   0          55m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-c7dp7                                           1/1     Running   0          55m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-4s7mb                                         1/1     Running   0          56m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-r8qk4                                         1/1     Running   0          56m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-44rzd cilium-wf497]
cmd: kubectl exec -n kube-system cilium-44rzd -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 189        Disabled           Disabled          5645       k8s:io.cilium.k8s.policy.cluster=default             fd02::151   10.0.1.65    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                
	 783        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                    ready   
	                                                            k8s:node-role.kubernetes.io/master                                                    
	                                                            reserved:host                                                                         
	 1229       Disabled           Disabled          43248      k8s:io.cilium.k8s.policy.cluster=default             fd02::1a1   10.0.1.112   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                 
	 1313       Disabled           Disabled          46119      k8s:io.cilium.k8s.policy.cluster=default             fd02::1f6   10.0.1.122   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                     
	 1463       Disabled           Enabled           2167       k8s:io.cilium.k8s.policy.cluster=default             fd02::1ef   10.0.1.149   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                    
	 2102       Disabled           Disabled          29176      k8s:io.cilium.k8s.policy.cluster=default             fd02::12e   10.0.1.166   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a                                                                     
	 2709       Disabled           Enabled           50630      k8s:io.cilium.k8s.policy.cluster=default             fd02::10b   10.0.1.62    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                        
	 2962       Disabled           Disabled          61618      k8s:io.cilium.k8s.policy.cluster=default             fd02::193   10.0.1.238   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                 
	 3148       Disabled           Enabled           57256      k8s:io.cilium.k8s.policy.cluster=default             fd02::1be   10.0.1.201   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                      
	 3206       Disabled           Disabled          4          reserved:health                                      fd02::16f   10.0.1.60    ready   
	 3236       Disabled           Enabled           44704      k8s:io.cilium.k8s.policy.cluster=default             fd02::1d7   10.0.1.29    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-denied-cnp                                                          
	 3406       Disabled           Disabled          11522      k8s:io.cilium.k8s.policy.cluster=default             fd02::16c   10.0.1.156   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-headless                                                 
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-wf497 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 219        Disabled           Enabled           39585      k8s:io.cilium.k8s.policy.cluster=default             fd02::33   10.0.0.185   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-allowed-cnp                                                        
	 312        Disabled           Disabled          7547       k8s:io.cilium.k8s.policy.cluster=default             fd02::c0   10.0.0.143   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 357        Disabled           Enabled           27084      k8s:io.cilium.k8s.policy.cluster=default             fd02::f7   10.0.0.234   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 665        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 826        Disabled           Disabled          30603      k8s:io.cilium.k8s.policy.cluster=default             fd02::2    10.0.0.118   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                          
	                                                            k8s:k8s-app=kube-dns                                                                 
	 839        Disabled           Disabled          9083       k8s:io.cilium.k8s.policy.cluster=default             fd02::82   10.0.0.154   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 949        Disabled           Disabled          22915      k8s:io.cilium.k8s.policy.cluster=default             fd02::65   10.0.0.39    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 1569       Disabled           Disabled          35287      k8s:io.cilium.k8s.policy.cluster=default             fd02::e0   10.0.0.23    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 1664       Enabled            Disabled          19015      k8s:io.cilium.k8s.policy.cluster=default             fd02::fd   10.0.0.105   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 1927       Disabled           Disabled          4          reserved:health                                      fd02::a6   10.0.0.125   ready   
	 2566       Disabled           Disabled          14412      k8s:io.cilium.k8s.policy.cluster=default             fd02::a2   10.0.0.8     ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 2852       Disabled           Disabled          9033       k8s:io.cilium.k8s.policy.cluster=default             fd02::24   10.0.0.106   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 3466       Disabled           Enabled           21489      k8s:io.cilium.k8s.policy.cluster=default             fd02::b    10.0.0.71    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
14:58:15 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
14:59:00 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|4cbcd32c_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9//2637/artifact/4cbcd32c_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9//2637/artifact/test_results_Cilium-PR-K8s-1.18-kernel-4.9_2637_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9/2637/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
1 task
@maintainer-s-little-helper
Copy link
Author

PR #24872 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-13T18:08:39.557447963Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.135 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-13T18:08:32.036415736Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.88 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-04-13T18:08:39.557447963Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.135 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-04-13T18:08:32.036415736Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.88 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 461
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
100:\tfrom all lookup local
+ true
+ '[' false = true ']'
++ awk '{print $2}'
Cilium pods: [cilium-lf8v2 cilium-rqppr]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-a-denied-cnp-64d7765ddf-znr4r                                   
pod-to-b-intra-node-hostport-54d55874b5-4x64j                          
pod-to-b-multi-node-nodeport-84fdc88d9f-k5b78                          
pod-to-c-multi-node-proxy-to-proxy-policy-6fff5cc8c4-v9zpc             
pod-to-external-1111-6b96d6cc7-v8gd9                                   
pod-to-external-fqdn-allow-google-cnp-6d8b6d59fc-5s4g5                 
coredns-7c74c644b-psqtx                                                
echo-b-5898f8c6c4-ds47q                                                
pod-to-c-intra-node-proxy-ingress-policy-8467c7ccf4-47nfd              
pod-to-a-intra-node-proxy-egress-policy-55bbdc8944-k46d7               
echo-c-6b4f857f7-w97dl                                                 
pod-to-a-9dc6d768c-lmhq6                                               
pod-to-b-intra-node-nodeport-5c8cd69ff5-495h9                          
pod-to-b-multi-node-clusterip-7b5854d46c-rmh5x                         
pod-to-b-multi-node-hostport-5cbc88fbcc-phfds                          
pod-to-c-intra-node-proxy-to-proxy-policy-7497c557df-gwsfz             
echo-a-56f9849b6b-shbjq                                                
pod-to-a-multi-node-proxy-egress-policy-5564db6bb7-spsl9               
pod-to-b-multi-node-headless-77b698d8f5-lnvnf                          
pod-to-c-multi-node-proxy-ingress-policy-9c4bb99d5-hnv2x               
pod-to-a-allowed-cnp-5ff69578c5-gnkxs                                  
Cilium agent 'cilium-lf8v2': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0
Cilium agent 'cilium-rqppr': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 58 Failed 0

Standard Error

Click to show. 
18:08:09 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
18:08:09 STEP: Ensuring the namespace kube-system exists
18:08:10 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
18:08:10 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
18:08:10 STEP: Installing Cilium
18:08:10 STEP: Waiting for Cilium to become ready
18:09:40 STEP: Restarting unmanaged pods coredns-7c74c644b-95cwk in namespace kube-system
18:09:47 STEP: Validating if Kubernetes DNS is deployed
18:09:47 STEP: Checking if deployment is ready
18:09:47 STEP: Kubernetes DNS is not ready: only 0 of 1 replicas are available
18:09:47 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
18:09:48 STEP: Waiting for Kubernetes DNS to become operational
18:09:48 STEP: Checking if deployment is ready
18:09:48 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:49 STEP: Checking if deployment is ready
18:09:49 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:50 STEP: Checking if deployment is ready
18:09:50 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:51 STEP: Checking if deployment is ready
18:09:51 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:52 STEP: Checking if deployment is ready
18:09:52 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:53 STEP: Checking if deployment is ready
18:09:53 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:54 STEP: Checking if deployment is ready
18:09:54 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:55 STEP: Checking if deployment is ready
18:09:55 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:56 STEP: Checking if deployment is ready
18:09:56 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:09:57 STEP: Checking if deployment is ready
18:09:57 STEP: Checking if kube-dns service is plumbed correctly
18:09:57 STEP: Checking if pods have identity
18:09:57 STEP: Checking if DNS can resolve
18:10:01 STEP: Validating Cilium Installation
18:10:01 STEP: Performing Cilium controllers preflight check
18:10:01 STEP: Performing Cilium health check
18:10:01 STEP: Checking whether host EP regenerated
18:10:01 STEP: Performing Cilium status preflight check
18:10:08 STEP: Performing Cilium service preflight check
18:10:08 STEP: Performing K8s service preflight check
18:10:14 STEP: Waiting for cilium-operator to be ready
18:10:14 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
18:10:14 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
18:10:14 STEP: Making sure all endpoints are in ready state
18:10:18 STEP: WaitforPods(namespace="default", filter="")
18:10:37 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-04-13T18:10:37Z====
18:10:37 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-13T18:08:39.557447963Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.135 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-13T18:08:32.036415736Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.88 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
18:10:37 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-d69c97b9b-bs6sv                                      0/1     Running   0          93m     10.0.0.164      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-t6wzt                                  1/1     Running   0          93m     10.0.0.142      k8s1   <none>           <none>
	 default             echo-a-56f9849b6b-shbjq                                      1/1     Running   0          26s     10.0.1.11       k8s2   <none>           <none>
	 default             echo-b-5898f8c6c4-ds47q                                      1/1     Running   0          26s     10.0.1.207      k8s2   <none>           <none>
	 default             echo-b-host-7b949c7b9f-h7wk9                                 1/1     Running   0          26s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-6b4f857f7-w97dl                                       1/1     Running   0          26s     10.0.1.106      k8s2   <none>           <none>
	 default             echo-c-host-6ccff4d6bb-wf4pd                                 1/1     Running   0          26s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6c4db976b5-8f8xz              1/1     Running   0          23s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-68d54bb4b7-9j57k               1/1     Running   0          23s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-9dc6d768c-lmhq6                                     1/1     Running   0          26s     10.0.1.219      k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-5ff69578c5-gnkxs                        1/1     Running   0          25s     10.0.1.20       k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-64d7765ddf-znr4r                         1/1     Running   0          25s     10.0.1.119      k8s2   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-55bbdc8944-k46d7     2/2     Running   0          25s     10.0.1.87       k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5564db6bb7-spsl9     2/2     Running   0          25s     10.0.0.116      k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-54d55874b5-4x64j                1/1     Running   0          22s     10.0.1.242      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-5c8cd69ff5-495h9                1/1     Running   0          22s     10.0.1.21       k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-7b5854d46c-rmh5x               1/1     Running   0          23s     10.0.0.245      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-77b698d8f5-lnvnf                1/1     Running   0          23s     10.0.0.188      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-5cbc88fbcc-phfds                1/1     Running   0          23s     10.0.0.232      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-84fdc88d9f-k5b78                1/1     Running   0          22s     10.0.0.34       k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-8467c7ccf4-47nfd    2/2     Running   0          24s     10.0.1.173      k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-7497c557df-gwsfz   2/2     Running   0          24s     10.0.1.208      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-9c4bb99d5-hnv2x     2/2     Running   0          24s     10.0.0.101      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-6fff5cc8c4-v9zpc   2/2     Running   0          24s     10.0.0.221      k8s1   <none>           <none>
	 default             pod-to-external-1111-6b96d6cc7-v8gd9                         1/1     Running   0          26s     10.0.1.57       k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-6d8b6d59fc-5s4g5       1/1     Running   0          25s     10.0.1.72       k8s2   <none>           <none>
	 kube-system         cilium-lf8v2                                                 1/1     Running   1          2m33s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-8459bfdd68-pvt4l                             1/1     Running   0          2m33s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-8459bfdd68-zd6ts                             1/1     Running   0          2m33s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-rqppr                                                 1/1     Running   1          2m33s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-7c74c644b-psqtx                                      1/1     Running   0          55s     10.0.0.44       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   1          101m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   2          101m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   4          101m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-m7hqq                                             1/1     Running   0          101m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-pdl8k                                             1/1     Running   0          93m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   3          101m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-2khr6                                           1/1     Running   0          93m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-lxcrr                                           1/1     Running   0          93m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-7vw2b                                         1/1     Running   0          93m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-94mrs                                         1/1     Running   0          93m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-lf8v2 cilium-rqppr]
cmd: kubectl exec -n kube-system cilium-lf8v2 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 42         Disabled           Enabled           17884      k8s:io.cilium.k8s.policy.cluster=default             fd02::180   10.0.1.208   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                    
	 233        Disabled           Disabled          4          reserved:health                                      fd02::117   10.0.1.249   ready   
	 707        Disabled           Disabled          43659      k8s:io.cilium.k8s.policy.cluster=default             fd02::16e   10.0.1.21    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                 
	 843        Disabled           Disabled          1922       k8s:io.cilium.k8s.policy.cluster=default             fd02::1cf   10.0.1.11    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-a                                                                       
	 959        Disabled           Enabled           2635       k8s:io.cilium.k8s.policy.cluster=default             fd02::1d4   10.0.1.87    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                      
	 964        Disabled           Enabled           40179      k8s:io.cilium.k8s.policy.cluster=default             fd02::12f   10.0.1.72    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                        
	 1009       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                    ready   
	                                                            reserved:host                                                                         
	 1018       Disabled           Disabled          34340      k8s:io.cilium.k8s.policy.cluster=default             fd02::192   10.0.1.242   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                 
	 1202       Disabled           Disabled          6204       k8s:io.cilium.k8s.policy.cluster=default             fd02::19e   10.0.1.207   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-b                                                                       
	 1424       Enabled            Disabled          7546       k8s:io.cilium.k8s.policy.cluster=default             fd02::182   10.0.1.106   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-c                                                                       
	 1557       Disabled           Disabled          13597      k8s:io.cilium.k8s.policy.cluster=default             fd02::153   10.0.1.219   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a                                                                     
	 1755       Disabled           Enabled           26993      k8s:io.cilium.k8s.policy.cluster=default             fd02::1db   10.0.1.119   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-denied-cnp                                                          
	 3319       Disabled           Disabled          12169      k8s:io.cilium.k8s.policy.cluster=default             fd02::130   10.0.1.173   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                     
	 3323       Disabled           Disabled          18242      k8s:io.cilium.k8s.policy.cluster=default             fd02::11e   10.0.1.57    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-1111                                                         
	 3400       Disabled           Enabled           22973      k8s:io.cilium.k8s.policy.cluster=default             fd02::193   10.0.1.20    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-allowed-cnp                                                         
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-rqppr -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 63         Disabled           Disabled          42702      k8s:io.cilium.k8s.policy.cluster=default             fd02::9    10.0.0.34    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                
	 736        Disabled           Disabled          4          reserved:health                                      fd02::76   10.0.0.67    ready   
	 957        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                   ready   
	                                                            k8s:node-role.kubernetes.io/master                                                   
	                                                            reserved:host                                                                        
	 1331       Disabled           Disabled          3487       k8s:io.cilium.k8s.policy.cluster=default             fd02::6e   10.0.0.245   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                               
	 1604       Disabled           Disabled          61455      k8s:io.cilium.k8s.policy.cluster=default             fd02::7e   10.0.0.101   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                    
	 1708       Disabled           Disabled          55334      k8s:io.cilium.k8s.policy.cluster=default             fd02::50   10.0.0.44    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                          
	                                                            k8s:k8s-app=kube-dns                                                                 
	 2017       Disabled           Disabled          2077       k8s:io.cilium.k8s.policy.cluster=default             fd02::ba   10.0.0.188   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-headless                                                
	 2182       Disabled           Disabled          17459      k8s:io.cilium.k8s.policy.cluster=default             fd02::60   10.0.0.232   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                
	 2483       Disabled           Enabled           27422      k8s:io.cilium.k8s.policy.cluster=default             fd02::5f   10.0.0.116   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                     
	 3673       Disabled           Enabled           11085      k8s:io.cilium.k8s.policy.cluster=default             fd02::1d   10.0.0.221   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
18:10:54 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
18:11:38 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|c0a29f88_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9//2809/artifact/c0a29f88_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9//2809/artifact/test_results_Cilium-PR-K8s-1.19-kernel-4.9_2809_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9/2809/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #24911 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-15T01:52:33.926527388Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.195 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-15T01:52:21.758029540Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.150 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-04-15T01:52:33.926527388Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.195 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-04-15T01:52:21.758029540Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.150 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 467
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
+ true
100:\tfrom all lookup local
Command execution failed
Waiting for k8s node information
+ '[' false = true ']'
Cilium pods: [cilium-ljgmc cilium-t8hh6]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-b-intra-node-hostport-d568f6bd4-r7c6c                           
pod-to-b-intra-node-nodeport-79d9fc898b-4whx9                          
pod-to-b-multi-node-headless-7bcf8566b-28nmb                           
coredns-86c74c674b-smc95                                               
pod-to-a-allowed-cnp-86b8c7bf44-fs28b                                  
pod-to-a-intra-node-proxy-egress-policy-65f4654f99-wsfj7               
pod-to-a-multi-node-proxy-egress-policy-69b8569968-p8bsq               
pod-to-b-multi-node-clusterip-86dbd49cdb-68ltb                         
pod-to-b-multi-node-nodeport-8697db657d-k2skj                          
pod-to-c-multi-node-proxy-ingress-policy-56889f84cd-9z8f2              
echo-b-5c5d7c49b5-xm2hl                                                
pod-to-b-multi-node-hostport-6d54854b6d-w4vvq                          
pod-to-c-intra-node-proxy-ingress-policy-fccbcc9bf-sv5t4               
pod-to-c-multi-node-proxy-to-proxy-policy-78647869d5-cxxl7             
echo-c-78dc7fc59d-6wcbb                                                
pod-to-a-69cc5b49b8-fj6r6                                              
pod-to-a-denied-cnp-5f9b6b964b-lvxfn                                   
pod-to-c-intra-node-proxy-to-proxy-policy-77b98c977d-pxzqs             
pod-to-external-1111-f466d786b-hbddw                                   
pod-to-external-fqdn-allow-google-cnp-7d5cc887cd-8nv5d                 
echo-a-78667c8fdf-n2djk                                                
Cilium agent 'cilium-ljgmc': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 58 Failed 0
Cilium agent 'cilium-t8hh6': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0

Standard Error

Click to show. 
01:52:00 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
01:52:00 STEP: Ensuring the namespace kube-system exists
01:52:00 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
01:52:00 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
01:52:01 STEP: Installing Cilium
01:52:01 STEP: Waiting for Cilium to become ready
01:53:33 STEP: Restarting unmanaged pods coredns-86c74c674b-kcj5h in namespace kube-system
01:53:40 STEP: Validating if Kubernetes DNS is deployed
01:53:40 STEP: Checking if deployment is ready
01:53:40 STEP: Checking if kube-dns service is plumbed correctly
01:53:40 STEP: Checking if pods have identity
01:53:40 STEP: Checking if DNS can resolve
01:53:44 STEP: Kubernetes DNS is up and operational
01:53:44 STEP: Validating Cilium Installation
01:53:44 STEP: Performing Cilium controllers preflight check
01:53:44 STEP: Performing Cilium health check
01:53:44 STEP: Checking whether host EP regenerated
01:53:44 STEP: Performing Cilium status preflight check
01:53:51 STEP: Performing Cilium service preflight check
01:53:51 STEP: Performing K8s service preflight check
01:53:58 STEP: Waiting for cilium-operator to be ready
01:53:58 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
01:53:58 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
01:53:58 STEP: Making sure all endpoints are in ready state
01:54:01 STEP: WaitforPods(namespace="default", filter="")
01:54:22 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-04-15T01:54:22Z====
01:54:22 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-15T01:52:33.926527388Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.195 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-15T01:52:21.758029540Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.150 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
01:54:23 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-54dbdc987-25nnb                                      0/1     Running   0          28m     10.0.0.143      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-6ff848df8b-srbvd                                  1/1     Running   0          28m     10.0.0.136      k8s1   <none>           <none>
	 default             echo-a-78667c8fdf-n2djk                                      1/1     Running   0          27s     10.0.0.148      k8s2   <none>           <none>
	 default             echo-b-5c5d7c49b5-xm2hl                                      1/1     Running   0          27s     10.0.0.104      k8s2   <none>           <none>
	 default             echo-b-host-7df9796db6-cd7f7                                 1/1     Running   0          27s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-78dc7fc59d-6wcbb                                      1/1     Running   0          27s     10.0.0.199      k8s2   <none>           <none>
	 default             echo-c-host-7779f885fc-5sdxr                                 1/1     Running   0          27s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6744fdbf6-ltsph               1/1     Running   0          25s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-87b4cc8cd-mqw46                1/1     Running   0          25s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-69cc5b49b8-fj6r6                                    1/1     Running   0          27s     10.0.0.234      k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-86b8c7bf44-fs28b                        1/1     Running   0          27s     10.0.1.65       k8s1   <none>           <none>
	 default             pod-to-a-denied-cnp-5f9b6b964b-lvxfn                         1/1     Running   0          27s     10.0.0.102      k8s2   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-65f4654f99-wsfj7     2/2     Running   0          27s     10.0.0.228      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-69b8569968-p8bsq     2/2     Running   0          26s     10.0.1.41       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-d568f6bd4-r7c6c                 1/1     Running   0          24s     10.0.0.124      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-79d9fc898b-4whx9                1/1     Running   0          24s     10.0.0.186      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-86dbd49cdb-68ltb               1/1     Running   0          25s     10.0.1.201      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-7bcf8566b-28nmb                 1/1     Running   0          25s     10.0.1.118      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-6d54854b6d-w4vvq                1/1     Running   0          25s     10.0.1.159      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-8697db657d-k2skj                1/1     Running   0          24s     10.0.1.229      k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-fccbcc9bf-sv5t4     2/2     Running   0          26s     10.0.0.173      k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-77b98c977d-pxzqs   2/2     Running   0          26s     10.0.0.45       k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-56889f84cd-9z8f2    2/2     Running   0          26s     10.0.1.213      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-78647869d5-cxxl7   2/2     Running   0          26s     10.0.1.244      k8s1   <none>           <none>
	 default             pod-to-external-1111-f466d786b-hbddw                         1/1     Running   0          27s     10.0.0.31       k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7d5cc887cd-8nv5d       1/1     Running   0          27s     10.0.0.237      k8s2   <none>           <none>
	 kube-system         cilium-ljgmc                                                 1/1     Running   1          2m27s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-67fffd48c4-8jgpr                             1/1     Running   0          2m27s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-67fffd48c4-twsnq                             1/1     Running   0          2m27s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-t8hh6                                                 1/1     Running   1          2m27s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-86c74c674b-smc95                                     1/1     Running   0          55s     10.0.0.121      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          36m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   1          36m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   3          36m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-899rq                                             1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-jnnfz                                             1/1     Running   0          36m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   4          36m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-n7plg                                           1/1     Running   0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-s5tbk                                           1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-8f6qh                                         1/1     Running   0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-8zqlc                                         1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-ljgmc cilium-t8hh6]
cmd: kubectl exec -n kube-system cilium-ljgmc -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 414        Disabled           Disabled          53636      k8s:io.cilium.k8s.policy.cluster=default             fd02::1f6   10.0.1.213   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                     
	 636        Disabled           Disabled          53545      k8s:io.cilium.k8s.policy.cluster=default             fd02::134   10.0.1.118   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-headless                                                 
	 919        Disabled           Disabled          10032      k8s:io.cilium.k8s.policy.cluster=default             fd02::1d1   10.0.1.159   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                 
	 1001       Disabled           Disabled          6932       k8s:io.cilium.k8s.policy.cluster=default             fd02::1a3   10.0.1.201   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                
	 1029       Disabled           Disabled          40972      k8s:io.cilium.k8s.policy.cluster=default             fd02::1a7   10.0.1.229   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                 
	 2167       Disabled           Enabled           11375      k8s:io.cilium.k8s.policy.cluster=default             fd02::195   10.0.1.244   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                    
	 2258       Disabled           Enabled           53251      k8s:io.cilium.k8s.policy.cluster=default             fd02::133   10.0.1.65    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-allowed-cnp                                                         
	 3633       Disabled           Disabled          4          reserved:health                                      fd02::1aa   10.0.1.20    ready   
	 3955       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                    ready   
	                                                            k8s:node-role.kubernetes.io/master                                                    
	                                                            reserved:host                                                                         
	 4048       Disabled           Enabled           3817       k8s:io.cilium.k8s.policy.cluster=default             fd02::1a2   10.0.1.41    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                      
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-t8hh6 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 8          Disabled           Enabled           29600      k8s:io.cilium.k8s.policy.cluster=default             fd02::a2   10.0.0.45    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 186        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 209        Disabled           Disabled          16809      k8s:io.cilium.k8s.policy.cluster=default             fd02::8a   10.0.0.104   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 305        Disabled           Disabled          7892       k8s:io.cilium.k8s.policy.cluster=default             fd02::6    10.0.0.186   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 465        Disabled           Disabled          12513      k8s:io.cilium.k8s.policy.cluster=default             fd02::3d   10.0.0.121   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                          
	                                                            k8s:k8s-app=kube-dns                                                                 
	 944        Disabled           Enabled           969        k8s:io.cilium.k8s.policy.cluster=default             fd02::15   10.0.0.102   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-denied-cnp                                                         
	 1032       Enabled            Disabled          17516      k8s:io.cilium.k8s.policy.cluster=default             fd02::9b   10.0.0.199   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 1119       Disabled           Disabled          39400      k8s:io.cilium.k8s.policy.cluster=default             fd02::e4   10.0.0.124   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 1182       Disabled           Disabled          4          reserved:health                                      fd02::73   10.0.0.163   ready   
	 1191       Disabled           Disabled          13252      k8s:io.cilium.k8s.policy.cluster=default             fd02::fe   10.0.0.234   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a                                                                    
	 1582       Disabled           Enabled           58792      k8s:io.cilium.k8s.policy.cluster=default             fd02::4f   10.0.0.237   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                       
	 1929       Disabled           Disabled          43538      k8s:io.cilium.k8s.policy.cluster=default             fd02::4b   10.0.0.31    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 2134       Disabled           Disabled          18394      k8s:io.cilium.k8s.policy.cluster=default             fd02::2b   10.0.0.148   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 3734       Disabled           Disabled          45248      k8s:io.cilium.k8s.policy.cluster=default             fd02::a8   10.0.0.173   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 4083       Disabled           Enabled           44279      k8s:io.cilium.k8s.policy.cluster=default             fd02::f7   10.0.0.228   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
01:54:39 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
01:55:25 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|7cc910d8_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9//2659/artifact/7cc910d8_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9//2659/artifact/test_results_Cilium-PR-K8s-1.18-kernel-4.9_2659_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9/2659/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #25139 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-26T16:53:15.306014054Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.128 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-26T16:53:12.191613870Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.254 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-04-26T16:53:15.306014054Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.128 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-04-26T16:53:12.191613870Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.254 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 4
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 468
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
100:\tfrom all lookup local
+ true
+ '[' false = true ']'
Waiting for k8s node information
Cilium pods: [cilium-n9qnj cilium-t7xxt]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-a-allowed-cnp-5ff69578c5-l4z9h                                  
pod-to-b-intra-node-hostport-54d55874b5-lxf6q                          
pod-to-b-intra-node-nodeport-5c8cd69ff5-rfldt                          
pod-to-b-multi-node-nodeport-84fdc88d9f-bc6dz                          
pod-to-c-intra-node-proxy-ingress-policy-8467c7ccf4-djptd              
pod-to-c-multi-node-proxy-to-proxy-policy-6fff5cc8c4-cw42w             
pod-to-b-multi-node-clusterip-7b5854d46c-592kv                         
pod-to-c-intra-node-proxy-to-proxy-policy-7497c557df-q5s8d             
pod-to-c-multi-node-proxy-ingress-policy-9c4bb99d5-68szr               
coredns-7c74c644b-wk8td                                                
echo-b-5898f8c6c4-7dlzb                                                
pod-to-a-denied-cnp-64d7765ddf-n42tk                                   
pod-to-a-intra-node-proxy-egress-policy-55bbdc8944-csht2               
pod-to-external-1111-6b96d6cc7-586tk                                   
pod-to-external-fqdn-allow-google-cnp-6d8b6d59fc-bsn72                 
echo-a-56f9849b6b-9vcz7                                                
echo-c-6b4f857f7-nmbfr                                                 
pod-to-a-9dc6d768c-x98q8                                               
pod-to-a-multi-node-proxy-egress-policy-5564db6bb7-wmcqv               
pod-to-b-multi-node-headless-77b698d8f5-hs9f5                          
pod-to-b-multi-node-hostport-5cbc88fbcc-hdmz4                          
Cilium agent 'cilium-n9qnj': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 70 Failed 0
Cilium agent 'cilium-t7xxt': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 66 Failed 0

Standard Error

Click to show. 
16:52:48 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
16:52:48 STEP: Ensuring the namespace kube-system exists
16:52:48 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
16:52:48 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
16:52:48 STEP: Installing Cilium
16:52:49 STEP: Waiting for Cilium to become ready
16:54:07 STEP: Restarting unmanaged pods coredns-7c74c644b-lm775 in namespace kube-system
16:54:14 STEP: Validating if Kubernetes DNS is deployed
16:54:14 STEP: Checking if deployment is ready
16:54:14 STEP: Kubernetes DNS is not ready: only 0 of 1 replicas are available
16:54:14 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
16:54:14 STEP: Waiting for Kubernetes DNS to become operational
16:54:14 STEP: Checking if deployment is ready
16:54:14 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:54:15 STEP: Checking if deployment is ready
16:54:15 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:54:16 STEP: Checking if deployment is ready
16:54:16 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:54:17 STEP: Checking if deployment is ready
16:54:17 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:54:18 STEP: Checking if deployment is ready
16:54:18 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:54:19 STEP: Checking if deployment is ready
16:54:19 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:54:20 STEP: Checking if deployment is ready
16:54:20 STEP: Checking if kube-dns service is plumbed correctly
16:54:20 STEP: Checking if DNS can resolve
16:54:20 STEP: Checking if pods have identity
16:54:24 STEP: Kubernetes DNS is not ready yet: CiliumEndpoint does not exist
16:54:24 STEP: Checking if deployment is ready
16:54:24 STEP: Checking if kube-dns service is plumbed correctly
16:54:24 STEP: Checking if pods have identity
16:54:24 STEP: Checking if DNS can resolve
16:54:28 STEP: Validating Cilium Installation
16:54:28 STEP: Performing Cilium controllers preflight check
16:54:28 STEP: Performing Cilium health check
16:54:28 STEP: Performing Cilium status preflight check
16:54:28 STEP: Checking whether host EP regenerated
16:54:36 STEP: Performing Cilium service preflight check
16:54:36 STEP: Performing K8s service preflight check
16:54:42 STEP: Waiting for cilium-operator to be ready
16:54:42 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
16:54:42 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
16:54:42 STEP: Making sure all endpoints are in ready state
16:54:45 STEP: WaitforPods(namespace="default", filter="")
16:55:30 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-04-26T16:55:30Z====
16:55:30 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-04-26T16:53:15.306014054Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.128 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-04-26T16:53:12.191613870Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.254 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
16:55:31 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-d69c97b9b-k67km                                      0/1     Running   0          98m     10.0.1.178      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-59tsw                                  1/1     Running   0          98m     10.0.1.171      k8s2   <none>           <none>
	 default             echo-a-56f9849b6b-9vcz7                                      1/1     Running   0          51s     10.0.0.58       k8s2   <none>           <none>
	 default             echo-b-5898f8c6c4-7dlzb                                      1/1     Running   0          51s     10.0.0.110      k8s2   <none>           <none>
	 default             echo-b-host-7b949c7b9f-pwtlr                                 1/1     Running   0          51s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-6b4f857f7-nmbfr                                       1/1     Running   0          51s     10.0.0.81       k8s2   <none>           <none>
	 default             echo-c-host-6ccff4d6bb-4jrht                                 1/1     Running   0          51s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6c4db976b5-mkt4k              1/1     Running   0          49s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-68d54bb4b7-6b6s9               1/1     Running   0          49s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-9dc6d768c-x98q8                                     1/1     Running   0          51s     10.0.0.84       k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-5ff69578c5-l4z9h                        1/1     Running   0          51s     10.0.0.167      k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-64d7765ddf-n42tk                         1/1     Running   0          51s     10.0.1.34       k8s1   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-55bbdc8944-csht2     2/2     Running   0          51s     10.0.0.90       k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5564db6bb7-wmcqv     2/2     Running   0          50s     10.0.1.25       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-54d55874b5-lxf6q                1/1     Running   0          48s     10.0.0.112      k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-5c8cd69ff5-rfldt                1/1     Running   0          48s     10.0.0.205      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-7b5854d46c-592kv               1/1     Running   0          49s     10.0.1.87       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-77b698d8f5-hs9f5                1/1     Running   0          49s     10.0.1.238      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-5cbc88fbcc-hdmz4                1/1     Running   0          49s     10.0.1.172      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-84fdc88d9f-bc6dz                1/1     Running   0          48s     10.0.1.28       k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-8467c7ccf4-djptd    2/2     Running   0          50s     10.0.0.166      k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-7497c557df-q5s8d   2/2     Running   0          50s     10.0.0.95       k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-9c4bb99d5-68szr     2/2     Running   0          50s     10.0.1.85       k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-6fff5cc8c4-cw42w   2/2     Running   0          50s     10.0.1.40       k8s1   <none>           <none>
	 default             pod-to-external-1111-6b96d6cc7-586tk                         1/1     Running   0          51s     10.0.0.30       k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-6d8b6d59fc-bsn72       1/1     Running   0          51s     10.0.1.232      k8s1   <none>           <none>
	 kube-system         cilium-n9qnj                                                 1/1     Running   1          2m47s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-66887dbc68-2s7l7                             1/1     Running   0          2m47s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-66887dbc68-fpl22                             1/1     Running   0          2m47s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-t7xxt                                                 1/1     Running   1          2m47s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-7c74c644b-wk8td                                      1/1     Running   0          82s     10.0.1.142      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   0          105m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   1          105m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   2          105m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-bp8h9                                             1/1     Running   0          104m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-rtn92                                             1/1     Running   0          98m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   2          105m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-8nwlq                                           1/1     Running   0          98m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-dkn4t                                           1/1     Running   0          98m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-7rzvs                                         1/1     Running   0          98m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-rdr4x                                         1/1     Running   0          98m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-n9qnj cilium-t7xxt]
cmd: kubectl exec -n kube-system cilium-n9qnj -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 588        Disabled           Disabled          8699       k8s:io.cilium.k8s.policy.cluster=default             fd02::c2   10.0.0.58    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-a                                                                      
	 732        Disabled           Enabled           36413      k8s:io.cilium.k8s.policy.cluster=default             fd02::2c   10.0.0.167   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-allowed-cnp                                                        
	 907        Disabled           Enabled           1442       k8s:io.cilium.k8s.policy.cluster=default             fd02::aa   10.0.0.95    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                   
	 1036       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                   ready   
	                                                            reserved:host                                                                        
	 1120       Enabled            Disabled          7373       k8s:io.cilium.k8s.policy.cluster=default             fd02::6    10.0.0.81    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-c                                                                      
	 1121       Disabled           Disabled          37394      k8s:io.cilium.k8s.policy.cluster=default             fd02::5    10.0.0.110   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=echo-b                                                                      
	 1475       Disabled           Disabled          41865      k8s:io.cilium.k8s.policy.cluster=default             fd02::c    10.0.0.112   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                
	 1803       Disabled           Disabled          6497       k8s:io.cilium.k8s.policy.cluster=default             fd02::35   10.0.0.166   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                    
	 2438       Disabled           Disabled          43771      k8s:io.cilium.k8s.policy.cluster=default             fd02::e6   10.0.0.84    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a                                                                    
	 3355       Disabled           Disabled          40423      k8s:io.cilium.k8s.policy.cluster=default             fd02::b3   10.0.0.30    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-external-1111                                                        
	 3688       Disabled           Enabled           3830       k8s:io.cilium.k8s.policy.cluster=default             fd02::a9   10.0.0.90    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                     
	 3708       Disabled           Disabled          4          reserved:health                                      fd02::e2   10.0.0.158   ready   
	 3921       Disabled           Disabled          7543       k8s:io.cilium.k8s.policy.cluster=default             fd02::67   10.0.0.205   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-t7xxt -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 30         Disabled           Disabled          37204      k8s:io.cilium.k8s.policy.cluster=default             fd02::13a   10.0.1.28    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                 
	 378        Disabled           Enabled           61248      k8s:io.cilium.k8s.policy.cluster=default             fd02::1a1   10.0.1.40    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                    
	 939        Disabled           Disabled          4          reserved:health                                      fd02::194   10.0.1.53    ready   
	 1175       Disabled           Disabled          1506       k8s:io.cilium.k8s.policy.cluster=default             fd02::146   10.0.1.87    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                
	 1729       Disabled           Disabled          38860      k8s:io.cilium.k8s.policy.cluster=default             fd02::14b   10.0.1.238   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-headless                                                 
	 2024       Disabled           Enabled           2178       k8s:io.cilium.k8s.policy.cluster=default             fd02::159   10.0.1.34    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-denied-cnp                                                          
	 2060       Disabled           Enabled           13182      k8s:io.cilium.k8s.policy.cluster=default             fd02::1c0   10.0.1.232   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                        
	 2631       Disabled           Disabled          29394      k8s:io.cilium.k8s.policy.cluster=default             fd02::162   10.0.1.85    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                     
	 2915       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                    ready   
	                                                            k8s:node-role.kubernetes.io/master                                                    
	                                                            reserved:host                                                                         
	 3125       Disabled           Enabled           12464      k8s:io.cilium.k8s.policy.cluster=default             fd02::1eb   10.0.1.25    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                      
	 3454       Disabled           Disabled          13922      k8s:io.cilium.k8s.policy.cluster=default             fd02::1cd   10.0.1.172   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                 
	 3979       Disabled           Disabled          9989       k8s:io.cilium.k8s.policy.cluster=default             fd02::12a   10.0.1.142   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                       
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                           
	                                                            k8s:k8s-app=kube-dns                                                                  
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
16:55:47 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
16:56:31 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|0873c255_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9//2832/artifact/0873c255_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9//2832/artifact/test_results_Cilium-PR-K8s-1.19-kernel-4.9_2832_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9/2832/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
2 tasks
@maintainer-s-little-helper
Copy link
Author

PR #25182 hit this flake with 94.08% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-05-08T11:05:55.394368956Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.92 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-05-08T11:05:55.282955726Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.80 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-05-08T11:05:55.394368956Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.92 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-05-08T11:05:55.282955726Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.80 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 4
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 471
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Waiting for k8s node information
100:\tfrom all lookup local
+ true
+ '[' false = true ']'
++ awk '{print $2}'
Cilium pods: [cilium-lww8g cilium-sv9d2]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
pod-to-c-multi-node-proxy-ingress-policy-74749c8794-sz95m              
echo-b-787dc99778-99h56                                                
echo-c-58cc9b67d9-q2s8l                                                
pod-to-a-denied-cnp-6887b57579-dxjr7                                   
pod-to-b-multi-node-nodeport-54446bdbb9-k6kl5                          
pod-to-c-intra-node-proxy-ingress-policy-5f95cf647c-9pf85              
echo-a-d576c5f8b-sw58x                                                 
pod-to-a-multi-node-proxy-egress-policy-5b5f7d94d8-2h8wv               
pod-to-b-multi-node-headless-64b6cbdd49-vkgwp                          
pod-to-c-multi-node-proxy-to-proxy-policy-9f68b7595-pxwzh              
pod-to-a-intra-node-proxy-egress-policy-74f796f479-27ncd               
pod-to-b-intra-node-hostport-7d656d7bb9-gs652                          
pod-to-b-intra-node-nodeport-569d7c647-lfbkk                           
pod-to-c-intra-node-proxy-to-proxy-policy-79584b8d68-mcfzv             
coredns-69b675786c-mj2l2                                               
pod-to-external-fqdn-allow-google-cnp-5ff4986c89-c68ch                 
pod-to-a-57695cc7ff-csdz5                                              
pod-to-a-allowed-cnp-7b6d5ff99f-sv8xn                                  
pod-to-b-multi-node-clusterip-fdf45bbbc-lknkm                          
pod-to-b-multi-node-hostport-57fc8854f5-vv2gs                          
pod-to-external-1111-56548587dc-v2rdq                                  
Cilium agent 'cilium-lww8g': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 58 Failed 0
Cilium agent 'cilium-sv9d2': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0

Standard Error

Click to show. 
11:05:22 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
11:05:22 STEP: Ensuring the namespace kube-system exists
11:05:22 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
11:05:22 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
11:05:22 STEP: Installing Cilium
11:05:23 STEP: Waiting for Cilium to become ready
11:06:54 STEP: Restarting unmanaged pods coredns-69b675786c-x6kpf in namespace kube-system
11:07:11 STEP: Validating if Kubernetes DNS is deployed
11:07:11 STEP: Checking if deployment is ready
11:07:11 STEP: Checking if kube-dns service is plumbed correctly
11:07:11 STEP: Checking if pods have identity
11:07:11 STEP: Checking if DNS can resolve
11:07:15 STEP: Kubernetes DNS is up and operational
11:07:15 STEP: Validating Cilium Installation
11:07:15 STEP: Performing Cilium controllers preflight check
11:07:15 STEP: Performing Cilium status preflight check
11:07:15 STEP: Performing Cilium health check
11:07:15 STEP: Checking whether host EP regenerated
11:07:23 STEP: Performing Cilium service preflight check
11:07:23 STEP: Performing K8s service preflight check
11:07:29 STEP: Waiting for cilium-operator to be ready
11:07:29 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
11:07:29 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
11:07:29 STEP: Making sure all endpoints are in ready state
11:07:32 STEP: WaitforPods(namespace="default", filter="")
11:07:54 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-05-08T11:07:54Z====
11:07:54 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-05-08T11:05:55.394368956Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.92 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-05-08T11:05:55.282955726Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.80 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
11:07:54 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-nhpt5                                     0/1     Running   0          93m     10.0.0.205      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-qbd4g                                  1/1     Running   0          93m     10.0.0.143      k8s1   <none>           <none>
	 default             echo-a-d576c5f8b-sw58x                                       1/1     Running   0          27s     10.0.0.143      k8s2   <none>           <none>
	 default             echo-b-787dc99778-99h56                                      1/1     Running   0          27s     10.0.0.223      k8s2   <none>           <none>
	 default             echo-b-host-675cd8cfff-22wsb                                 1/1     Running   0          27s     192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-58cc9b67d9-q2s8l                                      1/1     Running   0          27s     10.0.0.16       k8s2   <none>           <none>
	 default             echo-c-host-79c7cc6568-98dhr                                 1/1     Running   0          27s     192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6fd884bcf7-hl6lp              1/1     Running   0          24s     192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-79f7df47b9-qx5fn               1/1     Running   0          24s     192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-57695cc7ff-csdz5                                    1/1     Running   0          27s     10.0.0.177      k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-7b6d5ff99f-sv8xn                        1/1     Running   0          26s     10.0.0.64       k8s2   <none>           <none>
	 default             pod-to-a-denied-cnp-6887b57579-dxjr7                         1/1     Running   0          26s     10.0.0.30       k8s2   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-74f796f479-27ncd     2/2     Running   0          26s     10.0.0.123      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5b5f7d94d8-2h8wv     2/2     Running   0          26s     10.0.1.63       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-7d656d7bb9-gs652                1/1     Running   0          24s     10.0.0.60       k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-569d7c647-lfbkk                 1/1     Running   0          23s     10.0.0.150      k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-fdf45bbbc-lknkm                1/1     Running   0          25s     10.0.1.230      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-64b6cbdd49-vkgwp                1/1     Running   0          24s     10.0.1.166      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-57fc8854f5-vv2gs                1/1     Running   0          24s     10.0.1.156      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-54446bdbb9-k6kl5                1/1     Running   0          23s     10.0.1.93       k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-5f95cf647c-9pf85    2/2     Running   0          25s     10.0.0.65       k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-79584b8d68-mcfzv   2/2     Running   0          25s     10.0.0.253      k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-74749c8794-sz95m    2/2     Running   0          25s     10.0.1.102      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-9f68b7595-pxwzh    2/2     Running   0          25s     10.0.1.32       k8s1   <none>           <none>
	 default             pod-to-external-1111-56548587dc-v2rdq                        1/1     Running   0          27s     10.0.0.197      k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-5ff4986c89-c68ch       1/1     Running   0          26s     10.0.0.187      k8s2   <none>           <none>
	 kube-system         cilium-lww8g                                                 1/1     Running   1          2m36s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-ff85cfc49-brplf                              1/1     Running   0          2m36s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-ff85cfc49-rtxcp                              1/1     Running   0          2m36s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-sv9d2                                                 1/1     Running   1          2m36s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-69b675786c-mj2l2                                     1/1     Running   0          65s     10.0.1.100      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   1          100m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   1          100m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   1          100m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-6d5h5                                             1/1     Running   0          100m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-vvq2v                                             1/1     Running   0          94m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   2          100m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-5z9nb                                           1/1     Running   0          93m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-62fs5                                           1/1     Running   0          93m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-hrmpw                                         1/1     Running   0          94m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-khfzv                                         1/1     Running   0          94m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-lww8g cilium-sv9d2]
cmd: kubectl exec -n kube-system cilium-lww8g -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 72         Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                            ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                     
	                                                            k8s:node-role.kubernetes.io/master                                                                            
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                   
	                                                            reserved:host                                                                                                 
	 223        Disabled           Disabled          14485      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::1d4   10.0.1.230   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                       
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                                                        
	 267        Disabled           Disabled          15874      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::1e5   10.0.1.100   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 825        Disabled           Enabled           15327      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::15f   10.0.1.63    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                       
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                                              
	 1008       Disabled           Disabled          5384       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::172   10.0.1.102   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                       
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                                             
	 1185       Disabled           Disabled          40649      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::12d   10.0.1.166   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                       
	                                                            k8s:name=pod-to-b-multi-node-headless                                                                         
	 1887       Disabled           Enabled           41972      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::164   10.0.1.32    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                       
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                                            
	 1897       Disabled           Disabled          53286      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::1f3   10.0.1.93    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                       
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                                         
	 1975       Disabled           Disabled          4          reserved:health                                                              fd02::1bf   10.0.1.113   ready   
	 2567       Disabled           Disabled          19120      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default       fd02::17d   10.0.1.156   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                               
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                       
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                                         
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-sv9d2 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                              IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                   
	 147        Disabled           Disabled          6826       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::18   10.0.0.177   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-a                                                                                        
	 468        Disabled           Enabled           58265      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::f1   10.0.0.123   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                                         
	 666        Disabled           Disabled          9658       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::14   10.0.0.60    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                                    
	 755        Disabled           Disabled          17164      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::74   10.0.0.223   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=echo-b                                                                                          
	 781        Disabled           Enabled           33431      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::7d   10.0.0.64    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-a-allowed-cnp                                                                            
	 891        Disabled           Enabled           31385      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::24   10.0.0.253   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                                       
	 1020       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                       ready   
	                                                            reserved:host                                                                                            
	 1164       Disabled           Disabled          2877       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::34   10.0.0.143   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=echo-a                                                                                          
	 1194       Disabled           Disabled          28989      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::aa   10.0.0.65    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                                        
	 1468       Disabled           Disabled          4          reserved:health                                                          fd02::5c   10.0.0.208   ready   
	 1737       Enabled            Disabled          63219      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::bc   10.0.0.16    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=echo-c                                                                                          
	 2885       Disabled           Enabled           41198      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::23   10.0.0.187   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                                           
	 2998       Disabled           Disabled          56363      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::43   10.0.0.150   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                                    
	 3028       Disabled           Disabled          8094       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::3e   10.0.0.197   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-external-1111                                                                            
	 3339       Disabled           Enabled           18144      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default   fd02::1b   10.0.0.30    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:name=pod-to-a-denied-cnp                                                                             
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
11:08:10 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
11:08:54 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|668a92f9_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9//2619/artifact/63457bb5_K8sPolicyTest_Multi-node_policy_test_with_L7_policy_using_connectivity-check_to_check_datapath.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9//2619/artifact/668a92f9_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9//2619/artifact/test_results_Cilium-PR-K8s-1.21-kernel-4.9_2619_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/2619/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue May 8, 2023
Merged
@maintainer-s-little-helper
Copy link
Author

PR #25182 hit this flake with 92.92% similarity:

Click to show.

Test Name

K8sConformance Portmap Chaining Check connectivity-check compliance with portmap chaining

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-05-08T10:33:59.898781866Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.187 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-05-08T10:34:03.248444896Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.142 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
/home/jenkins/workspace/Cilium-PR-K8s-1.17-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:425

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️  Found "2023-05-08T10:33:59.898781866Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.187 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
⚠️  Found "2023-05-08T10:34:03.248444896Z level=error msg=\"Command execution failed\" cmd=\"[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.142 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]\" error=\"exit status 1\" subsys=datapath-loader" in logs 1 times
Number of "context deadline exceeded" in logs: 4
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 462
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Command execution failed
100:\tfrom all lookup local
+ true
+ '[' false = true ']'
10:\tfrom all fwmark 0xa00/0xf00 lookup 2005
Cilium pods: [cilium-5j92b cilium-xmxgl]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::echo-c default::pod-to-a-allowed-cnp default::pod-to-a-denied-cnp default::pod-to-a-intra-node-proxy-egress-policy default::pod-to-a-multi-node-proxy-egress-policy default::pod-to-c-intra-node-proxy-to-proxy-policy default::pod-to-c-multi-node-proxy-to-proxy-policy default::pod-to-external-fqdn-allow-google-cnp 
Endpoint Policy Enforcement:
Pod                                                          Ingress   Egress
echo-b-6c864d75d7-tzxzh                                                
echo-c-688dcdfd44-lnqc7                                                
pod-to-a-multi-node-proxy-egress-policy-5466dc57db-q928l               
pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-gc9xh              
coredns-5b7c49d4d8-xcs5h                                               
pod-to-a-denied-cnp-64b85f46b8-74d5f                                   
pod-to-b-intra-node-hostport-5c58cb8758-fdjr2                          
pod-to-b-intra-node-nodeport-7b59546c64-jtd98                          
pod-to-b-multi-node-clusterip-6c74fdb4b8-xr6cx                         
pod-to-b-multi-node-nodeport-658b99d8-b928r                            
pod-to-external-1111-694c4c87dd-kqrd9                                  
pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-fxt4d                 
echo-a-6954f488f5-pjfhn                                                
pod-to-a-57c7db48fd-sxnbm                                              
pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-cvkhh               
pod-to-b-multi-node-hostport-869554778-xgttl                           
pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-m4x27              
pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-zsdj7             
pod-to-a-allowed-cnp-548f548798-cv6jn                                  
pod-to-b-multi-node-headless-595bf6dfb-m7gq8                           
pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-9nlxr             
Cilium agent 'cilium-5j92b': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 78 Failed 0
Cilium agent 'cilium-xmxgl': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 58 Failed 0

Standard Error

Click to show. 
10:33:42 STEP: Running BeforeAll block for EntireTestsuite K8sConformance Portmap Chaining
10:33:42 STEP: Ensuring the namespace kube-system exists
10:33:42 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
10:33:42 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
10:33:43 STEP: Installing Cilium
10:33:43 STEP: Waiting for Cilium to become ready
10:34:27 STEP: Restarting unmanaged pods coredns-5b7c49d4d8-7t7q6 in namespace kube-system
10:34:44 STEP: Validating if Kubernetes DNS is deployed
10:34:44 STEP: Checking if deployment is ready
10:34:45 STEP: Checking if kube-dns service is plumbed correctly
10:34:45 STEP: Checking if pods have identity
10:34:45 STEP: Checking if DNS can resolve
10:34:48 STEP: Kubernetes DNS is up and operational
10:34:48 STEP: Validating Cilium Installation
10:34:48 STEP: Performing Cilium status preflight check
10:34:48 STEP: Performing Cilium health check
10:34:48 STEP: Performing Cilium controllers preflight check
10:34:48 STEP: Checking whether host EP regenerated
10:34:56 STEP: Performing Cilium service preflight check
10:34:56 STEP: Performing K8s service preflight check
10:34:57 STEP: Cilium is not ready yet: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-xmxgl': Exitcode: 1 
Err: exit status 1
Stdout:
 	 
Stderr:
 	 Error: Cannot get status/probe: Put "http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe": dial unix /var/run/cilium/health.sock: connect: no such file or directory
	 
	 command terminated with exit code 1
	 

10:34:57 STEP: Performing Cilium controllers preflight check
10:34:57 STEP: Performing Cilium health check
10:34:57 STEP: Performing Cilium status preflight check
10:34:57 STEP: Checking whether host EP regenerated
10:35:04 STEP: Performing Cilium service preflight check
10:35:04 STEP: Performing K8s service preflight check
10:35:05 STEP: Performing Cilium controllers preflight check
10:35:05 STEP: Performing Cilium health check
10:35:05 STEP: Performing Cilium status preflight check
10:35:05 STEP: Checking whether host EP regenerated
10:35:13 STEP: Performing Cilium service preflight check
10:35:13 STEP: Performing K8s service preflight check
10:35:19 STEP: Waiting for cilium-operator to be ready
10:35:19 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
10:35:19 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
10:35:19 STEP: Making sure all endpoints are in ready state
10:35:22 STEP: WaitforPods(namespace="default", filter="")
10:35:46 STEP: WaitforPods(namespace="default", filter="") => <nil>
=== Test Finished at 2023-05-08T10:35:46Z====
10:35:46 STEP: Running JustAfterEach block for EntireTestsuite K8sConformance Portmap Chaining
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-05-08T10:33:59.898781866Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.187 fd04::11 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
2023-05-08T10:34:03.248444896Z level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.1.142 fd04::12 tunnel vxlan 8472 <nil> cilium_host cilium_net 1500 false false false /run/cilium/cgroupv2 /sys/fs/bpf false true v3 3 false true]" error="exit status 1" subsys=datapath-loader
===================== TEST FAILED =====================
10:35:47 STEP: Running AfterFailed block for EntireTestsuite K8sConformance Portmap Chaining
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                                                         READY   STATUS    RESTARTS   AGE    IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-7fd557d749-fmj7d                                     0/1     Running   0          46m    10.0.0.76       k8s1   <none>           <none>
	 cilium-monitoring   prometheus-d87f8f984-54b5s                                   1/1     Running   0          46m    10.0.0.59       k8s1   <none>           <none>
	 default             echo-a-6954f488f5-pjfhn                                      1/1     Running   0          30s    10.0.1.171      k8s2   <none>           <none>
	 default             echo-b-6c864d75d7-tzxzh                                      1/1     Running   0          30s    10.0.1.95       k8s2   <none>           <none>
	 default             echo-b-host-fdb854bd-8hp29                                   1/1     Running   0          30s    192.168.56.12   k8s2   <none>           <none>
	 default             echo-c-688dcdfd44-lnqc7                                      1/1     Running   0          30s    10.0.1.48       k8s2   <none>           <none>
	 default             echo-c-host-5d6b79658c-thlzd                                 1/1     Running   0          30s    192.168.56.12   k8s2   <none>           <none>
	 default             host-to-b-multi-node-clusterip-6868bbd4f8-rjqcj              1/1     Running   0          28s    192.168.56.11   k8s1   <none>           <none>
	 default             host-to-b-multi-node-headless-69c9465576-jlqzr               1/1     Running   0          28s    192.168.56.11   k8s1   <none>           <none>
	 default             pod-to-a-57c7db48fd-sxnbm                                    1/1     Running   0          30s    10.0.1.224      k8s2   <none>           <none>
	 default             pod-to-a-allowed-cnp-548f548798-cv6jn                        1/1     Running   0          30s    10.0.0.205      k8s1   <none>           <none>
	 default             pod-to-a-denied-cnp-64b85f46b8-74d5f                         1/1     Running   0          30s    10.0.1.192      k8s2   <none>           <none>
	 default             pod-to-a-intra-node-proxy-egress-policy-7f46cf4857-cvkhh     2/2     Running   0          29s    10.0.1.121      k8s2   <none>           <none>
	 default             pod-to-a-multi-node-proxy-egress-policy-5466dc57db-q928l     2/2     Running   0          29s    10.0.0.52       k8s1   <none>           <none>
	 default             pod-to-b-intra-node-hostport-5c58cb8758-fdjr2                1/1     Running   0          27s    10.0.1.50       k8s2   <none>           <none>
	 default             pod-to-b-intra-node-nodeport-7b59546c64-jtd98                1/1     Running   0          27s    10.0.1.49       k8s2   <none>           <none>
	 default             pod-to-b-multi-node-clusterip-6c74fdb4b8-xr6cx               1/1     Running   0          28s    10.0.0.164      k8s1   <none>           <none>
	 default             pod-to-b-multi-node-headless-595bf6dfb-m7gq8                 1/1     Running   0          28s    10.0.0.19       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-hostport-869554778-xgttl                 1/1     Running   0          27s    10.0.0.89       k8s1   <none>           <none>
	 default             pod-to-b-multi-node-nodeport-658b99d8-b928r                  1/1     Running   0          27s    10.0.0.176      k8s1   <none>           <none>
	 default             pod-to-c-intra-node-proxy-ingress-policy-64c77b5979-gc9xh    2/2     Running   0          29s    10.0.1.90       k8s2   <none>           <none>
	 default             pod-to-c-intra-node-proxy-to-proxy-policy-5b6c8c6ddd-9nlxr   2/2     Running   0          28s    10.0.1.25       k8s2   <none>           <none>
	 default             pod-to-c-multi-node-proxy-ingress-policy-749956dc6d-m4x27    2/2     Running   0          29s    10.0.0.135      k8s1   <none>           <none>
	 default             pod-to-c-multi-node-proxy-to-proxy-policy-75dc998c79-zsdj7   2/2     Running   0          28s    10.0.0.206      k8s1   <none>           <none>
	 default             pod-to-external-1111-694c4c87dd-kqrd9                        1/1     Running   0          30s    10.0.1.118      k8s2   <none>           <none>
	 default             pod-to-external-fqdn-allow-google-cnp-7b6c5d747b-fxt4d       1/1     Running   0          29s    10.0.1.76       k8s2   <none>           <none>
	 kube-system         cilium-5j92b                                                 1/1     Running   1          2m9s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-7746c5494c-gnpmh                             1/1     Running   0          2m9s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7746c5494c-rd8js                             1/1     Running   0          2m9s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-xmxgl                                                 1/1     Running   1          2m9s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-5b7c49d4d8-xcs5h                                     1/1     Running   0          85s    10.0.1.252      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                                                    1/1     Running   1          55m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                                          1/1     Running   1          55m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1                                 1/1     Running   3          55m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-9whsq                                             1/1     Running   0          47m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-kktmq                                             1/1     Running   0          55m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                                          1/1     Running   2          55m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-hl5b7                                           1/1     Running   0          46m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-kt7gw                                           1/1     Running   0          46m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-4czrf                                         1/1     Running   0          47m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-z48nc                                         1/1     Running   0          47m    192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-5j92b cilium-xmxgl]
cmd: kubectl exec -n kube-system cilium-5j92b -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                
	 54         Disabled           Enabled           25738      k8s:io.cilium.k8s.policy.cluster=default             fd02::1c1   10.0.1.76    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-fqdn-allow-google-cnp                                        
	 222        Disabled           Disabled          9758       k8s:io.cilium.k8s.policy.cluster=default             fd02::149   10.0.1.118   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-external-1111                                                         
	 401        Disabled           Disabled          3342       k8s:io.cilium.k8s.policy.cluster=default             fd02::1b7   10.0.1.252   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                       
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                           
	                                                            k8s:k8s-app=kube-dns                                                                  
	 643        Disabled           Disabled          4          reserved:health                                      fd02::165   10.0.1.172   ready   
	 721        Disabled           Enabled           20630      k8s:io.cilium.k8s.policy.cluster=default             fd02::1ec   10.0.1.121   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-intra-node-proxy-egress-policy                                      
	 752        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                    ready   
	                                                            reserved:host                                                                         
	 838        Disabled           Disabled          21822      k8s:io.cilium.k8s.policy.cluster=default             fd02::163   10.0.1.90    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-intra-node-proxy-ingress-policy                                     
	 964        Disabled           Disabled          41334      k8s:io.cilium.k8s.policy.cluster=default             fd02::193   10.0.1.49    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-intra-node-nodeport                                                 
	 1302       Disabled           Disabled          6195       k8s:io.cilium.k8s.policy.cluster=default             fd02::14b   10.0.1.50    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-b-intra-node-hostport                                                 
	 1417       Disabled           Disabled          33984      k8s:io.cilium.k8s.policy.cluster=default             fd02::150   10.0.1.171   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-a                                                                       
	 1486       Disabled           Enabled           53893      k8s:io.cilium.k8s.policy.cluster=default             fd02::1a0   10.0.1.25    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-c-intra-node-proxy-to-proxy-policy                                    
	 1736       Enabled            Disabled          12650      k8s:io.cilium.k8s.policy.cluster=default             fd02::1eb   10.0.1.48    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-c                                                                       
	 2257       Disabled           Disabled          19299      k8s:io.cilium.k8s.policy.cluster=default             fd02::1ae   10.0.1.224   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a                                                                     
	 2793       Disabled           Enabled           36532      k8s:io.cilium.k8s.policy.cluster=default             fd02::1e4   10.0.1.192   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=pod-to-a-denied-cnp                                                          
	 3141       Disabled           Disabled          57650      k8s:io.cilium.k8s.policy.cluster=default             fd02::1f9   10.0.1.95    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                       
	                                                            k8s:io.kubernetes.pod.namespace=default                                               
	                                                            k8s:name=echo-b                                                                       
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-xmxgl -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                          IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                               
	 355        Disabled           Enabled           25239      k8s:io.cilium.k8s.policy.cluster=default             fd02::7f   10.0.0.205   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-allowed-cnp                                                        
	 460        Disabled           Disabled          52085      k8s:io.cilium.k8s.policy.cluster=default             fd02::31   10.0.0.89    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-hostport                                                
	 950        Disabled           Disabled          38834      k8s:io.cilium.k8s.policy.cluster=default             fd02::fe   10.0.0.135   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-multi-node-proxy-ingress-policy                                    
	 1310       Disabled           Disabled          4          reserved:health                                      fd02::58   10.0.0.51    ready   
	 1790       Disabled           Enabled           54725      k8s:io.cilium.k8s.policy.cluster=default             fd02::41   10.0.0.206   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-c-multi-node-proxy-to-proxy-policy                                   
	 2246       Disabled           Disabled          36958      k8s:io.cilium.k8s.policy.cluster=default             fd02::bc   10.0.0.19    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-headless                                                
	 2430       Disabled           Enabled           32314      k8s:io.cilium.k8s.policy.cluster=default             fd02::5    10.0.0.52    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-a-multi-node-proxy-egress-policy                                     
	 2590       Disabled           Disabled          3013       k8s:io.cilium.k8s.policy.cluster=default             fd02::6e   10.0.0.176   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-nodeport                                                
	 2887       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                   ready   
	                                                            k8s:node-role.kubernetes.io/master                                                   
	                                                            reserved:host                                                                        
	 3782       Disabled           Disabled          3189       k8s:io.cilium.k8s.policy.cluster=default             fd02::39   10.0.0.164   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                      
	                                                            k8s:io.kubernetes.pod.namespace=default                                              
	                                                            k8s:name=pod-to-b-multi-node-clusterip                                               
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
10:36:04 STEP: Running AfterEach for block EntireTestsuite K8sConformance Portmap Chaining
10:36:50 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|b1c197d9_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1347/artifact/b1c197d9_K8sConformance_Portmap_Chaining_Check_connectivity-check_compliance_with_portmap_chaining.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9//1347/artifact/test_results_Cilium-PR-K8s-1.17-kernel-4.9_1347_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9/1347/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@github-actions
Copy link

github-actions bot commented Jul 8, 2023

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jul 8, 2023
@github-actions
Copy link

This issue has not seen any activity since it was marked stale.
Closing.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gandro