Flake: external gateway LANE: [FAIL] External Gateway With Admin Policy Based External Route CRs BFD e2e non-vxlan external gateway through a dynamic hop Should validate TCP/UDP connectivity to an external gateway's loopback address via a pod with a dynamic hop [It] TCP ipv4

[tssurya]

Seen here: https://github.com/ovn-org/ovn-kubernetes/actions/runs/7813480320/job/21313562521?pr=4100

2024-02-07T11:45:37.6421280Z �[0mExternal Gateway �[38;5;243mWith Admin Policy Based External Route CRs �[0mBFD �[38;5;243me2e non-vxlan external gateway through a dynamic hop �[0mShould validate TCP/UDP connectivity to an external gateway's loopback address via a pod with a dynamic hop �[0m�[1mTCP ipv4�[0m
2024-02-07T11:45:37.6424588Z �[38;5;243m/home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/external_gateways.go:2030�[0m
2024-02-07T11:45:37.6426072Z   �[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m@ 02/07/24 11:45:37.641�[0m
2024-02-07T11:45:37.6427512Z   Feb  7 11:45:37.641: INFO: >>> kubeConfig: /home/runner/ovn.conf
2024-02-07T11:45:37.6429235Z   �[1mSTEP:�[0m Building a namespace api object, basename externalgw-pod-novxlan �[38;5;243m@ 02/07/24 11:45:37.642�[0m
2024-02-07T11:45:37.6459668Z   Feb  7 11:45:37.645: INFO: Skipping waiting for service account
2024-02-07T11:45:37.6594003Z   �[1mSTEP:�[0m Creating the gateway containers for the icmp test �[38;5;243m@ 02/07/24 11:45:37.658�[0m
2024-02-07T11:45:38.2250608Z   Feb  7 11:45:38.224: INFO: target ips are [10.249.10.1 10.249.10.2 10.249.10.3 10.249.10.4 10.249.10.5 10.249.10.6 10.249.10.7 10.249.10.8 10.249.10.9 10.249.10.10 10.249.10.11 10.249.10.12 10.249.10.13 10.249.10.14 10.249.10.15 10.249.10.16 10.249.10.17 10.249.10.18 10.249.10.19 10.249.10.20]
2024-02-07T11:45:38.2254372Z   Feb  7 11:45:38.224: INFO: target ipsv6 are [fc00:f853:ccd:e794::1 fc00:f853:ccd:e794::2 fc00:f853:ccd:e794::3 fc00:f853:ccd:e794::4 fc00:f853:ccd:e794::5 fc00:f853:ccd:e794::6 fc00:f853:ccd:e794::7 fc00:f853:ccd:e794::8 fc00:f853:ccd:e794::9 fc00:f853:ccd:e794::10 fc00:f853:ccd:e794::11 fc00:f853:ccd:e794::12 fc00:f853:ccd:e794::13 fc00:f853:ccd:e794::14 fc00:f853:ccd:e794::15 fc00:f853:ccd:e794::16 fc00:f853:ccd:e794::17 fc00:f853:ccd:e794::18 fc00:f853:ccd:e794::19 fc00:f853:ccd:e794::20]
2024-02-07T11:45:38.2505123Z   Feb  7 11:45:38.250: INFO: the pod side node is ovn-control-plane and the source node ip is 172.19.0.3 - fc00:f853:ccd:e798::3
2024-02-07T11:45:38.2506408Z   �[1mSTEP:�[0m Creating the source pod to reach the destination ips from �[38;5;243m@ 02/07/24 11:45:38.25�[0m
2024-02-07T11:45:38.2568041Z   W0207 11:45:38.254962   47918 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "e2e-exgw-src-ping-pod-container" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "e2e-exgw-src-ping-pod-container" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "e2e-exgw-src-ping-pod-container" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "e2e-exgw-src-ping-pod-container" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
2024-02-07T11:45:40.2643946Z   Feb  7 11:45:40.264: INFO: the pod source pod ip(s) are 10.244.0.95 - 
2024-02-07T11:45:40.2644957Z   �[1mSTEP:�[0m Setting up the destination ips to ex-gw-container1 �[38;5;243m@ 02/07/24 11:45:40.264�[0m
2024-02-07T11:45:41.2680848Z   �[1mSTEP:�[0m Adding a route from ex-gw-container1 to the src pod �[38;5;243m@ 02/07/24 11:45:41.267�[0m
2024-02-07T11:45:41.3281633Z   �[1mSTEP:�[0m Setting up the listeners on the gateway �[38;5;243m@ 02/07/24 11:45:41.327�[0m
2024-02-07T11:45:45.4296441Z   �[1mSTEP:�[0m Setting up the destination ips to ex-gw-container2 �[38;5;243m@ 02/07/24 11:45:45.429�[0m
2024-02-07T11:45:46.4197235Z   �[1mSTEP:�[0m Adding a route from ex-gw-container2 to the src pod �[38;5;243m@ 02/07/24 11:45:46.419�[0m
2024-02-07T11:45:46.4755124Z   �[1mSTEP:�[0m Setting up the listeners on the gateway �[38;5;243m@ 02/07/24 11:45:46.475�[0m
2024-02-07T11:46:01.5052274Z   �[1mSTEP:�[0m Create the external route policy with dynamic hops to manage the src app pod namespace �[38;5;243m@ 02/07/24 11:46:01.504�[0m
2024-02-07T11:46:01.5112702Z   W0207 11:46:01.510083   47918 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "e2e-gateway-pod1-container" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "e2e-gateway-pod1-container" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "e2e-gateway-pod1-container" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "e2e-gateway-pod1-container" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
2024-02-07T11:46:03.5270544Z   W0207 11:46:03.525798   47918 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "e2e-gateway-pod2-container" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "e2e-gateway-pod2-container" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "e2e-gateway-pod2-container" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "e2e-gateway-pod2-container" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
2024-02-07T11:46:05.5344596Z   Feb  7 11:46:05.533: INFO: Annotating the external gateway pod with annotation '[annotate pods e2e-gateway-pod1 --overwrite k8s.v1.cni.cncf.io/network-status=[{"name":"foo","interface":"net1","ips":["172.19.0.5"],"mac":"01:23:45:67:89:10","dns":{}}]]'
2024-02-07T11:46:05.5349956Z   Feb  7 11:46:05.534: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf --namespace=exgw-bfd-serving-6033 annotate pods e2e-gateway-pod1 --overwrite k8s.v1.cni.cncf.io/network-status=[{"name":"foo","interface":"net1","ips":["172.19.0.5"],"mac":"01:23:45:67:89:10","dns":{}}]'
2024-02-07T11:46:05.5886199Z   Feb  7 11:46:05.588: INFO: stderr: ""
2024-02-07T11:46:05.5887412Z   Feb  7 11:46:05.588: INFO: stdout: "pod/e2e-gateway-pod1 annotated\n"
2024-02-07T11:46:05.5893082Z   Feb  7 11:46:05.588: INFO: Annotating the external gateway pod with annotation '[annotate pods e2e-gateway-pod2 --overwrite k8s.v1.cni.cncf.io/network-status=[{"name":"foo","interface":"net1","ips":["172.19.0.6"],"mac":"01:23:45:67:89:10","dns":{}}]]'
2024-02-07T11:46:05.5897449Z   Feb  7 11:46:05.588: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf --namespace=exgw-bfd-serving-6033 annotate pods e2e-gateway-pod2 --overwrite k8s.v1.cni.cncf.io/network-status=[{"name":"foo","interface":"net1","ips":["172.19.0.6"],"mac":"01:23:45:67:89:10","dns":{}}]'
2024-02-07T11:46:05.6440647Z   Feb  7 11:46:05.643: INFO: stderr: ""
2024-02-07T11:46:05.6442417Z   Feb  7 11:46:05.643: INFO: stdout: "pod/e2e-gateway-pod2 annotated\n"
2024-02-07T11:46:05.6443734Z   Feb  7 11:46:05.644: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf create -f -'
2024-02-07T11:46:05.7531202Z   Feb  7 11:46:05.751: INFO: stderr: ""
2024-02-07T11:46:05.7533412Z   Feb  7 11:46:05.751: INFO: stdout: "adminpolicybasedexternalroute.k8s.ovn.org/default-route-policy created\n"
2024-02-07T11:46:05.7536716Z   Feb  7 11:46:05.752: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf get apbexternalroute default-route-policy -ojsonpath={.status.messages[-1:]}'
2024-02-07T11:46:05.8046089Z   Feb  7 11:46:05.804: INFO: stderr: ""
2024-02-07T11:46:05.8047715Z   Feb  7 11:46:05.804: INFO: stdout: "ovn-control-plane: configured external gateway IPs: 172.19.0.5,172.19.0.6"
2024-02-07T11:46:05.8051002Z   Feb  7 11:46:05.804: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf get apbexternalroute default-route-policy -ojsonpath={.status.status}'
2024-02-07T11:46:05.8525384Z   Feb  7 11:46:05.852: INFO: stderr: ""
2024-02-07T11:46:05.8526611Z   Feb  7 11:46:05.852: INFO: stdout: "Success"
2024-02-07T11:46:45.1073527Z   Feb  7 11:46:45.107: INFO: Hostname for ex-gw-container1 is e61e723b8e8f
2024-02-07T11:46:45.1073996Z 
2024-02-07T11:46:45.1674792Z   Feb  7 11:46:45.167: INFO: Hostname for ex-gw-container2 is 5dad5044df11
2024-02-07T11:46:45.1675544Z 
2024-02-07T11:46:45.1676165Z   Feb  7 11:46:45.167: INFO: Expected hostnames are map[5dad5044df11:{} e61e723b8e8f:{}]
2024-02-07T11:46:45.1679242Z   Feb  7 11:46:45.167: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf --namespace=externalgw-pod-novxlan-2724 exec e2e-exgw-src-ping-pod -- bash -c echo | nc -w 1 10.249.10.1 91'
2024-02-07T11:46:45.2813677Z   Feb  7 11:46:45.281: INFO: stderr: ""
2024-02-07T11:46:45.2814756Z   Feb  7 11:46:45.281: INFO: stdout: "e61e723b8e8f\n"
2024-02-07T11:46:45.2817850Z   Feb  7 11:46:45.281: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf --namespace=externalgw-pod-novxlan-2724 exec e2e-exgw-src-ping-pod -- bash -c echo | nc -w 1 10.249.10.1 91'
2024-02-07T11:46:45.3889608Z   Feb  7 11:46:45.388: INFO: stderr: ""
2024-02-07T11:46:45.3890391Z   Feb  7 11:46:45.388: INFO: stdout: "5dad5044df11\n"
2024-02-07T11:46:45.3891380Z   Feb  7 11:46:45.388: INFO: Received hostnames for protocol tcp are map[5dad5044df11:{} e61e723b8e8f:{}] 
2024-02-07T11:46:45.3893079Z   �[1mSTEP:�[0m Deleting one container �[38;5;243m@ 02/07/24 11:46:45.388�[0m
2024-02-07T11:46:45.5732371Z   �[1mSTEP:�[0m Waiting for BFD to sync �[38;5;243m@ 02/07/24 11:46:45.572�[0m
2024-02-07T11:46:48.6318435Z   Feb  7 11:46:48.631: INFO: Hostname for ex-gw-container1 is e61e723b8e8f
2024-02-07T11:46:48.6319157Z 
2024-02-07T11:46:48.6319907Z   �[1mSTEP:�[0m Checking hostname multiple times �[38;5;243m@ 02/07/24 11:46:48.631�[0m
2024-02-07T11:46:48.6321660Z   Feb  7 11:46:48.631: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf --namespace=externalgw-pod-novxlan-2724 exec e2e-exgw-src-ping-pod -- bash -c echo | nc -w 1 10.249.10.1 91'
2024-02-07T11:46:49.7431854Z   Feb  7 11:46:49.742: INFO: rc: 1
2024-02-07T11:46:49.7432870Z   Feb  7 11:46:49.743: INFO: Unexpected error: failed to reach 10.249.10.1 (tcp): 
2024-02-07T11:46:49.7433726Z       <exec.CodeExitError>: 
2024-02-07T11:46:49.7436135Z       error running /usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf --namespace=externalgw-pod-novxlan-2724 exec e2e-exgw-src-ping-pod -- bash -c echo | nc -w 1 10.249.10.1 91:
2024-02-07T11:46:49.7437919Z       Command stdout:
2024-02-07T11:46:49.7438192Z       
2024-02-07T11:46:49.7438420Z       stderr:
2024-02-07T11:46:49.7438725Z       command terminated with exit code 1
2024-02-07T11:46:49.7439073Z       
2024-02-07T11:46:49.7439542Z       error:
2024-02-07T11:46:49.7439792Z       exit status 1
2024-02-07T11:46:49.7440053Z       {
2024-02-07T11:46:49.7440465Z           Err: <*errors.errorString | 0xc001467050>{
2024-02-07T11:46:49.7443166Z               s: "error running /usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf --namespace=externalgw-pod-novxlan-2724 exec e2e-exgw-src-ping-pod -- bash -c echo | nc -w 1 10.249.10.1 91:\nCommand stdout:\n\nstderr:\ncommand terminated with exit code 1\n\nerror:\nexit status 1",
2024-02-07T11:46:49.7444698Z           },
2024-02-07T11:46:49.7445042Z           Code: 1,
2024-02-07T11:46:49.7445281Z       }
2024-02-07T11:46:49.7446385Z   �[38;5;9m[FAILED]�[0m in [It] - /home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/external_gateways.go:3139 �[38;5;243m@ 02/07/24 11:46:49.743�[0m
2024-02-07T11:46:50.4076261Z   Feb  7 11:46:50.407: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:41103 --kubeconfig=/home/runner/ovn.conf delete apbexternalroute default-route-policy'
2024-02-07T11:46:50.4790003Z   Feb  7 11:46:50.468: INFO: stderr: ""
2024-02-07T11:46:50.4793665Z   Feb  7 11:46:50.468: INFO: stdout: "adminpolicybasedexternalroute.k8s.ovn.org \"default-route-policy\" deleted\n"
2024-02-07T11:46:50.4795892Z   �[1mSTEP:�[0m Deleting the gateway containers �[38;5;243m@ 02/07/24 11:46:50.468�[0m
2024-02-07T11:46:50.6863722Z   �[1mSTEP:�[0m dump namespace information after failure �[38;5;243m@ 02/07/24 11:46:50.686�[0m
2024-02-07T11:46:50.6865862Z   �[1mSTEP:�[0m Collecting events from namespace "externalgw-pod-novxlan-2724". �[38;5;243m@ 02/07/24 11:46:50.686�[0m
2024-02-07T11:46:50.6889353Z   �[1mSTEP:�[0m Found 3 events. �[38;5;243m@ 02/07/24 11:46:50.688�[0m
2024-02-07T11:46:50.6891986Z   Feb  7 11:46:50.688: INFO: At 2024-02-07 11:45:38 +0000 UTC - event for e2e-exgw-src-ping-pod: {kubelet ovn-control-plane} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.26" already present on machine

[tssurya]

Seen here: #4100

[flavio-fernandes]

Seen here: https://github.com/ovn-org/ovn-kubernetes/actions/runs/9361290864/job/25768583286
PR affected: #4353

[flavio-fernandes]

Also seen here: https://github.com/ovn-org/ovn-kubernetes/actions/runs/9391848862/job/25865709365?pr=4353

@jordigilh @trozet : maybe these lanes don't matter and we should disable them?

[jordigilh]

Can we disable them for now? I don't have time to look at them right now (and don't know when I can commit to it) but disabling the whole line seems a bit too drastic for a few tests. Unless, of course, there are more flaky tests in this lane aside from these 2 gremlins 😄 , in which case then I guess that's the only course of action for now.