Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: new CLI 0.8.6's L7 tests failures on EKS with IPsec enabled #17139

Closed
1 of 4 tasks
nbusseneau opened this issue Aug 11, 2021 · 0 comments · Fixed by #22617
Closed
1 of 4 tasks

CI: new CLI 0.8.6's L7 tests failures on EKS with IPsec enabled #17139

nbusseneau opened this issue Aug 11, 2021 · 0 comments · Fixed by #22617
Assignees
@pchaigno
Labels
area/CI Continuous Integration testing issue or flake area/encryption Impacts encryption support such as IPSec, WireGuard, or kTLS. ci/flake This is a known failure that occurs in the tree. Please investigate me! integration/cloud Related to integration with cloud environments such as AKS, EKS, GKE, etc.

Comments

@nbusseneau
Copy link
Member

nbusseneau commented Aug 11, 2021 •
edited

CLI 0.8.6 added new L7 tests coverage, which consistently fails on EKS with IPsec enabled (see below for logs) when it should be working.

It is probable this new coverage uncovered an undetected bug in Cilium.

Next steps

Behavior

https://github.com/cilium/cilium/actions/runs/1114941992
Sysdump: cilium-sysdump-out.zip.zip
Logs:

[=] Test [client-egress-l7]
....
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http' to namespace 'cilium-test'..
  [-] Scenario [client-egress-l7/pod-to-pod]
  [.] Action [client-egress-l7/pod-to-pod/curl-0: cilium-test/client-7b7bf54b85-n594s (192.168.103.225) -> cilium-test/echo-other-node-697d5d69b7-4zrxz (192.168.157.144:8080)]
  📄 Following flows...
  [.] Action [client-egress-l7/pod-to-pod/curl-1: cilium-test/client-7b7bf54b85-n594s (192.168.103.225) -> cilium-test/echo-same-node-7967996674-zmn74 (192.168.122.247:8080)]
  📄 Following flows...
  [.] Action [client-egress-l7/pod-to-pod/curl-2: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> cilium-test/echo-other-node-697d5d69b7-4zrxz (192.168.157.144:8080)]
  📄 Following flows...
  [.] Action [client-egress-l7/pod-to-pod/curl-3: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> cilium-test/echo-same-node-7967996674-zmn74 (192.168.122.247:8080)]
  📄 Following flows...
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://192.168.122.247:8080" failed: command terminated with exit code 22
  ℹ️  curl stderr:
  curl: (22) The requested URL returned error: 503
  
  📄 Flow logs for peer cilium-test/client2-666976c95b-vlkxn:
  ❓ [0] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [1] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 L3-L4 FORWARDED (TCP Flags: SYN)
  ❓ [2] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-proxy FORWARDED (TCP Flags: SYN)
  ❓ [3] Aug 10 01:51:13.623: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-proxy FORWARDED (TCP Flags: SYN, ACK)
  ❓ [4] Aug 10 01:51:13.623: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [5] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [6] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-proxy FORWARDED (TCP Flags: ACK)
  ❓ [7] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [8] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-proxy FORWARDED (TCP Flags: ACK, PSH)
  ❓ [9] Aug 10 01:51:13.623: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-proxy FORWARDED (TCP Flags: ACK)
  ❓ [10] Aug 10 01:51:13.623: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [11] Aug 10 01:51:13.624: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 http-request FORWARDED (HTTP/1.1 GET http://192.168.122.247:8080/)
  ❓ [12] Aug 10 01:51:13.624: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [13] Aug 10 01:51:13.624: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [14] Aug 10 01:51:13.624: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [15] Aug 10 01:51:13.624: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [16] Aug 10 01:51:13.624: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: SYN, ACK)
  ❓ [17] Aug 10 01:51:13.625: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: SYN, ACK)
  ❓ [18] Aug 10 01:51:14.636: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [19] Aug 10 01:51:14.636: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [20] Aug 10 01:51:14.636: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [21] Aug 10 01:51:14.636: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [22] Aug 10 01:51:14.636: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: ACK)
  ❓ [23] Aug 10 01:51:14.637: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: ACK)
  ❓ [24] Aug 10 01:51:14.637: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [25] Aug 10 01:51:14.637: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: SYN, ACK)
  ❓ [26] Aug 10 01:51:14.638: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: SYN, ACK)
  ❓ [27] Aug 10 01:51:15.665: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [28] Aug 10 01:51:15.665: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [29] Aug 10 01:51:15.665: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [30] Aug 10 01:51:15.665: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [31] Aug 10 01:51:15.665: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: ACK)
  ❓ [32] Aug 10 01:51:15.665: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: ACK)
  ❓ [33] Aug 10 01:51:16.654: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [34] Aug 10 01:51:16.654: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: SYN, ACK)
  ❓ [35] Aug 10 01:51:16.654: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: SYN, ACK)
  ❓ [36] Aug 10 01:51:16.694: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [37] Aug 10 01:51:16.694: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [38] Aug 10 01:51:16.694: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [39] Aug 10 01:51:16.694: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [40] Aug 10 01:51:16.694: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: ACK)
  ❓ [41] Aug 10 01:51:16.694: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: ACK)
  
  📄 Flow logs for peer cilium-test/echo-same-node-7967996674-zmn74:
  ❓ [0] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [1] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 L3-L4 FORWARDED (TCP Flags: SYN)
  ❓ [2] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-proxy FORWARDED (TCP Flags: SYN)
  ❓ [3] Aug 10 01:51:13.623: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-proxy FORWARDED (TCP Flags: SYN, ACK)
  ❓ [4] Aug 10 01:51:13.623: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [5] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [6] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-proxy FORWARDED (TCP Flags: ACK)
  ❓ [7] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [8] Aug 10 01:51:13.623: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-proxy FORWARDED (TCP Flags: ACK, PSH)
  ❓ [9] Aug 10 01:51:13.623: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-proxy FORWARDED (TCP Flags: ACK)
  ❓ [10] Aug 10 01:51:13.623: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [11] Aug 10 01:51:13.624: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 http-request FORWARDED (HTTP/1.1 GET http://192.168.122.247:8080/)
  ❓ [12] Aug 10 01:51:13.624: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [13] Aug 10 01:51:13.624: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [14] Aug 10 01:51:13.624: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [15] Aug 10 01:51:13.624: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [16] Aug 10 01:51:13.624: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: SYN, ACK)
  ❓ [17] Aug 10 01:51:13.625: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: SYN, ACK)
  ❓ [18] Aug 10 01:51:14.636: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [19] Aug 10 01:51:14.636: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [20] Aug 10 01:51:14.636: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [21] Aug 10 01:51:14.636: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [22] Aug 10 01:51:14.636: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: ACK)
  ❓ [23] Aug 10 01:51:14.637: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: ACK)
  ❓ [24] Aug 10 01:51:14.637: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [25] Aug 10 01:51:14.637: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: SYN, ACK)
  ❓ [26] Aug 10 01:51:14.638: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: SYN, ACK)
  ❓ [27] Aug 10 01:51:15.665: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [28] Aug 10 01:51:15.665: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [29] Aug 10 01:51:15.665: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [30] Aug 10 01:51:15.665: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [31] Aug 10 01:51:15.665: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: ACK)
  ❓ [32] Aug 10 01:51:15.665: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: ACK)
  ❓ [33] Aug 10 01:51:16.654: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [34] Aug 10 01:51:16.654: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: SYN, ACK)
  ❓ [35] Aug 10 01:51:16.654: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: SYN, ACK)
  ❓ [36] Aug 10 01:51:16.694: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [37] Aug 10 01:51:16.694: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [38] Aug 10 01:51:16.694: cilium-test/client2-666976c95b-vlkxn:56676 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [39] Aug 10 01:51:16.694: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [40] Aug 10 01:51:16.694: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 to-stack FORWARDED (TCP Flags: ACK)
  ❓ [41] Aug 10 01:51:16.694: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:56676 from-network FORWARDED (TCP Flags: ACK)
  
  [-] Scenario [client-egress-l7/pod-to-world]
  [.] Action [client-egress-l7/pod-to-world/https-to-cilium-io-0: cilium-test/client-7b7bf54b85-n594s (192.168.103.225) -> cilium-io-https (cilium.io:443)]
  📄 Following flows...
  [.] Action [client-egress-l7/pod-to-world/http-to-cilium-io-0: cilium-test/client-7b7bf54b85-n594s (192.168.103.225) -> cilium-io-http (cilium.io:80)]
  📄 Following flows...
  [.] Action [client-egress-l7/pod-to-world/http-to-jenkins-cilium-0: cilium-test/client-7b7bf54b85-n594s (192.168.103.225) -> jenkins-cilium-io-http (jenkins.cilium.io:80)]
  📄 Following flows...
  [.] Action [client-egress-l7/pod-to-world/https-to-cilium-io-1: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> cilium-io-https (cilium.io:443)]
  📄 Following flows...
  [.] Action [client-egress-l7/pod-to-world/http-to-cilium-io-1: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> cilium-io-http (cilium.io:80)]
  📄 Following flows...
  [.] Action [client-egress-l7/pod-to-world/http-to-jenkins-cilium-1: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> jenkins-cilium-io-http (jenkins.cilium.io:80)]
  📄 Following flows...
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-only-dns' from namespace 'cilium-test'..
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-l7-http' from namespace 'cilium-test'..

[=] Test [echo-ingress-l7]
....
  ℹ️  📜 Applying CiliumNetworkPolicy 'echo-ingress-l7-http' to namespace 'cilium-test'..
  [-] Scenario [echo-ingress-l7/pod-to-pod]
  [.] Action [echo-ingress-l7/pod-to-pod/curl-0: cilium-test/client-7b7bf54b85-n594s (192.168.103.225) -> cilium-test/echo-other-node-697d5d69b7-4zrxz (192.168.157.144:8080)]
  📄 Following flows...
  [.] Action [echo-ingress-l7/pod-to-pod/curl-1: cilium-test/client-7b7bf54b85-n594s (192.168.103.225) -> cilium-test/echo-same-node-7967996674-zmn74 (192.168.122.247:8080)]
  📄 Following flows...
  [.] Action [echo-ingress-l7/pod-to-pod/curl-2: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> cilium-test/echo-other-node-697d5d69b7-4zrxz (192.168.157.144:8080)]
  📄 Following flows...
  [.] Action [echo-ingress-l7/pod-to-pod/curl-3: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> cilium-test/echo-same-node-7967996674-zmn74 (192.168.122.247:8080)]
  📄 Following flows...
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://192.168.122.247:8080" failed: command terminated with exit code 52
  ℹ️  curl stderr:
  curl: (52) Empty reply from server
  
  📄 Flow logs for peer cilium-test/client2-666976c95b-vlkxn:
  ❓ [0] Aug 10 01:52:03.426: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [1] Aug 10 01:52:03.426: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: SYN)
  ❓ [2] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [3] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [4] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 L3-L4 FORWARDED (TCP Flags: SYN)
  ❓ [5] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-proxy FORWARDED (TCP Flags: SYN)
  ❓ [6] Aug 10 01:52:03.427: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:57048 from-network FORWARDED (TCP Flags: SYN, ACK)
  ❓ [7] Aug 10 01:52:03.427: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:57048 from-stack FORWARDED (TCP Flags: SYN, ACK)
  ❓ [8] Aug 10 01:52:03.427: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:57048 to-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [9] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [10] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK)
  ❓ [11] Aug 10 01:52:03.428: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK)
  ❓ [12] Aug 10 01:52:03.428: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [13] Aug 10 01:52:03.428: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [14] Aug 10 01:52:03.428: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [15] Aug 10 01:52:03.634: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [16] Aug 10 01:52:03.634: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [17] Aug 10 01:52:03.634: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [18] Aug 10 01:52:03.842: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [19] Aug 10 01:52:03.842: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [20] Aug 10 01:52:03.842: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [21] Aug 10 01:52:04.269: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [22] Aug 10 01:52:04.270: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [23] Aug 10 01:52:04.270: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [24] Aug 10 01:52:05.101: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [25] Aug 10 01:52:05.101: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [26] Aug 10 01:52:05.102: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [27] Aug 10 01:52:06.765: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [28] Aug 10 01:52:06.765: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [29] Aug 10 01:52:06.766: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [30] Aug 10 01:52:10.189: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [31] Aug 10 01:52:10.189: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [32] Aug 10 01:52:10.190: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [33] Aug 10 01:52:16.845: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [34] Aug 10 01:52:16.845: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [35] Aug 10 01:52:16.846: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  
  📄 Flow logs for peer cilium-test/echo-same-node-7967996674-zmn74:
  ❓ [0] Aug 10 01:52:03.426: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: SYN)
  ❓ [1] Aug 10 01:52:03.426: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: SYN)
  ❓ [2] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: SYN)
  ❓ [3] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-stack FORWARDED (TCP Flags: SYN)
  ❓ [4] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 L3-L4 FORWARDED (TCP Flags: SYN)
  ❓ [5] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-proxy FORWARDED (TCP Flags: SYN)
  ❓ [6] Aug 10 01:52:03.427: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:57048 from-network FORWARDED (TCP Flags: SYN, ACK)
  ❓ [7] Aug 10 01:52:03.427: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:57048 from-stack FORWARDED (TCP Flags: SYN, ACK)
  ❓ [8] Aug 10 01:52:03.427: cilium-test/echo-same-node-7967996674-zmn74:8080 -> cilium-test/client2-666976c95b-vlkxn:57048 to-endpoint FORWARDED (TCP Flags: SYN, ACK)
  ❓ [9] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK)
  ❓ [10] Aug 10 01:52:03.427: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK)
  ❓ [11] Aug 10 01:52:03.428: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK)
  ❓ [12] Aug 10 01:52:03.428: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [13] Aug 10 01:52:03.428: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [14] Aug 10 01:52:03.428: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [15] Aug 10 01:52:03.634: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [16] Aug 10 01:52:03.634: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [17] Aug 10 01:52:03.634: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [18] Aug 10 01:52:03.842: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [19] Aug 10 01:52:03.842: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [20] Aug 10 01:52:03.842: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [21] Aug 10 01:52:04.269: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [22] Aug 10 01:52:04.270: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [23] Aug 10 01:52:04.270: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [24] Aug 10 01:52:05.101: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [25] Aug 10 01:52:05.101: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [26] Aug 10 01:52:05.102: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [27] Aug 10 01:52:06.765: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [28] Aug 10 01:52:06.765: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [29] Aug 10 01:52:06.766: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [30] Aug 10 01:52:10.189: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [31] Aug 10 01:52:10.189: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [32] Aug 10 01:52:10.190: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  ❓ [33] Aug 10 01:52:16.845: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
  ❓ [34] Aug 10 01:52:16.845: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
  ❓ [35] Aug 10 01:52:16.846: cilium-test/client2-666976c95b-vlkxn:57048 -> cilium-test/echo-same-node-7967996674-zmn74:8080 from-network FORWARDED (TCP Flags: ACK, PSH)
  
  ℹ️  📜 Deleting CiliumNetworkPolicy 'echo-ingress-l7-http' from namespace 'cilium-test'..
[...]
Connectivity test failed: 2 tests failed
📋 Test Report
❌ 2/11 tests failed (2/110 actions), 0 tests skipped, 0 scenarios skipped:
Test [client-egress-l7]:
  ❌ client-egress-l7/pod-to-pod/curl-3: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> cilium-test/echo-same-node-7967996674-zmn74 (192.168.122.247:8080)
Test [echo-ingress-l7]:
  ❌ echo-ingress-l7/pod-to-pod/curl-3: cilium-test/client2-666976c95b-vlkxn (192.168.145.252) -> cilium-test/echo-same-node-7967996674-zmn74 (192.168.122.247:8080)
@nbusseneau nbusseneau added area/CI Continuous Integration testing issue or flake area/encryption Impacts encryption support such as IPSec, WireGuard, or kTLS. integration/cloud Related to integration with cloud environments such as AKS, EKS, GKE, etc. ci/flake This is a known failure that occurs in the tree. Please investigate me! labels Aug 11, 2021
nbusseneau added a commit that referenced this issue Aug 11, 2021
@nbusseneau
workflows: add test exceptions for failing L7 tests on EKS with IPsec
9491df2 
CLI 0.8.6 added new L7 tests coverage, which consistently fails on EKS
with IPsec enabled when it should be working.

We add test temporary exceptions until we fix the issue.

More details in #17139.

Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
@nbusseneau nbusseneau mentioned this issue Aug 11, 2021
Merged
@nbusseneau nbusseneau self-assigned this Aug 11, 2021
@pchaigno pchaigno mentioned this issue Aug 11, 2021
Merged
This was referenced Aug 12, 2021
Merged
Merged
@jrajahalme jrajahalme mentioned this issue Aug 12, 2021
Closed
@errordeveloper errordeveloper mentioned this issue Aug 13, 2021
Merged
ti-mo pushed a commit that referenced this issue Aug 13, 2021
@nbusseneau @ti-mo
workflows: add test exceptions for failing L7 tests on EKS with IPsec
095bc07 
CLI 0.8.6 added new L7 tests coverage, which consistently fails on EKS
with IPsec enabled when it should be working.

We add test temporary exceptions until we fix the issue.

More details in #17139.

Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
@gandro gandro mentioned this issue Aug 16, 2021
Merged
This was referenced Aug 19, 2021
Merged
Merged
@nbusseneau nbusseneau mentioned this issue Sep 14, 2021
Merged
krishgobinath pushed a commit to krishgobinath/cilium that referenced this issue Oct 20, 2021
@nbusseneau
workflows: add test exceptions for failing L7 tests on EKS with IPsec
b6b59cb 
CLI 0.8.6 added new L7 tests coverage, which consistently fails on EKS
with IPsec enabled when it should be working.

We add test temporary exceptions until we fix the issue.

More details in cilium#17139.

Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
@nbusseneau nbusseneau mentioned this issue Nov 26, 2021
Closed
17 tasks
@pchaigno pchaigno mentioned this issue Dec 8, 2022
Merged
@pchaigno pchaigno assigned pchaigno and unassigned nbusseneau Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pchaigno pchaigno

Labels
area/CI Continuous Integration testing issue or flake area/encryption Impacts encryption support such as IPSec, WireGuard, or kTLS. ci/flake This is a known failure that occurs in the tree. Please investigate me! integration/cloud Related to integration with cloud environments such as AKS, EKS, GKE, etc.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

workflow: Reenable L7 tests on EKS + IPsec cilium/cilium
2 participants
@pchaigno @nbusseneau