CI: ClusterMesh: timeouts in client-egress-l7

[pchaigno]

I've seen a few variations of this, all with L7 on egress and clustermesh:

  ❌ client-egress-l7-named-port/pod-to-pod/curl-1: cilium-test/client2-bc59f56d5-w5w9c (10.242.1.182) -> cilium-test/echo-other-node-5c58b8cf8b-g4lwr (10.244.1.197:8080)
  ❌ client-egress-l7/pod-to-world/http-to-google.com-0: cilium-test/client2-bc59f56d5-bzr5n (10.242.1.229) -> google.com-http (google.com:80)
  ❌ client-egress-l7/pod-to-world/http-to-google.com-0: cilium-test/client2-bc59f56d5-qkj62 (10.242.1.156) -> google.com-http (google.com:80)

Link: https://github.com/cilium/cilium/actions/runs/4722694005/jobs/8377638774
Sysdump:
cilium-sysdumps-5.zip

Link: https://github.com/cilium/cilium/actions/runs/4719364276/jobs/8370097019
Sysdump:
cilium-sysdumps-1.zip

Link: https://github.com/cilium/cilium/actions/runs/4713940509/jobs/8359970494
Sysdump:
cilium-sysdumps-2.zip

[michi-covalent]

https://github.com/cilium/cilium/actions/runs/4827305584/jobs/8599940047

this one is with client-egress-l7-named-port but i'm guessing it's the same issue:

❌ 1/32 tests failed (1/224 actions), 1 tests skipped, 1 scenarios skipped:
Test [client-egress-l7-named-port]:
  ❌ client-egress-l7-named-port/pod-to-world/http-to-google.com-0: cilium-test/client2-76f4d7c5bc-k75nf (10.242.1.24) -> google.com-http (google.com:80)

cilium-sysdumps-7.zip

[qmonnet]

Hit on #26205

 ❌ 1/43 tests failed (1/472 actions), 13 tests skipped, 1 scenarios skipped:
Test [client-egress-l7-named-port]:
  ❌ client-egress-l7-named-port/pod-to-pod/curl-ipv4-2: cilium-test/client2-76f4d7c5bc-qbt49 (10.242.1.55) -> cilium-test/echo-other-node-545c9b778b-6nld8 (10.244.1.116:8080)

cilium-sysdumps.zip

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[christarazi]

I've seen

  ❌ client-egress-l7/pod-to-pod/curl-ipv6-3: cilium-test/client2-59b578d4bb-z72ww (fd00:10:242:1::bb2f) -> cilium-test/echo-other-node-596fc9c48f-6vqvd (fd00:10:244:1::e07a:8080)

in #27838.

Run: https://github.com/cilium/cilium/actions/runs/6042609538/job/16398064551

[joestringer]

Hit a similar issue with pod-to-pod on #28589 (1/10 failures)

[=] Test [client-egress-l7]
142
.......
143
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
144
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
145
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http' to namespace 'cilium-test'..
146
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http' to namespace 'cilium-test'..
147
  [-] Scenario [client-egress-l7/pod-to-pod]
148
  [.] Action [client-egress-l7/pod-to-pod/curl-ipv4-0: cilium-test/client-78f9dffc84-qwhrf (10.242.1.249) -> cilium-test/echo-same-node-79d996cb79-8qx7h (10.242.1.232:8080)]
149
  [.] Action [client-egress-l7/pod-to-pod/curl-ipv6-0: cilium-test/client-78f9dffc84-qwhrf (fd00:10:242:1::a4bb) -> cilium-test/echo-same-node-79d996cb79-8qx7h (fd00:10:242:1::551f:8080)]
150
  [.] Action [client-egress-l7/pod-to-pod/curl-ipv4-1: cilium-test/client-78f9dffc84-qwhrf (10.242.1.249) -> cilium-test/echo-other-node-9746df45c-cmttz (10.244.1.145:8080)]
151
  [.] Action [client-egress-l7/pod-to-pod/curl-ipv6-1: cilium-test/client-78f9dffc84-qwhrf (fd00:10:242:1::a4bb) -> cilium-test/echo-other-node-9746df45c-cmttz (fd00:10:244:1::45d4:8080)]
152
  [.] Action [client-egress-l7/pod-to-pod/curl-ipv4-2: cilium-test/client2-59b578d4bb-pgbtx (10.242.1.138) -> cilium-test/echo-same-node-79d996cb79-8qx7h (10.242.1.232:8080)]
153
  [.] Action [client-egress-l7/pod-to-pod/curl-ipv6-2: cilium-test/client2-59b578d4bb-pgbtx (fd00:10:242:1::b9de) -> cilium-test/echo-same-node-79d996cb79-8qx7h (fd00:10:242:1::551f:8080)]
154
  [.] Action [client-egress-l7/pod-to-pod/curl-ipv4-3: cilium-test/client2-59b578d4bb-pgbtx (10.242.1.138) -> cilium-test/echo-other-node-9746df45c-cmttz (10.244.1.145:8080)]
155
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://10.244.1.145:8080" failed: command terminated with exit code 28
156
  ℹ️  curl output:

https://github.com/cilium/cilium/actions/runs/6546405659/job/17776797446

[pippolo84]

Another hit from #28746

[=] Test [client-egress-l7-named-port]
.......
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  [-] Scenario [client-egress-l7-named-port/pod-to-pod]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-0: cilium-test/client-78f9dffc84-nh28l (10.242.1.159) -> cilium-test/echo-same-node-79d996cb79-pqnx2 (10.242.1.91:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-0: cilium-test/client-78f9dffc84-nh28l (fd00:10:242:1::faac) -> cilium-test/echo-same-node-79d996cb79-pqnx2 (fd00:10:242:1::7360:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-1: cilium-test/client-78f9dffc84-nh28l (10.242.1.159) -> cilium-test/echo-other-node-9746df45c-h4xh5 (10.244.1.128:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-1: cilium-test/client-78f9dffc84-nh28l (fd00:10:242:1::faac) -> cilium-test/echo-other-node-9746df45c-h4xh5 (fd00:10:244:1::845b:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-2: cilium-test/client2-59b578d4bb-ssvc6 (10.242.1.88) -> cilium-test/echo-same-node-79d996cb79-pqnx2 (10.242.1.91:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-2: cilium-test/client2-59b578d4bb-ssvc6 (fd00:10:242:1::a4a3) -> cilium-test/echo-same-node-79d996cb79-pqnx2 (fd00:10:242:1::7360:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-3: cilium-test/client2-59b578d4bb-ssvc6 (10.242.1.88) -> cilium-test/echo-other-node-9746df45c-h4xh5 (10.244.1.128:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://10.244.1.128:8080/" failed: command terminated with exit code 28

Run: https://github.com/cilium/cilium/actions/runs/6797757339/job/18480538566
Sysdump: cilium-sysdumps.zip

[thorn3r]

Another hit in #29643

..............
[=] Test [client-egress-l7-named-port] [47/59]
........
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  [-] Scenario [client-egress-l7-named-port/pod-to-pod]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-0: cilium-test/client-59bddc8c5-2kd52 (10.242.1.109) -> cilium-test/echo-same-node-5cd8c46654-48js6 (10.242.1.57:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-0: cilium-test/client-59bddc8c5-2kd52 (fd00:10:242:1::d1ac) -> cilium-test/echo-same-node-5cd8c46654-48js6 (fd00:10:242:1::aa3a:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-1: cilium-test/client-59bddc8c5-2kd52 (10.242.1.109) -> cilium-test/echo-other-node-54c86bc6b8-drgzc (10.244.1.105:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-1: cilium-test/client-59bddc8c5-2kd52 (fd00:10:242:1::d1ac) -> cilium-test/echo-other-node-54c86bc6b8-drgzc (fd00:10:244:1::1f19:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-2: cilium-test/client2-694d49cf6d-rv5j9 (10.242.1.247) -> cilium-test/echo-same-node-5cd8c46654-48js6 (10.242.1.57:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-2: cilium-test/client2-694d49cf6d-rv5j9 (fd00:10:242:1::e78d) -> cilium-test/echo-same-node-5cd8c46654-48js6 (fd00:10:242:1::aa3a:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-3: cilium-test/client2-694d49cf6d-rv5j9 (10.242.1.247) -> cilium-test/echo-other-node-54c86bc6b8-drgzc (10.244.1.105:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-3: cilium-test/client2-694d49cf6d-rv5j9 (fd00:10:242:1::e78d) -> cilium-test/echo-other-node-54c86bc6b8-drgzc (fd00:10:244:1::1f19:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[fd00:10:244:1::1f19]:8080" failed: error with exec request (pod=cilium-test/client2-694d49cf6d-rv5j9, container=client2): command terminated with exit code 28

Sysdump:
cilium-sysdumps.zip

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[github-actions]

This issue has not seen any activity since it was marked stale.
Closing.