CI: ClusterMesh (ci-multicluster) Setup & Test (4, disabled, ipv6, disabled, none) job failure

[michi-covalent]

CI failure

https://github.com/cilium/cilium/actions/runs/4767503942/jobs/8475815473

📋 Test Report
❌ 3/28 tests failed (8/174 actions), 5 tests skipped, 7 scenarios skipped:
Test [no-policies]:
  ❌ no-policies/pod-to-service/curl-0: cilium-test/client2-76f4d7c5bc-7zl8k (fd00:10:242:1::7e1d) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies/pod-to-service/curl-2: cilium-test/client-6965d549d5-qrm6g (fd00:10:242:1::ecff) -> cilium-test/echo-other-node (echo-other-node:8080)
Test [no-policies-extra]:
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-0: cilium-test/client-6965d549d5-qrm6g (fd00:10:242:1::ecff) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-3: cilium-test/client2-76f4d7c5bc-7zl8k (fd00:10:242:1::7e1d) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-local-nodeport/curl-0: cilium-test/client-6965d549d5-qrm6g (fd00:10:242:1::ecff) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-local-nodeport/curl-2: cilium-test/client2-76f4d7c5bc-7zl8k (fd00:10:242:1::7e1d) -> cilium-test/echo-other-node (echo-other-node:8080)
Test [allow-all-except-world]:
  ❌ allow-all-except-world/pod-to-service/curl-0: cilium-test/client-6965d549d5-qrm6g (fd00:10:242:1::ecff) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ allow-all-except-world/pod-to-service/curl-2: cilium-test/client2-76f4d7c5bc-7zl8k (fd00:10:242:1::7e1d) -> cilium-test/echo-other-node (echo-other-node:8080)

sysdumps:

cilium-sysdump-20230421-175904.zip
cilium-sysdump-20230421-180243.zip
cilium-sysdump-20230421-180318.zip
cilium-sysdump-20230421-175938.zip
cilium-sysdump-20230421-180014.zip
cilium-sysdump-20230421-180047.zip
cilium-sysdump-20230421-180122.zip
cilium-sysdump-20230421-180202.zip
cilium-sysdump-context1-final.zip
cilium-sysdump-context2-final.zip

[giorio94]

Hit the same in a different matrix entry (2, disabled, ipv4, wireguard, none): https://github.com/cilium/cilium/actions/runs/5134229230/jobs/9237943355

❌ 3/44 tests failed (8/289 actions), 12 tests skipped, 0 scenarios skipped:
Test [no-policies]:
  ❌ no-policies/pod-to-service/curl-0: cilium-test/client-6965d549d5-pvfts (10.242.1.46) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies/pod-to-service/curl-2: cilium-test/client2-76f4d7c5bc-cs8l8 (10.242.1.180) -> cilium-test/echo-other-node (echo-other-node:8080)
Test [no-policies-extra]:
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-0: cilium-test/client-6965d549d5-pvfts (10.242.1.46) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-2: cilium-test/client2-76f4d7c5bc-cs8l8 (10.242.1.180) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-local-nodeport/curl-0: cilium-test/client-6965d549d5-pvfts (10.242.1.46) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-local-nodeport/curl-2: cilium-test/client2-76f4d7c5bc-cs8l8 (10.242.1.180) -> cilium-test/echo-other-node (echo-other-node:8080)
Test [allow-all-except-world]:
  ❌ allow-all-except-world/pod-to-service/curl-1: cilium-test/client2-76f4d7c5bc-cs8l8 (10.242.1.180) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ allow-all-except-world/pod-to-service/curl-2: cilium-test/client-6965d549d5-pvfts (10.242.1.46) -> cilium-test/echo-other-node (echo-other-node:8080)

Sysdumps:

cilium-sysdump-20230531-144425.zip
cilium-sysdump-20230531-144504.zip
cilium-sysdump-20230531-144544.zip
cilium-sysdump-20230531-144623.zip
cilium-sysdump-20230531-144700.zip
cilium-sysdump-20230531-144734.zip
cilium-sysdump-20230531-144815.zip
cilium-sysdump-20230531-144853.zip
cilium-sysdump-context1-final.zip
cilium-sysdump-context2-final.zip

[giorio94]

Hit a similar one: https://github.com/cilium/cilium/actions/runs/5252822823/jobs/9489404395
Matrix entry: (3, disabled, ipv4, ipsec, iptables)

connectivity test failed: 1 tests failed
📋 Test Report
❌ 1/43 tests failed (1/277 actions), 13 tests skipped, 1 scenarios skipped:
Test [no-policies]:
  ❌ no-policies/pod-to-service/curl-0: cilium-test/client-6965d549d5-2f962 (10.242.1.200) -> cilium-test/echo-other-node (echo-other-node:8080)

Looking at the sysdump, this failure seems to be due to a race condition between the connectivity test and the propagation of the remote service information. Indeed, cilium service list reported no backends for the echo-other-node service (10.243.85.29):

ID   Frontend              Service Type   Backend                           
...
8    10.243.85.29:8080     ClusterIP                                        

As also confirmed by the Hubble flows:

Jun 13 08:07:20.712: cilium-test/client-6965d549d5-2f962:37474 (ID:74122) <> cilium-test/echo-other-node:8080 (world) from-endpoint FORWARDED (TCP Flags: SYN)
Jun 13 08:07:20.712: cilium-test/client-6965d549d5-2f962:37474 (ID:74122) <> cilium-test/echo-other-node:8080 (world) Service backend not found DROPPED (TCP Flags: SYN)
Jun 13 08:07:21.713: cilium-test/client-6965d549d5-2f962:37474 (ID:74122) <> cilium-test/echo-other-node:8080 (world) from-endpoint FORWARDED (TCP Flags: SYN)
Jun 13 08:07:21.713: cilium-test/client-6965d549d5-2f962:37474 (ID:74122) <> cilium-test/echo-other-node:8080 (world) Service backend not found DROPPED (TCP Flags: SYN)

All subsequent tests passed, and the service backends are correctly populated in the final sysdumps.

Sysdumps:
cilium-sysdump-20230613-080722.zip
cilium-sysdump-context1-final.zip
cilium-sysdump-context2-final.zip

[giorio94]

This issue should be fixed by cilium/cilium-cli#1758, as the CLI now waits for service propagation before starting the actual connectivity tests.