CI: ConformanceEKS - Error: Connectivity test failed: timeout reached waiting for pod IDs in ipcache of Cilium pod kube-system/cilium-lbwv4

[gandro]

CI failure

Observed in #16635

Error: Connectivity test failed: timeout reached waiting for pod IDs in ipcache of Cilium pod kube-system/cilium-lbwv4

• cilium-sysdump-out.zip.zip
• logs_125844.zip

This is likely different from #16693 as this is EKS and not AKS, and supposedly the root cause for #16693 has already been fixed in EKS.

[gandro]

The problem seems to be that the CEPs for the new endpoint could not be created because the old namespace was still getting deleted. The cilium-cli CEP check however passed because it still found the old ones.

Just two seconds before the timeout was reached:

cilium-lbwv4-cilium-agent-20210630-084005.log

cilium-lbwv4-cilium-agent-20210630-084005.log
368:2021-06-30T08:37:49.411351564Z level=error msg="Cannot create CEP" containerID=87f8270279 controller="sync-to-k8s-ciliumendpoint (32)" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=32 error="ciliumendpoints.cilium.io \"echo-same-node-7967996674-th842\" is forbidden: unable to create new content in namespace cilium-test because it is being terminated" identity=27408 ipv4=192.168.180.2 ipv6= k8sPodName=cilium-test/echo-same-node-7967996674-th842 subsys=endpointsynchronizer
369:2021-06-30T08:37:49.588050217Z level=error msg="Cannot create CEP" containerID=eaa61f8bf2 controller="sync-to-k8s-ciliumendpoint (2639)" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2639 error="ciliumendpoints.cilium.io \"client-7b7bf54b85-h2ll4\" is forbidden: unable to create new content in namespace cilium-test because it is being terminated" identity=16680 ipv4=192.168.79.236 ipv6= k8sPodName=cilium-test/client-7b7bf54b85-h2ll4 subsys=endpointsynchronizer
386:2021-06-30T08:37:50.429861129Z level=error msg="Cannot create CEP" containerID=87f8270279 controller="sync-to-k8s-ciliumendpoint (32)" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=32 error="ciliumendpoints.cilium.io \"echo-same-node-7967996674-th842\" is forbidden: unable to create new content in namespace cilium-test because it is being terminated" identity=27408 ipv4=192.168.180.2 ipv6= k8sPodName=cilium-test/echo-same-node-7967996674-th842 subsys=endpointsynchronizer
387:2021-06-30T08:37:50.597178176Z level=error msg="Cannot create CEP" containerID=eaa61f8bf2 controller="sync-to-k8s-ciliumendpoint (2639)" datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=2639 error="ciliumendpoints.cilium.io \"client-7b7bf54b85-h2ll4\" is forbidden: unable to create new content in namespace cilium-test because it is being terminated" identity=16680 ipv4=192.168.79.236 ipv6= k8sPodName=cilium-test/client-7b7bf54b85-h2ll4 subsys=endpointsynchronizer
391:2021-06-30T08:37:52.443959377Z level=error msg="Cannot create CEP" containerID=87f8270279 controller="sync-to-k8s-ciliumendpoint (32)" datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=32 error="ciliumendpoints.cilium.io \"echo-same-node-7967996674-th842\" is forbidden: unable to create new content in namespace cilium-test because it is being terminated" identity=27408 ipv4=192.168.180.2 ipv6= k8sPodName=cilium-test/echo-same-node-7967996674-th842 subsys=endpointsynchronizer
392:2021-06-30T08:37:52.605544806Z level=error msg="Cannot create CEP" containerID=eaa61f8bf2 controller="sync-to-k8s-ciliumendpoint (2639)" datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=2639 error="ciliumendpoints.cilium.io \"client-7b7bf54b85-h2ll4\" is forbidden: unable to create new content in namespace cilium-test because it is being terminated" identity=16680 ipv4=192.168.79.236 ipv6= k8sPodName=cilium-test/client-7b7bf54b85-h2ll4 subsys=endpointsynchronizer

This is likely fixed by cilium/cilium-cli@7c72164

[gandro]

This particulate failure here should be fixed by #16689