CI: K8sPolicyTest varied flakes

[raybejjani]

This seems like a race between adding this policy and regenerating. My current guess is that the way we wait for k8s netpols to regenerate is not as robust as what we do for CNPs.

These are all of the form:

Expected command: kubectl exec -n 202003300621k8spolicytestbasictestchecksallkindofkubernetespoli app2-64975875bc-j29hw -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 8 http://1.1.1.1/ -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'"
To have failed, but it was successful:

https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/10979/testReport/junit/Suite-k8s-1/11/K8sPolicyTest_Basic_Test_Denies_traffic_with_k8s_default_deny_ingress_egress_policy/

https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/10976/testReport/junit/Suite-k8s-1/18/K8sPolicyTest_Basic_Test_Denies_traffic_with_k8s_default_deny_egress_policy/

https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/10970/

https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/10967/

There may be cross-test effects. In one case a number of tests failed later with cilium preflight failures (no socket found) https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/10971/

[joestringer]

From weekly meeting: Maybe related to CEP status changes?

[raybejjani]

This may not be fixed fully :(

https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/11051/testReport/junit/Suite-k8s-1/11/K8sPolicyTest_Basic_Test_Denies_traffic_with_k8s_default_deny_egress_policy/

https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/11058/execution/node/192/log/?consoleFull