test: force restarting of Cilium pods

[nebril]

Fixes race between cilium being restarted and connectivity test.

[maintainer-s-little-helper]

Please set the appropriate release note label.

[nebril]

test-focus K8sFQDNTest.*

[nebril]

test-gke K8sFQDNTest.*

[nebril]

test-focus K8sFQDNTest.*

[nebril]

test-gke K8sFQDNTest.*

[coveralls]

Coverage decreased (-0.006%) to 36.874% when pulling b885df8 on pr/fqdn-delete-cilium into 60786b6 on master.

[nebril]

test-focus K8sFQDNTest.*

[nebril]

test-focus K8sFQDNTest.*

[nebril]

test-me-please

[nebril]

test-me-please

[nebril]

test-focus K8sFQDNTest.*

[nebril]

test-focus K8sFQDNTest.*

[nebril]

test-me-please

[nebril]

retest-runtime

[nebril]

retest-4.19

[nebril]

test-me-please

[nebril]

test-gke

[nebril]

test-me-please

[errordeveloper]

@nebril could you elaborate on why exactly connectivity check is interfering here, just trying to understand the context better.

[nebril]

@errordeveloper the connectivity check was not interfering, but the point of the test is to run the test while Cilium is recovering to validate that dns cache works during restarting Cilium pods.

[errordeveloper]

@nebril that sounds like it would add even more racy behaviour, it sound to me that it would be more reliable to delete the the daemonset instead, or taint and drain the nodes.

[nebril]

@errordeveloper AFAIU if we delete the daemonset, Cilium pods will uninstall cleanly, deleting bpf maps. If we drain the nodes, the same applies, and also how will we test the workload running on a drained node?

[errordeveloper]

if we delete the daemonset, Cilium pods will uninstall cleanly, deleting bpf maps

I thought the opposite was actually the case.

how will we test the workload running on a drained node?

You just need to have the right toleration set. It may the case that it's not a full drain that is needed, but a taint that Cilium doesn't tolerate followed by deletion of the Cilium pod(s).

[nebril]

@errordeveloper without the Cilium pod scheduled on a node, we end up with a node that doesn't handle networking via our cni plugin, which is not what we want to test afaiu.

[errordeveloper]

@nebril I believe missing pod will have the same effect as restarting pod in this case, and just to be clear, my view is that ad-hoc commands is exactly what we should stop doing in the tests. If this is a hack that fixes another hack, I get that :)