ci: Restart pods when toggling KPR switch

[brb]

Previously, in the graceful backend termination test we switched to
KPR=disabled and we didn't restart CoreDNS. Before the switch,
CoreDNS@k8s2 -> kube-apiserver@k8s1 was handled by the socket-lb, so the
outgoing packet was $CORE_DNS_IP -> $KUBE_API_SERVER_NODE_IP. The packet
should have been BPF masq-ed. After the switch, the BPF masq is no
longer in place, so the packets from CoreDNS are subject to the
iptables' masquerading (they can be either dropped by the invalid rule
or masqueraded to some other port). Combined with CoreDNS unable to
recover from connectivity errors [1], the CoreDNS was no longer able to
receive updates from the kube-apiserver, thus NXDOMAIN errors for the
new service name.

To avoid such flakes, forcefully restart the DNS pods if the KPR setting
change is detected.

[1]: #18018

Fix #17881

[brb]

/test

Job 'Cilium-PR-K8s-GKE' failed and has not been observed before, so may be related to your PR:

Click to show.

Test Name

K8sHealthTest cilium-health Checks status between nodes

Failure Output

FAIL: Expected

If it is a flake, comment /mlh new-flake Cilium-PR-K8s-GKE so I can create a new GitHub issue to track it.

[brb]

test-1.22-4.19

[brb]

k8s-1.21-kernel-5.4 hit #17010 (comment)

[brb]

test-1.21-5.4

[brb]

gke-stable hit #6728

[brb]

test-gke

[brb]

k8s-1.22-kernel-4.19 hit #18014 (currently investigating)

[brb]

test-1.22-4.19

[brb]

k8s-1.22-kernel-4.19 hit #18014.

[joestringer]

@brb should we document somewhere that users must restart all pods to properly apply that feature if they change the setting?

[brb]

@brb should we document somewhere that users must restart all pods to properly apply that feature if they change the setting?

IIRC @aanm has already documented it somewhere. Unfortunately, I cannot find it. @aanm ?

[aanm]

@brb should we document somewhere that users must restart all pods to properly apply that feature if they change the setting?

IIRC @aanm has already documented it somewhere. Unfortunately, I cannot find it. @aanm ?

@joestringer @brb Here

[nbusseneau]

optionChangeRequiresPodRedeploy does not exist in v1.9 because #16767 was not backported to v1.9, preventing this from being backported in #18147.

Are we hitting this issue in v1.9?

• If yes, I advocate for backporting test: Redeploy DNS after endpointRoutes reconfiguration #16767 as well.
• If not, I advocate for removing this PR from backports in v1.9 backports 2021-12-06 #18147 and removing backport label for 1.9 on this PR.

[pchaigno]

I think we should backport both. I can't think of a reason why the v1.9 tests wouldn't be affected by bug fixed in those two PRs. Note it's a bit hard to check because the issue can manifest in several different tests and it depends, at least partially, on the order in which tests are executed.

If we backport #16767, we'll also need to backport #16835.

[nbusseneau]

OK, then I'm adding both of these to the backport PR.