test: Delete DNS pods in AfterAll for datapath tests

[joestringer]

Commit a0e7712 ("test: Redeploy DNS after changing endpointRoutes")
didn't go quite far enough: It ensured that between individual tests in
a given file, the DNS pods would be redeployed during the next run if
there were significant enough datapath changes. However, the way it did
this was by storing state within the 'kubectl' variable, which is
recreated in each test file. So if the last test in one CI run enabled
endpoint routes mode, then the DNS pods would not be redeployed to
disable endpoint routes mode as part of the next test.

Fix it by just deleting the overall kube-dns deployment at the end of
any test files which reconfigure this datapath option. I assume that the
next test will set up DNS again if it's not available.

Reported-by: Chris Tarazi chris@isovalent.com
Fixes: 0e77127dcd7 ("test: Redeploy DNS after changing endpointRoutes")
Fixes: #16717

[christarazi]

LGTM, simple enough. This is a prime example of something that could easily reoccur again because someone isn't aware that this must be done if they configure endpoint-routes mode. However, it seems this would be unlikely to happen in the near future so we don't have to solve this problem now.

[joestringer]

test-me-please

[joestringer]

I manually tried this out by just running:

CNI_INTEGRATION=minikube K8S_VERSION=1.20 ginkgo --focus="K8sCustom*" -- -cilium.provision=false -cilium.kubeconfig=`echo ~/.kube/config` -cilium.image="quay.io/cilium/cilium-ci" -cilium.operator-image="quay.io/cilium/operator" -cilium.operator-suffix="-ci" -cilium.passCLIEnvironment=true

And sure enough, kube-dns was gone at the end along with Cilium.

I'm not the biggest fan of how the testsuite leaves your cluster in a broken state after it completes (rather than returning to where it started with whatever apps you previously had deployed), but I think that's already the case so if we want to fix that we can follow up separately.

[joestringer]

Cilium init pods went into crashloopbackoff, many of the jobs seem to have the same issue:

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/1018/testReport/junit/Suite-k8s-1/21/K8sHubbleTest_Hubble_Observe_Test_L3_L4_Flow/

	 kube-system         cilium-5jwhk                       0/1     Init:CrashLoopBackOff   5          4m2s    192.168.36.11   k8s1     <none>           <none>
	 kube-system         cilium-fz9m9                       0/1     Init:CrashLoopBackOff   5          4m2s    192.168.36.12   k8s2     <none>           <none>

[joestringer]

cilium-combined-logs-previous.txt shows that it's a problem with the mount init container:

2021-07-09T00:25:29.978691280Z cp: cannot stat '/usr/bin/cilium-mount': No such file or directory
2021-07-09T00:25:29.979226735Z rm: cannot remove '/hostbin/cilium-mount': No such file or directory
Error from server (BadRequest): previous terminated container "clean-cilium-state" in pod "cilium-5jwhk" not found

I'll try rebasing to see whether I just need to pull in #16815 to resolve the issue.

[joestringer]

test-me-please

[aditighag]

cilium-combined-logs-previous.txt shows that it's a problem with the mount init container:

2021-07-09T00:25:29.978691280Z cp: cannot stat '/usr/bin/cilium-mount': No such file or directory

2021-07-09T00:25:29.979226735Z rm: cannot remove '/hostbin/cilium-mount': No such file or directory

Error from server (BadRequest): previous terminated container "clean-cilium-state" in pod "cilium-5jwhk" not found

I'll try rebasing to see whether I just need to pull in #16815 to resolve the issue.

This init container change was introduced in #16815. Looks like there is a mismatch between cilium daemonset (up-to-date with master) used and image built on your PR (not up-to-date). Yeah, please rebase your PR.

[joestringer]

Also hit #16846 .

[joestringer]

Evidently not all of the tests ensure that DNS is deployed before running, so the current version of the fix breaks those tests because they're relying on DNS from previous runs.

Maybe I'll just do a complete Cilium + DNS forced redeploy in the AfterAll of the relevant test suites (K8sCustomCalls, K8sDatapathConfig). It may take a few minutes longer but should do the trick.

[joestringer]

OK, turns out Cilium was already getting removed at the end of the tests, we just weren't cleaning up the DNS pods. My previous iteration removed the DNS deployment, and apparently subsequent tests wouldn't ensure this was available. But if Cilium has been removed, it should be sufficient to simply delete the DNS pods and let them get rescheduled into the cluster (then hopefully the next iteration of Cilium will provision the networking for them).

[joestringer]

test-me-please

[joestringer]

Triage:

• CI 3.0 did not run because only test/ code changes were made here.
• Travis failed with known flake CI: Travis: Flake in ClusterMeshServicesTestSuite.TestClusterMeshServicesUpdate #12756
• Cilium-PR-K8s-1.16-net-next failed with what looks like CI: K8sServicesTest Checks service across nodes Tests NodePort BPF Tests with direct routing Tests LoadBalancer Connectivity to endpoint via LB #16399
• All other Ginkgo runs were successful.

This is addressing a known flake in the tree and is now only hitting other known flakes. Good to merge.

[nbusseneau]

Marking for backport to 1.9 following discussion here: #18031 (comment)