test: Wait for Istio POD termination before deleting istio-system or …

…cilium

Recently Istio PODs have been lingering in terminating state long
after deletion. It appears that kubelet on k8s2-1.17 is not happy that
istio-system namespace has been deleted. Deletion of the namespace
usually automatically deletes all the PODs in it. It may also be that
the termination fails to complete since the Cilium DS is being deleted
right after the istio-system namespace is deleted.

  k8s2 kubelet[3338]: W0225 ... 3338 status_manager.go:546] Failed to update status for pod "istio-citadel-7859cc956c-74474_istio-system(9a0a2875-076b-4670-a9b4-13f92f6c7240)": failed to patch status "... \"state\":{\"terminated\":{\"exitCode\":0,\"finishedAt\":null,\"startedAt\":null}}" for pod "istio-system"/"istio-citadel-7859cc956c-74474": namespaces "istio-system" not found

Signed-off-by: Jarno Rajahalme <jarno@covalent.io>

test/k8sT/istio.go

@@ -120,11 +120,14 @@ var _ = Describe("K8sIstioTest", func() {
 		By("Deleting the Istio CRDs")
 		_ = kubectl.Delete(istioCRDYAMLPath)
 
+		By("Waiting all terminating PODs to disappear")
+		err := kubectl.WaitCleanAllTerminatingPods(teardownTimeout)
+		ExpectWithOffset(1, err).To(BeNil(), "terminating Istio PODs are not deleted after timeout")
+
 		By("Deleting the istio-system namespace")
 		_ = kubectl.NamespaceDelete(istioSystemNamespace)
 
 		kubectl.DeleteCiliumDS()
-		kubectl.WaitCleanAllTerminatingPods(teardownTimeout)
 
 		kubectl.CloseSSHClient()
 	})