test: Only restart KubeDNS if required

Instead of always restarting the kube-dns deployment, split the
validation of the installation into a separate function so we can
validate the deployment and only restart kube-dns if we have to.

While doing so, replace the validation with a more efficient version
that invokes as few kubectl execs within loops as possible and
parallelizes operations were possible.

Given that Cilium is typically re-deployed before this logic is
executed, the slowest path is typically the service plumbing. Allow for
some aggressive timeout for the Kubernetes DNS service to be plumbed to
avoid restarting it in the common case.

Example output:
```
STEP: Checking if kube-dns service is plumbed correctly
STEP: Checking if pods have identity
STEP: Checking if DNS can resolve
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-b9dcp: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-b9dcp
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-fm9qm: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-fm9qm
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mxjvw: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-mxjvw
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-fm9qm: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-fm9qm
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mxjvw: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-mxjvw
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-b9dcp: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-b9dcp
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-fm9qm: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-fm9qm
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-b9dcp: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-b9dcp
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mxjvw: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-mxjvw
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-b9dcp: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-b9dcp
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mxjvw: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-mxjvw
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-fm9qm: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-fm9qm
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-b9dcp: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-b9dcp
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-fm9qm: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-fm9qm
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mxjvw: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-mxjvw
STEP: Kubernetes DNS is not ready: ClusterIP 10.71.240.10 not found in service list of cilium pod cilium-mxjvw
STEP: Restarting Kubernetes DSN (-l k8s-app=kube-dns)
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mxjvw: unable to find service backend 10.68.1.228:53 in datapath of cilium pod cilium-mxjvw
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-b9dcp: unable to find service backend 10.68.1.228:53 in cilium pod cilium-b9dcp
STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-fm9qm: unable to find service backend 10.68.1.228:53 in cilium pod cilium-fm9qm
STEP: Waiting for Kubernetes DNS to become operational
STEP: Checking if deployment is ready
STEP: Checking if kube-dns service is plumbed correctly
STEP: Checking if pods have identity
STEP: Checking if DNS can resolve
```

Signed-off-by: Thomas Graf <thomas@cilium.io>

test/helpers/cons.go

@@ -235,6 +235,9 @@ const (
 
 	// HelmTemplate is the location of the Helm templates to install Cilium
 	HelmTemplate = "../install/kubernetes/cilium"
+
+	// ServiceSuffix is the Kubernetes service suffix
+	ServiceSuffix = "svc.cluster.local"
 )
 
 var (