test/k8s: for services test, wait for all applied manifests to delete

In the K8sDatapathConfig tests "echo-svc" deployment Pods are failing to terminate while waiting to terminate all Pods.
These come from a deployment, that shouldn't be applied in this suite. Presumably failure to delete is being cause by the Deployment controller restarting the Pods as they're deleted.

To try to fix this, going to wait for applied yamls in K8sServices to be fully deleted, including finalizers.

Addresses: #25255

Signed-off-by: Tom Hadlaw <tom.hadlaw@isovalent.com>

test/k8s/services.go

@@ -8,6 +8,7 @@ import (
 	"fmt"
 	"net"
 	"strings"
+	"sync"
 	"time"
 
 	. "github.com/onsi/gomega"
@@ -89,9 +90,19 @@ var _ = SkipDescribeIf(helpers.RunsOn54Kernel, "K8sDatapathServicesTest", func()
 		})
 
 		AfterAll(func() {
+			wg := sync.WaitGroup{}
 			for _, yaml := range yamls {
-				kubectl.Delete(yaml)
+				wg.Add(1)
+				go func(yaml string) {
+					defer wg.Done()
+					// Ensure that all deployments are fully cleaned up before
+					// proceeding to the next test.
+					res := kubectl.DeleteAndWait(yaml, true)
+
+					Expect(res.WasSuccessful()).Should(BeTrue(), "Unable to cleanup yaml: %s", yaml)
+				}(yaml)
 			}
+			wg.Wait()
 			ExpectAllPodsTerminated(kubectl)
 		})