ci: Require cluster-wide connectivity before running tests

[gandro]

This PR introduces more in-depth checks on cilium-health in CI.

The first set of commits ensures that cilium-health properly reports connectivity failures in dual-stack setups. Before this PR, cilium-health would probe the connectivity of secondary endpoint addresses, but not report any failures in the health API and thus also not display the failures in cilium status or cilium-health status. To fix this, the API is extended to expose the status of the secondary endpoint addresses probes, and the formatting and output logic is improved to better display and report these kind of failures.

The second set of commits modifies CI to make use of this new functionality, i.e. ciliumHealthPreFlightCheck is extended such that we wait for all paths to be healthy before we start a test suite. This likely fixes a a few flakes in e.g. direct routing mode, where we started the test suite even though the direct routes for IPv6 endpoint addresses have not yet been installed.

Ref: #17895 (should fix that flake, but the test will not yet be unquarantined)

[gandro]

/test

[gandro]

/test

[gandro]

/test

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19/75/
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-net-next/2147/

Edit: Seems like the readiness check is not sufficient, as it does not consider node count. Will try to modify ciliumHealthPreFlightCheck instead

[gandro]

/test

Edit: Uncovered a bug in K8sDatapathConfig Host firewall * with endpoint routes

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.9/77/
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19/83/
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-5.4/74/

Reproducible on Vagrant via ping -s 8 <healthep>, the reply packet is dropped as invalid due its size. Will push a work around for now. Created #18177

[gandro]

/test

Runtime failed with a provisioning failure: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/810/

Edit: All Jenkins pipeline affected by these changes are green ✔️ Will re-run after pushing a few minor changes and mark ready for review.

[gandro]

/test

[gandro]

Addressed two nits, rebased and removed the commit which un-qurantines the test (and thus this PR needs no sign-off from team loadbalancer anymore). Main diff:

diff --git a/pkg/health/client/client.go b/pkg/health/client/client.go
index ab2b2f088466..fe93c9834d55 100644
--- a/pkg/health/client/client.go
+++ b/pkg/health/client/client.go
@@ -274,7 +274,7 @@ func GetAllHostAddresses(node *models.NodeStatus) []*models.PathStatus {
 }
 
 // GetEndpointPrimaryAddress returns the PrimaryAddress for the health endpoint
-// within node. If node.Host is nil, returns nil.
+// within node. If node.HealthEndpoint is nil, returns nil.
 func GetEndpointPrimaryAddress(node *models.NodeStatus) *models.PathStatus {
        if node.HealthEndpoint == nil {
                return nil
@@ -283,7 +283,8 @@ func GetEndpointPrimaryAddress(node *models.NodeStatus) *models.PathStatus {
        return node.HealthEndpoint.PrimaryAddress
 }
 
-// GetEndpointSecondaryAddresses returns the secondary host addresses (if any)
+// GetEndpointSecondaryAddresses returns the secondary health endpoint addresses
+// (if any)
 func GetEndpointSecondaryAddresses(node *models.NodeStatus) []*models.PathStatus {
        if node.HealthEndpoint == nil {
                return nil

[gandro]

/test

Job 'Cilium-PR-K8s-GKE' failed and has not been observed before, so may be related to your PR:

Click to show.

Test Name

K8sHealthTest cilium-health Checks status between nodes

Failure Output

FAIL: Kubernetes DNS did not become ready in time

If it is a flake, comment /mlh new-flake Cilium-PR-K8s-GKE so I can create a new GitHub issue to track it.

[gandro]

• GKE ( https://jenkins.cilium.io/job/Cilium-PR-K8s-GKE/7185/) failed with DNS errors, which looks very unrelated (but is similar to an existing flake, might be the same, might not): CI: K8sHealthTest cilium-health Checks status between nodes: Kubernetes DNS did not become ready in time #18258
• conformance-test-ipv6 (https://github.com/cilium/cilium/runs/4518035234?check_suite_focus=true)seems to have hit CI: Smoke test: Timed out waiting for the condition on pods #12279
• Multicluster (https://github.com/cilium/cilium/actions/runs/1577173392) failed during provisioning ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=This operation will exceed max secondary ranges per subnetwork (30) for subnet "default", consider reusing existing secondary ranges or use a different subnetwork.

[gandro]

/test-gke

[gandro]

/ci-multicluster