health: Fix cluster-health-port for health endpoint

[gandro]

To determine cluster health, Cilium exposes a HTTP server both on each
node, as well as on the artificial health endpoint running on each node.
The port used for this HTTP server is the same and can be configured via
cluster-health-port (introduced in #16926) and defaults to 4240.

This commit fixes a bug where the port specified by
cluster-health-port was not passed to the Cilium health endpoint
responder. Which meant that cilium-health-responder was always
listening on the default port instead of the one configured by the user,
while the probe tried to connect via cluster-health-port. This
resulted in the cluster being reported us unhealthy whenever
cluster-health-port was set to a non-default value (which is the case
our OpenShift OLM for v1.11):

Nodes:
  gandro-7bmc2-worker-2-blgxf.c.cilium-dev.internal (localhost):
    Host connectivity to 10.0.128.2:
      ICMP to stack:   OK, RTT=634.746µs
      HTTP to agent:   OK, RTT=228.066µs
    Endpoint connectivity to 10.128.11.73:
      ICMP to stack:   OK, RTT=666.83µs
      HTTP to agent:   Get "http://10.128.11.73:9940/hello": dial tcp 10.128.11.73:9940: connect: connection refused

Fixes: e624868 ("health: Add a flag to set HTTP port")

⚠️ Marking as release blocker since this is required for our v1.11 OpenShift OLM templates.

[tklauser]

Nice find!

Non-blocking nit in case another revision is needed: there is a tiny typo in the commit message of the first commit, 2nd paragraph (and the PR description as well): clsuter-health-port instead of cluster-health-port.

[gandro]

/test

[gandro]

For cross-reference:

cilium/cilium-health/responder/main.go

Line 40 in aef5002

flag.IntVar(&listen, "listen", healthDefaults.HTTPPathPort, "Port on which the responder listens")

[gandro]

Decided to add drive-by cleanup of the responder while at it.

[gandro]

/test

Job 'Cilium-PR-K8s-1.22-kernel-4.19' failed and has not been observed before, so may be related to your PR:

Click to show.

Test Name

K8sServicesTest Checks service across nodes Tests NodePort BPF Tests with direct routing Tests NodePort

Failure Output

FAIL: Request from k8s1 to service http://[fd04::11]:31022 failed

If it is a flake, comment /mlh new-flake Cilium-PR-K8s-1.22-kernel-4.19 so I can create a new GitHub issue to track it.

[gandro]

• ConformanceKind1.19 failed with a variant of ConformanceEKS (ci-eks) CI: no-policies/pod-to-world/http-to-jenkins-cilium: Fails during connectivity test after deploying with encryption enabled #16938 https://github.com/cilium/cilium/runs/4370353553?check_suite_focus=true

[joestringer]

Thanks!

[joestringer]

@gandro the individual test case failures look similar to #16938, however in all of the other cases I've been tracking recently, that issue only occurs in an EKS environment when a cilium-cli test run was successful without encryption, then encryption is enabled, then the cilium-cli tests are re-run. For that issue it suggests that the problem is either a test pollution issue or some sort of resource leak issue triggered by the previous CI run.

In the case in https://github.com/cilium/cilium/runs/4370353553?check_suite_focus=true , this is failing on the first try in a kind environment so I think we should treat it as a different failure.

I agree it doesn't seem related to the changes in this PR though, so I'm OK with going ahead to merge this PR regardless of that failure.

[joestringer]

Looking at the test-1.16-netnext job failure, I see four types of failures:

1. Found 1 io.cilium/app=operator logs matching list of errors that must be investigated:, appears to be CI: K8sBandwidthTest Checks Bandwidth Rate-Limiting Checks Pod to Pod bandwidth, vxlan tunneling: Operator Failed to update lock: use of closed network connection #18053
2. FAIL: failed to ensure kubectl version: failed to download kubectl, CI: failed to ensure kubectl version: failed to download kubectl #18060
3. BeforeAll failures which are typically triggered by some bad state in the cluster left behind by a previous test, eg because kubectl couldn't be set up
4. Test suite timed out after 2h50m0s, because too many things failed, taking too long.

The test-1.22-4.19 job failure looks like #18014 .

LGTM to merge.

[gandro]

Thanks a ton for triaging the failiures @joestringer 🙏 And yes, agreed that the Kind failure looks new :/