health-server prober bugfixes

[thorn3r]

This PR fixes 2 separate bugs in the health-server prober:

The 1st commit resets the cache if a client-side error is encountered:

    reset health-server's node cache on error
    
    The health-server prober maintains a cache of the nodes in the cluster
    to avoid requesting the entire list of cluster nodes on each interval.
    When calling the daemon API's /cluster/nodes endpoint, it subscribes as
    a client so that it only receives a diff since the last request, which
    it applies to the cache.
    
    If a client-side error is encountered when making this request, the
    cache is not updated and events are lost since the server is unaware.
    
    This commit flushes the prober's node cache if any error occurs and sets
    its clientID to 0. On the subsequent request, the prober will be
    subscribed as a new client and receive the full list of nodes.

The 2nd commit ensures we actually skip adding empty IP addresses to the prober's node cache:

    prevent adding empty IPs to health-server cache
    
    The logic in the prober.setNodes function should skip adding nodes to
    the prober's cache if the IP address is empty. This is dependant on
    resolveIP() returning a nil address if it is unable to resolve the IP,
    however this will only happen if net.ResolveIPAddr returns an error. In
    the case that the address is an empty string, net.ResolveIPAddr will
    return a pointer to an empty net.IPAddr{} and error will be nil. This
    allows a node with no IP addresses to be added to the cache.
    
    server.resolveIP() has been updated to skip the address if it is empty.
    This prevents the prober from trying to probe and report on the empty
    addresses.
    
    The following is the output of `cilium-health status` for a faux
    CiliumNode with no addresses before and after this commit (snipped for
    brevity):
    
    Before:
    ---
    root@kind-worker:/home/cilium# cilium-health status
    Probe time:   2023-12-08T14:58:11Z
    Nodes:
      kind-kind/kind-worker2:
        Host connectivity to :
          ICMP to stack:   Connection timed out
          Secondary connectivity to :
            ICMP to stack:   Connection timed out
        Endpoint connectivity to 10.244.1.202:
          ICMP to stack:   Connection timed out
          Secondary connectivity to fd00:10:244:1::c879:
            ICMP to stack:   Connection timed out
    ---
    
    After:
    ---
    root@kind-control-plane:/home/cilium# cilium-health status
    Probe time:   2023-12-08T14:44:35Z
    Nodes:
      kind-kind/kind-worker2:
        Endpoint connectivity to 10.244.1.202:
          ICMP to stack:   Connection timed out
          Secondary connectivity to fd00:10:244:1::c879:
            ICMP to stack:   Connection timed out
    ---

Fix bugs in health-server that cause the state in the prober's cache to drift and allow nodes with empty IP addresses to be added.

[thorn3r]

/test

[marseel]

LGTM, Thanks!

What was a bit confusing for me was that response contains "NodesAdded" and "NodesRemoved", but it seems like "NodesAdded" also contains "NodesUpdated":

cilium/daemon/cmd/status.go

Line 494 in 7df3194

for i, added := range c.NodesAdded {

🤯
so in case we are missing IP we should receive later on updated node

[dylandreimerink]

All seems good except for one unresolved conversation, is it blocking? https://github.com/cilium/cilium/pull/29745/files#r1431802011

[thorn3r]

@dylandreimerink thanks for pointing that out. Responded/resolved 👍