Fix bug where cilium-health reports connectivity failures to stale IPs

[kkourt]

NodeAdd and NodeUpdate update the node state for clients so that they
can return the changes when client requests so. If a node was added and
then updated, its old and new version would be on the added list and its
old on the removed list. Instead, we can just update the node on the
added list.

Note that the setNodes() function on pkg/health/server/prober.go first
deletes the removed nodes and then adds the new ones, which means that
the old version of the node would be added and remain as stale on the
health server.

This was found during investigation of issues with inconsistent health
reports when nodes are added/removed from the cluster (e.g., #11532),
and it seems to fix inconsistencies observed a small-scale test I did to
reproduce the issue.

Fixes #11532.

[christarazi]

test-me-please

[joestringer]

Thanks for digging into this! I have some questions below, even if this may address some use cases I don't understand how we don't cause regressions in other areas with the current version. Would be good to get a clearer outline of the exact set of steps we believe are happening and how this addresses that case, and make sure we're not missing other cases in the fix for this case.

I think this also highlights that if we can come up with some better ways to automate testing in this area, we would benefit greatly.

[kkourt]

Just re-read through a few times and I get it now :-)

I also wrote a more detailed explanation, hope it helps.

[pchaigno]

4.9 pipeline failed with a bunch of seemingly-unrelated errors, restarting to see if they are flakes.
Previous run available here: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9/303/testReport/.
retest-4.9

[joestringer]

Filed #12993 to follow up on 4.9 failures, seems like an unrelated issue.

[joestringer]

K8s-1.12-Kernel-netnext CI failure Suite-k8s-1.12.K8sServicesTest Checks service across nodes Tests NodePort BPF Tests with direct routing and DSR is a known flaky test, can retry but I'm guessing it'll pass

[joestringer]

retest-net-next

[joestringer]

4.9 kernel CI failures got worse this time, Up from 22 failures to 71 🤔 :
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9/304/testReport/

EDIT: Oh, I see. This time the K8sIstioTest Istio Bookinfo Demo test ran first out of the whole suite, so when it failed (& failed to clean up), it likely caused even more of the subsequent tests to fail this time.

[joestringer]

Observed that this is not the only PR to be affected by the 4.9 issues so do not plan to have that block this PR. They also look completely unrelated to the code changes here.

Only other failure is in net-next job, which is known failure #12994