nodediscovery: Fix bug where CiliumInternalIP was flapping

[gandro]

This fixes a bug in UpdateCiliumNodeResource where the CiliumInternalIP (aka cilium_host IP, aka router IP) was flapping in the node manager during restoration (i.e. during cilium-agent restarts).

In particular in cluster-pool mode, UpdateCiliumNodeResource is called before the cilium_host IP has been restored, as there are some circular dependencies: The restored IP can only be fully validated after the IPAM subsystem is ready, but that in turn can only happen if the CiliumNode object has been created. The UpdateCiliumNodeResource function however will only announce the cilium_host IP if it has been restored.

This commit attempts to break that cycle by not overwriting any already existing CiliumInternalIP in the CiliumNode resource.

Overall, this change is rather hacky, in particular it does not address the fact that other less crucial node information (like the health IP) also flaps (see also #28299). But since we want to backport this bugfix to older stable branches too, this change is intentionally kept as minimal as possible.

Example node event (as observed by other nodes) before this change:

2023-12-18T12:58:20.070330814Z level=debug msg="Received node update event from custom-resource" node="{\"Name\":\"kind-worker\",\"Cluster\":\"default\",\"IPAddresses\":[{\"Type\":\"InternalIP\",\"IP\":\"172.18.0.4\"},{\"Type\":\"InternalIP\",\"IP\":\"fc00:c111::4\"}],..." subsys=nodemanager
2023-12-18T12:58:20.208082226Z level=debug msg="Received node update event from custom-resource" node="{\"Name\":\"kind-worker\",\"Cluster\":\"default\",\"IPAddresses\":[{\"Type\":\"InternalIP\",\"IP\":\"172.18.0.4\"},{\"Type\":\"InternalIP\",\"IP\":\"fc00:c111::4\"},{\"Type\":\"CiliumInternalIP\",\"IP\":\"10.0.1.245\"}],..." subsys=nodemanager

After this change (note the CiliumInternalIP present in both events):

2023-12-18T15:38:23.695653876Z level=debug msg="Received node update event from custom-resource" node="{\"Name\":\"kind-worker\",\"Cluster\":\"default\",\"IPAddresses\":[{\"Type\":\"CiliumInternalIP\",\"IP\":\"10.0.1.245\"},{\"Type\":\"InternalIP\",\"IP\":\"172.18.0.4\"},{\"Type\":\"InternalIP\",\"IP\":\"fc00:c111::4\"}],..." subsys=nodemanager
2023-12-18T15:38:23.838604573Z level=debug msg="Received node update event from custom-resource" node="{\"Name\":\"kind-worker\",\"Cluster\":\"default\",\"IPAddresses\":[{\"Type\":\"InternalIP\",\"IP\":\"172.18.0.4\"},{\"Type\":\"InternalIP\",\"IP\":\"fc00:c111::4\"},{\"Type\":\"CiliumInternalIP\",\"IP\":\"10.0.1.245\"}],...}" subsys=nodemanager

Reported-by: Paul Chaignon paul.chaignon@gmail.com

[gandro]

I have reproduced this also on v1.14. It likely requires a manual backport on that branch. I'll also check v1.13 and v1.12 now.

[gandro]

Converting back to draft, after discussing offline with Marco that this also needs to be fixed in kvstore mode. Also, the bug is present in all supported Cilium versions.

[gandro]

Apparently kvstore should also be fixed by this commit. Will manually verify tomorrow

[giorio94]

Thanks! The fix looks reasonable to me, especially considering the requirement for backporting it to all stable versions. Moving forward, we'll likely want to refactor the IPAM CiliumNode creation logic to avoid triggering unnecessary updates before restoring the full local node information (and fix also the bouncing of the other IP addresses), but that looks more invasive as a change.

I've also double-checked the kvstore case in combination with cluster-pool IPAM mode, and the CiliumInternalIP does not flip anymore (it was caused by the CiliumNode to kvstore synchronization process implemented by the Cilium operator):

Full kvstore events received after restart

PUT
cilium/state/nodes/v1/default/kind-worker
{"Name":"kind-worker","Cluster":"default","IPAddresses":[{"Type":"CiliumInternalIP","IP":"10.0.1.199"},{"Type":"CiliumInternalIP","IP":"fd00::147"},{"Type":"InternalIP","IP":"172.19.0.3"},{"Type":"InternalIP","IP":"fc00:c111::3"}],"IPv4AllocCIDR":{"IP":"10.0.1.0","Mask":"////AA=="},"IPv4SecondaryAllocCIDRs":null,"IPv6AllocCIDR":{"IP":"fd00::100","Mask":"////////////////////AA=="},"IPv6SecondaryAllocCIDRs":null,"IPv4HealthIP":"","IPv6HealthIP":"","IPv4IngressIP":"","IPv6IngressIP":"","ClusterID":0,"Source":"custom-resource","EncryptionKey":0,"Labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"kind-worker","kubernetes.io/os":"linux"},"Annotations":null,"NodeIdentity":0,"WireguardPubKey":""}
PUT
cilium/state/nodes/v1/default/kind-worker
{"Name":"kind-worker","Cluster":"default","IPAddresses":[{"Type":"InternalIP","IP":"172.19.0.3"},{"Type":"InternalIP","IP":"fc00:c111::3"},{"Type":"CiliumInternalIP","IP":"10.0.1.199"},{"Type":"CiliumInternalIP","IP":"fd00::147"}],"IPv4AllocCIDR":{"IP":"10.0.1.0","Mask":"////AA=="},"IPv4SecondaryAllocCIDRs":null,"IPv6AllocCIDR":{"IP":"fd00::100","Mask":"////////////////////AA=="},"IPv6SecondaryAllocCIDRs":null,"IPv4HealthIP":"10.0.1.63","IPv6HealthIP":"fd00::11a","IPv4IngressIP":"","IPv6IngressIP":"","ClusterID":0,"Source":"local","EncryptionKey":0,"Labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"kind-worker","kubernetes.io/os":"linux"},"Annotations":{},"NodeIdentity":1,"WireguardPubKey":""}
PUT
cilium/state/nodes/v1/default/kind-worker
{"Name":"kind-worker","Cluster":"default","IPAddresses":[{"Type":"InternalIP","IP":"172.19.0.3"},{"Type":"InternalIP","IP":"fc00:c111::3"},{"Type":"CiliumInternalIP","IP":"10.0.1.199"},{"Type":"CiliumInternalIP","IP":"fd00::147"}],"IPv4AllocCIDR":{"IP":"10.0.1.0","Mask":"////AA=="},"IPv4SecondaryAllocCIDRs":null,"IPv6AllocCIDR":{"IP":"fd00::100","Mask":"////////////////////AA=="},"IPv6SecondaryAllocCIDRs":null,"IPv4HealthIP":"10.0.1.63","IPv6HealthIP":"fd00::11a","IPv4IngressIP":"","IPv6IngressIP":"","ClusterID":0,"Source":"custom-resource","EncryptionKey":0,"Labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"kind-worker","kubernetes.io/os":"linux"},"Annotations":null,"NodeIdentity":0,"WireguardPubKey":""}

[gandro]

/test