pkg/k8s: fallback on retrieving CiliumNode from kube-apiserver

Retrieving objects from caches can be useful to prevent doing useless
requests to kube-apiserver. In the unlikely event that the object
doesn't exist in the local cache Cilium can try to retrieve it from
kube-apiserver directly. For this particular case, with CiliumNode, it
is causing Cilium to fatal as it is unable to retrieve CiliumNode from
the cache, due subsystem initialization issues, thus we will fallback on
retrieving the object directly from kube-apiserver.

In this case, the subsystem initialization issue happened due to the
fact that CiliumNode watcher is blocked on its event handler by the
egressGatewayManager [1] which is blocked by the initialization of the
identity allocator [2]. Unfortunately, the identity allocator is only
initialized at a later stage causing the CiliumNode cache from being
populated with all of its nodes.

[1] https://github.com/cilium/cilium/blob/933bdcbec9319b0148b12688f720fbaaf55e0dba/pkg/k8s/watchers/cilium_node.go#L56
[2] https://github.com/cilium/cilium/blob/933bdcbec9319b0148b12688f720fbaaf55e0dba/pkg/egressgateway/manager.go#L83

Fixes: 69e4c69 ("k8s: optimize API calls made to kube-apiserver")
Signed-off-by: André Martins <andre@cilium.io>

pkg/k8s/watchers/cilium_node.go

@@ -168,7 +168,7 @@ func (k *K8sWatcher) GetCiliumNode(ctx context.Context, nodeName string) (*ciliu
 	}
 	k.ciliumNodeStoreMU.RUnlock()
 
-	if getFromAPIServer {
+	if !exists || getFromAPIServer {
 		// fallback to using the kube-apiserver
 		return k.clientset.CiliumV2().CiliumNodes().Get(ctx, nodeName, v1.GetOptions{})
 	}