[v1.9] daemon: Avoid blocking datapath on node discovery

[aanm]

Previously, the datapath relied on node discovery completing. With a
kvstore configured, this meant that node registration would also need to
complete first.

If the kvstore is deployed as pods intended to be managed by Cilium,
then this creates a chicken and egg problem. The kvstore pod cannot come
online because the datapath is blocked. The datapath is blocked because
the agent cannot register the node into the kvstore. And finally, the
node cannot be registered into kvstore because kvstore is not online.

Here's how the issue manifested itself:

$ kubectl -n cilium describe pods etcd-operator-59cf4cfb7c-288qx
Name:           etcd-operator-59cf4cfb7c-288qx
Namespace:      cilium
Priority:       0
Node:           gke-chris-form3-cluster-default-pool-3f30c3b5-zd0f/10.138.0.10
Start Time:     Fri, 15 Jan 2021 14:56:24 -0800
Labels:         io.cilium/app=etcd-operator
                pod-template-hash=59cf4cfb7c
...
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               83s   default-scheduler  Successfully assigned cilium/etcd-operator-59cf4
  Warning  FailedCreatePodSandBox  22s   kubelet            Failed create pod sandbox: rpc error: code = Unk
5bf622367" network for pod "etcd-operator-59cf4cfb7c-288qx": networkPlugin cni failed to set up pod "etcd-op
ent client after 30.000000 seconds timeout: Get "http:///var/run/cilium/cilium.sock/v1/config": dial unix /v
Is the agent running?
  Normal  SandboxChanged  21s  kubelet  Pod sandbox changed, it will be killed and re-created.

To break the chicken and egg problem with the datapath, we need to split
the "node discovery" ((*NodeDiscovery).StartDiscovery()) into two
separate phases: (1) local node state is initialized and (2) node
registration which includes syncing to the kvstore if that's configured.
The (*NodeDiscovery).Registered channel will close when (2) completes.
The new channel (*NodeDiscovery).LocalStateInitialized will close when
(1) completes. This is because the datapath only relies on (1) being
complete and doesn't depend on (2).

This will unblock the datapath from the implicit dependency on the
kvstore and allow the kvstore pods to come online (Cilium generates
endpoints for them), while node registration into the kvstore continues
on in the background.

Fixes: 43997f5 ("loader: Wait for node configuration to generate datapath")
Related: 7045103 ("daemon: Move KVStore initialization earlier")

Co-authored-by: André Martins andre@cilium.io
Co-authored-by: Joe Stringer joe@cilium.io
Co-authored-by: Paul Chaignon paul@cilium.io
Co-authored-by: Kornilios Kourtis kornilios@isovalent.com
Signed-off-by: Chris Tarazi chris@isovalent.com

Fix bug Cilium hangs with kvstore configured

[aanm]

test-backport-1.9

[aanm]

test-backport-1.9

[nebril]

test-missed-k8s

[aanm]

I've dropped the last commit from this PR which was incrementing the timeout for the k8s jobs. The CI had passed so there's no need to re-run it again.

[christarazi]

👍