Improve Cilium integration with managed Kubernetes providers

[aanm]

This PR introduces steps on how to install Cilium in cloud providers. The reason behind this PR is described in detail in #16602. Initially, the changes in this PR were supposed to be a few but the commit "pkg/k8s: replace GetNode instances with a k8s store" is necessary to make sure nodes that have the taint node.cilium.io/agent-not-ready set after Cilium started, will also have that taint removed without restart Cilium, by watching for Node events.

Marked this PR as needs-backport/1.10 as it tremendously improves the deployment of Cilium in cloud providers.

It should be easier to review the changes on a per-commit basis.

Provide new installation steps to deploy Cilium in managed kubernetes providers (GKE, EKS, AKS) to allow scale up and down node pools.

Fixes #7404
Fixes #16602
Fixes #15177
Fixes #16542

⚠️ Note for reviewers, this PR was ran with a test commit to test the new changes into the conformance GH workflows. All of them have passed except multicluster which is currently broken on master.

[aanm]

test-me-please

[aanm]

test-runtime

[aanm]

test-runtime

[aanm]

test-runtime

[aanm]

test-runtime

[aanm]

test-me-please

[thejosephstevens]

I'm holding off on deep diving in the workflow changes as I have a bit of a high-level concern to address first: I feel like this solution, while neat, feels like a complete kludge from a user perspective.
I feel like it complicates cluster setup a lot, especially on AKS and EKS where nodepools are cumbersome to manage, while also restricting the options available for setting up the clusters (e.g. the load balancer choices on AKS).
Are we really OK with these drawbacks? Do we want this to be the default Cilium installation experience?

I see it as an unfortunate price to pay. Right now the AKS integration is broken and although usability is worse with this PR, it guarantees that there won't be any Cilium issues in it. Once we have a better integration in AKS, and / or this is fixed Azure/aks-engine#4476 the usability will improve.

I would advocate for having this as an optional thing for advanced users, and not the default installation. In particular, the simple getting started guide suddenly becomes a whole lot more complex :/

The issue I see with it is that people assume the GSG is enough to run in production, which kind of should it be, and if they only follow the "simple" GSG, they might face the issues we are trying to avoid with this PR.

Speaking as a user of AKS and Cilium, I think @aanm 's assessment here aligns with what I look for as an operator. We've had to deal with weird operational issues around the default pool that they force you to launch for as long as we've been on AKS (almost two years now), so having to work around it while setting up a CNI is not a big surprise.

It's a lot more important to me that I can follow the GSG and get a prod-ready cluster, we already had to build tooling to deal with the managed cluster bootstrap anyway.

[smnmtzgr]

@aanm / @thejosephstevens / @nbusseneau / @christarazi

We are also users of AKS and Cilium together.

In my opinion this change makes it very "hard" to use Cilium with AKS. It will also become very hard to update Cilium (or get the backport for 1.10) within an already running AKS-cluster. Will there be a documentation-part on how to do this? We deploy AKS-clusters using terraform. This workaround will be very cumbersome to implement with terraform.

In my opinion improving the STARTUP_SCRIPT would have been the better method/thing to fix these issues because it doesnt break so much in the deployment/operation tasks the operators have. E.g. #16356 for AKS.

The best solution would be a better integration of Cilium into the cloud providers. For example it should be possible to deploy AKS Clusters with CNI=None. So that one can choose a "custom" CNI like Cilium which will be installed/used. So no problems would occur with Azure-CNI being active before Cilium is ready. E.g. like suggested here: Azure/AKS#2092
Sure, thats something you would have to discuss with the cloud providers, but it would be the best one.

In my eyes this big change/merge/backport is not really a good thing for us operators. Sure it makes Cilium more stable on cloud providers managed kubernetes offerings. But the price to pay is too high. As long as there is no direct integration into the cloud provider investigating time into the STARTUP_SCRIPT and make it a lot more fail-safe would have been the better way in my eyes.

@aanm with Azure/aks-engine#4476 you refer to "aks-engine". I think you should open the issue here: https://github.com/Azure/AKS/issues ... aks-engine is not the "managed AKS solution".

[mattstam]

This should be far easier to set up on AKS now: https://docs.microsoft.com/en-us/azure/aks/use-byo-cni?tabs=azure-cli and won't need the node taint hack.

[thejosephstevens]

This should be far easier to set up on AKS now: https://docs.microsoft.com/en-us/azure/aks/use-byo-cni?tabs=azure-cli and won't need the node taint hack.

How does this avoid needing the node taint? As best I can tell AKS is still expecting you to deploy a daemonset for the CNI so you would still have race conditions for scheduling on node scaleup.

[aanm]

FYI @mattstam @thejosephstevens the documentation for the BYO CNI is being done in #19379