[GEP-36] `SelfHostedShootExposure` in Cilium extension

[hown3d]

How to categorize this topic?

/area networking
/kind enhancement

/label teamsize/medium

What is the topic about?:
With GEP-36 we introduced a way to exposure the API server of Self hosted shoots via a new API SelfHostedShootExposure.

This topic is proposing to implement this API in the Cilium extension. We can leverage Cilium LB IPAM feature to allocate virtual IPs for Load balancers and advertise such IPs either via L2Announcements or BGP Advertisements.

PoC for doing this with Cilium:
https://github.com/hown3d/cilium-loadbalancer-apiserver

[timebertt]

ref gardener/enhancements#36

[timebertt]

/retitle [GEP-36] SelfHostedShootExposure in Cilium extension

[hown3d]

We had a bit of a rough start with using cilium in the gind setup.
When the API server is rolling towards managing itself, cilium looses connection towards the API server by using the default kubernetes service cluster IP (10.4.0.1).
When adding the API server IP of the gind-machine-0 as IP of the API server (using localhost as the api server is running on hostNetwork like cilium) to cilium it was able to start and connecting to the API server.

EDIT:
When API server is dying because the underlying ETCD is being changed and cilium looses connection, NetworkPolicy end up in a undeterministic state.
When setting up Cilium to connect to the API server using localhost, reconnects to it and reconciles NetworkPolicies, the connection is working again.
We want to add a CiliumNodeConfig resource that matches only control plane nodes to add a config entry to use localhost for connecting to the api server via localhost.

[timebertt]

@hown3d can you update this issue with the WIP branch, the discussed points, and planned/unplanned next steps?

[hown3d]

The current implementation lives at gardener/gardener-extension-networking-cilium@master...hown3d:gardener-extension-networking-cilium:selfhosteshootexposure

There were some neccessary things to add to cilium to make it work for SelfHostedShoots:

• Cilium needs to communicate directly to the API server either via localhost if it runs on control plane or directly with the control plane node IPs
• Cilium in kube-proxy replacement mode tries to fetch the API server URL via a lookup on Cluster.status.advertisedAddresses. This is currently not populated by gardenadm, so this is a temporary branch to do so: gardener/gardener@0b0dc9a
  However one must keep in mind that this could reside in a cyclic dependency: If Cilium is responsible to provide the LoadBalancer for the AdvertisedAddress it is not beneficial to let Cilium connect via this path to the API server. That's why we are in favor for the upper solution.

For SelfHostedShootExposure we were able to provide a simple implementation in the extension by using a Service of type LoadBalancer with the respective Cilium LoadBalancerClass and an EndpointSlice that contains the API server IPs. This is similar to the implementation in provider-local.
Currently we do implement only L2Announcements to distribute the virtual IP of the LoadBalancer via Gratious ARP. BGP support could be added in the Future.