Performance Improvements and Native Routing Mode for Cilium

[M4t7e]

This PR modifies the Cilium configuration to achieve the best possible performance and also native routing functionality for Cilium, allowing configuration of a fully integrated network model. For more background information, please have a look here: #911

Changes:

• Fixed decoupling issue between CCM IPAM and the Cilium IPAM. Now Cilium honors the IPAM from the CCM and Hetzner Network Routes are set accordingly to allow native routing directly to pod IPs as well. See also Update handling of custom cluster-cidr #902 (comment)
• Updated Cilium options according to the latest deprecation warnings: https://docs.cilium.io/en/latest/operations/upgrade/#deprecated-options
• The Cilium version is now maintained explicitly. This PR sets it to 1.14.0 and some settings require that as the minimum version.
• Enabled per endpoint routes instead of routing via the cilium_host interface.
• Native routing mode has been added so that even a fully integrated network model (flat model) can be configured with Cilium
• Enabled eBPF-based masquerading, considered the most efficient implementation.
• Added LoadBalancer & NodePort XDP Acceleration (eBPF is operating directly in the networking driver instead of a higher layer)
• Added option to enable the Egress Gateway for Cilium (disabled by default)

The native routing mode is disabled by default as we have been using tunnel mode so far. As we gain more experience with this setup, I would suggest changing the default to native. This also appears to be the setting Hetzner uses when testing their CCM with Cilium.

I've done my best to test many different cases, but I'd be very grateful for a thorough review 🙂

[mysticaltech]

@M4t7e Beautiful! That is a work of art my friend 🙏

[mysticaltech]

Will give 24h-48h for others in the team to review if they want, and will merge, this is huge! 🚀

[ifeulner]

@M4t7e cool stuff need to test this asap, but currently working on other areas. But thanks a lot!

[aleksasiriski]

@M4t7e Are there any known issues with native routing on Hetzner?

[M4t7e]

@aleksasiriski No issues I am aware of. With the new setup the Hetzner CCM takes care about all Pod routes and updates the Hetzner Network Routes accordingly. The old setup bypassed the Hetzner CCM entirely and that was leading to connectivity issues if a native routing setup (routingMode: native) was used.

Old (dedicated IPAM that bypasses HCCM):

ipam:
 operator:
  clusterPoolIPv4PodCIDRList:
   - ${var.cluster_ipv4_cidr}

New (native K8s IPAM with HCCM support):

ipam:
  mode: kubernetes
k8s:
  requireIPv4PodCIDR: true