Cilium - production ready technology adoption - phase 1 of 3

[zuzzas]

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Use case. Why is this important?

We've got stuff to do before Cilium becomes production ready.

Proposed Solution

• Enable Cilium to be default on new setups
• VXLAN Support: [cilium] improvements #1594
• Ensure modules support Cilium's Way of Networking
  • Node Local DNS
    • LocalRedirectPolicy viability
  • istio
    Everything except cilium's L7 policies work well.
• Enable VPA: [cilium] improvements #1594
• Make Hubble turned off by default, as it consumes large amounts of CPU (upstream issue).

To Do

• Bump cilium version to 1.14 #4574
• Build kernel checker. #[deckhouse] check kernel versions for enabled modules #2709
  • Check kernel version if cilium module enabled on deckhouse module start and block queue in case if kernel has wrong version.
  • Add metric and alerts about wrong kernel version.
  • Add deckhouseReleases requirements and disruptions for cilium minimal kernel version.
  • Add init container for cilium agents to prevent run agents on nodes with wrong kernels.
• Provide plan about migration to cilium 1.12
  • [cni-cilium] bump cilium version to v1.12.5 #3252
  • [cni-cilium] Unprivileged mode #2961
  • [cni-cilium] Run connectivity test #2873
• Check Hubble performance - create an issue in backlog ([cni-cilium] Cilium Hubble high cpu usage #3078)
• Cilium Service Mesh - create an issue in backlog ([cni-cilium] Service Mesh research #3084)
• Bundling CiliumNetworkPolicies with Deckhouse
  • Deckhouse host network ciliumClusterWideNetworkPolicies. ([cni-cilium] Deckhouse host-network ciliumClusterWideNetworkPolicies. #3286)
    • Possibly using entities instead of enumerating IPs and ports. Use of entities leads to unpredictable results.
    • Consider an option to review policies before applying them to a cluster during Deckhouse upgrades
• [cni-cilium] L7 CiliumNetworkPolicies do not work #2806
• Cilium doesn't recognize node IP changing Cilium does not pick up changed node IPs cilium/cilium#17500

Backlog

• keepalived and network-gateway ([network-gateway] Migrate SNAT DaemonSet to Cilium #1719)
• Possibly migrate existing clusters. Probably, manually. Here is a success story
• [cni-cilium] Cluster is stuck until validating/mutating configs for pods are deleted #2886
• [cni-cilium] Debugging guide ADVANCED_USAGE.md #2808
• [cni-cilium] Clarify configuration options: cleanState, bpfLBMode, tunnelMode, svcSourceRangeCheck #2807
• [cni-cilium] Disable rp_filter in cilium agent pods #2900
• We need monitoring for the better observability of Cilium
• [cni-cilium] Service Mesh research #3084
• [cni-cilium] Cilium Hubble high cpu usage #3078
• [cni-cilium] limiting identity-relevant labels #3116
• [cni-cilium] Getting started for BareMetal installations #3145
• [cni-cilium] CNI cilium grafana dashboard #3497
• Problem with hostNetwork sockets cilium/cilium#24417
• Cilium 1.11 high memory consumption cilium/cilium#23930
• NodePort BPF with L7 netpol is broken for N/S when request is sent to node running SVC backend  cilium/cilium#21955 LoadBalancer/NodePort problem with hostNetwork enppoints (fixed in main)
• DSR with ExternalTrafficPolicy do not work DSR does not work for service endpoints in host netns cilium/cilium#10789 fixed in main
• Investigate issues with hostPort ([cni-cilium] hostPort works unpredictably #3035) All works as expected.
• ingress-nginx ([ingress-nginx] Switch trafficPolicy: Cluster to reduce module's complexity when Cilium is in use #1718)

Links:

• https://www.datadoghq.com/blog/cilium-metrics-and-architecture/

What we should test in case of new cilium versions:

• nginx inlet hostWithFailover - test iptables hack is working.
• create and test iptables port forwarding to kubernetes api - it's temporary solution for external load balancer for kubernetes api (more details in https://flant.slack.com/archives/C78AQG5EY/p1676366607789619) (fixed in 1.13)
• test case when module node-local-dns is enabled, but node-local-dns pods is restarted. Dns redirection should work correctly.

[zuzzas]

cilium/cilium#20379