IPAM Multi-Pool Follow-ups

[gandro]

This issue tracks the remaining work items around the Multi-Pool IPAM mode:

• Prepare Cilium API for IPAM pools #24248
• Preparatory refactoring for IPAM pools #24247
• Add support for allocating PodCIDRs from multiple IPAM pools #22762
• IPAM pools followups #25498
• Add IPPool CRDs (ippool-crd branch)
  • PR: Support defining IPAM pools using CiliumPodIPPool CRD #25824
• Add mechanism to determine a workloads IPAM pool based on pod and/or namespace annotations (Slack thread)
  • PR: multi-pool: Determine IP pool based on ipam.cilium.io/ip-pool annotation #25511
• multi-pool: Support allocating from new IPAM pools on demand #25765
• ipam: Add ability to automatically create CiliumPodIPPool resources in multi-pool IPAM mode #25991
• Add documentation around how to configure and use multi-pool IPAM mode
  • docs: Deprecate cluster-pool-v2beta #25767
  • Document multi-pool IPAM mode #26308
    • Document limitations
      • Overlapping IPAM pools are not supported due to IPCache limitations
      • Tunnel mode not supported (related Remove TUNNEL_MAP lookup #20170)
      • IPSec not supported
      • Legacy Masquerading not supported (related Node clusterpool Endpoint CIDRs used in AWS ENI mode erroneously #22273)
      • kvstore not supported multi-pool: Document unsupported kvstore mode #26662
  • Add kind-based workflow for e2e test ci: Add workflow for testing multi-pool IPAM #26175
• Collect CiliumPodIPPool resources in cilium sysdump
  • sysdump: collect CiliumPodIPPools cilium-cli#1726

Best effort/cleanup:

• Convert to netip.Prefix earlier in allocator code ipam: Add ability to automatically create CiliumPodIPPool resources in multi-pool IPAM mode #25991 (comment)
  • PR: IPAM multipool followups #26138
• Fix issue where IP owner of non-default pool IPs does not show up in cilium status --verbose (PR ipam: report IP owner of non-default pool IPs in multi-pool IPAM #27968)
• Fix bug where auto-direct-node-routes parses spec.podCIDR in multi-pool IPAM mode (PR ipam: Fix invalid PodCIDR in CiliumNode in ENI/Azure/MultiPool mode #26663)
• Add check to ensure IPAM pools don't overlap
• Fix bug where Cilium agent would release CIDRs it considers unused because the endpoints using them weren't restored yet. (PR ipam/multipool: wait for restoration before releasing CIDRs #26668)
• Fix bug where status.ipam.operator-status.error is not cleared after a successful allocation. To reproduce, create Cilium installation without default pool, create default pool after one minute, observe nodes successfully allocating pod CIDRs from default pool, but cannot allocate from non-existing pool: default error is not cleared Multi-Pool: status.ipam.operator-status.error is not cleared after a successful allocation #28758
• IPAM: Validate Changes to CiliumPodIPPool #26966
• BGP CP: Add Support for Multi-Pool IPAM #26937
• CI: multi-pool-ipam-conformance-test - cilium agent does not become ready #27417

[gandro]

Removing the release blocker label from this. All necessary parts have been merged. The rest are smaller improvements that can be done in v1.15 as part of the stabilization of the feature

[chestack]

multiple CIDRs per cluster/node is a highly useful feature, especially for workloads from different tenants. Are there any plans to enable this feature for cloud-related(ENI, Alibaba...) ipam modes?

[gandro]

multiple CIDRs per cluster/node is a highly useful feature, especially for workloads from different tenants. Are there any plans to enable this feature for cloud-related(ENI, Alibaba...) ipam modes?

Not at the moment, there are no plans to support cloud-based IPAM modes.

One difficulty in particular is that this will require deeper changes to the protocol between cilium-operator and per-node cilium-agent in ENI mode, as at the allocation mechanism for ENI modes and multi-pool work fundamentally different. In ENI mode, the operator decides when and how many IPs a node receives, where as with multi-pool, the agent issues an explicit request. There are multiple options to explore, but such a change requires a design document first.

[gandro]

I'm closing this issue. There are remaining bugs, but those are better tracked via the area/multipool label.

[chaunceyjiang]

Hi @gandro It seems to me that Add check to ensure IPAM pools don't overlap has not been implemented yet. Is that correct? I would like to take up this issue.

[gandro]

Hi @gandro It seems to me that Add check to ensure IPAM pools don't overlap has not been implemented yet. Is that correct? I would like to take up this issue.

Yes, that has not yet been implemented. Any help here is very much appreciated 🎉 !
I've added a note about it in #26966 but it can be implemented separately.

One issue with it is that we're not sure how to best address it:

• I don't think we can check this with CEL, since it is a cross-resource check.
• Maybe we could do it via webhooks, but that's not something we can install via helm install
• Maybe we could have a check in the operator? The question is how we expose/reject conflicting pools. A simple first version is to just log it as an error, but this will probably not be seen by most users. A more sophisticated version could extend the CiliumPodIPPool CRD with a status field, where we then write a "ok"/"not ok" flag into the status field of the resource. We could then also teach the operator to only use "ok" pools for allocation.

I'm very open to other ideas as well!

[chaunceyjiang]

Perhaps we can build a webhook into operator. Similar to https://book.kubebuilder.io/cronjob-tutorial/webhook-implementation, This way, we don't need to install an additional webhook-server.

[gandro]

Perhaps we can build a webhook into operator. Similar to https://book.kubebuilder.io/cronjob-tutorial/webhook-implementation, This way, we don't need to install an additional webhook-server.

Ah, good point. Yeah, I think we have discussed a ValidatingAdmissionWebhook inside the operator in the past too, it would e.g. also be useful for #23152 - but I think there are some operational concerns about webhooks too (e.g. #23152 (comment)).

I'm not part of sig-k8s, so don't feel like I have the expertise to judge how complicated/tricky a webhook solution could be. On the other hand, having something which is opt-in might also allow us to gain a bit of experience with this.
It seems to me that adding webhook infrastructure might be something that warrants a CFP.

[chaunceyjiang]

ut I think there are some operational concerns about webhooks too (e.g. #23152 (comment)).

Oh, you are right.