Reclaim IPs in PodSubnet when subnet is full or close to full

[tyler-lloyd]

Component (Azure NPM or Azure CNI):

Azure CNI + pod subnet (CNS)

Describe in detail the feature/behavior/change you'd like to see:

When a pod subnet has completely or nearly exhausted the IPs available, there should be a way to reclaim some of the unused IPs on different nodes since nodes are allocated IPs in batches of 16 but it does not mean IPs are necessarily in use.

Orchestrator(e.g. Kubernetes, Docker):

Kubernetes

Operating System (Linux/Windows):

Both

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

[rbtr]

Collecting existing PRs for this feature:

• Creating new CRD for ClusterSubnetState. #1465
• Build out ClusterSubnetStatus client/api #1487
• Update poolmonitor to support batch size of 1 #1492
• Watch ClusterSubnetState and drive subnet exhaustion batch changes #1494
• Fix ClusterSubnetState JSON annotations and kubectl printcolumns #1591
• Align pool to batch during scale up #1593
• Optimize subnet exhaustion handling #1594

[olljanat]

@rbtr good summary. However I wonder that if it have been discussed here as option that Azure CNI would allow users to define different subnet for pod than what is used by AKS nodes?

Because as far I can see AKS nodes have ip-masq-agent-v2 enabled for all traffic which target is outside of cluster-cidr. That way users would be able to create 100.64.0.0/10 subnet for this purpose which based on this looks to be supported.

That would be very useful for all of those companies who have a lot of private IPv4 addresses in use already.

[rbtr]

@olljanat pods and nodes can have separate subnets already:

"The new dynamic IP allocation capability in Azure CNI solves this problem by allotting pod IPs from a subnet separate from the subnet hosting the AKS cluster."

That is actually the only mode in which this feature will work 😉

[olljanat]

That is actually the only mode in which this feature will work 😉

@rbtr Thanks. That is already useful. However the fact that those need to be different subnets inside of same vnet it problematic because on then also pods subnets get automatically included to network peering so even if I use those slices from that 100.64.0.0/10 those need to be unique inside of whole Azure landing zone setup. Would be nice to be able to create it similarly like kubenet works but I need setup which works also on Windows nodes.

[rbtr]

@olljanat maybe you are looking for something like AzCNI Overlay which is just now in public preview? 🙂

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

Issue closed due to inactivity.