Azure CNI support for dedicated subnets

[palma21]

Support specifying different subnet for the pod CIDR address space, different than the node CIDR, and dynamic IP allocation.

https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni#dynamic-allocation-of-ips-and-enhanced-subnet-support-preview

[mkosieradzki]

Do you plan to allow configuration of different subnets for different pods?

What has happened with multitenancy in Azure CNI? Azure/azure-container-networking@37eed02?short_path=2ac40eb#diff-2ac40ebea97cc7ab31e945192f9f1411

[aanandr]

This capability will allow you to use multiple subnets for Pods where each Pod subnet is used within 1 or more node pools. @mkosieradzki - i didnt understand your question on multitenant CNI

[lastcoolnameleft]

How does this issue differ from #1338 ? This one show in Development for Azure Public and the other shows as Private Preview.

[palma21]

1338 is about creating nodepools in different subnets, the nodes and pods of those nodepools still use the same CIDR.

This feature is about giving pods different CIDRs than those of the nodes. And also dynamic allocation of pod IPs.

[jabbera]

Any idea when this is coming to EastUS2?

[aanandr]

@jabbera - we are aiming to make this capability available in all regions by mid-May.

[jabbera]

@aanandr many thanks! Can't wait.

[jabbera]

Question. Can I use the node pool subnet for this to just enable the dynamic allocation of addresses or do you have to use a different subnet?

[djsly]

@aanandr / @palma21 : is it expected to have Windows support (hybrid clusters) for GA ?

We are interested in test the Dynamic IP feature today and are waiting on
1- more region support
2- running on clusters with windows node pools

for Dynamic IP support, do we need use a dedicated pod subnet ?

@jonstoker

[aanandr]

@djsly - we will have more regions supported in a couple of more months. Widows support is also in the roadmap but i dont have a clear ETA.

[djsly]

@aanandr is it dependant on container support on windows ?

[aanandr]

@djsly - the expansion to other regions is not dependent on supporting the capability on Windows. We are rolling the changes to other regions and that will take some time. Sorry if my answer was confusing.

[djsly]

@djsly - the expansion to other regions is not dependent on supporting the capability on Windows. We are rolling the changes to other regions and that will take some time. Sorry if my answer was confusing.

I meant, the support for windows, what is blocking it, is it a networking dependancy, do you guys need to wait for window to move to containerd, etc ?

[aanandr]

Windows support requires some changes in our CNIv2 implementation. The current implementation only support Linux.

[jabbera]

It should really be documented that this requires tearing down your cluster to enable.

[jabbera]

@aanandr @palma21 If you use an ARM Template to set the pod subnet you cannot re-rerun the template. This is a deal breaker for us.

{
  "code": "DeploymentFailed",
  "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
  "details": [
    {
      "code": "ScaleVMSSAgentPoolFailed",
      "message": "We are unable to serve this request due to an internal error, Correlation ID: c78f1e47-115e-4216-8d62-e0360bc3fa7e, Operation ID: 7f3ac56f-4cf2-4f88-ac11-0fe5fdc11c45, Timestamp: 2021-06-22T13:04:18Z."
    }
  ]
}

[aanandr]

@jabbera - sorry for the delayed response. Can you describe what you were trying to exactly do?

[jabbera]

@aanandr This seems to be unrelated to the pod subnet preview. I'm now getting this error on clusters that don't have the pod subnet preview enabled.

[aanandr]

@jabbera - the original issue was related to Pod subnet creation. Can you please describe the exact problem?

[jabbera]

@aanandr hi! I deployed a cluster with the pod subnet feature turned on. I ran into some small issues so I decided to delete my cluster. The cluster managed resource group is deleted but the cluster continues to stick around, cannot be deleted, and won't release my subnet from my vnet so I can reclaim my address space.

[aanandr]

@palma21 - can you look into this?

[jabbera]

@palma21 2106220040006618 for the ticket. The cluster finally got deleted last night but the subnet is still hung up:

[genscape-agodfrey]

This feature going to GA is very important to the continued use of AKS for us. I would love to see an update on when we can expect this.

[DavidZisky]

@aanandr yes, I was referring to a separate subnet for pods and hosts in AzureCNI. Most of my clients with hybrid cloud scenarios could use that but many of them have a policy of not using any preview feature :(

P.S.
For me, the main use case is that in many hybrid cloud scenarios people assign subnets to AKS from on-prem range/systems. Therefore these subnets are usually small therefore assigning IP from such subnet to each pod is often seen as "waste". A common solution to that is to use kubenet but due to the fact that in order to create AKS with kubenet you need elevated privileges, it's swapping one problem to another.

[denniszielke]

@aanandr yes I want to split my cluster across different vnet. This is relevant because I want to isolate some workloads, some should use non routable ip space and some should have access to routable access to other vnets.

[aanandr]

@aanandr yes, I was referring to a separate subnet for pods and hosts in AzureCNI. Most of my clients with hybrid cloud scenarios could use that but many of them have a policy of not using any preview feature :(

P.S. For me, the main use case is that in many hybrid cloud scenarios people assign subnets to AKS from on-prem range/systems. Therefore these subnets are usually small therefore assigning IP from such subnet to each pod is often seen as "waste". A common solution to that is to use kubenet but due to the fact that in order to create AKS with kubenet you need elevated privileges, it's swapping one problem to another.

Thanks @DavidZisky - we are working on a few key features that are required before this can go GA. As of now it is looking like we will GA in late Q1 of CY22.
I wanted to understand the second portion of your message. Are you saying that the ideal solution would be one where you dont have to assign VNet IPs to Pods? I dont think KubeNet needs any special privileges for deployment.

[aanandr]

@aanandr yes I want to split my cluster across different vnet. This is relevant because I want to isolate some workloads, some should use non routable ip space and some should have access to routable access to other vnets.

@denniszielke - this seems to be about spreading a single cluster (viz cluster nodes) across multiple VNets and running Pods that use a combination of non-VNet IPs and VNet IPs (maybe in separate node pools).

[DavidZisky]

@aanandr yes, I was referring to a separate subnet for pods and hosts in AzureCNI. Most of my clients with hybrid cloud scenarios could use that but many of them have a policy of not using any preview feature :(
P.S. For me, the main use case is that in many hybrid cloud scenarios people assign subnets to AKS from on-prem range/systems. Therefore these subnets are usually small therefore assigning IP from such subnet to each pod is often seen as "waste". A common solution to that is to use kubenet but due to the fact that in order to create AKS with kubenet you need elevated privileges, it's swapping one problem to another.

Thanks @DavidZisky - we are working on a few key features that are required before this can go GA. As of now it is looking like we will GA in late Q1 of CY22. I wanted to understand the second portion of your message. Are you saying that the ideal solution would be one where you dont have to assign VNet IPs to Pods? I dont think KubeNet needs any special privileges for deployment.

Great, thanks for the estimate GA!

As for your question - kubenet needs write permissions for the subnet/routing table. In highly regulated environments this is a problem because this means that a user who wants to create kubenet based AKS needs to have that write permission for the route tables (also roleassignments/write). AzureCNI doesn't require these permissions so it can be used when there is a requirement to give users as little permissions as possible. But then since AzureCNI assigns IP addresses to pods from Azure vnet this creates another problem because in hybrid environments the CIDRs are often assigned from some on-prem systems and they are small.

So to answer your question ideal solution would be either 1) make it possible to deploy AKS with kubenet without the need for extra write permissions or 2) make something to AzureCNI so it doesn't assign IP addresses from vnet to each pod. This would mean either make it optionally work as kubenet so NAT pod IPs or the current solution of assigning separate subnet for hosts and pods also solves the problem in theory (because then I can assign the "official" CIDR that I get from on-prem system to nodes and create some separate Azure only subnet to pods).

Hope I explained it well. Is it clear now? :D

[aanandr]

@aanandr yes, I was referring to a separate subnet for pods and hosts in AzureCNI. Most of my clients with hybrid cloud scenarios could use that but many of them have a policy of not using any preview feature :(
P.S. For me, the main use case is that in many hybrid cloud scenarios people assign subnets to AKS from on-prem range/systems. Therefore these subnets are usually small therefore assigning IP from such subnet to each pod is often seen as "waste". A common solution to that is to use kubenet but due to the fact that in order to create AKS with kubenet you need elevated privileges, it's swapping one problem to another.

Thanks @DavidZisky - we are working on a few key features that are required before this can go GA. As of now it is looking like we will GA in late Q1 of CY22. I wanted to understand the second portion of your message. Are you saying that the ideal solution would be one where you dont have to assign VNet IPs to Pods? I dont think KubeNet needs any special privileges for deployment.

Great, thanks for the estimate GA!

As for your question - kubenet needs write permissions for the subnet/routing table. In highly regulated environments this is a problem because this means that a user who wants to create kubenet based AKS needs to have that write permission for the route tables (also roleassignments/write). AzureCNI doesn't require these permissions so it can be used when there is a requirement to give users as little permissions as possible. But then since AzureCNI assigns IP addresses to pods from Azure vnet this creates another problem because in hybrid environments the CIDRs are often assigned from some on-prem systems and they are small.

So to answer your question ideal solution would be either 1) make it possible to deploy AKS with kubenet without the need for extra write permissions or 2) make something to AzureCNI so it doesn't assign IP addresses from vnet to each pod. This would mean either make it optionally work as kubenet so NAT pod IPs or the current solution of assigning separate subnet for hosts and pods also solves the problem in theory (because then I can assign the "official" CIDR that I get from on-prem system to nodes and create some separate Azure only subnet to pods).

Hope I explained it well. Is it clear now? :D

Hi @DavidZisky - this is clear. Thanks a lot for sharing your thoughts here. You are right about KubeNet (it skipped my mind). I had a few thoughts to share and some questions. Your feedback will enable us understand future requirements

1. The separate Pod subnet will have an IP space that belongs to the larger space configured on the VNet. So in both cases its the official CIDR being used, correct?
2. In the KubeNet case how do you pick the address space to use for Pods?

[DavidZisky]

Hi @DavidZisky - this is clear. Thanks a lot for sharing your thoughts here. You are right about KubeNet (it skipped my mind). I had a few thoughts to share and some questions. Your feedback will enable us understand future requirements

1. The separate Pod subnet will have an IP space that belongs to the larger space configured on the VNet. So in both cases its the official CIDR being used, correct?
2. In the KubeNet case how do you pick the address space to use for Pods?

So, first I'll answer the second question:

2. Subnet for pods is NATed by kubenet so it doesn't really matter what it is. We usually use 10.x.x.x/x for all clusters.

and then the first:

1. Hmm, maybe. I only discovered that feature this week didn't think of the actual implementation. Now that I think about it it may actually not solve the problem. My idea was that I'll have one "official (from on-prem) subnet and one unofficial but now that I think about it just like you said these both subnets will still have to be from the same vnet so maybe it doesn't solve the problem. Because I can't have 192.168.x.x subnet for nodes and 10.x.x.x for pods in this case. The point is to be able to assign non-routable (in vnet) address space for pods (for example like kubenet is NATing the pods IPs to host IPs). I know this is against the whole idea of AzureCNI but I see the same problem in many companies. They use kubenet to not "waste" IPs but kubenet requires permissions which ideally they would avoid.

[kxs-jnadeau]

@DavidZisky What If we define the pod and worker subnets in their respective address spaces under the same vnet ? In that case it seems possible to have a 192.168.X.X subnet for workers but a 10.X.X.X subnet for PODs and only route/peer the worker one (or do I miss something) ?

[aanandr]

Hi @DavidZisky - this is clear. Thanks a lot for sharing your thoughts here. You are right about KubeNet (it skipped my mind). I had a few thoughts to share and some questions. Your feedback will enable us understand future requirements

1. The separate Pod subnet will have an IP space that belongs to the larger space configured on the VNet. So in both cases its the official CIDR being used, correct?
2. In the KubeNet case how do you pick the address space to use for Pods?

So, first I'll answer the second question:

1. Subnet for pods is NATed by kubenet so it doesn't really matter what it is. We usually use 10.x.x.x/x for all clusters.

and then the first:

1. Hmm, maybe. I only discovered that feature this week didn't think of the actual implementation. Now that I think about it it may actually not solve the problem. My idea was that I'll have one "official (from on-prem) subnet and one unofficial but now that I think about it just like you said these both subnets will still have to be from the same vnet so maybe it doesn't solve the problem. Because I can't have 192.168.x.x subnet for nodes and 10.x.x.x for pods in this case. The point is to be able to assign non-routable (in vnet) address space for pods (for example like kubenet is NATing the pods IPs to host IPs). I know this is against the whole idea of AzureCNI but I see the same problem in many companies. They use kubenet to not "waste" IPs but kubenet requires permissions which ideally they would avoid.

Hi @DavidZisky - sorry for the delayed response.

1. Reg. picking addr space for Pods - yes, SNATing happens, but imagine a case where that address range overlaps with a range in a peered VNet. If a Pod tries to access an IP in that range it is difficult to route traffic to the actual desired destination (it could be another Pod or an endpt in the other VNet)
2. If you configure the 192.168.x.x. and 10.x.x.x both on the VNet then you can use subnets from each of these spaces for nodes and Pods.

[kxs-jnadeau]

Update on GA timeline ? Also when will we see the windows support in the roadmap ?

[abossard]

Hi @palma21 I'm also curious about any ETA. Thanks

[manali1st]

Is there any plan or timeline for supporting Windows worker nodes for this Dynamic IPs feature?
https://aka.ms/aks/dynamic-cni

[aanandr]

@manali1st - yes it is planned. We are tentatively looking at the later part of the second half of this year.

[kxs-jnadeau]

@aanandr Is the windows support delaying the GA status for Linux ? I've been waiting for months for this feature to go GA on my Linux only clusters.

[aanandr]

@kxs-jnadeau - no, Windows support doesnt impact Linux GA. We plan to GA Linux in a couple of months.

[cpkuo]

I'm sorry if this has been answered somewhere in this thread but I could not find it. Does the PodSubnet feature allow for pods within the same cluster to have different subnets? For instance Pod A to be in a subnet different from Pod B? Both pods are in a different subnet from the cluster? I have a multitenant application where each pod is assigned to a tenant and some tenants might want private link or have their pod's vnet connected to their customer-owned vent.

[phealy]

Azure CNI dynamic IP allocation and enhanced subnet support is now GA.

[manali1st]

@phealy : Thanks for sharing update.
Question : I think this dynamic IP allocation not yet available for windows containers. But can we use this dynamic IP allocation for hybrid AKS cluster? having windows and linux node/pod in same AKS cluster and use dynamic IPs for linux pods?

[mosabami]

@phealy can we move this to GA on the GitHub project please?

[BahriNipun]

Hello, what are the egress traffic requirements of the pod subnet ?

Does the same egress traffic rules applies to pod subnet as mentioned here- https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic

[aanandr]

@BahriNipun - if your question was whether the egress rules in that link will all just work seamlessly with the new CNI then yes you are right - they will. If not then can you please clarify?

[mikeeq]

Any ideas when dynamic IP allocation will be supported on Windows nodes?

[aanandr]

Any ideas when dynamic IP allocation will be supported on Windows nodes?

We are aiming to preview it within the next 6 months.

[naioja]

For AKS clusters deployed with CNI dynamic IP allocation I can see Azure Front Door PLS not working correctly, any hints what the issue could be?

[kaarthis]

https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni-dynamic-ip-allocation#prerequisites PLS is explicitly noted here as not supported , however we are aware of the gap and working to support it soon.