title | titleSuffix | description | author | ms.subservice | ms.custom | ms.topic | ms.date | ms.author |
---|---|---|---|---|---|---|---|---|
Create a managed or user-assigned NAT gateway for your Azure Kubernetes Service (AKS) cluster |
Azure Kubernetes Service |
Learn how to create an AKS cluster with managed NAT integration and user-assigned NAT gateway. |
asudbring |
aks-networking |
devx-track-azurecli |
how-to |
01/10/2024 |
allensu |
While you can route egress traffic through an Azure Load Balancer, there are limitations on the number of outbound flows of traffic you can have. Azure NAT Gateway allows up to 64,512 outbound UDP and TCP traffic flows per IP address with a maximum of 16 IP addresses.
This article shows you how to create an Azure Kubernetes Service (AKS) cluster with a managed NAT gateway and a user-assigned NAT gateway for egress traffic. It also shows you how to disable OutboundNAT on Windows.
- Make sure you're using the latest version of Azure CLI.
- Make sure you're using Kubernetes version 1.20.x or above.
- Managed NAT gateway is incompatible with custom virtual networks.
-
Create an AKS cluster with a new managed NAT gateway using the
az aks create
command with the--outbound-type managedNATGateway
,--nat-gateway-managed-outbound-ip-count
, and--nat-gateway-idle-timeout
parameters. If you want the NAT gateway to operate out of a specific availability zone, specify the zone using--zones
. -
If no zone is specified when creating a managed NAT gateway, then NAT gateway is deployed to "no zone" by default. When NAT gateway is placed in no zone, Azure places the resource in a zone for you. For more information on non-zonal deployment model, see non-zonal NAT gateway.
-
A managed NAT gateway resource can't be used across multiple availability zones.
az aks create \ --resource-group myResourceGroup \ --name myNatCluster \ --node-count 3 \ --outbound-type managedNATGateway \ --nat-gateway-managed-outbound-ip-count 2 \ --nat-gateway-idle-timeout 4 \ --generate-ssh-keys
-
Update the outbound IP address or idle timeout using the
az aks update
command with the--nat-gateway-managed-outbound-ip-count
or--nat-gateway-idle-timeout
parameter.az aks update \ --resource-group myResourceGroup \ --name myNatCluster\ --nat-gateway-managed-outbound-ip-count 5
This configuration requires bring-your-own networking (via Kubenet or Azure CNI) and that the NAT gateway is preconfigured on the subnet. The following commands create the required resources for this scenario.
-
Create a resource group using the
az group create
command.az group create --name myResourceGroup \ --location southcentralus
-
Create a managed identity for network permissions and store the ID to
$IDENTITY_ID
for later use.IDENTITY_ID=$(az identity create \ --resource-group myResourceGroup \ --name myNatClusterId \ --location southcentralus \ --query id \ --output tsv)
-
Create a public IP for the NAT gateway using the
az network public-ip create
command.az network public-ip create \ --resource-group myResourceGroup \ --name myNatGatewayPip \ --location southcentralus \ --sku standard
-
Create the NAT gateway using the
az network nat gateway create
command.az network nat gateway create \ --resource-group myResourceGroup \ --name myNatGateway \ --location southcentralus \ --public-ip-addresses myNatGatewayPip
[!Important] A single NAT gateway resource can't be used across multiple availability zones. To ensure zone-resiliency, it is recommended to deploy a NAT gateway resource to each availability zone and assign to subnets containing AKS clusters in each zone. For more information on this deployment model, see NAT gateway for each zone. If no zone is configured for NAT gateway, the default zone placement is "no zone", in which Azure places NAT gateway into a zone for you.
-
Create a virtual network using the
az network vnet create
command.az network vnet create \ --resource-group myResourceGroup \ --name myVnet \ --location southcentralus \ --address-prefixes 172.16.0.0/20
-
Create a subnet in the virtual network using the NAT gateway and store the ID to
$SUBNET_ID
for later use.SUBNET_ID=$(az network vnet subnet create \ --resource-group myResourceGroup \ --vnet-name myVnet \ --name myNatCluster \ --address-prefixes 172.16.0.0/22 \ --nat-gateway myNatGateway \ --query id \ --output tsv)
-
Create an AKS cluster using the subnet with the NAT gateway and the managed identity using the
az aks create
command.az aks create \ --resource-group myResourceGroup \ --name myNatCluster \ --location southcentralus \ --network-plugin azure \ --vnet-subnet-id $SUBNET_ID \ --outbound-type userAssignedNATGateway \ --assign-identity $IDENTITY_ID \ --generate-ssh-keys
Windows OutboundNAT can cause certain connection and communication issues with your AKS pods. An example issue is node port reuse. In this example, Windows OutboundNAT uses ports to translate your pod IP to your Windows node host IP, which can cause an unstable connection to the external service due to a port exhaustion issue.
Windows enables OutboundNAT by default. You can now manually disable OutboundNAT when creating new Windows agent pools.
- Existing AKS cluster with v1.26 or above. If you're using Kubernetes version 1.25 or older, you need to update your deployment configuration.
- You can't set cluster outbound type to LoadBalancer. You can set it to Nat Gateway or UDR:
- NAT Gateway: NAT Gateway can automatically handle NAT connection and is more powerful than Standard Load Balancer. You might incur extra charges with this option.
- UDR (UserDefinedRouting): You must keep port limitations in mind when configuring routing rules.
- If you need to switch from a load balancer to NAT Gateway, you can either add a NAT gateway into the VNet or run
az aks upgrade
to update the outbound type.
Note
UserDefinedRouting has the following limitations:
- SNAT by Load Balancer (must use the default OutboundNAT) has "64 ports on the host IP".
- SNAT by Azure Firewall (disable OutboundNAT) has 2496 ports per public IP.
- SNAT by NAT Gateway (disable OutboundNAT) has 64512 ports per public IP.
- If the Azure Firewall port range isn't enough for your application, you need to use NAT Gateway.
- Azure Firewall doesn't SNAT with Network rules when the destination IP address is in a private IP address range per IANA RFC 1918 or shared address space per IANA RFC 6598.
-
Manually disable OutboundNAT for Windows when creating new Windows agent pools using the
az aks nodepool add
command with the--disable-windows-outbound-nat
flag.[!NOTE] You can use an existing AKS cluster, but you might need to update the outbound type and add a node pool to enable
--disable-windows-outbound-nat
.az aks nodepool add \ --resource-group myResourceGroup \ --cluster-name myNatCluster \ --name mynp \ --node-count 3 \ --os-type Windows \ --disable-windows-outbound-nat
For more information on Azure NAT Gateway, see Azure NAT Gateway.