Loadbalancer Public IP with Zones enabled

[dkanbier]

Ask your question here

I have configured an AKS cluster using Availability Zones following this document.

The Standard LoadBalancer called kubernetes this creates by default has a frontend-ip configuration using a public-ip which has Availability Zones 1, 2, 3 enabled.

When running the azqr tool it reports the LoadBalancer to not have Availability Zones enabled. I was wondering if this is expected behavior or if I've made an error in the configuration.

Possible cause for azqr reporting the issue:

When showing the public-ip using az network public-ip show I can see the Zones field defined correctly. However when I query the same public-ip using az network lb frontend-ip show I do not see a Zones field.

When trying use az network lb frontend-ip show for an internal-ip I do get a Zones field. The azqr tool reports no issues on a LB using this internal-ip.

[nlighten]

The test just seems to look at the FrontendIPConfigurations and the zone info listed there. When using a public ip you need look the zone configuration of the public ip address.

https://github.com/Azure/azqr/blob/main/internal/scanners/lb/rules.go#L36

Looks like a bug to me.

[cmendible]

Hi @dkanbier can you please share de JSON for the Load Balancer and the Public IP so I can take a look? (Remember to remove or mask your subscription ID or any other value you want to keep private)

[nlighten]

A frontendIPConfigurations with a public ip:

  "frontendIPConfigurations": [
    {
      "etag": "W/\"7a579181-9309-4228-9c2a-2b9c7d4e276a\"",
      "id": "/subscriptions/***/resourceGroups/MC_aks/providers/Microsoft.Network/loadBalancers/kubernetes/frontendIPConfigurations/582505be-f971-490b-8b85-7ebad9bfdba0",
      "name": "582505be-f971-490b-8b85-7ebad9bfdba0",
      "outboundRules": [
        {
          "id": "/subscriptions/***/resourceGroups/MC_aks/providers/Microsoft.Network/loadBalancers/kubernetes/outboundRules/aksOutboundRule",
          "resourceGroup": "MC_aks"
        }
      ],
      "privateIPAllocationMethod": "Dynamic",
      "provisioningState": "Succeeded",
      "publicIPAddress": {
        "id": "/subscriptions/***/resourceGroups/MC_aks/providers/Microsoft.Network/publicIPAddresses/582505be-f971-490b-8b85-7ebad9bfdba0",
        "resourceGroup": "MC_aks"
      },
      "resourceGroup": "MC_aks",
      "type": "Microsoft.Network/loadBalancers/frontendIPConfigurations"
    }
  ],

versus one with only private ip:

  "frontendIPConfigurations": [
    {
      "etag": "W/\"5f1c68bc-5633-4fd7-85da-1b4948de4b81\"",
      "id": "/subscriptions/***/resourceGroups/mc_aks/providers/Microsoft.Network/loadBalancers/kubernetes-internal/frontendIPConfigurations/af4b5ab1abce344deb03ae8d21e4a9ad-Aks02IngressSubnet",
      "loadBalancingRules": [
        {
          "id": "/subscriptions/***/resourceGroups/mc_aks/providers/Microsoft.Network/loadBalancers/kubernetes-internal/loadBalancingRules/af4b5ab1abce344deb03ae8d21e4a9ad-Aks02IngressSubnet-TCP-80",
          "resourceGroup": "mc_aks"
        },
        {
          "id": "/subscriptions/***/resourceGroups/mc_aks/providers/Microsoft.Network/loadBalancers/kubernetes-internal/loadBalancingRules/af4b5ab1abce344deb03ae8d21e4a9ad-Aks02IngressSubnet-TCP-8443",
          "resourceGroup": "mc_aks"
        }
      ],
      "name": "af4b5ab1abce344deb03ae8d21e4a9ad-Aks02IngressSubnet",
      "privateIPAddress": "10.0.10.149",
      "privateIPAddressVersion": "IPv4",
      "privateIPAllocationMethod": "Static",
      "provisioningState": "Succeeded",
      "resourceGroup": "mc_aks",
      "subnet": {
        "id": "/subscriptions/***/resourceGroups/spoke/providers/Microsoft.Network/virtualNetworks/vnet-cncr-nonprod-frc/subnets/AksIngressSubnet",
        "resourceGroup": "spoke"
      },
      "type": "Microsoft.Network/loadBalancers/frontendIPConfigurations",
      "zones": [
        "1",
        "2",
        "3"
      ]
    }
  ],  
``

[dkanbier]

Hi @cmendible , thanks for the reply. The root cause in my opinion is what @nlighten is mentioning. The frontendIPConfigurations object of a Public IP does not contain a zones field as opposed to a Private IP which does return a zones field.

When you directly get the Public IP object, you do get a zones field. This happens both when using the azure-sdk-for-go or azure-cli like I tried to explain in my initial post.

azqr uses the frontendIPConfiguration object to determine if a LB is zone redundant or not, hence I think it will always report it's not when the LB is using a Public IP.

LoadBalancer JSON:

{
    "name": "kubernetes",
    "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes",
    "etag": "W/\"1fcbefe6-13be-43aa-a09d-f26c7736c332\"",
    "type": "Microsoft.Network/loadBalancers",
    "location": "westeurope",
    "tags": {
        "aks-managed-cluster-name": "deleteAKS",
        "aks-managed-cluster-rg": "deleteMe"
    },
    "properties": {
        "provisioningState": "Succeeded",
        "resourceGuid": "16aeb038-c9c6-4535-973c-d37fd6e8eb09",
        "frontendIPConfigurations": [
            {
                "name": "428f3d97-c114-4fcf-b58a-cb7b127a7e92",
                "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/frontendIPConfigurations/428f3d97-c114-4fcf-b58a-cb7b127a7e92",
                "etag": "W/\"1fcbefe6-13be-43aa-a09d-f26c7736c332\"",
                "type": "Microsoft.Network/loadBalancers/frontendIPConfigurations",
                "properties": {
                    "provisioningState": "Succeeded",
                    "privateIPAllocationMethod": "Dynamic",
                    "publicIPAddress": {
                        "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/publicIPAddresses/428f3d97-c114-4fcf-b58a-cb7b127a7e92"
                    }
                }
            }
        ],
        "backendAddressPools": [
            {
                "name": "aksOutboundBackendPool",
                "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool",
                "etag": "W/\"1fcbefe6-13be-43aa-a09d-f26c7736c332\"",
                "properties": {
                    "provisioningState": "Succeeded",
                    "backendIPConfigurations": [
                        {
                            "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/2/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
                        },
                        {
                            "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/3/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
                        },
                        {
                            "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/4/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
                        }
                    ]
                },
                "type": "Microsoft.Network/loadBalancers/backendAddressPools"
            },
            {
                "name": "kubernetes",
                "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes",
                "etag": "W/\"1fcbefe6-13be-43aa-a09d-f26c7736c332\"",
                "properties": {
                    "provisioningState": "Succeeded",
                    "backendIPConfigurations": [
                        {
                            "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/2/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
                        },
                        {
                            "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/3/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
                        },
                        {
                            "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/4/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
                        }
                    ]
                },
                "type": "Microsoft.Network/loadBalancers/backendAddressPools"
            }
        ],
        "loadBalancingRules": [],
        "probes": [],
        "inboundNatRules": [],
        "inboundNatPools": []
    },
    "sku": {
        "name": "Standard"
    }
}

Public IP JSON:

{
    "name": "428f3d97-c114-4fcf-b58a-cb7b127a7e92",
    "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/publicIPAddresses/428f3d97-c114-4fcf-b58a-cb7b127a7e92",
    "etag": "W/\"62b30655-2450-4daa-91b1-ccd8c8ec1fff\"",
    "location": "westeurope",
    "tags": {
        "aks-managed-cluster-name": "deleteAKS",
        "aks-managed-cluster-rg": "deleteMe",
        "aks-managed-type": "aks-slb-managed-outbound-ip"
    },
    "properties": {
        "provisioningState": "Succeeded",
        "resourceGuid": "4ca22ea6-3e5c-4e7a-9967-fe7bdb138c7c",
        "ipAddress": "***",
        "publicIPAddressVersion": "IPv4",
        "publicIPAllocationMethod": "Static",
        "idleTimeoutInMinutes": 4,
        "ipTags": [],
        "ipConfiguration": {
            "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/frontendIPConfigurations/428f3d97-c114-4fcf-b58a-cb7b127a7e92"
        }
    },
    "type": "Microsoft.Network/publicIPAddresses",
    "sku": {
        "name": "Standard"
    }
}

Please note that in both JSONs there is no mention of zones. However if you query the Public IP directly it will show a zones field:

az network public-ip show -n 428f3d97-c114-4fcf-b58a-cb7b127a7e92 --resource-group MC_deleteMe_deleteAKS_westeurope

{
  "etag": "W/\"62b30655-2450-4daa-91b1-ccd8c8ec1fff\"",
  "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/publicIPAddresses/428f3d97-c114-4fcf-b58a-cb7b127a7e92",
  "idleTimeoutInMinutes": 4,
  "ipAddress": "***",
  "ipConfiguration": {
    "id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/frontendIPConfigurations/428f3d97-c114-4fcf-b58a-cb7b127a7e92",
    "resourceGroup": "MC_deleteMe_deleteAKS_westeurope"
  },
  "ipTags": [],
  "location": "westeurope",
  "name": "428f3d97-c114-4fcf-b58a-cb7b127a7e92",
  "provisioningState": "Succeeded",
  "publicIPAddressVersion": "IPv4",
  "publicIPAllocationMethod": "Static",
  "resourceGroup": "MC_deleteMe_deleteAKS_westeurope",
  "resourceGuid": "4ca22ea6-3e5c-4e7a-9967-fe7bdb138c7c",
  "sku": {
    "name": "Standard",
    "tier": "Regional"
  },
  "tags": {
    "aks-managed-cluster-name": "deleteAKS",
    "aks-managed-cluster-rg": "deleteMe",
    "aks-managed-type": "aks-slb-managed-outbound-ip"
  },
  "type": "Microsoft.Network/publicIPAddresses",
  "zones": [
    "3",
    "2",
    "1"
  ]
}

[cmendible]

Thank you both! Seems like current validation works only for private IP's and will fail for Public IP's.

I'll have to query Public IP's with zones and add the results to the scan context in order to fix this rule.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[cmendible]

Hey @dkanbier @nlighten can you try the binary from here: https://github.com/Azure/azqr/actions/runs/6768369641 and check if the issue is fixed?