AGIC Add-on enablement with ARM template

[petegrimsdale]

What happened:
When attempting to create application gateway and AKS cluster with AGIC addon in the same ARM template the validation of the Ingress Controller add-on fails as it expects the application gateway to exist when using applicationGatewayId in the add-on config.
The cluster resource is dependent on the application gateway provisioning:

       {
            "type": "Microsoft.ContainerService/managedClusters",
            "apiVersion": "2020-12-01",
            "name": "[parameters('clusterName')]",
            "location": "[resourceGroup().location]",
            "dependsOn": [
                "[resourceId('Microsoft.Network/virtualNetworks', variables('vnetName'))]",
                "[resourceId('Microsoft.Network/applicationGateways', variables('applicationGatewayName'))]"

Addon definition:

            "addonProfiles": {
                "omsagent": {
                    "enabled": true,
                    "config": {
                        "logAnalyticsWorkspaceResourceID": "[resourceId('Microsoft.OperationalInsights/workspaces', variables('uniqueResourceName'))]"
                    }
                },
                "ingressApplicationGateway": {
                    "enabled": true,
                    "config": {
                        "applicationGatewayId": "[resourceId('Microsoft.Network/applicationGateways',variables('applicationGatewayName'))]"
                    }
                }
            },

The error given when the template is validated is:

{"error":{"code":"InvalidTemplateDeployment","message":"The template deployment 'aksagic_aks' is not valid according to the validation procedure. The tracking id is 'c5cea8fe-9c66-4439-a65d-2dc1857c08cc'. See inner errors for details.","details":[{"code":"IngressAppGwAddonConfigApplicationGatewayNotFound","message":"Provisioning of resource(s) for container service demoCluster in resource group rg-aksdemo failed. Message: {\n "code": "IngressAppGwAddonConfigApplicationGatewayNotFound",\n "message": "IngressApplicationGateway addon cannot find Application Gateway '/subscriptions/37910d7c-da82-4bba-aea7-2b2f7cba76e1/resourceGroups/rg-aksdemo/providers/Microsoft.Network/applicationGateways/aksagicdeeb-gw'."\n }. Details: "}]}}

What you expected to happen:
The Ingress Controller Addon should be able to leverage an application gateway that is going to be deployed within the ARM template even though at the time of validation of the template the application gateway resource does not exist

How to reproduce it (as minimally and precisely as possible):
Define application gateway and AKS with Ingress controller addon within the same ARM template

Environment:

• Kubernetes version (use kubectl version): 1.18.14
• Size of cluster (how many worker nodes are in the cluster?): 3 nodes
• General description of workloads in the cluster: none deployed at this stage

The addon works correctly within the ARM template when setting the Addon to create a new application gateway by defining the name and the subnet CIDR. ( Incidentally subnetId does not work )

[ghost]

Hi petegrimsdale, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[AresiusXP]

I'm having the same issue right now.

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[curious-toast]

Having the same issue with this, would love to see an update! :]

[palma21]

When attempting to create application gateway and AKS cluster with AGIC addon in the same ARM template the validation of the Ingress Controller add-on fails as it expects the application gateway to exist when using applicationGatewayId in the add-on config.

That is correct because you passed it an ID, does the ID you passed on not exist? If so you can't refer to an existing AGIC and need to either allow for a new on to be created or pass a valid existing one.

Which from the 2 above is your goal?

[ghost]

@akshaysngupta, @mscatyao would you be able to assist?

Issue Details

---

What happened:
When attempting to create application gateway and AKS cluster with AGIC addon in the same ARM template the validation of the Ingress Controller add-on fails as it expects the application gateway to exist when using applicationGatewayId in the add-on config.
The cluster resource is dependent on the application gateway provisioning:

       {
            "type": "Microsoft.ContainerService/managedClusters",
            "apiVersion": "2020-12-01",
            "name": "[parameters('clusterName')]",
            "location": "[resourceGroup().location]",
            "dependsOn": [
                "[resourceId('Microsoft.Network/virtualNetworks', variables('vnetName'))]",
                "[resourceId('Microsoft.Network/applicationGateways', variables('applicationGatewayName'))]"

Addon definition:

            "addonProfiles": {
                "omsagent": {
                    "enabled": true,
                    "config": {
                        "logAnalyticsWorkspaceResourceID": "[resourceId('Microsoft.OperationalInsights/workspaces', variables('uniqueResourceName'))]"
                    }
                },
                "ingressApplicationGateway": {
                    "enabled": true,
                    "config": {
                        "applicationGatewayId": "[resourceId('Microsoft.Network/applicationGateways',variables('applicationGatewayName'))]"
                    }
                }
            },

The error given when the template is validated is:

{"error":{"code":"InvalidTemplateDeployment","message":"The template deployment 'aksagic_aks' is not valid according to the validation procedure. The tracking id is 'c5cea8fe-9c66-4439-a65d-2dc1857c08cc'. See inner errors for details.","details":[{"code":"IngressAppGwAddonConfigApplicationGatewayNotFound","message":"Provisioning of resource(s) for container service demoCluster in resource group rg-aksdemo failed. Message: {\n "code": "IngressAppGwAddonConfigApplicationGatewayNotFound",\n "message": "IngressApplicationGateway addon cannot find Application Gateway '/subscriptions/37910d7c-da82-4bba-aea7-2b2f7cba76e1/resourceGroups/rg-aksdemo/providers/Microsoft.Network/applicationGateways/aksagicdeeb-gw'."\n }. Details: "}]}}

What you expected to happen:
The Ingress Controller Addon should be able to leverage an application gateway that is going to be deployed within the ARM template even though at the time of validation of the template the application gateway resource does not exist

How to reproduce it (as minimally and precisely as possible):
Define application gateway and AKS with Ingress controller addon within the same ARM template

Environment:

• Kubernetes version (use kubectl version): 1.18.14
• Size of cluster (how many worker nodes are in the cluster?): 3 nodes
• General description of workloads in the cluster: none deployed at this stage

The addon works correctly within the ARM template when setting the Addon to create a new application gateway by defining the name and the subnet CIDR. ( Incidentally subnetId does not work )

Author:	petegrimsdale
Assignees:	-
Labels:	Needs Attention :wave:, action-required, addon/agic, azure/application-gateway, triage
Milestone:	-

[robbiezhang]

It's failed in the preflight validation request because the AppGW is not yet created. @akshaysngupta I think it's a bug in the AGIC addon validation logic, that we should not check the existence of the AppGW for preflight validation.

[mauve]

Any progress?

[petegrimsdale]

@robbiezhang Is there a plan / timeline to get this issue addressed as a number of deployments will expect to create the Application Gateway in the same automation template as the AKS cluster with the AGIC addon

[bmbadr]

I passed the validation andnew AKS is deploued and AGIC is enabled. I used Bicep template (will work also in ARM obviously). I didn't mentioned the AppGW ID so the deployment should create a new one using the default name of "ingress-appgateway". AKS tried to create new AppGW but it could create only a new Managed Identity, and it shows this message in the portal:
"Ingress controller has not created the application gateway yet. It may take up to 15 mins before the application gateway gets created." but nothing else happened

here is what I used:

param enableApplicationGateway bool = false
param appgwSubnet string

addonProfiles: {
  ingressApplicationGateway:{
    enabled: enableApplicationGateway
    config: {
      subnetPrefix: appgwSubnet
    }
  }
}

[AresiusXP]

I passed the validation andnew AKS is deploued and AGIC is enabled. I used Bicep template (will work also in ARM obviously). I didn't mentioned the AppGW ID so the deployment should create a new one using the default name of "ingress-appgateway". AKS tried to create new AppGW but it could create only a new Managed Identity, and it shows this message in the portal:
"Ingress controller has not created the application gateway yet. It may take up to 15 mins before the application gateway gets created." but nothing else happened

here is what I used:

param enableApplicationGateway bool = false
param appgwSubnet string

addonProfiles: {
  ingressApplicationGateway:{
    enabled: enableApplicationGateway
    config: {
      subnetPrefix: appgwSubnet
    }
  }
}

The point of the issue here is that we want to specify the AppGw created within the same ARM template, not leave it to be automatically created.

[thepaulmacca]

I've just come across this issue myself and found this thread...really keen to get this resolved as it's the last thing preventing me from a complete deployment (like others here)

[miwithro]

@akshaysngupta are there any updates?

[petegrimsdale]

@palma21 can you help get an update on this issue

[thepaulmacca]

Just thought I'd update this thread to help others here;

If using Bicep - if you have your AGW and AKS cluster in separate modules, you can add an existing resource to your AKS module:

resource appGateway 'Microsoft.Network/applicationGateways@2021-02-01' existing = {
  name: appGatewayName
}

Then enable the add-on like this, and it'll now pass preflight validation and do the deployment successfully:

ingressApplicationGateway: {
  enabled: true
  config: {
    applicationGatewayId: appGateway.id
  }
}

[mananpreetsingh]

We are blocked by this to create a solution template offer in the marketplace. There is definitely an issue. This example template does pass validation though: https://github.com/Azure/azure-quickstart-templates/blob/19e2aa29b16e017d599661f7869756ebfd217bab/quickstarts/microsoft.network/aks-application-gateway-ingress-controller/azuredeploy.json#L1797

It seems like if you specify aciconnectorlinux addon before ingress addon then validation gets successful.

          "aciConnectorLinux": {
            "enabled": "[parameters('aciConnectorLinuxEnabled')]"
          },

Not sure what is the use of aciconnector addon here.

[mananpreetsingh]

Workaround that I found for this bug:

• Create a workspace:

    {
      "type": "Microsoft.OperationalInsights/workspaces",
      "apiVersion": "2021-06-01",
      "name": "[parameters('logAnalyticsWorkspaceName')]",
      "location": "[parameters('location')]",
      "properties": {
        "sku": {
          "name": "[parameters('logAnalyticsSku')]"
        },
        "retentionInDays": "[parameters('logAnalyticsRetentionInDays')]"
      }
    }

• Then while creating AKS cluster in ARM enable the diagnosticsetting in AKS cluster resource:

      "resources": [
        {
          "type": "providers/diagnosticSettings",
          "apiVersion": "2021-05-01-preview",
          "name": "Microsoft.Insights/default",
          "dependsOn": [
            "[variables('aksClusterId')]",
            "[variables('workspaceId')]"
          ],
          "properties": {
            "workspaceId": "[variables('workspaceId')]",
            "logs": [
              {
                "category": "kube-apiserver",
                "enabled": true
              },
              {
                "category": "kube-audit",
                "enabled": true
              },
              {
                "category": "kube-audit-admin",
                "enabled": true
              },
              {
                "category": "kube-controller-manager",
                "enabled": true
              },
              {
                "category": "kube-scheduler",
                "enabled": true
              },
              {
                "category": "cluster-autoscaler",
                "enabled": true
              },
              {
                "category": "guard",
                "enabled": true
              }
            ],
            "metrics": [
              {
                "category": "AllMetrics",
                "enabled": true
              }
            ]
          }
        }
      ]

[curious-toast]

Any updates on this?

[evanrappe]

Just thought I'd update this thread to help others here;

If using Bicep - if you have your AGW and AKS cluster in separate modules, you can add an existing resource to your AKS module:

resource appGateway 'Microsoft.Network/applicationGateways@2021-02-01' existing = {
  name: appGatewayName
}

Then enable the add-on like this, and it'll now pass preflight validation and do the deployment successfully:

ingressApplicationGateway: {
  enabled: true
  config: {
    applicationGatewayId: appGateway.id
  }
}

I cannot make this work, has anyone else?

My scenario is greenfield: AGIC in child module, AKS in main template with dependsOn for the AGIC module, ingressApplicationGateway value references the ID of the separate "existing resource" declaration pointing to the agw name used in the AGIC module.

Still the template fails the preflight checks because the ingressApplicationGateway fields cannot find the AGW I'm referencing.

[Tautcius]

Same problem here.

[thepaulmacca]

Just thought I'd update this thread to help others here;
If using Bicep - if you have your AGW and AKS cluster in separate modules, you can add an existing resource to your AKS module:

resource appGateway 'Microsoft.Network/applicationGateways@2021-02-01' existing = {
  name: appGatewayName
}

Then enable the add-on like this, and it'll now pass preflight validation and do the deployment successfully:

ingressApplicationGateway: {
  enabled: true
  config: {
    applicationGatewayId: appGateway.id
  }
}

I cannot make this work, has anyone else?

My scenario is greenfield: AGIC in child module, AKS in main template with dependsOn for the AGIC module, ingressApplicationGateway value references the ID of the separate "existing resource" declaration pointing to the agw name used in the AGIC module.

Still the template fails the preflight checks because the ingressApplicationGateway fields cannot find the AGW I'm referencing.

I'm using the NGINX ingress now, but I would remove the dependsOn as I imagine that's what's causing it to fail. I don't remember setting that

[phealy]

My scenario is greenfield: AGIC in child module, AKS in main template with dependsOn for the AGIC module, ingressApplicationGateway value references the ID of the separate "existing resource" declaration pointing to the agw name used in the AGIC module.

You cannot deploy the AKS cluster in the same deployment that triggers the Application Gateway because the parameters for the AKS deployment are validated at deployment start, and the Application Gateway doesn't exist at that time. You can put AKS into a module and pass the AGW resource ID in as a parameter; that way, by the time the AKS deployment starts (which is when the module is reached), the Application Gateway is finished deploying and extant.

[JoshuaAnickat]

Workaround that I found for this bug:

• Create a workspace:

    {
      "type": "Microsoft.OperationalInsights/workspaces",
      "apiVersion": "2021-06-01",
      "name": "[parameters('logAnalyticsWorkspaceName')]",
      "location": "[parameters('location')]",
      "properties": {
        "sku": {
          "name": "[parameters('logAnalyticsSku')]"
        },
        "retentionInDays": "[parameters('logAnalyticsRetentionInDays')]"
      }
    }

• Then while creating AKS cluster in ARM enable the diagnosticsetting in AKS cluster resource:

      "resources": [
        {
          "type": "providers/diagnosticSettings",
          "apiVersion": "2021-05-01-preview",
          "name": "Microsoft.Insights/default",
          "dependsOn": [
            "[variables('aksClusterId')]",
            "[variables('workspaceId')]"
          ],
          "properties": {
            "workspaceId": "[variables('workspaceId')]",
            "logs": [
              {
                "category": "kube-apiserver",
                "enabled": true
              },
              {
                "category": "kube-audit",
                "enabled": true
              },
              {
                "category": "kube-audit-admin",
                "enabled": true
              },
              {
                "category": "kube-controller-manager",
                "enabled": true
              },
              {
                "category": "kube-scheduler",
                "enabled": true
              },
              {
                "category": "cluster-autoscaler",
                "enabled": true
              },
              {
                "category": "guard",
                "enabled": true
              }
            ],
            "metrics": [
              {
                "category": "AllMetrics",
                "enabled": true
              }
            ]
          }
        }
      ]

No idea why this worked, but it did...

[cd-romulus]

passed the validation andnew AKS is deploued and AGIC is enabled. I used Bicep template (will work also in ARM obviously). I didn't mentioned the AppGW ID so the deployment should create a new one using the default name of "ingress-appgateway". AKS tried to create new AppGW but it could create only a new Managed Identity, and it shows this message in the portal:
"Ingress controller has not created the application gateway yet. It may take up to 15 mins before the application gateway gets created." but nothing else happened

@bmbadr By any chance, did you solve this? I am trying to use an auto-created application gateway but it seems neither the subnet, not the application gateway itself gets created.

EDIT: Never mind, it turned out auto-creating the application gateway didn't work from the portal either. I now create the subnet and application gateway explicitly from the template and assign it to the AGIC.

[northynorth]

Just thought I'd update this thread to help others here;
If using Bicep - if you have your AGW and AKS cluster in separate modules, you can add an existing resource to your AKS module:

resource appGateway 'Microsoft.Network/applicationGateways@2021-02-01' existing = {
  name: appGatewayName
}

Then enable the add-on like this, and it'll now pass preflight validation and do the deployment successfully:

ingressApplicationGateway: {
  enabled: true
  config: {
    applicationGatewayId: appGateway.id
  }
}

I cannot make this work, has anyone else?

My scenario is greenfield: AGIC in child module, AKS in main template with dependsOn for the AGIC module, ingressApplicationGateway value references the ID of the separate "existing resource" declaration pointing to the agw name used in the AGIC module.

Still the template fails the preflight checks because the ingressApplicationGateway fields cannot find the AGW I'm referencing.

I did manage to make this work, but I had to have both the AGW and AKS cluster in separate modules called from main. The key was to have the appGatewayName passed into the AKS module as an output of the AGW module

module aksCluster './modules/akscluster.bicep' = {
	params: {
		appGatewayName: appGw.outputs.name
	}
}

[evanrappe]

Just thought I'd update this thread to help others here;
If using Bicep - if you have your AGW and AKS cluster in separate modules, you can add an existing resource to your AKS module:

resource appGateway 'Microsoft.Network/applicationGateways@2021-02-01' existing = {
  name: appGatewayName
}

Then enable the add-on like this, and it'll now pass preflight validation and do the deployment successfully:

ingressApplicationGateway: {
  enabled: true
  config: {
    applicationGatewayId: appGateway.id
  }
}

I cannot make this work, has anyone else?
My scenario is greenfield: AGIC in child module, AKS in main template with dependsOn for the AGIC module, ingressApplicationGateway value references the ID of the separate "existing resource" declaration pointing to the agw name used in the AGIC module.
Still the template fails the preflight checks because the ingressApplicationGateway fields cannot find the AGW I'm referencing.

I did manage to make this work, but I had to have both the AGW and AKS cluster in separate modules called from main. The key was to have the appGatewayName passed into the AKS module as an output of the AGW module

module aksCluster './modules/akscluster.bicep' = {
	params: {
		appGatewayName: appGw.outputs.name
	}
}

Also got this working as @northynorth describes. Nested AKS + AGW modules called from main, passing the AGW resource ID (output from the AGW module) as a parameter to the AKS module.

[sylar217]

What's the point of having a dependsOn section then? Can't we deploy AKS agic add on with a custom App gateway from an arm-template then?

[brwilkinson]

@evanrappe I cannot make it work.

@thepaulmacca I am using existing resource in my AKS template, however the validation still fails.

Also using modules like below... however the dependson validation is not making it to the child modules.

I also feel like there may have been some regression on this? Since I haven't run into this issue in some time, I had thought these problems were long behind us with the AGIC/IngressApplicationGateway/addonProfiles by now.

[EldarBorge]

I made it work by referencing the resourceId in Bicep for the yet to be created Application Gateway.

var appGwName = 'name'
var appGwId = resourceId('Microsoft.Network/applicationGateways', appGwName)

resource appGw 'Microsoft.Network/applicationGateways@2021-05-01' = {
  name: appGwName
  ...
}

module aks 'aks.bicep' = {
  name: 'aks'
  params: {
    appGwId: appGwId
  }
}

I'm not sure if having AKS in a module is required for this solution to work. At the very least you need to make sure the AppGw is created prior to using the variable, as it references a non existing AppGw if not.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ghost]

This issue will now be closed because it hasn't had any activity for 7 days after stale. petegrimsdale feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.