title	description	services	author	ms.service	ms.topic	ms.date	ms.author
Create an ingress controller with an existing Application Gateway	This article provides information on how to deploy an Application Gateway Ingress Controller with an existing Application Gateway.	application-gateway	caya	application-gateway	how-to	11/4/2019	caya

Install an Application Gateway Ingress Controller (AGIC) using an existing Application Gateway

The Application Gateway Ingress Controller (AGIC) is a pod within your Kubernetes cluster. AGIC monitors the Kubernetes Ingress resources, and creates and applies Application Gateway config based on the status of the Kubernetes cluster.

Outline:

Prerequisites
Azure Resource Manager Authentication (ARM)
- Option 1: Set up aad-pod-identity and create Azure Identity on ARMs
- Option 2: Using a Service Principal
Install Ingress Controller using Helm
Multi-cluster / Shared Application Gateway: Install AGIC in an environment, where Application Gateway is shared between one or more AKS clusters and/or other Azure components.

Prerequisites

This document assumes you already have the following tools and infrastructure installed:

AKS with Advanced Networking enabled
Application Gateway v2 in the same virtual network as AKS
AAD Pod Identity installed on your AKS cluster
Cloud Shell is the Azure shell environment, which has az CLI, kubectl, and helm installed. These tools are required for the commands below.

Please backup your Application Gateway's configuration before installing AGIC:

using Azure portal navigate to your Application Gateway instance
from Export template click Download

The zip file you downloaded will have JSON templates, bash, and PowerShell scripts you could use to restore App Gateway should that become necessary

Install Helm

Helm is a package manager for Kubernetes. We will leverage it to install the application-gateway-kubernetes-ingress package. Use Cloud Shell to install Helm:

Install Helm and run the following to add application-gateway-kubernetes-ingress helm package:

RBAC enabled AKS cluster

kubectl create serviceaccount --namespace kube-system tiller-sa
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller-sa
helm init --tiller-namespace kube-system --service-account tiller-sa

RBAC disabled AKS cluster

helm init

Add the AGIC Helm repository:

helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

Azure Resource Manager Authentication

AGIC communicates with the Kubernetes API server and the Azure Resource Manager. It requires an identity to access these APIs.

Set up AAD Pod Identity

AAD Pod Identity is a controller, similar to AGIC, which also runs on your AKS. It binds Azure Active Directory identities to your Kubernetes pods. Identity is required for an application in a Kubernetes pod to be able to communicate with other Azure components. In the particular case here, we need authorization for the AGIC pod to make HTTP requests to ARM.

Follow the AAD Pod Identity installation instructions to add this component to your AKS.

Next we need to create an Azure identity and give it permissions ARM. Use Cloud Shell to run all of the following commands and create an identity:

Create an Azure identity in the same resource group as the AKS nodes. Picking the correct resource group is important. The resource group required in the command below is not the one referenced on the AKS portal pane. This is the resource group of the aks-agentpool virtual machines. Typically that resource group starts with MC_ and contains the name of your AKS. For instance: MC_resourceGroup_aksABCD_westus
```
az identity create -g <agent-pool-resource-group> -n <identity-name>
```
For the role assignment commands below we need to obtain principalId for the newly created identity:
```
az identity show -g <resourcegroup> -n <identity-name>
```
Give the identity Contributor access to your Application Gateway. For this you need the ID of the Application Gateway, which will look something like this: /subscriptions/A/resourceGroups/B/providers/Microsoft.Network/applicationGateways/C

Get the list of Application Gateway IDs in your subscription with: az network application-gateway list --query '[].id'
```
az role assignment create \
    --role Contributor \
    --assignee <principalId> \
    --scope <App-Gateway-ID>
```
Give the identity Reader access to the Application Gateway resource group. The resource group ID would look like: /subscriptions/A/resourceGroups/B. You can get all resource groups with: az group list --query '[].id'
```
az role assignment create \
    --role Reader \
    --assignee <principalId> \
    --scope <App-Gateway-Resource-Group-ID>
```

Using a Service Principal

It is also possible to provide AGIC access to ARM via a Kubernetes secret.

Create an Active Directory Service Principal and encode with base64. The base64 encoding is required for the JSON blob to be saved to Kubernetes.

az ad sp create-for-rbac --sdk-auth | base64 -w0

Add the base64 encoded JSON blob to the helm-config.yaml file. More information on helm-config.yaml is in the next section.

armAuth:
    type: servicePrincipal
    secretJSON: <Base64-Encoded-Credentials>

Install Ingress Controller as a Helm Chart

In the first few steps, we install Helm's Tiller on your Kubernetes cluster. Use Cloud Shell to install the AGIC Helm package:

Add the application-gateway-kubernetes-ingress helm repo and perform a helm update

helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

Download helm-config.yaml, which will configure AGIC:

wget https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml -O helm-config.yaml

Or copy the YAML file below:

# This file contains the essential configs for the ingress controller helm chart

# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3

################################################################################
# Specify which application gateway the ingress controller will manage
#
appgw:
    subscriptionId: <subscriptionId>
    resourceGroup: <resourceGroupName>
    name: <applicationGatewayName>

    # Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
    # This prohibits AGIC from applying config for any host/path.
    # Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
    shared: false

################################################################################
# Specify which kubernetes namespace the ingress controller will watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in Ingress Controller observing all acessible namespaces.
#
# kubernetes:
#   watchNamespace: <namespace>

################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
armAuth:
    type: aadPodIdentity
    identityResourceID: <identityResourceId>
    identityClientID:  <identityClientId>

## Alternatively you can use Service Principal credentials
# armAuth:
#    type: servicePrincipal
#    secretJSON: <<Generate this value with: "az ad sp create-for-rbac --sdk-auth | base64 -w0" >>

################################################################################
# Specify if the cluster is RBAC enabled or not
rbac:
    enabled: false # true/false

# Specify aks cluster related information. THIS IS BEING DEPRECATED.
aksClusterConfiguration:
    apiServerAddress: <aks-api-server-address>

Edit helm-config.yaml and fill in the values for appgw and armAuth.
```
nano helm-config.yaml
```
[!NOTE] The <identity-resource-id> and <identity-client-id> are the properties of the Azure AD Identity you setup in the previous section. You can retrieve this information by running the following command: az identity show -g <resourcegroup> -n <identity-name>, where <resourcegroup> is the resource group in which the top level AKS cluster object, Application Gateway and Managed Identify are deployed.

Install Helm chart application-gateway-kubernetes-ingress with the helm-config.yaml configuration from the previous step

helm install -f <helm-config.yaml> application-gateway-kubernetes-ingress/ingress-azure

Alternatively you can combine the helm-config.yaml and the Helm command in one step:

helm install ./helm/ingress-azure \
     --name ingress-azure \
     --namespace default \
     --debug \
     --set appgw.name=applicationgatewayABCD \
     --set appgw.resourceGroup=your-resource-group \
     --set appgw.subscriptionId=subscription-uuid \
     --set appgw.shared=false \
     --set armAuth.type=servicePrincipal \
     --set armAuth.secretJSON=$(az ad sp create-for-rbac --sdk-auth | base64 -w0) \
     --set rbac.enabled=true \
     --set verbosityLevel=3 \
     --set kubernetes.watchNamespace=default \
     --set aksClusterConfiguration.apiServerAddress=aks-abcdefg.hcp.westus2.azmk8s.io

Check the log of the newly created pod to verify if it started properly

Refer to this how-to guide to understand how you can expose an AKS service over HTTP or HTTPS, to the internet, using an Azure Application Gateway.

Multi-cluster / Shared Application Gateway

By default AGIC assumes full ownership of the Application Gateway it is linked to. AGIC version 0.8.0 and later can share a single Application Gateway with other Azure components. For instance, we could use the same Application Gateway for an app hosted on Virtual Machine Scale Set as well as an AKS cluster.

Please backup your Application Gateway's configuration before enabling this setting:

using Azure portal navigate to your Application Gateway instance
from Export template click Download

The zip file you downloaded will have JSON templates, bash, and PowerShell scripts you could use to restore Application Gateway

Example Scenario

Let's look at an imaginary Application Gateway, which manages traffic for two web sites:

dev.contoso.com - hosted on a new AKS, using Application Gateway and AGIC
prod.contoso.com - hosted on an Azure Virtual Machine Scale Set

With default settings, AGIC assumes 100% ownership of the Application Gateway it is pointed to. AGIC overwrites all of App Gateway's configuration. If we were to manually create a listener for prod.contoso.com (on Application Gateway), without defining it in the Kubernetes Ingress, AGIC will delete the prod.contoso.com config within seconds.

To install AGIC and also serve prod.contoso.com from our Virtual Machine Scale Set machines, we must constrain AGIC to configuring dev.contoso.com only. This is facilitated by instantiating the following CRD:

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: prod-contoso-com
spec:
  hostname: prod.contoso.com
EOF

The command above creates an AzureIngressProhibitedTarget object. This makes AGIC (version 0.8.0 and later) aware of the existence of Application Gateway config for prod.contoso.com and explicitly instructs it to avoid changing any configuration related to that hostname.

Enable with new AGIC installation

To limit AGIC (version 0.8.0 and later) to a subset of the Application Gateway configuration modify the helm-config.yaml template. Under the appgw: section, add shared key and set it to to true.

appgw:
    subscriptionId: <subscriptionId>    # existing field
    resourceGroup: <resourceGroupName>  # existing field
    name: <applicationGatewayName>      # existing field
    shared: true                        # <<<<< Add this field to enable shared Application Gateway >>>>>

Apply the Helm changes:

Ensure the AzureIngressProhibitedTarget CRD is installed with:

kubectl apply -f https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/ae695ef9bd05c8b708cedf6ff545595d0b7022dc/crds/AzureIngressProhibitedTarget.yaml

Update Helm:

helm upgrade \
    --recreate-pods \
    -f helm-config.yaml \
    ingress-azure application-gateway-kubernetes-ingress/ingress-azure

As a result your AKS will have a new instance of AzureIngressProhibitedTarget called prohibit-all-targets:

kubectl get AzureIngressProhibitedTargets prohibit-all-targets -o yaml

The object prohibit-all-targets, as the name implies, prohibits AGIC from changing config for any host and path. Helm install with appgw.shared=true will deploy AGIC, but will not make any changes to Application Gateway.

Broaden permissions

Since Helm with appgw.shared=true and the default prohibit-all-targets blocks AGIC from applying any config.

Broaden AGIC permissions with:

Create a new AzureIngressProhibitedTarget with your specific setup:

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: your-custom-prohibitions
spec:
  hostname: your.own-hostname.com
EOF

Only after you have created your own custom prohibition, you can delete the default one, which is too broad:
```
kubectl delete AzureIngressProhibitedTarget prohibit-all-targets
```

Enable for an existing AGIC installation

Let's assume that we already have a working AKS, Application Gateway, and configured AGIC in our cluster. We have an Ingress for prod.contosor.com and are successfully serving traffic for it from AKS. We want to add staging.contoso.com to our existing Application Gateway, but need to host it on a VM. We are going to reuse the existing Application Gateway and manually configure a listener and backend pools for staging.contoso.com. But manually tweaking Application Gateway config (via portal, ARM APIs or Terraform) would conflict with AGIC's assumptions of full ownership. Shortly after we apply changes, AGIC will overwrite or delete them.

We can prohibit AGIC from making changes to a subset of configuration.

Create an AzureIngressProhibitedTarget object:

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: manually-configured-staging-environment
spec:
  hostname: staging.contoso.com
EOF

View the newly created object:

kubectl get AzureIngressProhibitedTargets

Modify Application Gateway config via portal - add listeners, routing rules, backends etc. The new object we created (manually-configured-staging-environment) will prohibit AGIC from overwriting Application Gateway configuration related to staging.contoso.com.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ingress-controller-install-existing.md

ingress-controller-install-existing.md

Install an Application Gateway Ingress Controller (AGIC) using an existing Application Gateway

Outline:

Prerequisites

Install Helm

Azure Resource Manager Authentication

Set up AAD Pod Identity

Using a Service Principal

Install Ingress Controller as a Helm Chart

Multi-cluster / Shared Application Gateway

Example Scenario

Enable with new AGIC installation

Broaden permissions

Enable for an existing AGIC installation

Files

ingress-controller-install-existing.md

Latest commit

History

ingress-controller-install-existing.md

File metadata and controls

Install an Application Gateway Ingress Controller (AGIC) using an existing Application Gateway

Outline:

Prerequisites

Install Helm

Azure Resource Manager Authentication

Set up AAD Pod Identity

Using a Service Principal

Install Ingress Controller as a Helm Chart

Multi-cluster / Shared Application Gateway

Example Scenario

Enable with new AGIC installation

Broaden permissions

Enable for an existing AGIC installation