Enable Application Gateway Ingress Controller with Helm using AAD Pod Identity

This article demonstrates how to:

Create an AKS cluster, an Azure Container Registry (ACR) instance, and an Azure Application Gateway (AAG).
Integrate the ACR and the AAG with the AKS.
Run your Java, Java EE, Jakarta EE, or MicroProfile application on the Open Liberty or WebSphere Liberty runtime.
Build the application Docker image using Open Liberty container images.
Deploy the containerized application to an AKS cluster using the Open Liberty Operator.

The Open Liberty Operator simplifies the deployment and management of applications running on Kubernetes clusters. With Open Liberty Operator, you can also perform more advanced operations, such as gathering traces and dumps.

For more details on Open Liberty, see the Open Liberty project page. For more details on IBM WebSphere Liberty, see the WebSphere Liberty product page.

[!INCLUDE quickstarts-free-trial-note]

[!INCLUDE azure-cli-prepare-your-environment.md]

This article requires the latest version of Azure CLI. If using Azure Cloud Shell, the latest version is already installed.
If running the commands in this guide locally (instead of Azure Cloud Shell):
- Prepare a local machine with Unix-like operating system installed (for example, Ubuntu, macOS, Windows Subsystem for Linux).
- Install a Java SE implementation (for example, AdoptOpenJDK OpenJDK 8 LTS/OpenJ9).
- Install Maven 3.5.0 or higher.
- Install Docker for your OS.
- Install jq.

Set up the infrastructure

In this section, you will create user-assigned manage identites, a virtual network, an AKS cluster, an AAG instance, and install Azure Application Gateway Controller (AGIC) using Helm.

# Create resource groups
UAMI_RG_NAME=<uami-rg-name>
VNET_RG_NAME=<vnet-rg-name>
AG_RG_NAME=<ag-rg-name>
AKS_RG_NAME=<aks-rg-name>
az group create --name ${UAMI_RG_NAME} --location eastus
az group create --name ${VNET_RG_NAME} --location eastus
az group create --name ${AG_RG_NAME} --location eastus
az group create --name ${AKS_RG_NAME} --location eastus

# Create identities for AKS cluster
CLUSTER_IDENTITY_NAME=cluster-identity
KUBELET_IDENTITY_NAME=kubelet-identity
az identity create --name ${CLUSTER_IDENTITY_NAME} --resource-group ${UAMI_RG_NAME}
az identity create --name ${KUBELET_IDENTITY_NAME} --resource-group ${UAMI_RG_NAME}
clusterIdentityId=$(az identity show -n ${CLUSTER_IDENTITY_NAME} -g ${UAMI_RG_NAME} -o tsv --query "id")
kubeletIdentityId=$(az identity show -n ${KUBELET_IDENTITY_NAME} -g ${UAMI_RG_NAME} -o tsv --query "id")

# Create vNet and subnets
VNET_NAME=myVnet
AG_SUBNET_NAME=agSubnet
AKS_SUBNET_NAME=aksSubnet
az network vnet create -n ${VNET_NAME} -g ${VNET_RG_NAME} --address-prefix 10.0.0.0/8
az network vnet subnet create -n ${AG_SUBNET_NAME} --address-prefixes 10.241.0.0/16 --vnet-name ${VNET_NAME} -g ${VNET_RG_NAME}
az network vnet subnet create -n ${AKS_SUBNET_NAME} --address-prefixes 10.240.0.0/16 --vnet-name ${VNET_NAME} -g ${VNET_RG_NAME}
agSubnetId=$(az network vnet subnet show -n ${AG_SUBNET_NAME} --vnet-name ${VNET_NAME} -g ${VNET_RG_NAME} --query "id" --output tsv)
aksSubnetId=$(az network vnet subnet show -n ${AKS_SUBNET_NAME} --vnet-name ${VNET_NAME} -g ${VNET_RG_NAME} --query "id" --output tsv)

# Create Application Gateway
PUBLIC_IP_NAME=myPublicIp
az network public-ip create -n ${PUBLIC_IP_NAME} -g ${AG_RG_NAME} --allocation-method Static --sku Standard
APPGW_NAME=myApplicationGateway
az network application-gateway create \
    --name ${APPGW_NAME} \
    --resource-group ${AG_RG_NAME} \
    --location eastus \
    --sku Standard_v2 \
    --public-ip-address ${PUBLIC_IP_NAME} \
    --subnet ${agSubnetId} \
    --frontend-port 80 \
    --priority 1000
appgwId=$(az network application-gateway show -n ${APPGW_NAME} -g ${AG_RG_NAME} -o tsv --query "id")

# Create AKS cluster
AKS_CLUSTER_NAME=akscluster
az aks create \
    --resource-group ${AKS_RG_NAME} \
    --name ${AKS_CLUSTER_NAME} \
    --node-count 1 \
    --enable-managed-identity \
    --assign-identity ${clusterIdentityId} \
    --assign-kubelet-identity ${kubeletIdentityId} \
    --network-plugin azure \
    --vnet-subnet-id ${aksSubnetId} \
    --docker-bridge-address 172.17.0.1/16 \
    --dns-service-ip 10.2.0.10 \
    --service-cidr 10.2.0.0/24 \
    --generate-ssh-keys
az aks get-credentials -n ${AKS_CLUSTER_NAME} -g ${AKS_RG_NAME} --overwrite-existing

# Role assignments required by AAD Pod Identity
export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
export RESOURCE_GROUP=${AKS_RG_NAME}
export CLUSTER_NAME=${AKS_CLUSTER_NAME}
export IDENTITY_RESOURCE_GROUP=${UAMI_RG_NAME}
curl -s https://raw.githubusercontent.com/Azure/aad-pod-identity/master/hack/role-assignment.sh | bash

# Deploy AAD Pod Identity
kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
# For AKS clusters, deploy the MIC and AKS add-on exception by running -
kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/mic-exception.yaml

# Assign roles to the uami used in the AGIC
principalId=$(az identity show -n ${CLUSTER_IDENTITY_NAME} -g ${UAMI_RG_NAME} --query principalId -o tsv)
az role assignment create --role Contributor --assignee ${principalId} --scope ${appgwId}
az role assignment create --role Reader --assignee ${principalId} --scope $(az group show -n ${AG_RG_NAME} --query id -o tsv)

# Grant azure ingress permission
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-azure-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: ingress-azure
  namespace: default
EOF

# Install AGIC
rm -f appgw-helm-config.yaml
cat >> appgw-helm-config.yaml <<EOF
# Based on https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml
verbosityLevel: 3
appgw:
    subscriptionId: @SUB_ID@
    resourceGroup: @APPGW_RG_NAME@
    name: @APPGW_NAME@
    usePrivateIP: false
    shared: false
kubernetes:
    watchNamespace: @WATCH_NAMESPACE@
armAuth:
    type: aadPodIdentity
    identityResourceID: @IDENTITY_RESOURCE_ID@
    identityClientID: @IDENTITY_CLIENT_ID@
rbac:
    create: true
EOF

AGIC_WATCH_NAMESPACE='""'
clusterIdentityClientId=$(az identity show -n ${CLUSTER_IDENTITY_NAME} -g ${UAMI_RG_NAME} -o tsv --query clientId)
sed -i -e "s:@SUB_ID@:${SUBSCRIPTION_ID}:g" appgw-helm-config.yaml
sed -i -e "s:@APPGW_RG_NAME@:${AG_RG_NAME}:g" appgw-helm-config.yaml
sed -i -e "s:@APPGW_NAME@:${APPGW_NAME}:g" appgw-helm-config.yaml
sed -i -e "s:@WATCH_NAMESPACE@:${AGIC_WATCH_NAMESPACE}:g" appgw-helm-config.yaml
sed -i -e "s:@IDENTITY_RESOURCE_ID@:${clusterIdentityId}:g" appgw-helm-config.yaml
sed -i -e "s:@IDENTITY_CLIENT_ID@:${clusterIdentityClientId}:g" appgw-helm-config.yaml

# If you haven't installed Helm, pls install it first
# Install Helm from `apt`, more details pls see https://helm.sh/docs/intro/install/
curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
sudo apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

# Add the application-gateway-kubernetes-ingress helm repo and perform a helm update
helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

azureAppgwIngressVersion="1.5.1"
helm install ingress-azure \
    -f appgw-helm-config.yaml \
    application-gateway-kubernetes-ingress/ingress-azure \
    --version ${azureAppgwIngressVersion}

rm -f appgw-helm-config.yaml

Create an ACR instance

Use the az acr create command to create the ACR instance. The following example creates an ACR instance named youruniqueacrname. Make sure youruniqueacrname is unique within Azure.

RESOURCE_GROUP_NAME=${AKS_RG_NAME}
REGISTRY_NAME=youruniqueacrname
az acr create --resource-group $RESOURCE_GROUP_NAME --name $REGISTRY_NAME --sku Basic --admin-enabled

After a short time, you should see a JSON output that contains:

  "provisioningState": "Succeeded",
  "publicNetworkAccess": "Enabled",
  "resourceGroup": "<aks-rg-name>",

Connect to the ACR instance

You will need to sign in to the ACR instance before you can push an image to it. Run the following commands to verify the connection:

LOGIN_SERVER=$(az acr show -n $REGISTRY_NAME --query 'loginServer' -o tsv)
USER_NAME=$(az acr credential show -n $REGISTRY_NAME --query 'username' -o tsv)
PASSWORD=$(az acr credential show -n $REGISTRY_NAME --query 'passwords[0].value' -o tsv)

docker login $LOGIN_SERVER -u $USER_NAME -p $PASSWORD

You should see Login Succeeded at the end of command output if you have logged into the ACR instance successfully.

Add a user node pool to the AKS cluster

To run your application on a user node pool, you will need to add it beforehand. Run the following commands to add a user nood pool:

NODE_LABEL_KEY=sku
NODE_LABEL_VALUE=gpu
az aks nodepool add \
    --resource-group $RESOURCE_GROUP_NAME \
    --cluster-name $CLUSTER_NAME \
    --name labelnp \
    --node-count 1 \
    --labels ${NODE_LABEL_KEY}=${NODE_LABEL_VALUE}

# Optional: list node pools
az aks nodepool list -g $RESOURCE_GROUP_NAME --cluster-name $CLUSTER_NAME

Install Open Liberty Operator

After creating and connecting to the cluster, install the Open Liberty Operator by running the following commands.

OPERATOR_NAMESPACE=default
WATCH_NAMESPACE='""'

# Install Custom Resource Definitions (CRDs) for OpenLibertyApplication
kubectl apply -f https://raw.githubusercontent.com/OpenLiberty/open-liberty-operator/main/deploy/releases/0.8.2/kubectl/openliberty-app-crd.yaml

# Install cluster-level role-based access to watch all namespaces
curl -L https://raw.githubusercontent.com/OpenLiberty/open-liberty-operator/main/deploy/releases/0.8.2/kubectl/openliberty-app-rbac-watch-all.yaml \
    | sed -e "s/OPEN_LIBERTY_OPERATOR_NAMESPACE/${OPERATOR_NAMESPACE}/" \
    | kubectl apply -f -

# Install the operator on the user node pool
rm -rf openliberty-app-operator.yaml
wget https://raw.githubusercontent.com/OpenLiberty/open-liberty-operator/main/deploy/releases/0.8.2/kubectl/openliberty-app-operator.yaml -O openliberty-app-operator.yaml
cat <<EOF >>openliberty-app-operator.yaml
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: ${NODE_LABEL_KEY}
                operator: In
                values:
                - ${NODE_LABEL_VALUE}
EOF

cat openliberty-app-operator.yaml \
    | sed -e "s/OPEN_LIBERTY_WATCH_NAMESPACE/${WATCH_NAMESPACE}/" \
    | kubectl apply -n ${OPERATOR_NAMESPACE} -f -

rm -rf openliberty-app-operator.yaml

Build application image

To deploy and run your Liberty application on the AKS cluster, containerize your application as a Docker image using Open Liberty container images or WebSphere Liberty container images.

Clone the sample code for this guide. The sample is on GitHub.
Locate to your local clone and run cd agic-aks to change to its sub directory agic-aks.
Run mvn clean package to package the application.
Run mvn liberty:dev to test the application. You should see The defaultServer server is ready to run a smarter planet. in the command output if successful. Use CTRL-C to stop the application.

Retrieve values for properties artifactId and version defined in the pom.xml.

artifactId=$(mvn -q -Dexec.executable=echo -Dexec.args='${project.artifactId}' --non-recursive exec:exec)
version=$(mvn -q -Dexec.executable=echo -Dexec.args='${project.version}' --non-recursive exec:exec)

Run cd target to change directory to the build of the sample.

Run one of the following commands to build the application image and push it to the ACR instance.

Build with Open Liberty base image if you prefer to use Open Liberty as a lightweight open source Java™ runtime:

# Build and tag application image. This will cause the ACR instance to pull the necessary Open Liberty base images.
az acr build -t ${artifactId}:${version} -r $REGISTRY_NAME .

Build with WebSphere Liberty base image if you prefer to use a commercial version of Open Liberty:

# Build and tag application image. This will cause the ACR instance to pull the necessary WebSphere Liberty base images.
az acr build -t ${artifactId}:${version} -r $REGISTRY_NAME --file=Dockerfile-wlp .

Deploy application on the AKS cluster

Follow steps below to deploy the Liberty application on the AKS cluster.

Create a namespace for the sample.

APPLICATION_NAMESPACE=javaee-app-sample-namespace
kubectl create namespace ${APPLICATION_NAMESPACE}

Create a pull secret so that the AKS cluster is authenticated to pull image from the ACR instance.

PULL_SECRET_NAME=javaee-app-sample-pull-secret
kubectl create secret docker-registry ${PULL_SECRET_NAME} \
   --docker-server=${LOGIN_SERVER} \
   --docker-username=${USER_NAME} \
   --docker-password=${PASSWORD} \
   --namespace=${APPLICATION_NAMESPACE}

Verify the current working directory is agic-aks/target of your local clone.

Run the following commands to deploy your Liberty application with 3 replicas to the AKS cluster. Command output is also shown inline.

# Create OpenLibertyApplication "javaee-app-sample"
APPLICATION_NAME=javaee-app-sample
REPLICAS=3

cat openlibertyapplication.yaml \
    | sed -e "s/\${APPLICATION_NAME}/${APPLICATION_NAME}/g" \
    | sed -e "s/\${APPLICATION_NAMESPACE}/${APPLICATION_NAMESPACE}/g" \
    | sed -e "s/\${REPLICAS}/${REPLICAS}/g" \
    | sed -e "s/\${LOGIN_SERVER}/${LOGIN_SERVER}/g" \
    | sed -e "s/\${PULL_SECRET_NAME}/${PULL_SECRET_NAME}/g" \
    | sed -e "s/\${NODE_LABEL_KEY}/${NODE_LABEL_KEY}/g" \
    | sed -e "s/\${NODE_LABEL_VALUE}/${NODE_LABEL_VALUE}/g" \
    | kubectl apply -f -

openlibertyapplication.apps.openliberty.io/javaee-app-sample created

# Check if OpenLibertyApplication instance is created
kubectl get openlibertyapplication ${APPLICATION_NAME} -n ${APPLICATION_NAMESPACE}

NAME                   IMAGE                                                   EXPOSED   RECONCILED   AGE
javaee-app-sample      youruniqueacrname.azurecr.io/javaee-cafe:1.0.0          True         59s

# Check if deployment created by Operator is ready
kubectl get deployment ${APPLICATION_NAME} -n ${APPLICATION_NAMESPACE} --watch

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
javaee-app-sample        0/3     3            0           20s

Wait until you see 3/3 under the READY column and 3 under the AVAILABLE column, use CTRL-C to stop the kubectl watch process.

Run the following commands to deploy Ingress resource for routing client requests to your deployed application. Command output is also shown inline.

# Create Ingress "javaee-app-sample-ingress"
APPLICATION_INGRESS=javaee-app-sample-ingress

cat appgw-cluster-ingress.yaml \
    | sed -e "s/\${APPLICATION_INGRESS}/${APPLICATION_INGRESS}/g" \
    | sed -e "s/\${APPLICATION_NAMESPACE}/${APPLICATION_NAMESPACE}/g" \
    | sed -e "s/\${APPLICATION_NAME}/${APPLICATION_NAME}/g" \
    | kubectl apply -f -

ingress.networking.k8s.io/javaee-app-sample-ingress created

# Check if Ingress instance is created
kubectl get ingress ${APPLICATION_INGRESS} -n ${APPLICATION_NAMESPACE}

NAME                        CLASS    HOSTS   ADDRESS        PORTS   AGE
javaee-app-sample-ingress   <none>   *       20.62.178.13   80      17s

Test the application

To get public IP address of the Ingress, use the kubectl get ingress command with the --watch argument.

kubectl get ingress ${APPLICATION_INGRESS} -n ${APPLICATION_NAMESPACE} --watch

NAME                        CLASS    HOSTS   ADDRESS        PORTS   AGE
javaee-app-sample-ingress   <none>   *       20.62.178.13   80      5m49s

Once the ADDRESS represents to an actual public IP address, use CTRL-C to stop the kubectl watch process.

Open a web browser to the external IP address of your Ingress (20.62.178.13 for the above example) to see the application home page. You should see the pod name of your application replicas displayed at the top-left of the page.

Clean up the resources

To avoid Azure charges, you should clean up unnecessary resources. When the cluster is no longer needed, use the az group delete command to remove the resource group, container service, container registry, and all related resources.

az group delete --name ${AKS_RG_NAME} --yes --no-wait
az group delete --name ${AG_RG_NAME} --yes --no-wait
az group delete --name ${VNET_RG_NAME} --yes --no-wait
az group delete --name ${UAMI_RG_NAME} --yes --no-wait

References

Deploying Azure Application Gateway - Part 1: Intro And Basic Deployment
Deploying Azure Application Gateway - Part 2: End To End Encryption, Path Routing, And Header Rewriting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

agic-helm-identity.md

agic-helm-identity.md

Enable Application Gateway Ingress Controller with Helm using AAD Pod Identity

Set up the infrastructure

Create an ACR instance

Connect to the ACR instance

Add a user node pool to the AKS cluster

Install Open Liberty Operator

Build application image

Deploy application on the AKS cluster

Test the application

Clean up the resources

References

Files

agic-helm-identity.md

Latest commit

History

agic-helm-identity.md

File metadata and controls

Enable Application Gateway Ingress Controller with Helm using AAD Pod Identity

Set up the infrastructure

Create an ACR instance

Connect to the ACR instance

Add a user node pool to the AKS cluster

Install Open Liberty Operator

Build application image

Deploy application on the AKS cluster

Test the application

Clean up the resources

References