Ingress controller not picking changes to new/existing Ingress resources

[sidd-pandey]

Describe the bug
The deployed ingress control does not pick existing or newly created Ingress resources. The Ingress resources are properly annotated. On killing controller pod, the newly spawned pod is able to detect existing Ingress and update the Application Gateway accordingly. But after the initial sync up with Application Gateway, the pod will stop updating/listening for the Ingress existing/new resources.

AKS Kubernetes Version: 1.14.8
Helm Chart version: 1.0.0
Docker image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0

Things I have tried:

• Deploying using Helm 3 and also as manual deployment yaml.
• Changing the service account to cluster-admin, to be sure it has proper permissions to watch resources

To Reproduce
Create an Ingress resource. It does not get synced up with Application Gateway. I have been testing out with a simple ingress resource to expose kubernetes dashboard on the private frontend configuration.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: kube-system
  name: kubernetes-dashboard-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/use-private-ip: "true"
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 80

Ingress Controller details
Controller Description

Name:           ingress-azure-6b585dff57-qhww6
Namespace:      default
Priority:       0
Node:           [some-node]/[ip]
Start Time:     Thu, 23 Jan 2020 10:58:10 +0800
Labels:         aadpodidbinding=ingress-azure
                app=ingress-azure
                pod-template-hash=6b585dff57
Annotations:    prometheus.io/port: 8123
                prometheus.io/scrape: true
Status:         Running
IP:             [ip]
IPs:            <none>
Controlled By:  ReplicaSet/ingress-azure-6b585dff57
Containers:
  ingress-azure:
    Container ID:   docker://[container-id]
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:c295f99ae66443c5a392fd894620fcd1fc313b9efdec96d13f166fefb29780a9
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 23 Jan 2020 10:58:13 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CONTEXT_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:           ingress-azure-6b585dff57-qhww6 (v1:metadata.name)
      AGIC_POD_NAMESPACE:      default (v1:metadata.namespace)
    Mounts:
      /etc/appgw/azure.json from azure (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from sa-ingress-azure-token-7sg6b (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  sa-ingress-azure-token-7sg6b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sa-ingress-azure-token-7sg6b
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                                   Message
  ----     ------     ----               ----                                   -------
  Normal   Scheduled  41m                default-scheduler                      Successfully assigned default/ingress-azure-6b585dff57-qhww6 to [node]
  Normal   Pulling    41m                kubelet, node  Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0"
  Normal   Pulled     41m                kubelet, node  Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0"
  Normal   Created    41m                kubelet, node  Created container ingress-azure
  Normal   Started    41m                kubelet, node  Started container ingress-azure
  Warning  Unhealthy  41m                kubelet, node  Liveness probe failed: Get http://[ip]:8123/health/alive: dial tcp [ip]:8123: connect: connection refused
  Warning  Unhealthy  41m (x3 over 41m)  kubelet, node  Readiness probe failed: Get http://[ip]:8123/health/ready: dial tcp [ip]:8123: connect: connection refused

Controller Logs

ERROR: logging before flag.Parse: I0123 02:58:13.524079       1 main.go:302] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
I0123 02:58:13.573767       1 environment.go:168] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0123 02:58:13.573797       1 main.go:132] App Gateway Details: Subscription:[id], Resource Group: [rg], Name: aks-application-gateway
I0123 02:58:13.573807       1 auth.go:90] Creating authorizer from Azure Managed Service Identity
I0123 02:58:49.783010       1 main.go:179] Ingress Controller will observe all namespaces.
I0123 02:58:49.834831       1 context.go:129] k8s context run started
I0123 02:58:49.834868       1 context.go:168] Waiting for initial cache sync
I0123 02:58:49.935052       1 context.go:176] Initial cache sync done
I0123 02:58:49.935076       1 context.go:177] k8s context run finished
I0123 02:58:49.935175       1 worker.go:35] Worker started
I0123 02:58:49.935264       1 httpserver.go:57] Starting API Server on :8123
I0123 02:58:50.057359       1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I0123 03:00:40.815597       1 mutate_app_gateway.go:182] Applied App Gateway config in 1m50.758202898s
I0123 03:00:40.815628       1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I0123 03:00:40.816362       1 mutate_app_gateway.go:203] END AppGateway deployment

Logs and events are not really that helpful. Let me know there is some other way I can debug this on my end.

• Any Azure support tickets associated with this issue.

[akshaysngupta]

@sidd-pandey You should try the following

1. Look at the logs by setting log verbosity to 5.
2. Look at the k8s events by see if you see any event associated to the ingress created by AGIC.

[sidd-pandey]

@akshaysngupta I tried checking the events and logs at verbosity 5. Nothing really comes up.
All I can see in logs is the JSON template which is getting applied initially. But after I create the ingress resource, no new events or changes to logs are seen.

I am sharing the events here:
kubectl get events --sort-by=.metadata.creationTimestamp --all-namespaces

default             98s         Normal    Scheduled           pod/ingress-azure-1580181398-7d959bb57f-cwbh2        Successfully assigned default/ingress-azure-1580181398-7d959bb57f-cwbh2 to [node-name]
default             96s         Normal    Pulling             pod/ingress-azure-1580181398-7d959bb57f-cwbh2        Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0"
default             88s         Normal    Pulled              pod/ingress-azure-1580181398-7d959bb57f-cwbh2        Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0"
default             74s         Normal    Created             pod/ingress-azure-1580181398-7d959bb57f-cwbh2        Created container ingress-azure
default             74s         Normal    Started             pod/ingress-azure-1580181398-7d959bb57f-cwbh2        Started container ingress-azure
default             62s         Warning   Unhealthy           pod/ingress-azure-1580181398-7d959bb57f-cwbh2        Readiness probe failed: Get http://[ip]:8123/health/ready: dial tcp [ip]:8123: connect: connection refused
default             98s         Normal    SuccessfulCreate    replicaset/ingress-azure-1580181398-7d959bb57f       Created pod: ingress-azure-1580181398-7d959bb57f-cwbh2
default             98s         Normal    ScalingReplicaSet   deployment/ingress-azure-1580181398                  Scaled up replica set ingress-azure-1580181398-7d959bb57f to 1
default             59s         Normal    binding applied     v1.azureidentitybinding/ingress-azure-1580181398     Binding ingress-azure-1580181398 applied on node [node-name] for pod ingress-azure-1580181398-7d959bb57f-c
wbh2-default-ingress-azure-1580181398

Just to add there is no AzureIngressProhibitedTarget type resource which can prohibit the sync up with with application gateway

kubectl get AzureIngressProhibitedTarget --all-namespaces
error: the server doesn't have a resource type "AzureIngressProhibitedTarget"

After killing the pod and following the new logs, most of the stuff is same, but we have following new lines:

I0128 03:38:00.501685       1 health_probes.go:56] Created default HTTPS probe defaultprobe-Http
I0128 03:38:00.501691       1 ingress_rules.go:112] Found backend:kube-system/kubernetes-dashboard
I0128 03:38:00.501788       1 health_probes.go:70] Created probe pb-kube-system-kubernetes-dashboard-80-kubernetes-dashboard-ingress for ingress kube-system/kubernetes-dashboard-ingress and service kube-system/kubernetes-dashboard
I0128 03:38:00.501807       1 backendhttpsettings.go:124] resolving port name [] for service [kube-system/kubernetes-dashboard] and service port [80] for Ingress [kubernetes-dashboard-ingress]
I0128 03:38:00.501828       1 backendhttpsettings.go:185] Created backend http settings bp-kube-system-kubernetes-dashboard-80-9090-kubernetes-dashboard-ingress for ingress kube-system/kubernetes-dashboard-ingress and service kube-system/kubernetes-dashboard
I0128 03:38:00.501842       1 backendaddresspools.go:37] Created default backend pool defaultaddresspool
I0128 03:38:00.501855       1 backendaddresspools.go:48] Created backend pool pool-kube-system-kubernetes-dashboard-80-bp-9090 for service kube-system/kubernetes-dashboard
I0128 03:38:00.501887       1 frontend_listeners.go:117] Processing Rules for Ingress: kube-system/kubernetes-dashboard-ingress
I0128 03:38:00.501948       1 requestroutingrules.go:112] Bound basic rule rr-80 to listener fl-80-privateip(, 80), backend pool pool-kube-system-kubernetes-dashboard-80-bp-9090 and backend http settings bp-kube-system-kubernetes-dashboard-80-9090-kubernetes-dashboard-ingress

But as I mentioned in the issue, there is no issue with initial sync up with newly spawned pods, but after this, it won't pick any changes to this ingress resource or apply new ingress resources.

[akshaysngupta]

This is probably because we are skipping events for kube-system.

application-gateway-kubernetes-ingress/pkg/k8scontext/context.go

Line 42 in d4ee68e

"kube-system": nil,

Is there a use-case for exposing k8s dashboard through AGIC ? Would be great if you can explain your scenario.

[sidd-pandey]

Basically I want to compare the ease of use and reliability when using the managed Azure Application Gateway + Ingress Controller versus Azure Load Balancer + Ambassador.
The primary use cases would be to expose web applications for internal use. We have Azure Virtual Network gateway set up to access these private endpoints.
There is nothing specific about exposing the dashboard. I picked it as a test because it is already installed in AKS clusters. Basically for now, I just want to expose the dashboard (which is read-only) so that I don't have to do port-forwarding, and anyone with VPN can access, without kube config.
I believe that the AGIC should be capable of picking new ingress resources and configuration changes on the fly, not only on start up. Let me know if that's not the case.

[akshaysngupta]

Yes, Ingress Controller will pick up changes on the fly.
If you want to experiment, you can also pick this example.
https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml

[sidd-pandey]

Thanks for sharing a simpler example. I will test this, probably next week, and report back how it goes.