Running as root irrespective of the authentication mechanism to Azure API

[mikhailadvani]

Describe the bug

In version 1.3.0 of the helm chart security context of runAsUser: 0 was added which forces to run as root.

To Reproduce
Steps to reproduce the behavior:

1. Exec into the application-gateway-ingress-controller pod.
2. Run id

The user is root.

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.

Name:         application-gateway-ingress-controller-ingress-azure-84ffbm8596
Namespace:    default
Priority:     0
Node:         aks-cpt-34239724-vmss000000/10.X.X.X
Start Time:   Fri, 22 Jan 2021 11:53:47 +0100
Labels:       aadpodidbinding=application-gateway-ingress-controller-ingress-azure
              app=ingress-azure
              pod-template-hash=84ffb89c75
              release=application-gateway-ingress-controller
Annotations:  checksum/config: 185a0b2d30a1c71680c4c9df01daf6e9a82915f5aaa78992ca1c55db27afe87e
              prometheus.io/port: 8123
              prometheus.io/scrape: true
Status:       Running
IP:           10.X.X.X
IPs:
  IP:           10.X.X.X
Controlled By:  ReplicaSet/application-gateway-ingress-controller-ingress-azure-84ffb89c75
Containers:
  ingress-azure:
    Container ID:   docker://fdf8d6071c133d7d9eb80769bcb8a5cc280fa4924a3e02072d9b446f8ca28df5
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.3.0
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:28cbb5581b775523821742119f57b25dd584697b6c1c05c5ddeabf2fb59f37c7
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 22 Jan 2021 11:54:03 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      application-gateway-ingress-controller-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  application-gateway-ingress-controller-ingress-azure-84ffbm8596 (v1:metadata.name)
      AGIC_POD_NAMESPACE:             cpt (v1:metadata.namespace)
    Mounts:
      /etc/appgw/ from azure (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from application-gateway-ingress-controller-sa-ingress-azure-tobqg7t (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/
    HostPathType:  Directory
  application-gateway-ingress-controller-sa-ingress-azure-tobqg7t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  application-gateway-ingress-controller-sa-ingress-azure-tobqg7t
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  nodegroup=default
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

• Output of `kubectl logs .

ERROR: logging before flag.Parse: I0121 14:19:07.005226       1 utils.go:115] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
ERROR: logging before flag.Parse: I0121 14:19:07.005350       1 main.go:78] Unable to load cloud provider config '/etc/appgw/azure.json'. Error: Reading Az Context file "/etc/appgw/azure.json" failed: open /etc/appgw/azure.json: permission denied
I0121 14:19:07.076331       1 environment.go:246] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0121 14:19:07.076666       1 main.go:128] Appication Gateway Details: Subscription="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" Resource Group="k8s" Name="k8s-appgateway"
I0121 14:19:07.076692       1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I0121 14:19:07.076954       1 httpserver.go:57] Starting API Server on :8123
I0121 14:19:38.578923       1 main.go:175] Ingress Controller will observe all namespaces.
I0121 14:19:38.618298       1 context.go:138] k8s context run started
I0121 14:19:38.618333       1 context.go:188] Waiting for initial cache sync
I0121 14:19:38.818479       1 context.go:201] Initial cache sync done
I0121 14:19:38.818500       1 context.go:202] k8s context run finished
I0121 14:19:38.818611       1 worker.go:39] Worker started

• Any Azure support tickets associated with this issue.

Proposed Solution

1. Populate the securityContext from the values.yaml instead of hardcoding it in the deployment.yaml with the same default value of runAsUser: 0 so that the current implementation does not break.
2. Mount the hostPath conditionally if and only if armAuth is not defined in values.
3. Add documentation stating that users with armAuth defined should ideally override runAsUser to 65534(nobody)