appgw.ingress.kubernetes.io/health-probe-status-code does nothing in application gateway

[LudoNieuwenhuizen]

Hi there,

We have image 1.4.0 of the ag-k8s-ingress running in our cluster, and have setup multiple ingress rules using this setup. For most of our endpoints, we add a health-probe path with the appgw.ingress.kubernetes.io/health-probe-path: annotation, and this leads to the correct health probe settings. So far everything is good.
For some of our endpoints, we want to allow certain additional response codes, say 404. To implement this, we added the following annotation:
appgw.ingress.kubernetes.io/health-probe-status-code: 200-299, 404
While the configuration is accepted by aks, this does not lead to any change in the related health probe's settings.

Ingress setup extract:

Name:             pivt-demo-backends-ingress-azure
Namespace:        ingress-azure-allowlisted
Address:          <redacted>
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  pivt-demo-backends-azure-tls terminates <redaceted-url>
Rules:
  Host                        Path  Backends
  ----                        ----  --------
  <redaceted-url>
                              /   uat-ag-allowlisted-ingress-nginx-controller:80 (10.11.3.119:80)
Annotations:                  appgw.ingress.kubernetes.io/backend-protocol: http
                              appgw.ingress.kubernetes.io/health-probe-interval: 20
                              appgw.ingress.kubernetes.io/health-probe-path: /api/health
                              appgw.ingress.kubernetes.io/health-probe-port: 80
                              appgw.ingress.kubernetes.io/health-probe-status-code: 200-299, 404
                              appgw.ingress.kubernetes.io/health-probe-timeout: 2
                              appgw.ingress.kubernetes.io/health-probe-unhealthy-threshold: 5
                              appgw.ingress.kubernetes.io/request-timeout: 20
                              appgw.ingress.kubernetes.io/ssl-redirect: true
                              cert-manager.io/cluster-issuer: cluster-issuer-<redacted>
                              kubernetes.io/ingress.class: azure/application-gateway
                              kubernetes.io/tls-acme: true

part of agic config:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/config: 5755a388347f1aad9099bfd5cb90d9b0631043ae7d7a9a5bbf0b5b7d00331d09
    prometheus.io/port: "8123"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-06-30T12:28:20Z"
  generateName: prd-ag-allowlisted-ingress-azure-67b6c59db5-
  labels:
    aadpodidbinding: prd-ag-allowlisted-ingress-azure
    app: ingress-azure
    pod-template-hash: 67b6c59db5
    release: prd-ag-allowlistedspec:
  containers:
  - env:
    - name: AZURE_CLOUD_PROVIDER_LOCATION
      value: /etc/appgw/azure.json
    - name: AGIC_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: AGIC_POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    envFrom:
    - configMapRef:
        name: prd-ag-allowlisted-cm-ingress-azure
    image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.4.0
    imagePullPolicy: Always
<...>

Screenshot of health probe:

Note that the 'use probe mathcing conditions' is not set to allow at all. I expected this to be set to the range 200-299, 404.

I needed, I can provide additional information.
We can manually edit the health probes to allow the 404, but after each update to the application gateway endoints, all of them are reset all the time.
I checked if there is a v.1.4.1 or similar available to fix this, but I could not find it.

Help and suggestions are very welcome!!

[akshaysngupta]

change annotation from health-probe-status-code -> health-probe-status-codes

https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/annotations.md#health-probe-status-codes