Failed to watch *v1beta1.AzureApplicationGatewayRewrite

[wkostn]

After upgrading AGIC from version 1.5.0 to 1.7.2 the following error/warning is displayed in the logs of the agic pod:
Failed to watch *v1beta1.AzureApplicationGatewayRewrite: failed to list *v1beta1.AzureApplicationGatewayRewrite: the server could not find the requested resource (get azureapplicationgatewayrewrites.appgw.ingress.azure.io)

Probably similar to #1495

The update on the Application Gateway is successful but we would like to get rid of this error. Any advice where this error is coming from and how to fix this?

To Reproduce
Upgrade agic to version 1.7.2 on AKS cluster version 1.27.3

Ingress Controller details

** DESCRIBE OUTPUT **
Name: agic-dev-ingress-azure-7787495b7d-mqgp9
Namespace: agic-dev
Priority: 0
Service Account: agic-dev-sa-ingress-azure
Node: aks-systempool-42583204-vmss00000x/10.75.161.5
Start Time: Thu, 14 Sep 2023 11:11:21 +0200
Labels: aadpodidbinding=agic-dev-ingress-azure
app=ingress-azure
pod-template-hash=7787495b7d
release=agic-dev
Annotations: checksum/config: e6c88ae7319e0feb757a871e31a7cbba825490bfad28955cf856f86a5de90415
kubectl.kubernetes.io/restartedAt: 2023-05-25T08:09:14Z
prometheus.io/port: 8123
prometheus.io/scrape: true
Status: Running
IP: 10.75.161.8
IPs:
IP: 10.75.161.8
Controlled By: ReplicaSet/agic-dev-ingress-azure-7787495b7d
Containers:
ingress-azure:
Container ID: containerd://c2f13bd7b9e37c52f68a7a7506cc44d74ade561153ed3e9f8c56757c6a5969f9
Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2
Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:eeb1d42ebfb872478d9b0b16f6936ea938d6e5eed4a59cde332b8757556a5e1f
Port:
Host Port:
State: Running
Started: Thu, 14 Sep 2023 11:11:26 +0200
Ready: True
Restart Count: 0
Limits:
cpu: 200m
memory: 300Mi
Requests:
cpu: 100m
memory: 150Mi
Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
agic-dev-cm-ingress-azure ConfigMap Optional: false
Environment:
AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json
AGIC_POD_NAME: agic-dev-ingress-azure-7787495b7d-mqgp9 (v1:metadata.name)
AGIC_POD_NAMESPACE: agic-dev (v1:metadata.namespace)
Mounts:
/etc/appgw/ from azure (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6c5rk (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
azure:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/
HostPathType: Directory
kube-api-access-6c5rk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.azure.com/mode=system
Tolerations: CriticalAddonsOnly op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 35m default-scheduler Successfully assigned agic-dev/agic-dev-ingress-azure-7787495b7d-mqgp9 to aks-systempool-42583204-vmss00000x
Normal Pulling 35m kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2"
Normal Pulled 35m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2" in 4.357888201s (4.357896101s including waiting)
Normal Created 35m kubelet Created container ingress-azure
Normal Started 35m kubelet Started container ingress-azure
Warning Unhealthy 35m (x2 over 35m) kubelet Readiness probe failed: Get "http://10.75.161.8:8123/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 35m kubelet Liveness probe failed: Get "http://10.75.161.8:8123/health/alive": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

** AGIC LOGS **
E0914 09:52:03.395285 1 requestroutingrules.go:277] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/e84d14c9-e150-4500-93b2-5aa2829e295d/resourceGroups//providers/Microsoft.Network/applicationGateways//redirectConfigurations/sslr-fl-20c873062004285c8255ab97d9341396
I0914 09:52:03.901124 1 mutate_app_gateway.go:166] BEGIN AppGateway deployment
I0914 09:52:08.590516 1 client.go:220] OperationID='eeb1e7fc-f1c0-44de-a0a8-178e15088b8e'
I0914 09:52:19.006191 1 mutate_app_gateway.go:174] Applied generated Application Gateway configuration
I0914 09:52:19.006224 1 mutate_app_gateway.go:189] cache: Updated with latest applied config.
I0914 09:52:19.095722 1 mutate_app_gateway.go:193] END AppGateway deployment
I0914 09:52:19.095770 1 controller.go:152] Completed last event loop run in: 29.906588424s
I0914 09:52:39.638688 1 reflector.go:255] Listing and watching *v1beta1.AzureApplicationGatewayRewrite from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
E0914 09:52:39.643193 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Failed to watch *v1beta1.AzureApplicationGatewayRewrite: failed to list *v1beta1.AzureApplicationGatewayRewrite: the server could not find the requested resource (get azureapplicationgatewayrewrites.appgw.ingress.azure.io)

[jlopezv31]

Hi,

When you update, the azureapplicationgatewayrewrites.appgw.ingress.azure.io object is not created in your cluster. I created it manually and I don't have problems. https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/helm/ingress-azure/crds/azureapplicationgatewayrewrite.yaml