Possible race condition between processing Ingress and Secret resource

[DenisBiondic]

Describe the bug
When using the application gateway with cert-manager, a random issue occurs where the 443 port / route will not be mapped in the application gateway. The application gateway ingress controller simply reports "secret is malformed, tls.key or tls.crt is not defined" (check logs below). Restarting the controller resolves the issue, which seems to me that the watching for these tls secrets is not implemented properly, there is a timing issue somewhere.

Perhaps someone else had a similar issue?

To Reproduce
-- no reliable steps found yet, occurs sporadically, every 1 in 10 deployments --

Ingress Controller details
Controller Version 1.0.0

I0129 12:47:24.645533 1 mutate_app_gateway.go:150] cache: Config has NOT changed! No need to connect to ARM.
I0129 13:49:26.780663 1 mutate_app_gateway.go:150] cache: Config has NOT changed! No need to connect to ARM.
I0129 13:49:45.001575 1 mutate_app_gateway.go:150] cache: Config has NOT changed! No need to connect to ARM.
I0129 13:49:52.247375 1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
E0129 13:49:52.376453 1 secretstore.go:75] secret [-] is malformed, tls.key or tls.crt is not defined
I0129 13:50:12.849423 1 mutate_app_gateway.go:182] Applied App Gateway config in 20.60201513s
I0129 13:50:12.849454 1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I0129 13:50:12.850154 1 mutate_app_gateway.go:203] END AppGateway deployment
I0129 13:50:13.957389 1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I0129 13:50:34.413684 1 mutate_app_gateway.go:182] Applied App Gateway config in 20.456241831s
I0129 13:50:34.413707 1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I0129 13:50:34.414836 1 mutate_app_gateway.go:203] END AppGateway deployment
I0129 13:50:35.556534 1 mutate_app_gateway.go:150] cache: Config has NOT changed! No need to connect to ARM.
E0131 09:12:05.004845 1 secretstore.go:75] secret [-] is malformed, tls.key or tls.crt is not defined
I0131 09:12:05.343003 1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I0131 09:12:25.846581 1 mutate_app_gateway.go:182] Applied App Gateway config in 20.503528948s
I0131 09:12:25.846606 1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I0131 09:12:25.852946 1 mutate_app_gateway.go:203] END AppGateway deployment
I0131 09:12:26.981341 1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I0131 09:12:47.330796 1 mutate_app_gateway.go:182] Applied App Gateway config in 20.349421891s
I0131 09:12:47.330828 1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I0131 09:12:47.331807 1 mutate_app_gateway.go:203] END AppGateway deployment
I0131 09:12:48.701208 1 mutate_app_gateway.go:150] cache: Config has NOT changed! No need to connect to ARM.
I0131 09:17:30.568515 1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I0131 09:17:50.989366 1 mutate_app_gateway.go:182] Applied App Gateway config in 20.420826306s
I0131 09:17:50.989389 1 mutate_app_gateway.go:198] cache: Updated with latest applied config.

[akshaysngupta]

@DenisBiondic This seems to a race condition between processing the ingress and secret object.

[sergiomcalzada]

Same problem here, restarting the pod solves the problem