access_log /dev/stdout main annotation causing issue #13862
Description
We are using nginx ingress has load balancer which work both has load balance and request handler
We have namespace called ingress-controller which is deployed using helm chart
We have around 80+ namespaces which commonly using single IP address has nginx load balancer
Each application has its own ingress service which will communicate to external world with main ingress controller.
One of the team members added below annotations to debug his application
annotations:
nginx.ingress.kubernetes.io/configuration-snippet: |
error_log /dev/stderr debug; # For debug-level logging
access_log /dev/stdout main;
Now due to this annotation whenever pod of any namespace within the cluster got RESTARTED then corrresponding application URL throwing 502 gateway error, because it still pointing OLD POD ENDPOINT instead of new endpoints.
This started failing for all the application, then somehow, we identified and removed given annotation, then all the applications started working as expected.
Question:
- How to avoid this in future?
- Why does local ingress service annotation affect globally?
- What is the root cause of this issue?
- How to identify any newly added annotation in their local ingress service?
Helm chart we are using ngress-nginx-4.12.1 App version 1.12.1
Configmap data that exists
data:
allow-snippet-annotations: "true"
annotations-risk-level: Critical
enable-vts-status: "true"
proxy-body-size: 2000m
use-forwarded-headers: "true"