Metric nginx_ingress_controller_config_last_reload_successful does not work correctly

[raorudzhov]

What happened:
This is a re-open of #7738.
The metric nginx_ingress_controller_config_last_reload_successful does not work correctly.
When I create an invalid Ingress resource (cluster-side validation is disabled for CVE mitigation), NGINX starts complaining about the configuration in the logs and the metric nginx_ingress_controller_config_last_reload_successful becomes 0.
When I delete the invalid configuration, NGINX stops complaining, but the metric still remains 0.

P.S. ⚠️ This issue is version-independent.
It affects all versions of ingress-nginx, up to and including the latest release.

What you expected to happen:
The metric nginx_ingress_controller_config_last_reload_successful should be set back to 1 after removing the invalid Ingress resource and restoring a valid configuration.

Kubernetes version (use kubectl version):
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.31.11

Environment:

• Cloud provider or hardware configuration: BareMetall
• OS (e.g. from /etc/os-release): Debian GNU/Linux 12 (bookworm)
• Kernel (e.g. uname -a): 6.1.0-38-amd64
• Basic cluster related info:

NAME   STATUS   ROLES                  AGE   VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
master01   Ready    control-plane,master   95m   v1.31.11   192.168.1.111   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-38-amd64   containerd://1.7.27

How to reproduce this issue:

1. Create an invalid Ingress resource;

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      bla-bla-bla
spec:
  ingressClassName: nginx
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp
                port:
                  number: 80

2. See in logs that NGINX complains about invalid configuration;

Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
nginx: [emerg] unknown directive "bla-bla-bla" in /tmp/nginx/nginx-cfg3530923120:6208
nginx: configuration file /tmp/nginx/nginx-cfg3530923120 test failed
-------------------------------------------------------------------------------

3. Check metrics inside the pod, value is 0;

curl 127.0.0.1:10254/metrics -s | grep reload_successful{
nginx_ingress_controller_config_last_reload_successful{controller_class="ingress-nginx.deckhouse.io/nginx",controller_namespace="ingress-nginx",controller_pod="controller-nginx-rwmf5"} 0

4. Delete the invalid Ingress resource;
5. Check metrics again, value is still 0.

curl 127.0.0.1:10254/metrics -s | grep reload_successful{
nginx_ingress_controller_config_last_reload_successful{controller_class="ingress-nginx.deckhouse.io/nginx",controller_namespace="ingress-nginx",controller_pod="controller-nginx-rwmf5"} 0

Anything else we need to know:
I found the place in the code that prevents the metric from being updated because the configuration was never applied:
https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/controller/controller.go#L195

I plan to create a fork and contribute a fix for this issue.

[Gacko]

/triage accepted
/kind bug
/priority backlog
/assign