inconsistent label cardinality on nginx_ingress_controller_requests prometheus metrics

[ekovacs]

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

/ $ ./nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.1.1
  Build:         git-167f30dbb
  Repository:    git@github.com:kubernetes/ingress-nginx.git
  nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.12-gke.1500", GitCommit:"d32c0db9a3ccd0ac73b0b3abd0532505217b376e", GitTreeState:"clean", BuildDate:"2021-11-17T09:30:02Z", GoVersion:"go1.15.15b5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release):

NAME="Container-Optimized OS"
ID=cos
PRETTY_NAME="Container-Optimized OS from Google"
HOME_URL="https://cloud.google.com/container-optimized-os/docs"
BUG_REPORT_URL="https://cloud.google.com/container-optimized-os/docs/resources/support-policy#contact_us"
KERNEL_COMMIT_ID=b752fda5f686c995a03732a2d3f6624d476abeca
GOOGLE_METRICS_PRODUCT_ID=26
GOOGLE_CRASH_ID=Lakitu
VERSION=89
VERSION_ID=89
BUILD_ID=16108.534.18

• Kernel (e.g. uname -a):

Linux gke-integration-gke-high-ram-v1-20-12-5b58ff93-0czh 5.4.144+ #1 SMP Wed Nov 3 09:56:10 PDT 2021 x86_64 Intel(
R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux

• Install tools:
  • it is a GKE managed cluster
• Basic cluster related info:
  • kubectl version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.12-gke.1500", GitCommit:"d32c0db9a3ccd0ac73b0b3abd0532505217b376e", GitTreeState:"clean", BuildDate:"2021-11-17T09:30:02Z", GoVersion:"go1.15.15b5", Compiler:"gc", Platform:"linux/amd64"}

• kubectl get nodes -o wide: 14 node cluster, with multiple nodepools

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

What happened:

When trying out the latest unreleased version (#8201 or beyond) of the ingress-controller, i encountered an issue:

E0204 11:21:32.975934       8 socket.go:282] "Error fetching requests metric" err="inconsistent label cardinality: expected 8 label values but got 5 in prometheus.Labels{\"canary\":\"\", \"ingress\":\"ingress-for-foo-component\", 
  \"namespace\":\"default\", \"service\":\"foo-component-svc\", \"status\":\"302\"}" 

which prevented the collection of incoming nginx_ingress_controller_requests metrics.

What you expected to happen:

i expected to still collect metrics of nginx_ingress_controller_requests and i expected no error message like the one above.

How to reproduce it:

Build a docker image from #8201 and deploy it to the cluster.
Deploy prometheus according to the guide at: https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/#prometheus-and-grafana-installation

Anything else we need to know:

[k8s-ci-robot]

@ekovacs: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/triage-accepted