nginx_ingress_controller_request_duration_seconds_count associates request with wrong ingress

[jeanbaptiste-brasselet]

What happened:

I have three ingress using the same host with three path in one namespace like this:

HOST PATH NAMESPACE SERVICE PORT INGRESS
example.host.com /public/v1/test1 test service1 80 public-service1
example.host.com /public/v1/test2 test service2 80 public-service2
example.host.com /public/v1/test3 test service3 80 public-service3

For some reason when I look into nginx_ingress_controller_request_duration_seconds_count I see lines like this one :

nginx_ingress_controller_request_duration_seconds_count{container="controller", controller_class="k8s.io/ingress-public", controller_namespace="ingress-controller", controller_pod="ingress-public-ingress-nginx-controller-d5c797b8b-krnst", endpoint="metrics", exported_namespace="test", host="example.host.com, ingress="public-service3", instance="10.56.2.4:10254", job="ingress-public-ingress-nginx-controller-metrics", method="GET", namespace="ingress-controller", path="/", pod="ingress-public-ingress-nginx-controller-d5c797b8b-krnst", service="ingress-public-ingress-nginx-controller-metrics", status="404"}

As you can see it associates a request done on / with public-service3 ingress so when I am trying to do some metrics I got a lot of 404 error reported for this specific ingress when I have none in reality.

I don't really understand why this association exist.

What you expected to happen:

I would have expected the ingress to be null or undefined since this path is not defined anywhere.

[k8s-ci-robot]

@jeanbaptiste-brasselet: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

There id a metricsbyhost flag somewhere. Please search in docs on the userguide section, monitoring page. Will try to send link later. Thanks, ; Long

…

On Thu, 20 Oct, 2022, 8:34 PM Kubernetes Prow Robot, < ***@***.***> wrote: @jeanbaptiste-brasselet <https://github.com/jeanbaptiste-brasselet>: This issue is currently awaiting triage. If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance. The triage/accepted label can be added by org members by writing /triage accepted in a comment. Instructions for interacting with me using PR comments are available here <https://git.k8s.io/community/contributors/guide/pull-requests.md>. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra <https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:> repository. — Reply to this email directly, view it on GitHub <#9194 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWV3GFX6V4ZIKCIUQETWEFNPLANCNFSM6AAAAAARKIASAQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[longwuyuan]

/remove-kind bug

You did not say you have a wildcard host but i wonder how this flag https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/#wildcard-ingresses impacts your use case of no /

[jeanbaptiste-brasselet]

I think you misunderstand my issue (Indeed I am not using any wildward). I have metrics for the path / on my hostname. The problem is for some reason it says it belongs to the ingress public-service3 when the only path declared for this ingress is /public/v1/test3.

As a matter of fact I have no ingress that matches the / path that's why it always return 404 and I would have expected to see ingress: '' in the metric not ingress: 'public-service3'

Another information after more testing if I do requests on path that are not in my ingresses like /toto /tata in the end all those thing are added to the path / in the metrics with ingress: 'public-service3' as a value.

Any idea ?

[jeanbaptiste-brasselet]

I think I get what is happening, it seems there is a default backend at the last resort to all the ingress you create and this default backend send back 404.

If I am right I wonder what I can do to have those error not associated with the ingress and it's default backend. Do I need to create a backend for this myself with an ingress ?

[bumarcell]

Create a backend with an svc and configure nginx to use it as a default-backend