Performance Issue: Ingress creation in a cluster with many existing ingresses get slow admission webhook response

[bysph]

What happened:
When creating a new ingress in a cluster with over 1000 ingresses, the operation takes more than 3 seconds.

Even worse, if I create ingresses concurrently (150+), the response time of the ingress controller will increase to 10+ seconds, which will cause the API server time out.

In real-world scenarios, we will concurrently submit a large number of Spark tasks(Because they are scheduled to run at specific times), this issue can result in certain task submissions failing.

What you expected to happen:
I believe that processing time should ideally be within a few hundred milliseconds.

I have reviewed the admission webhook's code and noticed that it validates not only the new ingress but also all existing ingresses, potentially causing longer processing times.

The PR for the admission webhook (#3802) indeed confirms this intentional design. I'd like to understand the reasoning behind this decision and whether you have considered change to the another option mentioned by the original author.

Another option to get better performances could be to drop the test with all pre-existing ingresses, which could probably be good enough to detect most configuration errors (syntax errors mainly).

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
v0.43.0

Kubernetes version (use kubectl version):
v1.19.3

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bysph]

@tjamet Hello, could you please help me take a look at this issue?

[longwuyuan]

what is the output of kubectl -n ingess-nginx describe po | grep -i image:

[bysph]

kubectl -n ingess-nginx describe po | grep -i image:

{our-internal-harbor-address}/library/ingress-nginx/controller:v0.43.0@sha256:2b29d459bb978bc773fcfc824a837bb52d6223aba342411d22d61522e58f811b

[longwuyuan]

Can you install a release from this project

…

On Wed, 13 Mar, 2024, 5:51 pm bysph, ***@***.***> wrote: kubectl -n ingess-nginx describe po | grep -i image: ***@***.*** :2b29d459bb978bc773fcfc824a837bb52d6223aba342411d22d61522e58f811b — Reply to this email directly, view it on GitHub <#11115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWTBADBKD6YCYMV6I3TYYBAETAVCNFSM6AAAAABET33BEGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJUGI3DCNZSGY> . You are receiving this because you commented.Message ID: ***@***.***>

[longwuyuan]

/assign

[Gacko]

Ingress NGINX is always doing a full test of all Ingresses, so compiling all of them together and checking if the resulting nginx.conf is valid. You can disable this behavior via the --disable-full-test flag: https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/

[strongjz]

controller:v0.43.0 is an extremely old version, please upgrade.

There are several things we are trying to do to help with this issue. It is a known issue

#11046

#10884

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[longwuyuan]

Please re-open the issue when ready to re-engage and move forward on this. Thanls. For now we are closing the issue because there is no traction for long time and open issues are piling up

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

Please re-open the issue when ready to re-engage and move forward on this. Thanls. For now we are closing the issue because there is no traction for long time and open issues are piling up

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.