Add annotation for per-Ingress upstream-keepalive-connections

[robertxiaoxiao]

What do you want to happen?

I'm hoping for an annotation variant of "upstream-keepalive-connections" per ingress. we want to customized different upstream section configs like native Nginx config when using ingress controller in AKS. Now upstream section is integrated as one interface as below.

What happened

Now we cannot customize upstream section per ingress. My team hit a 421 issue when we want to separate requests in multi ingress to different upstream with different keep_alive configs. One is enabled keepalive and other one is disabled for some reasons.

What you expected to happen

we have found another closed but not resolved issue several years ago. #2441

Anything else we need to know

we tried several method to fix upstream section config per ingress but all failed. now we use native Openresty to implement proxy logics. this would be a nice thing to make available in ingress nginx controller.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[robertxiaoxiao]

I am trying to use lua scripts to workaround in my issue as docs below. will update if that workaround works.
https://github.com/openresty/lua-nginx-module?tab=readme-ov-file#tcpsocksetkeepalive

[Gacko]

I'm not sure if this is even possible since NGINX does not provide that on its own:

https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

This can only be configured inside the context of an upstream.