Annnotation updates on LoadBalancer service are not propagated to corresponding AWS LoadBalancer listener

[yogeek]

What happened?

I have a type: LoadBalancer service with the following annotations

serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
          service.beta.kubernetes.io/aws-load-balancer-internal: "true"
          service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "Name=k8s-istio-nlb-internal,Project=Test,Application=K8S"
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "${CERT_ARN}"
          service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "${SSL_NEGO_POLICY}"
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "${SSL_PORT}"
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
          service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "${BACKEND_PROTO}"

I want to set the ALPN policy, so I add the following annotation :

          service.beta.kubernetes.io/aws-load-balancer-alpn-policy: "HTTP2Preferred"

But no change is propagated to the AWS LoadBalancer listener

(same goes with a tag modification)

What did you expect to happen?

Updating service annotation should update the AWS LoadBalancer

How can we reproduce it (as minimally and precisely as possible)?

Add or update an annotation on a K8S LoadBalancer Service and check in AWS that the change is not applied.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.15", GitCommit:"1d79bc3bcccfba7466c44cc2055d6e7442e140ea", GitTreeState:"clean", BuildDate:"2022-09-21T12:18:10Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.15", GitCommit:"1d79bc3bcccfba7466c44cc2055d6e7442e140ea", GitTreeState:"clean", BuildDate:"2022-09-21T12:12:26Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

AWS

OS version

No response

Install tools

kubeadm on AWS EC2

Container runtime (CRI) and version (if applicable)

No response

Related plugins (CNI, CSI, ...) and versions (if applicable)

No response

[k8s-ci-robot]

@yogeek: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yogeek]

/sig cloud-provider

[zemliany]

@yogeek my apologies for bothering. that's not clear what are you using as a controller and how you exactly run k8s, it's vanilla k8s on EC2 or it's EKS. Why I'm asking? The matter is, we faced with the similar issues on our project with NLB and ALPN policies support of that has been introduced in 2020 https://aws.amazon.com/about-aws/whats-new/2020/05/network-load-balancer-now-supports-tls-alpn-policies/ and perhaps our little findings will be helpful to you somehow.

In case if you using latest version aws-load-balancer-controller https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/deploy/installation/ in your cluster, I think this annotation should be supported if believe to documentation https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/service/annotations/#alpn-policy

If you don't use aws-load-balancer-controller in your cluster, then you probably use legacy-clould-providers for provision services
https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/legacy-cloud-providers#things-you-should-not-do
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/legacy-cloud-providers/aws/aws.go

So annotation what you mentioned - it doesn't exists for legacy-cloud-providers now. There were Pull Requests
#92653
#96861

to add support of ALPN annotation for NLB in legacy providers, but in accordance to statements that new features not allowed, only bugfixes

Things you should NOT do
Add new features/integrations to a cloud provider in this repo. Changes sync here should only be incremental bug fixes.

https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/legacy-cloud-providers#things-you-should-not-do

Seems, that add support of that feature not possible, but not sure at 100%

[sftim]

I agree with @Zemlyanoy - Kubernetes is behaving as (poorly) documented; if you want to use the unofficial / deprecated service.beta.kubernetes.io/aws-load-balancer-alpn-policy annotation then you must use https://kubernetes-sigs.github.io/aws-load-balancer-controller/

PS the annotation is either unofficial or it's deprecated, but I'm not sure which,

[yogeek]

@Zemlyanoy as mentioned in my issue description, my cluster is deployed with kubeadm on AWS EC2 instances (so no EKS)

Thank you for the details, and indeed we had to switch to the AWS Load Balancer Controller to be able to configure the ALPN with annotation.

The other issue I mentioned is that with the legacy controller, after the creation of a LoadBalancer, any modification to an annotation (modifying a tag for example) is not propagated to the cloud LoadBalancer..
But I guess I will open a bug for this behavior.

As you said the documentation is not very clear on which LoadBalancer controller to use between :

• legacy controller in tree
• legacy controller out of tree
• AWS LoadBalancer controller (https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/)

Hopefully, this article helped to clarify the current situation https://baptistout.net/posts/two-kubernetes-controllers-for-managing-aws-nlb/

Thanks for your help @Zemlyanoy @sftim