[EKS] [request]: Support ServiceNodeExclusion feature gate

[stevehipwell]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
It would be great if EKS could support the ServiceNodeExclusion feature gate despite it currently (since v1.8) being in alpha.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We would like to be able to use an internal NLB in our cluster and have it only target our ingress nodes. The primary issue with the current setup is the NLB hairpinning issue (potentially an old AMI). We get this even when the communicating containers are running on different nodes. If the ServiceNodeExclusion was supported we could use the alpha.service-controller.kubernetes.io/exclude-balancer annotation to only allow the NLB to target our ingress nodes.

Are you currently working around this issue?
We're currently using an ELB for load balancers that need to be addressable from within the cluster.

Additional context
N/A

Attachments
N/A

[meringu]

👍 We're after this to avoid having traffic transit our Windows nodes on the way into the cluster.

What I'd really love is container native load-balancing, where the PodIPs get registered directly in the target group.

@stevehipwell does disabling external SNAT fix the NLB hairpinning issue? We are about to test it out ourselves: https://docs.aws.amazon.com/eks/latest/userguide/external-snat.html

[stevehipwell]

@meringu it didn't appear to when we went through this Autumn 2019 but that doesn't mean it won't now. I'd be interested in your findings?

[meringu]

@stevehipwell, we've just managed to achieve this by using a separate security group for the nodes we want in the ELB.

1. Create your own security group for the ELB with no egress rules
2. Annotate the Kubernetes service so the controller manager assigns the security group to the ELB
3. Duplicate your node security group (we called ours "other") and omit the kubernetes.io/cluster/<cluster-name> tag. This stops the controller manager adding new rules to this group, or an egress rule to this group from the ELB group
4. Assign the original node security group to the ingress nodes and the other security group to the rest of your nodes. The controller-manager should only add security group rules to the ingress node security group

All nodes will still be registered in the ELB, however this ensures the healthcheck fails for the non-ingress nodes, never sending traffic there (as long as at least one ingress node is healthy)

[meringu]

@stevehipwell, I've checked the order, looks like the service controller adds the security group to the ELB before adding egress rules to the ELB security group. As such I'd recommend that in step 1. the ELB security group is created with egress to the ingress nodes security group. Otherwise there will be small downtime.

[stevehipwell]

Thanks for this @meringu! We had wondered if there was a way to use SGs to achieve this and it's great to hear that there is a way. Could you confirm which EKS version you were using?

[meringu]

1.14, but it looks like the behaviour is the same in the master branch of Kubernetes at the moment. I think it is unlikely to change.

[tabern]

Per K8s docs this is beta in Kubernetes version 1.19. Solved by #1053

Closing this issue as EKS does not support alpha features/feature gates.

[M00nF1sh]

This can be better addressed via kubernetes/kubernetes#90943 as well