Allow NodeGroup/ASG targeting for ?LB instances

[bassco]

What would you like to be added:

Our cluster has been created with kops.

We operate multiple instance groups in our cluster to partition the different workloads we run.

I would like an annotation on the aws-load-balancer for targeted instances that will be assigned to the ELB/ALB/NLB TargetGroup using a tag.

E.g.:

service.beta.kubernetes.io/aws-load-balancer-node-groups: "asg-node-group-comma-separated-list"

Where node-group-comma-separated-list is a list of the ASG groupName tags to filter the host instance ids that will be added to the Load Balancer.

E.g. In our environment our 4 node groups have a tag: aws:autoscaling:groupName set to

• nodes
• ml-cpu
• ml-gpu
• search

Using the above list of instance group ASG names; to target Pods of the ml-* ASGs I would use the following annotation.

service.beta.kubernetes.io/aws-load-balancer-node-groups: "ml-cpu,ml-gpu"

Currently, I suspect that the tag k8s.io/role/node with a value of 1 is used to populate the instances on the LB.

Why is this needed:

When creating an ELB/ALB or NLB - the complete node instance list associated with the cluster is assigned to the TargetGroup. It is really inefficient to perform health checks against instances that will never host a Pod of the service type you are creating the LB for.

/kind feature

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[bassco]

Anyone able to read through this and provide feedback?

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ari-becker]

/reopen

Not sure why this was allowed to close - this is a must-have feature for large clusters where the number of nodes in the cluster reaches the quota for the number of targets permitted per load balancer. Scaling further requires limiting the load balancer's targets to a specific subset of servers, and configuring the deployment/statefulset to only schedule on those nodes. Currently the only workaround is to create the loadbalancer by hand outside of Kubernetes.

[k8s-ci-robot]

@ari-becker: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Not sure why this was allowed to close - this is a must-have feature for large clusters where the number of nodes in the cluster reaches the quota for the number of targets permitted per load balancer. Scaling further requires limiting the load balancer's targets to a specific subset of servers, and configuring the deployment/statefulset to only schedule on those nodes. Currently the only workaround is to create the loadbalancer by hand outside of Kubernetes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ari-becker]

@bassco as the author, do you mind re-opening?

[leakingtapan]

/reopen

[k8s-ci-robot]

@leakingtapan: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ari-becker]

An additional gotcha that popped up for me:

Let's say that you have 450 nodes and a service that you'd like to expose with an external load balancer.
If you expose a single port on the service - a load balancer is created that targets all 450 servers, and it works.
If you expose multiple additional ports - despite the number of servers not changing, each additional port is considered a whole new target. So if you expose three ports you now have 3 * 450 = 1350 targets, which is above the limit, and AWS will simply refuse to add the listeners for the new ports, complaining about TooManyTargets.

[foobarfran]

This would be super useful.
I can contribute with a pull request for this feature if it helps

[bassco]

That would be fantastic if you could, @foobarfran

[leakingtapan]

/remove-lifecycle rotten

[foobarfran]

/assign @foobarfran

[foobarfran]

The feature for this issue is already merged in kubernetes/kubernetes#90943

/close

[k8s-ci-robot]

@foobarfran: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

The feature for this issue is already merged in kubernetes/kubernetes#90943

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[leakingtapan]

/close as the PR is merged in legacy provider

[bassco]

@foobarfran - legend!
Feature will be released in v1.19, for those that reach this comment and don't follow the MR