Support AWS Network Load Balancer

[tamalsaha]

Can Kubernetes support the new AWS NLB for LoadBalancer type Service? https://aws.amazon.com/elasticloadbalancing/details/ . I am assuming that users can apply some annotation on the Service that will tell aws cloud provider to use NLB instead of classic ELB to expose Kubernetes service.

Feature Description

• One-line feature description (can be used as a release note): Use AWS Network Load Balancer for LoadBalancer type Service for aws cloud provider
• Primary contact (assignee): @justinsb
• Responsible SIGs: @kubernetes/sig-aws-misc
• Design proposal link (community repo):
• KEP: 20190501-aws-nlb-beta.md
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):

  • Alpha release target (1.9)

  • Beta release target (1.15)

    • [tests] Create an e2e integration test
    • [bug fix] LoadBalancerSourceRanges not working (Defining a custom loadBalancerSourceRanges in a AWS NLB service is not respected kubernetes#57212)
    • [bug fix] Failure to update SG's after an update (Failure to update security groups for NLBs after rolling update kubernetes#60825)
    • [bug fix] SG rules are removed erroneously (AWS Security Group rules are removed when adding/removing worker nodes kubernetes#64148)
    • [feature] Support IP address targeting (Enable Network Load Balancers to register nodes by IP address kubernetes#57249)
    • [feature] Cross-zone load balancing (AWS NLB: Support cross-zone load balancing annotation kubernetes#61064)

  • Stable release target (1.xy)
    v1: targeting Kubenetes v1.13 (estimated release in December)

    • [edge case / bug fix] externalTrafficPolicy issues with alternate DHCP ("externalTrafficPolicy": "Local" on AWS does not work if the dhcp of the vpc is not set exactly to <region>.compute.internal kubernetes#61486)
    • [edge case / bug fix] Incorrect behavior with older instances (nlb on aws should filter instance types for target groups kubernetes#66044)
    • [feature] Support EIP (Support EIP Allocations with AWS NLB kubernetes#63959)
    • [feature] PROXY protocol support (Add PROXY protocol support for AWS Network Load Balancers kubernetes#57250)
    • [stretch feature] More configurability to health checks
    • [stretch feature] Better tagging support

[cmluciano]

@kubernetes/sig-aws-misc

[jrideout]

see kubernetes-retired/kube-aws#945

[idvoretskyi]

@tamalsaha

Design proposal link (community repo):

Is the design proposal publicly available?

[tamalsaha]

@idvoretskyi , I don't have a design proposal.

[idvoretskyi]

@tamalsaha it is required.

cc @kubernetes/sig-aws-misc

[erickt]

FYI, @micahhausler has a PR to implement support NLBs as a service load balancer: kubernetes/kubernetes#53400

[micahhausler]

In addition to the cloudprovider portion to create the NLB/security rules, there will need to be additional work on the kube-proxy to correctly get the packet to the individual pod with the remote IP intact.

Also, there is an issue tracking NLB support on kubernetes/kubernetes#52173

[justinsb]

Great progress - thanks @micahhausler! For IP preservation, AIUI NLB doesn't support proxy protocol, which was the go-to trick previously.

The good news is that this is much more similar to GCE's load balancer, as I understand it. So we should be able to apply the same approach as is used on GCE. @thockin any pointers for us?

[micahhausler]

Two additional features that could be later added for NLB are:

• Supporting the Elastic IP addresses for NLB
• Supporting advanced health checks for NLB

[Mikulas]

For IP preservation, AIUI NLB doesn't support proxy protocol, which was the go-to trick previously.

It seems NLB does not always pass the original address though:

You can configure a target group so that you register targets by instance ID or IP address. If you specify targets using an instance ID, the source IP addresses of the clients are preserved and provided to your applications. If you specify targets by IP address, the source IP addresses are the private IP addresses of the load balancer nodes.
http://docs.aws.amazon.com/elasticloadbalancing/latest/network/elb-ng.pdf page 6

Will NLB support in k8s be limited to instances that can be registered by instance ID? Those seem to be types that support IPv6, explicitly excluding m3

You cannot register instances by instance ID if they have the following instance types: C1, CC1, CC2, CG1, CG2, CR1, G1, G2, HI1, HS1, M1, M2, M3, and T1. You can register instances of these types by IP address.
http://docs.aws.amazon.com/elasticloadbalancing/latest/network/elb-ng.pdf page 25

[micahhausler]

I'll be giving a demo of what I've currently got working with NLB at the sig-aws meeting later today, I'll write up more after the meeting.

[aledbf]

@micahhausler it will be recorded?

[micahhausler]

Yes, they do record them, but I don't know where links to recordings are. @justinsb probably knows.

[idvoretskyi]

@justinsb @kubernetes/sig-aws-misc can you please add the design proposal link to the feature description?

[zacharysarah]

@tamalsaha 👋 Please indicate in the 1.9 feature tracking board
whether this feature needs documentation. If yes, please open a PR and add a link to the tracking spreadsheet. Thanks in advance!

[micahhausler]

Already have a doc PR ready for merge kubernetes/website#6260

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[lalomartins]

I'm new here, but I don't think this is stale. Implementation is in progress, in alpha cycle, I'm using it, and it mostly works. Maybe issues with a stage/* label should be immune to stale?

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[micahhausler]

/remove-lifecycle rotten

[msedzins]

Hey @justinsb , Enhancement shadow for the v1.19 release cycle here. Just following up on my earlier update to inform you of the
upcoming Enhancement Freeze scheduled on Tuesday, May 19.

Regards,
Mirek

[palnabarun]

@justinsb -- Unfortunately the deadline for the 1.19 Enhancement freeze has passed. For now, this is being removed from the milestone and 1.19 tracking sheet. If there is a need to get this in, please file an enhancement exception.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[frittentheke]

/remove-lifecycle stale

[olemarkus]

What is missing to bring to this to GA?

[kikisdeliveryservice]

Hi @justinsb

Enhancements Lead here. Any plans for this to graduate in 1.20?

Thanks,
Kirsten

[kikisdeliveryservice]

Hi @M00nF1sh @justinsb @micahhausler ,

Any updates on this, as a reminder Enhancements Freeze is October 6th and we expect that:

• The KEP must be merged in an implementable state
• The KEP must have test plans
• The KEP must have graduation criteria.

Also the link to your KEP in the description seems to 404?

Thanks
Kirsten

[nckturner]

Since the legacy-cloud-provider won't be merging feature PRs after 1.20, and it looks like the original KEP has been deleted, it sounds like this PR should be closed and we should be tracking cloud provider specific enhancements elsewhere?

For the out-of-tree NLB integration, we will be eventually moving it to the aws-load-balancer-controller, so if we keep this KEP we should update it to reflect that change.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[frittentheke]

/remove-lifecycle stale

[ehashman]

Since the legacy-cloud-provider won't be merging feature PRs after 1.20, and it looks like the original KEP has been deleted,

It's here: https://github.com/kubernetes/enhancements/tree/master/keps/provider-aws/423-network-load-balancer

[thockin]

Is there any reason this should not be closed ?

[nckturner]

I’d vote for tracking it in the cloud-provider-aws repository.

…

On Sat, May 22, 2021 at 12:24 PM Tim Hockin ***@***.***> wrote: Is there any reason this should not be closed ? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#423 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJGJEMWLZGHNHEJBJYWA7TTPAAHBANCNFSM4D2HD6TA> .

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.