Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add instance connection draining for NLBs #12966

Merged
merged 2 commits into from
Dec 21, 2021

Conversation

heybronson
Copy link
Contributor

This builds on #12902 to add graceful connection draining for associated targetGroups.

Today, one can define targetGroupArns in an instanceGroup spec to associate instances in a corresponding instanceGroup with. When instances in the instanceGroup are cycled via rolling-updates, those instances aren't first properly deregistered with the corresponding targetGroupArns, causing dropped connections.

This PR fixes this issue by ensuring that we're properly draining instances from their corresponding classic load balancers and NLB targetGroups before terminating the instance. This is a big improvement to the overall client experience when running rolling-updates on production clusters.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 14, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @heybronson. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 14, 2021
@k8s-ci-robot k8s-ci-robot added the area/provider/aws Issues or PRs related to aws provider label Dec 14, 2021
@heybronson
Copy link
Contributor Author

cc @olemarkus @johngmyers

@johngmyers
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 15, 2021
@rifelpet
Copy link
Member

For reference, this deregistration occurs after the node has been drained of its non-daemonset pods but kube-proxy is still running, so this specifically helps services exposed via kube-proxy.

This doesn't help target groups targeting pod IPs (which can be gracefully deregistered via finalizers and the aws-load-balancer-controller during the drain process) nor does it help targeting pods listening on the hosts' nodePorts directly (since they will have been drained already and their connections disconnected)

@heybronson
Copy link
Contributor Author

heybronson commented Dec 15, 2021

For reference, this deregistration occurs after the node has been drained of its non-daemonset pods

@rifelpet That's correct, this deregistration happens after node draining, so certain service configurations are already covered. However, this helps in scenarios where traffic is routed to daemonset pods running on a node, since those aren't drained. For example, Nginx-ingress in daemonset mode benefits from this change.

@heybronson
Copy link
Contributor Author

/assign @johngmyers

@heybronson
Copy link
Contributor Author

/assign @olemarkus

@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. do-not-merge/contains-merge-commits and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 19, 2021
@heybronson
Copy link
Contributor Author

friendly bump

Copy link
Member

@olemarkus olemarkus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 20, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 21, 2021
@k8s-ci-robot k8s-ci-robot merged commit 0441f2f into kubernetes:master Dec 21, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Dec 21, 2021
@rifelpet rifelpet modified the milestones: v1.23, v1.24 Dec 25, 2021
@heybronson heybronson deleted the drain-nlb branch January 3, 2022 17:11
@heybronson
Copy link
Contributor Author

@rifelpet can we potentially keep this in the 1.23 release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/provider/aws Issues or PRs related to aws provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants