wait for instances to drain from classic LB

[heybronson]

Now that we've enabled connection draining by default for all classic LBs (#12881), we need to ensure that we're waiting for connectionDraining to complete before terminating the instance.

Deleting the instance without waiting doesn't allow for connections to drain. We need to call DeregisterInstancesFromLoadBalancer which will ensure that there aren't any ongoing connections to the instance before we terminate.

Gathering the corresponding loadBalancers by describing the underlying ASG.

Confirmed that rolling-updates continue to behave as expected and 0% of requests are dropped from corresponding load balancers.

[k8s-ci-robot]

Hi @heybronson. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[johngmyers]

/ok-to-test

[rifelpet]

Ideally we could cache the DescribeAutoScalingGroups response since it isn't likely to change between deleteInstance calls on instances from the same InstanceGroup, but the frequency that the Describe call is being made (once every few minutes) shouldn't have a material impact on AWS API call volume.

[johngmyers]

So this is only load balancers that are associated with the ASG, so probably only the ones directly created by kOps. Not the ones created by KCM/LBC. Unfortunate there doesn't seem to be a call to enumerate the load balancers that an instance is registered with.

And this only helps with rolling update, not cluster-autoscaler scale-down. I'm still hoping for someone to implement graceful node shutdown.

[heybronson]

/assign @johngmyers

[heybronson]

friendly bump

[heybronson]

/assign @olemarkus

[olemarkus]

/lgtm

Will approve unless someone objects in the next day or so.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johngmyers, olemarkus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johngmyers,olemarkus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[heybronson]

/test pull-kops-verify-golangci-lint