Request AWS ASGs in batches

[KierranM]

fixes #6045

Breaks up DescribeAutoScalingGroups API calls into batches of 50 as the input can't handle more than that.

Prior to this change running a command like kops delete ig <ig name> --yes -v 10 against a cluster with more than 50 tagged ASGs would result in:

I1108 08:15:11.520881   23423 loader.go:357] Config loaded from file /Users/kierran.mcpherson/.kube/config
Using cluster from kubectl context: my-cluster

I1108 08:15:11.520993   23423 factory.go:68] state store s3://my-bucket
I1108 08:15:12.865010   23423 s3context.go:198] Checking default bucket encryption "my-bucket"
I1108 08:15:12.865051   23423 s3context.go:203] Calling S3 GetBucketEncryption Bucket="my-bucket"
I1108 08:15:13.329914   23423 s3context.go:182] Found bucket "my-bucket" in region "ap-southeast-2" with default encryption set to true
I1108 08:15:13.329945   23423 s3fs.go:219] Reading file "s3://my-bucket/my-cluster/config"
I1108 08:15:13.497121   23423 s3fs.go:219] Reading file "s3://my-bucket/my-cluster/instancegroup/my-instancegroup"
I1108 08:15:13.668360   23423 aws_cloud.go:984] Querying EC2 for all valid zones in region "ap-southeast-2"
I1108 08:15:13.668565   23423 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
InstanceGroup "my-instancegroup" found for deletion
I1108 08:15:13.888626   23423 aws_cloud.go:424] Listing all Autoscaling groups matching cluster tags
I1108 08:15:13.890431   23423 request_logger.go:45] AWS request: autoscaling/DescribeTags
I1108 08:15:14.373755   23423 request_logger.go:45] AWS request: autoscaling/DescribeTags
I1108 08:15:14.454668   23423 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups

error finding CloudInstanceGroups: unable to find autoscale groups: error listing autoscaling groups: ValidationError: The number of group names that may be passed in is limited to 50
	status code: 400, request id: **********

With the changes made it chunks up the found ASGs into batches of 50 and calls it repeatedly:

I1108 08:15:24.925929   23459 loader.go:359] Config loaded from file /Users/kierran.mcpherson/.kube/config
Using cluster from kubectl context: my-cluster

I1108 08:15:24.926071   23459 factory.go:68] state store s3://my-bucket
I1108 08:15:26.450775   23459 s3context.go:194] found bucket in region "ap-southeast-2"
I1108 08:15:26.451023   23459 s3fs.go:220] Reading file "s3://my-bucket/my-cluster/config"
I1108 08:15:27.233866   23459 s3fs.go:220] Reading file "s3://my-bucket/my-cluster/instancegroup/my-instancegroup"
I1108 08:15:27.360543   23459 aws_cloud.go:1094] Querying EC2 for all valid zones in region "ap-southeast-2"
I1108 08:15:27.360724   23459 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
InstanceGroup "my-instancegroup" found for deletion
I1108 08:15:28.360377   23459 aws_cloud.go:461] Listing all Autoscaling groups matching cluster tags
I1108 08:15:28.360856   23459 request_logger.go:45] AWS request: autoscaling/DescribeTags
I1108 08:15:29.456171   23459 request_logger.go:45] AWS request: autoscaling/DescribeTags
I1108 08:15:29.578594   23459 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I1108 08:15:30.213484   23459 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I1108 08:15:30.426671   23459 delete.go:51] Deleting "my-instancegroup"
I1108 08:15:30.426696   23459 aws_cloud.go:354] Deleting autoscaling group "my-instancegroup.my-cluster"
I1108 08:15:30.426787   23459 request_logger.go:45] AWS request: autoscaling/DeleteAutoScalingGroup
I1108 08:15:30.573036   23459 aws_cloud.go:367] Deleting autoscaling launch configuration "my-instancegroup.my-cluster-20181009231549"
I1108 08:15:30.573175   23459 request_logger.go:45] AWS request: autoscaling/DeleteLaunchConfiguration
I1108 08:15:30.751226   23459 aws_cloud.go:377] deleted aws autoscaling group: "my-instancegroup.my-cluster"
I1108 08:15:30.751529   23459 s3fs.go:93] removing file s3://my-bucket/my-cluster/instancegroup/my-instancegroup

Deleted InstanceGroup: "my-instancegroup"

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

Hi @KierranM. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[KierranM]

I have signed the CLA now :)

[KierranM]

/check-cla

[mikesplain]

/check-cla

[mikesplain]

@KierranM Can you make sure you made the commit with the same email address that you used to sign the CLA? Luckily once you get this setup, you're good for all Kubernetes projects 😄 Thanks!

[KierranM]

/check-cla

[KierranM]

I've updated the email in my CLA profile, and I've confirmed I have my Github identity in the Social Networks part of the CLA site.

I originally joined up as an employee with my work email but that's not tied to my Github account

[chrisz100]

The check checks against the email that the git commit added as metadata - so what you configured with git config - not what’s here on github. Can you check this?

[KierranM]

/check-cla

[justinsb]

nit: might be good to pull out the 50 as a variable or a constant; it would make it easier to change in future and also let us put a comment like 'DescribeAutoScalingGroups is limited to 50 names at a time`. But just a nit :-)

I also thought there was a function to do this batching in the go stdlib, but I can't find it at the moment.

[justinsb]

That's a lot of ASGs :-) Thanks @KierranM

/approve
/lgtm
/ok-to-test

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb, KierranM

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment