Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch up requests with more than 10 target group changes #10435

Merged
merged 3 commits into from
Oct 10, 2019

Conversation

beezly
Copy link
Contributor

@beezly beezly commented Oct 9, 2019

The AWS API only supports adding/removing 10 requests at a time.

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Closes #256

Release note for CHANGELOG:

Batch up requests for Attach/DetachLoadBalancerTargetGroups into groups of 10

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccXXX'

...

@beezly beezly requested a review from a team October 9, 2019 12:57
@ghost ghost added size/S Managed by automation to categorize the size of a PR. service/autoscaling Issues and PRs that pertain to the autoscaling service. labels Oct 9, 2019
@bflad bflad added the bug Addresses a defect in current functionality. label Oct 9, 2019
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @beezly 👋 Thanks so much for submitting this fix! This appears to be on the right track, but it requires a covering acceptance test (example described below) which uncovered that we need to wait until each batch is fully applied, otherwise it will still return the same error.

Once the acceptance testing and batch waiting is added, we should be able to get this in. Thanks again.

Failing test prior to code fix:

--- FAIL: TestAccAWSAutoScalingGroup_TargetGroupArns (81.59s)
    testing.go:569: Step 2 error: errors during apply:

        Error: Error updating Load Balancers Target Groups for AutoScaling Group (tf-asg-2019101000215510330000000d), error: ValidationError: Trying to update too many Load Balancers/Target Groups at once. The limit is 10

Associated testing code:

// Reference: https://github.com/terraform-providers/terraform-provider-aws/issues/256
func TestAccAWSAutoScalingGroup_TargetGroupArns(t *testing.T) {
	var group autoscaling.Group
	resourceName := "aws_autoscaling_group.test"
	rName := acctest.RandomWithPrefix("tf-acc-test")

	resource.ParallelTest(t, resource.TestCase{
		PreCheck:     func() { testAccPreCheck(t) },
		Providers:    testAccProviders,
		CheckDestroy: testAccCheckAWSAutoScalingGroupDestroy,
		Steps: []resource.TestStep{
			{
				Config: testAccAWSAutoScalingGroupConfig_TargetGroupArns(rName, 11),
				Check: resource.ComposeTestCheckFunc(
					testAccCheckAWSAutoScalingGroupExists(resourceName, &group),
					resource.TestCheckResourceAttr(resourceName, "target_group_arns.#", "11"),
				),
			},
			{
				ResourceName:      resourceName,
				ImportState:       true,
				ImportStateVerify: true,
				ImportStateVerifyIgnore: []string{
					"force_delete",
					"initial_lifecycle_hook",
					"name_prefix",
					"tag",
					"tags",
					"wait_for_capacity_timeout",
					"wait_for_elb_capacity",
				},
			},
			{
				Config: testAccAWSAutoScalingGroupConfig_TargetGroupArns(rName, 0),
				Check: resource.ComposeTestCheckFunc(
					testAccCheckAWSAutoScalingGroupExists(resourceName, &group),
					resource.TestCheckResourceAttr(resourceName, "target_group_arns.#", "0"),
				),
			},
			{
				Config: testAccAWSAutoScalingGroupConfig_TargetGroupArns(rName, 11),
				Check: resource.ComposeTestCheckFunc(
					testAccCheckAWSAutoScalingGroupExists(resourceName, &group),
					resource.TestCheckResourceAttr(resourceName, "target_group_arns.#", "11"),
				),
			},
		},
	})
}

func testAccAWSAutoScalingGroupConfig_TargetGroupArns(rName string, tgCount int) string {
	return fmt.Sprintf(`
data "aws_ami" "test" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn-ami-hvm-*-x86_64-gp2"]
  }
}

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_launch_template" "test" {
  image_id      = data.aws_ami.test.id
  instance_type = "t3.micro"
  name          = %[1]q
}

resource "aws_vpc" "test" {
  cidr_block = "10.0.0.0/16"

  tags = {
    Name = %[1]q
  }
}

resource "aws_subnet" "test" {
  availability_zone = data.aws_availability_zones.available.names[0]
  cidr_block        = "10.0.0.0/24"
  vpc_id            = aws_vpc.test.id

  tags = {
    Name = %[1]q
  }
}

resource "aws_lb_target_group" "test" {
  count = %[2]d

  port     = 80
  protocol = "HTTP"
  vpc_id   = "${aws_vpc.test.id}"
}

resource "aws_autoscaling_group" "test" {
  force_delete        = true
  max_size            = 0
  min_size            = 0
  target_group_arns   = length(aws_lb_target_group.test) > 0 ? aws_lb_target_group.test[*].arn : []
  vpc_zone_identifier = [aws_subnet.test.id]

  launch_template {
    id = aws_launch_template.test.id
  }
}
`, rName, tgCount)
}

New test passes after adding the batch waiting:

--- PASS: TestAccAWSAutoScalingGroup_TargetGroupArns (187.93s)

aws/resource_aws_autoscaling_group.go Show resolved Hide resolved
aws/resource_aws_autoscaling_group.go Show resolved Hide resolved
@bflad bflad self-assigned this Oct 10, 2019
bflad added a commit that referenced this pull request Oct 10, 2019
… by 10 to prevent API and rate limiting errors

Reference: #256
Reference: #10435
Reference: https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_AttachLoadBalancers.html
Reference: https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_DetachLoadBalancers.html

This change is complementary to the target group batching. The AutoScaling API for `AttachLoadBalancers` and `DetachLoadBalancers` only allow 10 elements at a time. In addition to the batching to split the API requests, we must wait for the batch to fully complete before moving onto the next batch, otherwise the API returns a rate limiting error:

```
--- FAIL: TestAccAWSAutoScalingGroup_LoadBalancers (360.22s)
    testing.go:569: Step 2 error: errors during apply:

        Error: Error updating Load Balancers for AutoScaling Group (tf-asg-2019101000090127270000000d), error: ValidationError: Trying to update too many Load Balancers/Target Groups at once. The limit is 10
```

Output from acceptance testing:

```
--- PASS: TestAccAWSAutoScalingGroup_LoadBalancers (443.84s)
```
@ghost ghost added size/L Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. and removed size/S Managed by automation to categorize the size of a PR. labels Oct 10, 2019
Co-Authored-By: Brian Flad <bflad417@gmail.com>
@beezly
Copy link
Contributor Author

beezly commented Oct 10, 2019

@bflad thanks for the review! I've added in the new test you suggested and put the waitFor functions in. The new acceptance test now passes...

$ AWS_PROFILE=ac make testacc TEST=./aws TESTARGS='-run=TestAccAWSAutoScalingGroup_TargetGroupArns' ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSAutoScalingGroup_TargetGroupArns -timeout 120m go: finding github.com/terraform-providers/terraform-provider-tls v2.1.1+incompatible go: finding github.com/terraform-providers/terraform-provider-tls v2.1.1+incompatible === RUN TestAccAWSAutoScalingGroup_TargetGroupArns === PAUSE TestAccAWSAutoScalingGroup_TargetGroupArns === CONT TestAccAWSAutoScalingGroup_TargetGroupArns --- PASS: TestAccAWSAutoScalingGroup_TargetGroupArns (187.36s) PASS ok github.com/terraform-providers/terraform-provider-aws/aws 187.431s

@bflad bflad added this to the v2.32.0 milestone Oct 10, 2019
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, @beezly, thanks for the update 🚀

--- PASS: TestAccAWSAutoScalingGroup_serviceLinkedRoleARN (47.18s)
--- PASS: TestAccAWSAutoScalingGroup_namePrefix (55.63s)
--- PASS: TestAccAWSAutoScalingGroup_autoGeneratedName (81.21s)
--- PASS: TestAccAWSAutoScalingGroup_withMetrics (81.10s)
--- PASS: TestAccAWSAutoScalingGroup_VpcUpdates (99.23s)
--- PASS: TestAccAWSAutoScalingGroup_terminationPolicies (105.44s)
--- PASS: TestAccAWSAutoScalingGroup_classicVpcZoneIdentifier (108.41s)
--- PASS: TestAccAWSAutoScalingGroup_emptyAvailabilityZones (110.63s)
--- PASS: TestAccAWSAutoScalingGroup_LaunchTemplate_IAMInstanceProfile (71.64s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_InstancesDistribution_OnDemandAllocationStrategy (49.02s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy (49.92s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_InstancesDistribution_SpotInstancePools (41.75s)
--- PASS: TestAccAWSAutoScalingGroup_withPlacementGroup (181.76s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_LaunchTemplate_LaunchTemplateSpecification_LaunchTemplateName (52.19s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_InstancesDistribution_OnDemandPercentageAboveBaseCapacity (85.01s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_LaunchTemplate_Override_InstanceType (38.75s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_InstancesDistribution_OnDemandBaseCapacity (100.72s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_InstancesDistribution_SpotMaxPrice (77.26s)
--- PASS: TestAccAWSAutoScalingGroup_launchTemplate_update (165.60s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_InstancesDistribution_SpotAllocationStrategy (111.32s)
--- PASS: TestAccAWSAutoScalingGroup_suspendingProcesses (220.69s)
--- PASS: TestAccAWSAutoScalingGroup_initialLifecycleHook (225.24s)
--- PASS: TestAccAWSAutoScalingGroup_TargetGroupArns (233.59s)
--- PASS: TestAccAWSAutoScalingGroup_MixedInstancesPolicy_LaunchTemplate_LaunchTemplateSpecification_Version (104.85s)
--- PASS: TestAccAWSAutoScalingGroup_launchTemplate (256.90s)
--- PASS: TestAccAWSAutoScalingGroup_ALB_TargetGroups (257.96s)
--- PASS: TestAccAWSAutoScalingGroup_basic (267.00s)
--- PASS: TestAccAWSAutoScalingGroup_enablingMetrics (291.22s)
--- PASS: TestAccAWSAutoScalingGroup_ALB_TargetGroups_ELBCapacity (317.17s)
--- PASS: TestAccAWSAutoScalingGroup_WithLoadBalancer_ToTargetGroup (330.08s)
--- PASS: TestAccAWSAutoScalingGroup_tags (331.60s)
--- PASS: TestAccAWSAutoScalingGroup_WithLoadBalancer (443.13s)

@bflad bflad merged commit 9dfcf44 into hashicorp:master Oct 10, 2019
bflad added a commit that referenced this pull request Oct 10, 2019
@ghost
Copy link

ghost commented Oct 10, 2019

This has been released in version 2.32.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@FernandoMiguel
Copy link
Contributor

:parrot.gif:

@ghost
Copy link

ghost commented Nov 9, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Nov 9, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/autoscaling Issues and PRs that pertain to the autoscaling service. size/L Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[aws_autoscaling_group] Attaching/Detaching a lot of loadbalancers fails
3 participants