Autoscaling Group - Target groups are getting removed

[vishwakumba]

Issue Description

We upgraded aws terraform version from 2.70.0 to 3.0.0 on 03.08.20 for our terraform project. We also changed the EC2 Instance type, which usually is a small change for us in our Production Env. We were expecting the launch configuration(LC) and autoscaling groups(ASG) to be updated. The new aws terraform provider 3.0.0, updated the LC and ASG but also removed the attached target groups to the autoscaling group. This resulted in the Load balancer producing a lot of errors as it did not find out valid target groups.

Running terraform apply the second time, seems to add the missing autoscaling attachments to the target groups.

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.13

• provider.aws v3.0.0 and v3.1.0

Affected Resource(s)

• aws_autoscaling_group
• aws_launch_configuration
• aws_lb_target_group
• aws_autoscaling_attachment

Terraform Configuration Files

data "template_file" "user_data" {
  template = "${file("${path.module}/user-data.sh")}"

  vars = {   
    t_deploy_region    = var.deploy_region
    t_env_name         = var.env_name
    t_role             = "dummy-role"
  }
}

resource "aws_launch_configuration" "my_lc" {
  name_prefix = "${var.service_name}-${var.env_name}-${var.my_tag}-app-"
  image_id    = data.aws_ami.rhel.id
  iam_instance_profile = var.iam_instance_profile
  instance_type        = var.instance_type
  user_data            = data.template_file.user_data.rendered
  key_name             = var.ssh_key_name

  security_groups = [aws_security_group.my_security_group.id]

  root_block_device {
    volume_size = "10"
    volume_type = "gp2"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "my_asg" {
  name_prefix               = "${var.service_name}-${var.env_name}-${var.my_tag}-app"
  min_size                  = 1
  max_size                  = 1
  vpc_zone_identifier       = var.subnet_ids
  launch_configuration      = aws_launch_configuration.my_lc.id
  health_check_type         = "EC2"
  health_check_grace_period = 300
  termination_policies      = ["OldestInstance"]

  tag {
    key                 = "Name"
    value               = "${var.service_name}-${var.env_name}-${var.my_tag}-app"
    propagate_at_launch = true
  }
}

resource "aws_lb" "my_lb" {
  name               = "${var.service_name}-${var.env_name}-${var.my_tag}-app-lb"
  internal           = true
  subnets            = var.subnet_ids
  security_groups    = [aws_security_group.my_security_group.id]
  load_balancer_type = "application"

  tags = {
    Name = "${var.service_name}-${var.env_name}-${var.my_tag}-app-lb"
  }
}

resource "aws_lb_listener" "my_listener" {
  load_balancer_arn = aws_lb.my_lb.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    target_group_arn = aws_lb_target_group.my_tg.arn
    type             = "forward"
  }
}

resource "aws_lb_target_group" "my_tg" {
  name     = "${var.service_name}-${var.env_name}-${var.my_tag}-app-tg"
  port     = 80
  protocol = "HTTPS"
  vpc_id   = var.vpc_id

  health_check {
    healthy_threshold   = 2
    unhealthy_threshold = 2
    timeout             = 5
    path                = "/"
    protocol            = "HTTP"
    interval            = 30
    matcher             = 200
  }

  tags = {
    Name = "${var.service_name}-${var.env_name}-${var.my_tag}-app-tg"
  }
}

resource "aws_autoscaling_attachment" "my_asg_tg_attachment" {
  autoscaling_group_name = "${aws_autoscaling_group.my_asg.id}"
  alb_target_group_arn   = "${aws_lb_target_group.my_tg.arn}"
}

And the terraform plan output:

# aws_autoscaling_group.my_asg will be updated in-place
  ~ resource "aws_autoscaling_group" "my_asg" {
        arn                       = "arn:aws:autoscaling:eu-west-2:XXXXXXXXXXXX:autoScalingGroup:c4054f7a-057a-448e-bf99-24888adcfdca:autoScalingGroupName/hello-dev-dummy-app20200810092325278800000002"
        availability_zones        = [
            "eu-west-2b",
            "eu-west-2c",
        ]
        default_cooldown          = 300
        desired_capacity          = 1
        enabled_metrics           = []
        force_delete              = false
        health_check_grace_period = 300
        health_check_type         = "EC2"
        id                        = "hello-dev-dummy-app20200810092325278800000002"
      ~ launch_configuration      = "hello-dev-dummy-app-20200810092856999400000002" -> (known after apply)
        load_balancers            = []
        max_instance_lifetime     = 0
        max_size                  = 1
        metrics_granularity       = "1Minute"
        min_size                  = 1
        name                      = "hello-dev-dummy-app20200810092325278800000002"
        name_prefix               = "hello-dev-dummy-app"
        protect_from_scale_in     = false
        service_linked_role_arn   = "arn:aws:iam::XXXXXXXXXXXX::role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        suspended_processes       = []
      ~ target_group_arns         = [
          - "arn:aws:elasticloadbalancing:eu-west-2:XXXXXXXXXXXX::targetgroup/hello-dev-dummy-app-tg/e6d773958b40edb8",
        ]
        termination_policies      = [
            "OldestInstance",
        ]
        vpc_zone_identifier       = [
            "subnet-e217ee8b",
            "subnet-f1410bbc",
        ]
        wait_for_capacity_timeout = "10m"

        tag {
            key                 = "Name"
            propagate_at_launch = true
            value               = "hello-dev-dummy-app"
        }
    }

  # aws_launch_configuration.my_lc must be replaced
+/- resource "aws_launch_configuration" "my_lc" {
      ~ arn                              = "arn:aws:autoscaling:eu-west-2:XXXXXXXXXXXX::launchConfiguration:278cec73-e879-42aa-85a7-5727ac849e4a:launchConfigurationName/hello-dev-dummy-app-20200810092856999400000002" -> (known after apply)
        associate_public_ip_address      = false
      ~ ebs_optimized                    = false -> (known after apply)
        enable_monitoring                = true
        iam_instance_profile             = "dummy-dev-dummy-instance-profile"
      ~ id                               = "hello-dev-dummy-app-20200810092856999400000002" -> (known after apply)
        image_id                         = "ami-0a777b190599bebc2"
      ~ instance_type                    = "t3.medium" -> "t2.medium" # forces replacement
        key_name                         = "MasterKey"
      ~ name                             = "hello-dev-dummy-app-20200810092856999400000002" -> (known after apply)
        name_prefix                      = "hello-dev-dummy-app-"
        security_groups                  = [
            "sg-0dcdf2ca6cc7da614",
        ]
        user_data                        = "4090d22b8884048dec1e6212e0ac29cb995ae582"
      - vpc_classic_link_security_groups = [] -> null

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + no_device             = (known after apply)
          + snapshot_id           = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      ~ root_block_device {
            delete_on_termination = true
          ~ encrypted             = false -> (known after apply)
          ~ iops                  = 0 -> (known after apply)
            volume_size           = 10
            volume_type           = "gp2"
        }
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Expected Behavior

Target groups containing the EC2 Instances serving requests should have been retained in the Autoscaling Group

Actual Behavior

Target groups containing the EC2 Instances serving requests have been removed from the Autoscaling Group.

Using aws terraform provider 3.0.0 or 3.1.0 seems to remove the target groups associated.
This piece of code works fine in version 2.70.0

Steps to Reproduce

1. Modify the EC2 Instance type or AMI or some other small change in the Launch configuration.
2. Do a terraform plan and it shows that it is removing the target groups attached to the autoscaling group
3. Apply terraform

Note: Running terraform apply, the second time, seems to add the missing autoscaling attachments.

[shamil]

Have same issue.

resource "aws_autoscaling_attachment" "example" {
  autoscaling_group_name = aws_autoscaling_group.example.id
  elb                    = aws_elb.example.id
}

Once applied, terraform removes this load_balancer from the aws_autoscaling_group.example resource

[shamil]

Note: Running terraform apply, the second time, seems to add the missing autoscaling attachments.

@vishwakumba I think if you apply third time it will remove the target_group_arns again (this is what happens to me)

For now I ignore any changes in load_balancers in the asg resource, like below:

resource "aws_autoscaling_group" "example" {
  lifecycle {
    ignore_changes = [load_balancers]
  }
}

[vxe]

Hi any progress on this?

[steeef]

I can confirm this is still an issue with the latest provider version, 3.3.0.

[steeef]

I figured this out with @shamil's comment and after re-reading updated documentation:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_attachment

Similar to aws_security_group and aws_security_group_rule, you can define target group attachments inline in aws_autoscaling_groups or separately in aws_autoscaling_attachments. However, if you use aws_autoscaling_attachment, you need to set ignore_changes in a lifecycle block in the respective aws_autoscaling_group definition. Once I did that it worked as expected.

I should have read https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-3-upgrade#resource-aws_autoscaling_group a bit closer.

[cgkades]

I've added this to the autoscale group definition

  lifecycle {
    create_before_destroy = true
    ignore_changes = [ load_balancers, target_group_arns ]
  }

and it's still trying to remove the target groups

  # aws_autoscaling_group.my_asg will be updated in-place
  ~ resource "aws_autoscaling_group" "my_asg" {
        id                        = "asg"
        name                      = "asg"
      ~ target_group_arns         = [
          - "arn:aws:elasticloadbalancing:us-east-1:<removed>:targetgroup/targetgroup-443/0adc49f123b29cb0",
          - "arn:aws:elasticloadbalancing:us-east-1:<removed>:targetgroup/targetgroup-80/9155f929e96e398a",
        ]

[cgkades]

I deleted destroyed everything and rebuilt, it's now honoring the ignore

[jgelens]

I have a similar setup and added the lifecycle rules as explained and the change still shows up. I also tried rebuilding the entire ASG and related resources as @cgkades did, but that did not fix the issue for me. Is there anything else I could try? :/

[cgkades]

I have a similar setup and added the lifecycle rules as explained and the change still shows up. I also tried rebuilding the entire ASG and related resources as @cgkades did, but that did not fix the issue for me. Is there anything else I could try? :/

@jgelens we also made sure it wasn't generating a different name every time it ran. Can you paste your ASG section?

[jgelens]

@cgkades The names I use are static.

resource "aws_autoscaling_group" "default" {
  name                      = "${var.env}-asg-${var.name}"
  launch_configuration      = aws_launch_configuration.default.name
  wait_for_capacity_timeout = 0

  max_size = var.asg_max_size
  min_size = var.asg_min_size

  vpc_zone_identifier = var.subnets
  suspended_processes = var.suspended_processes

  lifecycle {
    create_before_destroy = true
    ignore_changes = [load_balancers, target_group_arns]
  }
}

resource "aws_autoscaling_attachment" "vpn_api_alb" {
  autoscaling_group_name = module.service_vpn_api.asg_id
  alb_target_group_arn   = module.alb_vpn_api.alb_target_group_arn
}

locals {
  alb_name = "${var.env}-${var.name}"
}

resource "aws_alb" "default" {
  name     = local.alb_name
  internal = var.alb_internal
  security_groups = [module.sg.sg_id]
  subnets         = var.subnets
  ip_address_type = "ipv4"
  idle_timeout    = var.alb_idle_timeout
}

[saikumarkalakota]

Hi.
Even i am facing similar issue. Can someone plz let me know the resolution?

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!