aws_autoscaling_group in deposed state, but shouldn't be

[spy-tech]

The following applies to Terraform v0.8.5-dev (cb4ffd4+CHANGES)

My understanding of "deposed" is that it's a resource which will be deleted if one does a subsequent terraform apply, which would be a huge problem.

In the terraform.tfstate file, I see the autoscaling group with a particular ID with the deposed state:

                    "deposed": [
                        {
                            "id": "ecs-asg-prod-ecs-prod-00ad9b82fc2aaba998ca643aae",
                            "attributes": {
                                "availability_zones.#": "0",
                                "desired_capacity": "2",
                                "force_delete": "false",
                                "health_check_grace_period": "300",
                                "id": "ecs-asg-prod-ecs-prod-00ad9b82fc2aaba998ca643aae",
                                "launch_configuration": "ecs-prod-00ad9b82fc2aaba998ca643aae",
                                "load_balancers.#": "0",
                                "max_size": "10",
                                "metrics_granularity": "1Minute",
                                "min_size": "1",
                                "name": "ecs-asg-prod-ecs-prod-00ad9b82fc2aaba998ca643aae",
                                "protect_from_scale_in": "false",
                                "target_group_arns.#": "1",
                                <snip>
                                "wait_for_capacity_timeout": "10m",
                                "wait_for_elb_capacity": "1"
                            },

If I compare the id with the one that should remain in the AWS Console, I see that they match, so it shouldn't be deleted.

Relevant: #10753

[mitchellh]

See #10753 for more details on what deposed means.

Deposed only happens if you attempted a create before destroy apply earlier (that's how it shifts to "deposed"). Further, if a CBD resource depends on that ASG, it'll automatically make that resource CBD and cause deposition.

As #10753 states correctly though: we need to do a way better job documenting this! Sorry!

[spy-tech]

@mitchellh Thanks for your reply and my apologies for what appears to all be the same issue. I know that I want to keep the machines associated with the ASG id above. Will the ASG and thus the associated machines be deleted or not when I would run terraform apply or not?

If the answer is "yes", how can I get a tfstate file which does meet my expectations, which is that everything is fine and nothing needs to happen?

[mitchellh]

If it is deposed then the ASG and associated machines likely will.

You can move the "deposed" instance into the "primary" slot in the state. The terraform state commands currently don't have any way to target a deposed resource so you'd have to do this manually.

This is a strange state to get in though and one I haven't experienced before [in the community]. It'd help to understand how this happened and maybe we can build guardrails to prevent this in the future.

[mitchellh]

Note if you back up your state file and never run terraform apply (or destroy), no other Terraform command will actually make changes to your infra so you can safely run plan for example.

[spy-tech]

@mitchellh Create an ASG with a health check which fails, then it times out after 10 minutes and you get into this state. That's the only weird thing I did, at least.

[mitchellh]

Yes, that makes sense: if you have create before destroy, and the new creation fails, your old one is "deposed" (cause it was going to be destroyed anyways due to CBD) .You can see that from the explanation in #10753. That is the correct behavior for CBD.

"Deposed" is just a state that instances go between the create and destroy.

[spy-tech]

@mitchellh I don't want this resource to be destroyed and I would like it to work like Terraform expects it to work, but I am confused at this moment. Will something else be created first, before this is destroyed? It's really not clear what Terraform's plan is supposed to communicate to me in this case. Before 0.8 there was no such communication problem.

[mitchellh]

Yes, if CBD is set, a new ASG is created first, Terraform [by default] will wait for instances to populate that ASG, and then the old one will be destroyed.

Note that even if you don't set CBD on the ASG, if any dependent (child) resource depends on the ASG has CBD and requires CBD, the ASG will "inherit" the CBD. Its hard to explain why that is necesasry, but if you manually sketch out how to do a CBD safely you'll see that that is necessary.

[spy-tech]

@mitchellh In this case there is the difficulty that I run ECS containers which are supposed to first scale up (with the new machines created by the new ASG), but currently no new ECS containers are being launched, because the desired (not the maximum number) has already been deployed. This creates a "no progress"-situation. Do you have any suggestion how to get around this?

[artburkart]

I think one use case to consider:

1. apply an asg with a health check that succeeds
2. apply a new asg with instances whose health checks will fail
3. delete the new lingering asg manually
4. try deploying again, except with the original asg (as an attempt to clear the deposed state through an apply)
5. Watch it create the new asg successfully, then immediately delete it and tell you the apply was a success

EDIT: I think this is the workflow that got me into this situation anyway:

    arn:                                "arn:aws:autoscaling:us-east-1:REDACTED:autoScalingGroup:f716c42f-d811-49c2-937f-1ea58868ee2a:autoScalingGroupName/qa-east-app-00279dd8f897bd1e4b622c6927" => "<computed>"
    availability_zones.#:               "3" => "<computed>"
    default_cooldown:                   "300" => "<computed>"
    desired_capacity:                   "2" => "2"
    force_delete:                       "false" => "false"
    health_check_grace_period:          "180" => "180"
    health_check_type:                  "EC2" => "EC2"
    launch_configuration:               "qa-east-app-00279dd8f897bd1e4b622c6927" => "qa-east-app-00b145b6da90b03289a4d79f33"
    load_balancers.#:                   "1" => "1"
    load_balancers.3149143247:          "qa-east-app-internal-elb" => "qa-east-app-internal-elb"
    max_size:                           "5" => "5"
    metrics_granularity:                "1Minute" => "1Minute"
    min_size:                           "2" => "2"
    name:                               "qa-east-app-00279dd8f897bd1e4b622c6927" => "qa-east-app-00b145b6da90b03289a4d79f33" (forces new resource)
    protect_from_scale_in:              "false" => "false"
    tag.#:                              "6" => "6"
    tag.1615450184.key:                 "Name" => "Name"
    tag.1615450184.propagate_at_launch: "true" => "true"
    tag.1615450184.value:               "qa-east-app" => "qa-east-app"
    tag.2949978428.key:                 "env" => "env"
    tag.2949978428.propagate_at_launch: "true" => "true"
    tag.2949978428.value:               "qa-east" => "qa-east"
    tag.3477109309.key:                 "service" => "service"
    tag.3477109309.propagate_at_launch: "true" => "true"
    tag.3477109309.value:               "app" => "app"
    tag.3530196379.key:                 "type" => "type"
    tag.3530196379.propagate_at_launch: "true" => "true"
    tag.3530196379.value:               "app" => "app"
    tag.575090145.key:                  "provisioned-by" => "provisioned-by"
    tag.575090145.propagate_at_launch:  "true" => "true"
    tag.575090145.value:                "terraform-asg" => "terraform-asg"
    target_group_arns.#:                "0" => "<computed>"
    vpc_zone_identifier.#:              "3" => "3"
    vpc_zone_identifier.1806423168:     "subnet-a58f5288" => "subnet-a58f5288"
    vpc_zone_identifier.2279651228:     "subnet-b3eb3ae8" => "subnet-b3eb3ae8"
    vpc_zone_identifier.965220095:      "subnet-ff28eec3" => "subnet-ff28eec3"
    wait_for_capacity_timeout:          "10m" => "10m"
    wait_for_elb_capacity:              "2" => "2"

- module.app.aws_launch_configuration.lc (deposed)

aws_launch_configuration.lc: Refreshing state... (ID: qa-east-app-00b145b6da90b03289a4d79f33)
aws_autoscaling_group.asg: Refreshing state... (ID: qa-east-app-00279dd8f897bd1e4b622c6927)
module.app.aws_autoscaling_group.asg: Creating...
  arn:                                "" => "<computed>"
  availability_zones.#:               "" => "<computed>"
  default_cooldown:                   "" => "<computed>"
  desired_capacity:                   "" => "2"
  force_delete:                       "" => "false"
  health_check_grace_period:          "" => "180"
  health_check_type:                  "" => "EC2"
  launch_configuration:               "" => "qa-east-app-00b145b6da90b03289a4d79f33"
  load_balancers.#:                   "" => "1"
  load_balancers.3149143247:          "" => "qa-east-app-internal-elb"
  max_size:                           "" => "5"
  metrics_granularity:                "" => "1Minute"
  min_size:                           "" => "2"
  name:                               "" => "qa-east-app-00b145b6da90b03289a4d79f33"
  protect_from_scale_in:              "" => "false"
  tag.#:                              "" => "6"
  tag.1615450184.key:                 "" => "Name"
  tag.1615450184.propagate_at_launch: "" => "true"
  tag.1615450184.value:               "" => "qa-east-app"
  tag.2737026993.key:                 "" => "ec2box-user"
  tag.2737026993.propagate_at_launch: "" => "true"
  tag.2737026993.value:               "" => "centos"
  tag.2949978428.key:                 "" => "env"
  tag.2949978428.propagate_at_launch: "" => "true"
  tag.2949978428.value:               "" => "qa-east"
  tag.3477109309.key:                 "" => "service"
  tag.3477109309.propagate_at_launch: "" => "true"
  tag.3477109309.value:               "" => "app"
  tag.3530196379.key:                 "" => "type"
  tag.3530196379.propagate_at_launch: "" => "true"
  tag.3530196379.value:               "" => "app"
  tag.575090145.key:                  "" => "provisioned-by"
  tag.575090145.propagate_at_launch:  "" => "true"
  tag.575090145.value:                "" => "terraform-asg"
  target_group_arns.#:                "" => "<computed>"
  vpc_zone_identifier.#:              "" => "3"
  vpc_zone_identifier.1806423168:     "" => "subnet-a58f5288"
  vpc_zone_identifier.2279651228:     "" => "subnet-b3eb3ae8"
  vpc_zone_identifier.965220095:      "" => "subnet-ff28eec3"
  wait_for_capacity_timeout:          "" => "10m"
  wait_for_elb_capacity:              "" => "2"
module.app.aws_autoscaling_group.asg: Still creating... (10s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (20s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (30s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (40s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (50s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (1m0s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (1m10s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (1m20s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (1m30s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (1m40s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (1m50s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (2m0s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (2m10s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (2m20s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (2m30s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (2m40s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (2m50s elapsed)
module.app.aws_autoscaling_group.asg: Still creating... (3m0s elapsed)
module.app.aws_autoscaling_group.asg: Creation complete (ID: qa-east-app-00b145b6da90b03289a4d79f33)
module.app.aws_autoscaling_group.asg (deposed #1): Destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33)
module.app.aws_autoscaling_group.asg (deposed #0): Destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33)
module.app.aws_autoscaling_group.asg (deposed #2): Destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 10s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 10s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 10s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 20s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 20s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 20s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 30s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 30s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 30s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 40s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 40s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 40s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 50s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 50s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 50s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m0s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m0s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 1m0s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m10s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m10s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 1m10s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m20s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m20s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 1m20s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m30s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m30s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 1m30s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m40s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m40s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Still destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927, 1m40s elapsed)
module.app.aws_autoscaling_group.asg (deposed #2): Destruction complete
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m50s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 1m50s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 2m0s elapsed)
module.app.aws_autoscaling_group.asg (deposed #0): Still destroying... (ID: qa-east-app-00b145b6da90b03289a4d79f33, 2m0s elapsed)
module.app.aws_autoscaling_group.asg (deposed #1): Destruction complete
module.app.aws_autoscaling_group.asg (deposed #0): Destruction complete
module.app.aws_launch_configuration.lc (deposed #0): Destroying... (ID: qa-east-app-00279dd8f897bd1e4b622c6927)
module.app.aws_launch_configuration.lc (deposed #0): Destruction complete```

[tamsky]

I'd offer that a manual action you've stated across a few different bugs [*] is the root cause of the undesired behavior you're observing:

In #11459 (comment)
you wrote:

I think one use case to consider:
...
3. delete the new lingering asg manually

Summary of the references below:
Don't do stuff to the ASG, instead do stuff to the Launch Config.

• References:

• Incorrectly destroying aws_autoscaling_group resource that isn't marked deposed terraform-provider-aws#2438 (comment)
• Autoscaling group already exists after failure due to AWS limits terraform-provider-aws#309 (comment)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.