Can not delete database because of dependency on ECS

[boskiv]

Hi there,

Terraform Version

ubuntu@ip-172-31-67-74:~/worker/tenant$ terraform -v
Terraform v0.11.7
+ provider.aws v1.18.0
+ provider.postgresql v0.1.1
+ provider.template v1.0.0

Affected Resource(s)

Please list the resources as a list, for example:

• postgresql_database.tenant

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

resource "aws_ecs_service" "selected" {
  name                              = "${terraform.workspace}-service"
  cluster                           = "${data.aws_ecs_cluster.selected.id}"
  task_definition                   = "${aws_ecs_task_definition.selected.arn}"
  desired_count                     = 1
  health_check_grace_period_seconds = 300
  launch_type                       = "FARGATE"

  network_configuration {
    security_groups = ["${data.aws_security_group.selected.id}"]
    subnets         = ["${data.aws_subnet_ids.default.ids}"]

    assign_public_ip = true
  }

  load_balancer {
    target_group_arn = "${aws_lb_target_group.tenant.arn}"
    container_name   = "selected"
    container_port   = 8080
  }

  depends_on = ["aws_ecs_task_definition.selected"]
}

...
resource "postgresql_database" "tenant" {
  provider          = "postgresql.rds"
  name              = "${terraform.workspace}"
  owner             = "${terraform.workspace}"
  allow_connections = true
  depends_on        = ["postgresql_role.tenant", "aws_ecs_service.selected"]
}
...

Debug Output

When I try to destroy terraform i have got an error:

Error: Error applying plan:

1 error(s) occurred:

* postgresql_database.tenant (destroy): 1 error(s) occurred:

* postgresql_database.tenant: Error dropping database: pq: database "test06" is being accessed by other users

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Expected Behavior

Database deletes after ECS Service

Actual Behavior

Database deletes before ECS Service

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform destroy

[boskiv]

terraform graph -type=plan-destroy
digraph {
	compound = "true"
	newrank = "true"
	subgraph "root" {
		"[root] aws_cloudwatch_log_group.selected" [label = "aws_cloudwatch_log_group.selected", shape = "box"]
		"[root] aws_ecs_service.selected" [label = "aws_ecs_service.selected", shape = "box"]
		"[root] aws_ecs_task_definition.selected" [label = "aws_ecs_task_definition.selected", shape = "box"]
		"[root] aws_lb_target_group.tenant" [label = "aws_lb_target_group.tenant", shape = "box"]
		"[root] data.aws_ecs_cluster.selected" [label = "data.aws_ecs_cluster.selected", shape = "box"]
		"[root] data.aws_iam_role.ecs" [label = "data.aws_iam_role.ecs", shape = "box"]
		"[root] data.aws_iam_role.selected" [label = "data.aws_iam_role.selected", shape = "box"]
		"[root] data.aws_security_group.selected" [label = "data.aws_security_group.selected", shape = "box"]
		"[root] data.aws_subnet_ids.default" [label = "data.aws_subnet_ids.default", shape = "box"]
		"[root] data.aws_vpc.default" [label = "data.aws_vpc.default", shape = "box"]
		"[root] data.template_file.container_definitions" [label = "data.template_file.container_definitions", shape = "box"]
		"[root] postgresql_database.tenant" [label = "postgresql_database.tenant", shape = "box"]
		"[root] postgresql_role.tenant" [label = "postgresql_role.tenant", shape = "box"]
		"[root] aws_cloudwatch_log_group.selected" -> "[root] aws_ecs_service.selected"
		"[root] aws_cloudwatch_log_group.selected" -> "[root] aws_ecs_task_definition.selected"
		"[root] aws_cloudwatch_log_group.selected" -> "[root] data.template_file.container_definitions"
		"[root] aws_cloudwatch_log_group.selected" -> "[root] postgresql_database.tenant"
		"[root] aws_ecs_service.selected" -> "[root] postgresql_database.tenant"
		"[root] aws_ecs_task_definition.selected" -> "[root] aws_ecs_service.selected"
		"[root] aws_ecs_task_definition.selected" -> "[root] postgresql_database.tenant"
		"[root] aws_lb_target_group.tenant" -> "[root] aws_ecs_service.selected"
		"[root] aws_lb_target_group.tenant" -> "[root] postgresql_database.tenant"
		"[root] data.aws_ecs_cluster.selected" -> "[root] aws_ecs_service.selected"
		"[root] data.aws_ecs_cluster.selected" -> "[root] postgresql_database.tenant"
		"[root] data.aws_iam_role.ecs" -> "[root] aws_ecs_service.selected"
		"[root] data.aws_iam_role.ecs" -> "[root] aws_ecs_task_definition.selected"
		"[root] data.aws_iam_role.ecs" -> "[root] postgresql_database.tenant"
		"[root] data.aws_iam_role.selected" -> "[root] aws_ecs_service.selected"
		"[root] data.aws_iam_role.selected" -> "[root] aws_ecs_task_definition.selected"
		"[root] data.aws_iam_role.selected" -> "[root] postgresql_database.tenant"
		"[root] data.aws_security_group.selected" -> "[root] aws_ecs_service.selected"
		"[root] data.aws_security_group.selected" -> "[root] postgresql_database.tenant"
		"[root] data.aws_subnet_ids.default" -> "[root] aws_ecs_service.selected"
		"[root] data.aws_subnet_ids.default" -> "[root] postgresql_database.tenant"
		"[root] data.aws_vpc.default" -> "[root] aws_ecs_service.selected"
		"[root] data.aws_vpc.default" -> "[root] aws_lb_target_group.tenant"
		"[root] data.aws_vpc.default" -> "[root] data.aws_subnet_ids.default"
		"[root] data.aws_vpc.default" -> "[root] postgresql_database.tenant"
		"[root] data.template_file.container_definitions" -> "[root] aws_ecs_service.selected"
		"[root] data.template_file.container_definitions" -> "[root] aws_ecs_task_definition.selected"
		"[root] data.template_file.container_definitions" -> "[root] postgresql_database.tenant"
		"[root] postgresql_role.tenant" -> "[root] postgresql_database.tenant"
		"[root] root" -> "[root] aws_cloudwatch_log_group.selected"
		"[root] root" -> "[root] data.aws_ecs_cluster.selected"
		"[root] root" -> "[root] data.aws_iam_role.ecs"
		"[root] root" -> "[root] data.aws_iam_role.selected"
		"[root] root" -> "[root] data.aws_security_group.selected"
		"[root] root" -> "[root] data.aws_vpc.default"
		"[root] root" -> "[root] postgresql_role.tenant"
	}
}

[tomelliff]

Why is the database creation dependent on the ECS service? That feels like the wrong way round to me. Surely the ECS service wants the database to be created first so that it can use it?

Changing the dependency around would automatically fix this as Terraform would know that it needs to destroy the ECS service first (freeing connections to the database) before destroying the database.

[cyrilgdn]

@boskiv I allow myself to close this issue based on @tomelliff 's comment (thanks @tomelliff for your help).

Feel free to open it back if we missed something.