Unable to apply autoscaling targets while changing billing mode from PAY_PER_REQUEST to PROVISIONED for Global Tables

[prudhvi]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

TERRAFORM_VERSION=0.14.11
TERRAFORM_AWS_PROVIDER_VERSION=3.73.0

Affected Resource(s)

• aws_dynamodb_table
• aws_appautoscaling_target

Terraform Configuration Files

resource "aws_dynamodb_table" "west" {
  provider = aws.west

  hash_key         = var.hash_key
  range_key        = var.range_key
  name             = "${var.service}.${var.name}"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  billing_mode   = var.billing_mode
  read_capacity  = var.billing_mode == "PROVISIONED" ? var.read_capacity : 0
  write_capacity = var.billing_mode == "PROVISIONED" ? var.write_capacity : 0

  server_side_encryption {
    enabled = true
  }

  point_in_time_recovery {
    enabled = true
  }

  dynamic "replica" {
    for_each = var.disable_global_table ? [] : [1]
    content {
      region_name = "us-east-1"
    }
  }

  dynamic "attribute" {
    for_each = var.attributes
    content {
      name = attribute.value.name
      type = attribute.value.type
    }
  }

  ttl {
    enabled        = var.ttl_attribute != ""
    attribute_name = var.ttl_attribute
  }

  dynamic "local_secondary_index" {
    for_each = var.local_secondary_index
    content {
      name               = local_secondary_index.value.name
      range_key          = local_secondary_index.value.range_key
      projection_type    = local_secondary_index.value.projection_type
      non_key_attributes = local_secondary_index.value.non_key_attributes
    }
  }

  dynamic "global_secondary_index" {
    for_each = var.global_secondary_index
    content {
      name               = global_secondary_index.value.name
      hash_key           = global_secondary_index.value.hash_key
      range_key          = global_secondary_index.value.range_key
      write_capacity     = global_secondary_index.value.write_capacity
      read_capacity      = global_secondary_index.value.read_capacity
      projection_type    = global_secondary_index.value.projection_type
      non_key_attributes = global_secondary_index.value.non_key_attributes
    }
  }

  tags = {
    Service     = var.service
    Environment = var.environment
  }
}


resource "aws_appautoscaling_target" "read_west" {
  provider           = aws.west
  max_capacity       = lookup(var.auto_scaling, "read", null) != null ? var.auto_scaling["read"].max : var.read_capacity
  min_capacity       = lookup(var.auto_scaling, "read", null) != null ? var.auto_scaling["read"].min : var.read_capacity
  resource_id        = "table/${aws_dynamodb_table.west.name}"
  scalable_dimension = "dynamodb:table:ReadCapacityUnits"
  service_namespace  = "dynamodb"
  count              = var.billing_mode == "PROVISIONED" && (lookup(var.auto_scaling, "read", null) != null || length(var.scheduled_auto_scaling) > 0)  ? 1 : 0
  depends_on = [aws_dynamodb_table.west]
}

resource "aws_appautoscaling_policy" "dynamodb_table_read_policy_west" {
  provider           = aws.west
  name               = "DynamoDBReadCapacityUtilization:${aws_appautoscaling_target.read_west[count.index].resource_id}"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.read_west[count.index].resource_id
  scalable_dimension = aws_appautoscaling_target.read_west[count.index].scalable_dimension
  service_namespace  = aws_appautoscaling_target.read_west[count.index].service_namespace
  count              = var.billing_mode == "PROVISIONED" && lookup(var.auto_scaling, "read", null) != null ? 1 : 0

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "DynamoDBReadCapacityUtilization"
    }

    target_value = var.auto_scaling["read"].target_utilization
  }

  depends_on = [aws_dynamodb_table.west, aws_appautoscaling_target.read_west]
}

resource "aws_appautoscaling_scheduled_action" "dynamodb_table_read_policy_west" {
  provider           = aws.west
  name               = "DynamoDBAutoScalingScheduledAction${var.scheduled_auto_scaling[count.index].name}"
  resource_id        = aws_appautoscaling_target.read_west[0].resource_id
  scalable_dimension = aws_appautoscaling_target.read_west[0].scalable_dimension
  service_namespace  = aws_appautoscaling_target.read_west[0].service_namespace
  schedule           = var.scheduled_auto_scaling[count.index].schedule
  timezone           = var.scheduled_auto_scaling[count.index].timezone
  scalable_target_action {
    max_capacity       = var.scheduled_auto_scaling[count.index].read.max
    min_capacity       = var.scheduled_auto_scaling[count.index].read.min
  }
  count = var.billing_mode == "PROVISIONED" ? length(var.scheduled_auto_scaling) : 0
  depends_on = [aws_dynamodb_table.west, aws_appautoscaling_target.read_west]
}

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Debug Output

Error: Error creating application autoscaling target: ValidationException: Validation failed for scalable target. Reason: PAY_PER_REQUEST table mode is not scalable.

  on main.tf line 92, in resource "aws_appautoscaling_target" "read_west":
  92: resource "aws_appautoscaling_target" "read_west" {



Error: Error creating application autoscaling target: ValidationException: Validation failed for scalable target. Reason: PAY_PER_REQUEST table mode is not scalable.

  on main.tf line 139, in resource "aws_appautoscaling_target" "write_west":
 139: resource "aws_appautoscaling_target" "write_west" {



Error: Error creating application autoscaling target: ValidationException: Validation failed for scalable target. Reason: PAY_PER_REQUEST table mode is not scalable.

  on main.tf line 187, in resource "aws_appautoscaling_target" "gsi_index_read_west":
 187: resource "aws_appautoscaling_target" "gsi_index_read_west" {



Error: Error creating application autoscaling target: ValidationException: Validation failed for scalable target. Reason: PAY_PER_REQUEST table mode is not scalable.

  on main.tf line 235, in resource "aws_appautoscaling_target" "gsi_index_write_west":
 235: resource "aws_appautoscaling_target" "gsi_index_write_west" {

Expected Behavior

Should change the table billing mode and also attach the autoscaling settings that are supplied in terraform

Actual Behavior

Failed to change the table from PAY_PER_REQUEST to PROVISIONED billing and add autoscaling at the same time. Updated the billing mode to PROVISIONED but failed to attach the autoscaling settings. Even after adding depends_on for autoscaling resources so that table update finishes before applying the autoscaling settings it's not working.

My guess is that depends_on doesn't actually wait for the table to become ACTIVE before applying the autoscaling targets

Steps to Reproduce

1. terraform apply

[prudhvi]

One of the debug log contains this warning message, not sure if it's related

-----------------------------------------------------: timestamp=2022-01-27T06:04:07.827Z
2022-01-27T06:04:07.828Z [INFO]  plugin.terraform-provider-aws_v3.73.0_x5: 2022/01/27 06:04:07 [DEBUG] [aws-sdk-go] {"Tags":[{"Key":"Environment","Value":"staging"},{"Key":"Service","Value":"prudhvi-test"}]}: timestamp=2022-01-27T06:04:07.827Z
2022/01/27 06:04:07 [WARN] Provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value for aws_dynamodb_table.west, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .write_capacity: was cty.NumberIntVal(500), but now cty.NumberIntVal(0)
      - .read_capacity: was cty.NumberIntVal(500), but now cty.NumberIntVal(0)
      - .billing_mode: was cty.StringVal("PROVISIONED"), but now cty.StringVal("PAY_PER_REQUEST")
    

[hanscg]

We still encountered this with provider version 4.67