Inconsistent final plan with aws_cloudwatch_metric_alarm

[rankol]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

0.12.31

Affected Resource(s)

• aws_cloudwatch_metric_alarm

Terraform Configuration Files

Child module main.tf where I've removed some additional code not related to the error message. If the additional main.tf code is needed let me know.

resource "aws_cloudwatch_metric_alarm" "metric_alarm" {

  for_each = var.cloudwatch_alarms_defined

  dynamic "metric_query" {
    for_each = lookup(each.value, "cloudwatch_metric_name") == "" ? lookup(each.value, "cloudwatch_metric_queries_defined", {}) : {}

    content {
      id          = lookup(metric_query.value, "metric_id", "")
      label       = lookup(metric_query.value, "metric_label", "")
      return_data = lookup(metric_query.value, "metric_return_data", "true")

      dynamic "metric" {
        for_each = lookup(each.value, "cloudwatch_metric_name") == "" ? lookup(each.value, "cloudwatch_metric_queries_defined", {}) : {}

        content {
          dimensions  = lookup(metric.value, "metric_dimensions", {})
          metric_name = lookup(metric.value, "metric_name", "")
          namespace   = lookup(metric.value, "metric_namespace", "")
          period      = lookup(metric.value, "metric_period", "")
          stat        = lookup(metric.value, "metric_stat", "")
          unit        = lookup(metric.value, "metric_unit", "")
        }
      }
    }
  }
}

Child module variables.tf

variable "cloudwatch_alarms_defined" {
  description = "Map of objects containing the alarm(s) to provision."
  type = map(object({
    cloudwatch_alarm_name                            = string
    cloudwatch_comparison_operator                   = string
    cloudwatch_evaluation_periods                    = string
    cloudwatch_metric_name                           = string
    cloudwatch_namespace                             = string
    cloudwatch_period                                = string
    cloudwatch_statistic                             = string
    cloudwatch_threshold                             = string
    cloudwatch_threshold_metric_id                   = string
    cloudwatch_actions_enabled                       = bool
    cloudwatch_alarm_actions                         = set(string)
    cloudwatch_alarm_description                     = string
    cloudwatch_datapoints_to_alarm                   = string
    cloudwatch_dimensions                            = map(string)
    cloudwatch_insufficient_data_actions             = set(string)
    cloudwatch_ok_actions                            = set(string)
    cloudwatch_unit                                  = string
    cloudwatch_extended_statistic                    = string
    cloudwatch_treat_missing_data                    = string
    cloudwatch_evaluate_low_sample_count_percentiles = string
    cloudwatch_tags                                  = map(string)
    cloudwatch_metric_queries_defined = map(object({
      metric_id          = string
      metric_label       = string
      metric_return_data = string
      metric_dimensions  = map(string)
      metric_name        = string
      metric_namespace   = string
      metric_period      = string
      metric_stat        = string
      metric_unit        = string
    }))
  }))
  default = {}
}

Parent module main.tf for cloudwatch_alarms_defined variable. There are additional parameters being passed in that are not related to the error message for metric_query.

cloudwatch_alarms_defined = {
    alarm1 = {
      cloudwatch_metric_queries_defined                = {}
    }
  }
}

Expected Behavior

I was not expecting for the output below of an inconsistent plan error showing up. I recently refactored this code to include the for_each loop in the child module main.tf. Previously there was no for_each loop or lookup function being used.

Actual Behavior

I received an error regarding an inconsistent final plan during the apply stage. This is ran through a Jenkins pipeline. I have not seen this before the refactoring of the module and I have not attempted to recreate this issue at the moment but I am planning on doing so.

Error: Provider produced inconsistent final plan

When expanding the plan for module.aws_cloudwatch_metric_alarm.aws_cloudwatch_metric_alarm.metric_alarm["alarm1"]
to include new values learned so far during apply, provider
"registry.terraform.io/-/aws" produced an invalid new value for .metric_query: planned set element
cty.ObjectVal(map[string]cty.Value{"account_id":cty.StringVal(""),
"expression":cty.StringVal(""), "id":cty.UnknownVal(cty.String),
"label":cty.UnknownVal(cty.String),
"metric":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"dimensions":cty.UnknownVal(cty.Map(cty.String)),
"metric_name":cty.UnknownVal(cty.String),
"namespace":cty.UnknownVal(cty.String), "period":cty.UnknownVal(cty.Number),
"stat":cty.UnknownVal(cty.String), "unit":cty.UnknownVal(cty.String)})}),
"return_data":cty.UnknownVal(cty.Bool)}) does not correlate with any element
in actual.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Steps to Reproduce

1. terraform plan
2. terraform apply

Important Factoids

1. This is run in Jenkins
2. Child module was recently refactored to include a for_each loop
3. I am passing in cloudwatch_metric_queries_defined = {} since I am not looking to use this actual field but it is required to be in there due to the definition of the cloudwatch_alarms_defined variable.

[majoras-masque]

Thought I'd chime in in case it helps: I got the same error, though with different circumstances. For me, my alarm is dependent on an RDS cluster instance resource which is also created via for_each. My metric_query.expression child block value relates to values dependent on the instance class of the host. In short: when my instance class changes, my alarm will change too. I was getting this error initially, however: later I went and manually changed the instance class in the console, then re-applied: this time the alarm made the change successfully. It's almost as if AWS' API is making value checks for the expression dependent on the host its monitoring, though that sounds pretty out there :/

If it's helpful, I can provide my code and error as well: I've saved them in a Gist if needed, but can paste into a comment here.

Edit:
For us, we resolved our issue by setting the apply_immediately to true for the RDS cluster instance. Seems like there was some verification happening on the alarm for the value of metric_query.expression against the actual instance. interesting that it produced the same error as the initial posting on this issue though.