Maintenance exclusion is not removed from google_container_cluster

[pdemagny]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

❯ terraform -v
Terraform v1.0.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v3.81.0
+ provider registry.terraform.io/hashicorp/google-beta v3.81.0

Affected Resource(s)

• google_container_cluster

Terraform Configuration Files

Notes:

• I have removed most of the unrelated stuff for clarity.
• Maintenance exclusions have been added with this exact code, but with the 3.76.0 google provider.

In my main configuration i call a module like this:

locals {
  env = {
    defaults = {
      prefix = "tfl-${terraform.workspace}"
    }
    preview = {}
    staging = {}
    prod = {}
    defaults_lists = {
      gke_c1_maintenance_exclusions = []
    }
    preview_lists = {}
    staging_lists = {}
    prod_lists = {}
  }

  workspace = merge(local.env["defaults"], local.env[terraform.workspace])
  workspace_lists = merge(
    local.env["defaults_lists"],
    local.env[format("%v_lists", terraform.workspace)],
  )
  
  region = "europe-west1"
  zone   = "europe-west1-c"
}

module "gke_c1" {
  source = "../modules/gcp-container-cluster"

  providers = {
    google-beta = google-beta.google-beta-be
  }

  inputs = {
    project  = "my-project-id"
    name     = "${local.workspace.prefix}-main"
    location = terraform.workspace == "prod" ? local.region : local.zone
    master = {
      maintenance_exclusions        = local.workspace_lists.gke_c1_maintenance_exclusions
    }
  }
}

And here is the module:

variable "inputs_default" {
  type = object({
    project  = string
    name     = string
    location = string
    master = object({
      maintenance_start_time = string
      maintenance_end_time   = string
      maintenance_recurrence = string
      maintenance_exclusions = list(object({ name = string, start_time = string, end_time = string }))
    })
  })
  default = {
    project  = ""
    name     = ""
    location = null
    master = {
      maintenance_start_time = "04:30"
      maintenance_end_time   = ""
      maintenance_recurrence = ""
      maintenance_exclusions = []
    }
  }
}

variable "inputs" {
  description = "Map of module specific inputs to be merged with inputs_default. Allowed keys are the same."
}

locals {
  merged_inputs                           = merge(var.inputs_default, var.inputs)
  merged_master_inputs                    = merge(var.inputs_default.master, var.inputs.master)
  cluster_maintenance_window_is_recurring = local.merged_master_inputs.maintenance_recurrence != "" && local.merged_master_inputs.maintenance_end_time != "" ? [1] : []
  cluster_maintenance_window_is_daily     = length(local.cluster_maintenance_window_is_recurring) > 0 ? [] : [1]
}

resource "google_container_cluster" "gke_master" {
  provider = google-beta

  project     = local.merged_inputs.project
  name        = local.merged_inputs.name
  location    = local.merged_inputs.location

  maintenance_policy {
    dynamic "recurring_window" {
      for_each = local.cluster_maintenance_window_is_recurring
      content {
        start_time = local.merged_master_inputs.maintenance_start_time
        end_time   = local.merged_master_inputs.maintenance_end_time
        recurrence = local.merged_master_inputs.maintenance_recurrence
      }
    }

    dynamic "daily_maintenance_window" {
      for_each = local.cluster_maintenance_window_is_daily
      content {
        start_time = local.merged_master_inputs.maintenance_start_time
      }
    }

    dynamic "maintenance_exclusion" {
      for_each = local.merged_master_inputs.maintenance_exclusions
      content {
        exclusion_name = maintenance_exclusion.value.name
        start_time     = maintenance_exclusion.value.start_time
        end_time       = maintenance_exclusion.value.end_time
      }
    }
  }
}

And the plan:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.gke_c1.google_container_cluster.gke_master has been changed
  ~ resource "google_container_cluster" "gke_master" {
        id                          = "projects/my-project-id/locations/europe-west1/clusters/tfl-prod-main"
        name                        = "tfl-prod-main"
        # (29 unchanged attributes hidden)

      ~ maintenance_policy {

          + maintenance_exclusion {
              + end_time       = "2021-07-19T00:00:00Z"
              + exclusion_name = "part1"
              + start_time     = "2021-07-03T00:00:00Z"
            }
          + maintenance_exclusion {
              + end_time       = "2021-08-12T00:00:00Z"
              + exclusion_name = "part2"
              + start_time     = "2021-07-31T00:00:00Z"
            }
          + maintenance_exclusion {
              + end_time       = "2021-08-24T00:00:00Z"
              + exclusion_name = "part3"
              + start_time     = "2021-08-12T01:00:00Z"
            }
            # (1 unchanged block hidden)
        }

        # (15 unchanged blocks hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.gke_c1.google_container_cluster.gke_master will be updated in-place
  ~ resource "google_container_cluster" "gke_master" {
        id                          = "projects/my-project-id/locations/europe-west1/clusters/tfl-prod-main"
        name                        = "tfl-prod-main"
        # (29 unchanged attributes hidden)

      ~ maintenance_policy {

          - maintenance_exclusion {
              - end_time       = "2021-07-19T00:00:00Z" -> null
              - exclusion_name = "part1" -> null
              - start_time     = "2021-07-03T00:00:00Z" -> null
            }
          - maintenance_exclusion {
              - end_time       = "2021-08-12T00:00:00Z" -> null
              - exclusion_name = "part2" -> null
              - start_time     = "2021-07-31T00:00:00Z" -> null
            }
          - maintenance_exclusion {
              - end_time       = "2021-08-24T00:00:00Z" -> null
              - exclusion_name = "part3" -> null
              - start_time     = "2021-08-12T01:00:00Z" -> null
            }
            # (1 unchanged block hidden)
        }

        # (15 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Expected Behavior

When removing maintenance exclusions windows from my configuration, they should also disappear from the GKE resource, and not still being visible in the console and constantly reappearing when i do a plan.

Actual Behavior

Maintenance exclusions are removed from the state, but not from the GKE cluster, so they still appear as external addition in the plan, and the plan wants to delete them again ...

Steps to Reproduce

1.1 Create a GKE cluster.
1.2 Add maintenance exclusions to this cluster using Terraform.
2.1 Remove these maintenance exclusions from your configuration.
2.2 Run terraform plan, it says it will be removing the maintenance exclusions.
2.3 Run terraform apply. Your maintenance exclusions are removed from the state, but not from the GKE cluster.
3.1 Run terraform plan again, Terraform detects the maintenance exclusions that still exists on the GKE cluster as external changes and wants to add them back ...

References

• Maintenance exclusion is not removed from google_container_cluster #8566

I guess the main difference with the issue referenced above is the usage of for_each in the maintenance_exclusion dynamic block.
Looking at the terraform-google-kubernetes-engine module i see they use the same logic for maintenance_exclusion.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.