gke cluster module isn't idempotent

[ideasculptor]

I have a template that calls the gke-cluster module. No matter what I put for variables in the template, every time I run terragrunt apply, it tells me it needs to destroy the old cluster and create a new one.

  # module.gke_cluster.google_container_cluster.cluster must be replaced
-/+ resource "google_container_cluster" "cluster" {
      ~ additional_zones            = [] -> (known after apply)
      ~ cluster_autoscaling         = [] -> (known after apply)
      ~ cluster_ipv4_cidr           = "10.4.0.0/16" -> (known after apply)
        description                 = "GKE cluster for mgmt environment"
      + enable_binary_authorization = (known after apply)
        enable_kubernetes_alpha     = false
        enable_legacy_abac          = false
      + enable_tpu                  = (known after apply)
      ~ endpoint                    = "{redacted}" -> (known after apply)
      ~ id                          = "{redacted}" -> (known after apply)
        initial_node_count          = 1
      ~ instance_group_urls         = [
          - "https://www.googleapis.com/compute/v1/projects/{redacted}/zones/us-west1-c/instanceGroups/gke-{redacted}-private-pool-4aab0243-grp",
        ] -> (known after apply)
      ~ ip_allocation_policy        = [
          ~ {
              ~ cluster_ipv4_cidr_block       = "10.4.0.0/16" -> (known after apply)
                cluster_secondary_range_name  = "public-services"
              ~ create_subnetwork             = false -> null
              ~ node_ipv4_cidr_block          = "10.0.0.0/16" -> (known after apply)
              ~ services_ipv4_cidr_block      = "10.4.0.0/16" -> (known after apply)
                services_secondary_range_name = "public-services"
              ~ subnetwork_name               = "" -> null
                use_ip_aliases                = true
            },
        ]
        location                    = "us-west1-c"
        logging_service             = "logging.googleapis.com/kubernetes"
      ~ master_version              = "1.13.7-gke.24" -> (known after apply)
        min_master_version          = "1.13.7-gke.24"
        monitoring_service          = "monitoring.googleapis.com/kubernetes"
        name                        = "{redacted}"
      ~ network                     = "projects/{redacted}/global/networks/mgmt-network" -> "https://www.googleapis.com/compute/v1/{redacted}/global/networks/mgmt-network"
      ~ node_locations              = [] -> (known after apply)
      ~ node_version                = "1.13.7-gke.24" -> (known after apply)
        project                     = "{redacted}"
      + region                      = (known after apply)
        remove_default_node_pool    = true
      - resource_labels             = {} -> null
      ~ subnetwork                  = "projects/{redacted}/regions/us-west1/subnetworks/mgmt-subnetwork-public" -> "https://www.googleapis.com/compute/v1/projects/{redacted}/regions/us-west1/subnetworks/mgmt-subnetwork-public"
      ~ zone                        = "us-west1-c" -> (known after apply)

        addons_config {
            horizontal_pod_autoscaling {
                disabled = false
            }

            http_load_balancing {
                disabled = false
            }

            kubernetes_dashboard {
                disabled = true
            }

            network_policy_config {
                disabled = false
            }
        }

      ~ maintenance_policy {
          ~ daily_maintenance_window {
              ~ duration   = "PT4H0M0S" -> (known after apply)
                start_time = "05:00"
            }
        }

      ~ master_auth {
          + client_certificate     = (known after apply)
          + client_key             = (sensitive value)
          ~ cluster_ca_certificate = "{redacted}" -> (known after apply)

            client_certificate_config {
                issue_client_certificate = false
            }
        }

        master_authorized_networks_config {
            cidr_blocks {
                cidr_block   = "0.0.0.0/0"
                display_name = "all-for-testing"
            }
        }

        network_policy {
            enabled  = true
            provider = "CALICO"
        }

      ~ node_config {
          ~ disk_size_gb      = 10 -> (known after apply)
          ~ disk_type         = "pd-standard" -> (known after apply)
          ~ guest_accelerator = [] -> (known after apply)
          ~ image_type        = "COS" -> (known after apply)
          - labels            = {
              - "private-pool-example" = "true"
            } -> null # forces replacement
          ~ local_ssd_count   = 0 -> (known after apply)
          ~ machine_type      = "n1-standard-1" -> (known after apply)
          ~ metadata          = {
              - "disable-legacy-endpoints" = "true"
            } -> (known after apply)
          ~ oauth_scopes      = [
              - "https://www.googleapis.com/auth/cloud-platform",
            ] -> (known after apply)
            preemptible       = false
            service_account   = "{redacted}@{redacted}.iam.gserviceaccount.com"
          - tags              = [
              - "private",
              - "private-pool-example",
            ] -> null # forces replacement
        }

      ~ node_pool {
          ~ initial_node_count  = 1 -> (known after apply)
          ~ instance_group_urls = [
              - "https://www.googleapis.com/compute/v1/projects/{redacted}/zones/us-west1-c/instanceGroupManagers/gke-{redacted}-private-pool-4aab0243-grp",
            ] -> (known after apply)
          ~ max_pods_per_node   = 0 -> (known after apply)
          ~ name                = "private-pool" -> (known after apply)
          + name_prefix         = (known after apply)
          ~ node_count          = 1 -> (known after apply)
          ~ version             = "1.13.7-gke.24" -> (known after apply)

          ~ autoscaling {
              ~ max_node_count = 5 -> (known after apply)
              ~ min_node_count = 1 -> (known after apply)
            }

          ~ management {
              ~ auto_repair  = true -> (known after apply)
              ~ auto_upgrade = true -> (known after apply)
            }

          ~ node_config {
              ~ disk_size_gb      = 10 -> (known after apply)
              ~ disk_type         = "pd-standard" -> (known after apply)
              ~ guest_accelerator = [] -> (known after apply)
              ~ image_type        = "COS" -> (known after apply)
              ~ labels            = {
                  - "private-pool-example" = "true"
                } -> (known after apply)
              ~ local_ssd_count   = 0 -> (known after apply)
              ~ machine_type      = "n1-standard-1" -> (known after apply)
              ~ metadata          = {
                  - "disable-legacy-endpoints" = "true"
                } -> (known after apply)
              + min_cpu_platform  = (known after apply)
              ~ oauth_scopes      = [
                  - "https://www.googleapis.com/auth/cloud-platform",
                ] -> (known after apply)
              ~ preemptible       = false -> (known after apply)
              ~ service_account   = "{redacted}@{redacted}.iam.gserviceaccount.com" -> (known after apply)
              ~ tags              = [
                  - "private",
                  - "private-pool-example",
                ] -> (known after apply)

              + taint {
                  + effect = (known after apply)
                  + key    = (known after apply)
                  + value  = (known after apply)
                }

              + workload_metadata_config {
                  + node_metadata = (known after apply)
                }
            }
        }

      ~ private_cluster_config {
          - enable_private_endpoint = false -> null
            enable_private_nodes    = true
            master_ipv4_cidr_block  = "10.2.255.0/28"
          ~ private_endpoint        = "10.2.255.2" -> (known after apply)
          ~ public_endpoint         = "{redacted}" -> (known after apply)
        }
    }

It seems that, if the plan is to be believed, that the tags on the default node pool must
be explicitly set to the same values as the tags in non-default node pool, so that subsequent runs don't require updates. I haven't tested it via a fork yet.

The module forced a new cluster when I specified no node_pool outside of the module AND when I specified a node_pool outside the module. I haven't tried a node_pool with no tags, because the network won't work for that, since it needs a 'private' tag, at minimum.

[ideasculptor]

I tried 2 node_pools, just to see what would happen, and it still forces a destroy -> create with every run. I'd love to know how to make it so that the module is stable from run to run.

[yorinasub17]

Do you have alternative_default_service_account set by chance?

[yorinasub17]

Nvmd opened #60 with the fix.

[yorinasub17]

This should be fixed in https://github.com/gruntwork-io/terraform-google-gke/releases/tag/v0.3.5

[ideasculptor]

Awesome. Thanks. So fast, I didn't even see the response before it was closed!