Separating GKE Standard and Autopilot Modules

FABRIC-AND-CFT.md

@@ -161,4 +161,4 @@ Even with all the above points, it may be hard to make a decision. While the mod
 
 * Since modules work well together within their ecosystem, select logical boundaries for using Fabric or CFT. For example use CFT for deploying resources within projects but use Fabric for managing project creation and IAM.
 * Use strengths of each collection of modules to your advantage. Empower application teams to define their infrastructure as code using off the shelf CFT modules. Using Fabric, bootstrap your platform team with a collection of tailor built modules for your organization.
-* Lean into module composition and dependency inversion that both Fabric and CFT modules follow. For example, you can create a GKE cluster using either [Fabric](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/modules/gke-cluster#gke-cluster-module) or [CFT](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine) GKE module and then use either [Fabric](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/modules/gke-hub#variables) or [CFT](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/master/modules/fleet-membership) for setting up GKE Hub by passing in outputs from the GKE module.
+* Lean into module composition and dependency inversion that both Fabric and CFT modules follow. For example, you can create a GKE cluster using either [Fabric](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/modules/gke-cluster-standard#gke-cluster-module) or [CFT](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine) GKE module and then use either [Fabric](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/modules/gke-hub#variables) or [CFT](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/master/modules/fleet-membership) for setting up GKE Hub by passing in outputs from the GKE module.

README.md

@@ -31,7 +31,7 @@ Currently available modules:
 
 - **foundational** - [billing budget](./modules/billing-budget), [Cloud Identity group](./modules/cloud-identity-group/), [folder](./modules/folder), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [organization](./modules/organization), [project](./modules/project), [projects-data-source](./modules/projects-data-source)
 - **networking** - [DNS](./modules/dns), [DNS Response Policy](./modules/dns-response-policy/), [Cloud Endpoints](./modules/endpoints), [address reservation](./modules/net-address), [NAT](./modules/net-cloudnat), [Global Load Balancer (classic)](./modules/net-glb/), [L4 ILB](./modules/net-ilb), [L7 ILB](./modules/net-ilb-l7), [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC firewall policy](./modules/net-vpc-firewall-policy), [VPC peering](./modules/net-vpc-peering), [VPN dynamic](./modules/net-vpn-dynamic), [HA VPN](./modules/net-vpn-ha), [VPN static](./modules/net-vpn-static), [Service Directory](./modules/service-directory)
-- **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid), [GKE cluster](./modules/gke-cluster), [GKE hub](./modules/gke-hub), [GKE nodepool](./modules/gke-nodepool)
+- **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid), [GKE cluster](./modules/gke-cluster-standard), [GKE hub](./modules/gke-hub), [GKE nodepool](./modules/gke-nodepool)
 - **data** - [BigQuery dataset](./modules/bigquery-dataset), [Bigtable instance](./modules/bigtable-instance), [Cloud SQL instance](./modules/cloudsql-instance), [Data Catalog Policy Tag](./modules/data-catalog-policy-tag), [Datafusion](./modules/datafusion), [Dataproc](./modules/dataproc), [GCS](./modules/gcs), [Pub/Sub](./modules/pubsub)
 - **development** - [API Gateway](./modules/api-gateway), [Apigee](./modules/apigee), [Artifact Registry](./modules/artifact-registry), [Container Registry](./modules/container-registry), [Cloud Source Repository](./modules/source-repository)
 - **security** - [Binauthz](./modules/binauthz/), [KMS](./modules/kms), [SecretManager](./modules/secret-manager), [VPC Service Control](./modules/vpc-sc)

blueprints/apigee/hybrid-gke/gke.tf

@@ -15,7 +15,7 @@
  */
 
 module "cluster" {
-  source     = "../../../modules/gke-cluster"
+  source     = "../../../modules/gke-cluster-standard"
   project_id = module.project.project_id
   name       = "cluster"
   location   = var.region

blueprints/cloud-operations/network-dashboard/src/main.py

@@ -80,8 +80,9 @@ def do_discovery(resources):
           resources[result.type][result.id][result.key] = result.data
         else:
           resources[result.type][result.id] = result.data
-  LOGGER.info('discovery end {}'.format(
-      {k: len(v) for k, v in resources.items() if not isinstance(v, str)}))
+  LOGGER.info('discovery end {}'.format({
+      k: len(v) for k, v in resources.items() if not isinstance(v, str)
+  }))
 
 
 def do_init(resources, discovery_root, monitoring_project, folders=None,

blueprints/gke/autopilot/cluster.tf

@@ -15,7 +15,7 @@
  */
 
 module "cluster" {
-  source     = "../../../modules/gke-cluster"
+  source     = "../../../modules/gke-cluster-autopilot"
   project_id = module.project.project_id
   name       = "cluster"
   location   = var.region
@@ -29,18 +29,18 @@ module "cluster" {
     master_authorized_ranges = var.cluster_network_config.master_authorized_cidr_blocks
     master_ipv4_cidr_block   = var.cluster_network_config.master_cidr_block
   }
-  enable_features = {
-    autopilot = true
-  }
-  monitoring_config = {
-    enenable_components = ["SYSTEM_COMPONENTS"]
-    managed_prometheus  = true
-  }
-  cluster_autoscaling = {
-    auto_provisioning_defaults = {
-      service_account = module.node_sa.email
-    }
-  }
+  # enable_features = {
+  #   autopilot = true
+  # }
+  # monitoring_config = {
+  #   enenable_components = ["SYSTEM_COMPONENTS"]
+  #   managed_prometheus  = true
+  # }
+  # cluster_autoscaling = {
+  #   auto_provisioning_defaults = {
+  #     service_account = module.node_sa.email
+  #   }
+  # }
   release_channel = "RAPID"
   depends_on = [
     module.project

blueprints/gke/binauthz/main.tf

@@ -83,7 +83,7 @@ module "nat" {
 }
 
 module "cluster" {
-  source     = "../../../modules/gke-cluster"
+  source     = "../../../modules/gke-cluster-standard"
   project_id = module.project.project_id
   name       = "${var.prefix}-cluster"
   location   = var.zone

blueprints/gke/multi-cluster-mesh-gke-fleet-api/README.md

@@ -53,7 +53,7 @@ Once done testing, you can clean up resources by running `terraform destroy`.
 | name | description | modules | resources |
 |---|---|---|---|
 | [ansible.tf](./ansible.tf) | Ansible generated files. |  | <code>local_file</code> |
-| [gke.tf](./gke.tf) | GKE cluster and hub resources. | <code>gke-cluster</code> · <code>gke-hub</code> · <code>gke-nodepool</code> |  |
+| [gke.tf](./gke.tf) | GKE cluster and hub resources. | <code>gke-cluster-standard</code> · <code>gke-hub</code> · <code>gke-nodepool</code> |  |
 | [main.tf](./main.tf) | Project resources. | <code>project</code> |  |
 | [variables.tf](./variables.tf) | Module variables. |  |  |
 | [vm.tf](./vm.tf) | Management server. | <code>compute-vm</code> |  |
@@ -75,7 +75,6 @@ Once done testing, you can clean up resources by running `terraform destroy`.
 | [region](variables.tf#L99) | Region. | <code>string</code> |  | <code>&#34;europe-west1&#34;</code> |
 
 <!-- END TFDOC -->
-
 ## Test
 
 ```hcl

blueprints/gke/multi-cluster-mesh-gke-fleet-api/gke.tf

@@ -18,7 +18,7 @@
 
 module "clusters" {
   for_each   = var.clusters_config
-  source     = "../../../modules/gke-cluster"
+  source     = "../../../modules/gke-cluster-standard"
   project_id = module.fleet_project.project_id
   name       = each.key
   location   = var.region

blueprints/gke/multitenant-fleet/README.md

@@ -234,7 +234,7 @@ module "gke" {
 
 | name | description | modules |
 |---|---|---|
-| [gke-clusters.tf](./gke-clusters.tf) | GKE clusters. | <code>gke-cluster</code> |
+| [gke-clusters.tf](./gke-clusters.tf) | GKE clusters. | <code>gke-cluster-standard</code> |
 | [gke-hub.tf](./gke-hub.tf) | GKE hub configuration. | <code>gke-hub</code> |
 | [gke-nodepools.tf](./gke-nodepools.tf) | GKE nodepools. | <code>gke-nodepool</code> |
 | [main.tf](./main.tf) | Project and usage dataset. | <code>bigquery-dataset</code> · <code>project</code> |

blueprints/gke/multitenant-fleet/gke-clusters.tf

@@ -17,7 +17,7 @@
 # tfdoc:file:description GKE clusters.
 
 module "gke-cluster" {
-  source                   = "../../../modules/gke-cluster"
+  source                   = "../../../modules/gke-cluster-standard"
   for_each                 = var.clusters
   name                     = each.key
   project_id               = module.gke-project-0.project_id

blueprints/networking/hub-and-spoke-peering/main.tf

@@ -240,7 +240,7 @@ module "service-account-gce" {
 ################################################################################
 
 module "cluster-1" {
-  source     = "../../../modules/gke-cluster"
+  source     = "../../../modules/gke-cluster-standard"
   name       = "${var.prefix}-cluster-1"
   project_id = module.project.project_id
   location   = "${var.region}-b"

blueprints/networking/shared-vpc-gke/main.tf

@@ -197,7 +197,7 @@ module "vm-bastion" {
 ################################################################################
 
 module "cluster-1" {
-  source     = "../../../modules/gke-cluster"
+  source     = "../../../modules/gke-cluster-standard"
   count      = var.cluster_create ? 1 : 0
   name       = "cluster-1"
   project_id = module.project-svc-gke.project_id

modules/README.md

@@ -63,7 +63,8 @@ These modules are used in the examples included in this repository. If you are u
 - [VM/VM group](./compute-vm)
 - [MIG](./compute-mig)
 - [COS container](./cloud-config-container/cos-generic-metadata/) (coredns/mysql/nva/onprem/squid)
-- [GKE cluster](./gke-cluster)
+- [GKE autopilot cluster](./gke-cluster-autopilot)
+- [GKE standard cluster](./gke-cluster-standard)
 - [GKE hub](./gke-hub)
 - [GKE nodepool](./gke-nodepool)
 

modules/gke-cluster-autopilot/README.md

@@ -0,0 +1,132 @@
+# GKE cluster Autopilot  module
+
+This module allows simplified creation and management of GKE Autopilot clusters. Some sensible defaults are set initially, in order to allow less verbose usage for most use cases.
+
+## Example
+
+### GKE Cluster
+
+```hcl
+module "cluster-1" {
+  source     = "./fabric/modules/gke-cluster-autopilot"
+  project_id = "myproject"
+  name       = "cluster-1"
+  location   = "europe-west1"
+  vpc_config = {
+    network    = var.vpc.self_link
+    subnetwork = var.subnet.self_link
+    secondary_range_names = {
+      pods     = "pods"
+      services = "services"
+    }
+    master_authorized_ranges = {
+      internal-vms = "10.0.0.0/8"
+    }
+    master_ipv4_cidr_block = "192.168.0.0/28"
+  }
+  private_cluster_config = {
+    enable_private_endpoint = true
+    master_global_access    = false
+  }
+  labels = {
+    environment = "dev"
+  }
+}
+# tftest modules=1 resources=1 inventory=basic.yaml
+```
+
+
+### Cloud DNS
+
+This example shows how to [use Cloud DNS as a Kubernetes DNS provider](https://cloud.google.com/kubernetes-engine/docs/how-to/cloud-dns) for GKE Standard clusters.
+
+```hcl
+module "cluster-1" {
+  source     = "./fabric/modules/gke-cluster-autopilot"
+  project_id = var.project_id
+  name       = "cluster-1"
+  location   = "europe-west1"
+  vpc_config = {
+    network               = var.vpc.self_link
+    subnetwork            = var.subnet.self_link
+    secondary_range_names = { pods = "pods", services = "services" }
+  }
+  enable_features = {
+    dns = {
+      provider = "CLOUD_DNS"
+      scope    = "CLUSTER_SCOPE"
+      domain   = "gke.local"
+    }
+  }
+}
+# tftest modules=1 resources=1 inventory=dns.yaml
+```
+
+
+### Backup for GKE
+
+This example shows how to [enable the Backup for GKE agent and configure a Backup Plan](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/concepts/backup-for-gke) for GKE Standard clusters.
+
+```hcl
+module "cluster-1" {
+  source     = "./fabric/modules/gke-cluster-autopilot"
+  project_id = var.project_id
+  name       = "cluster-1"
+  location   = "europe-west1"
+  vpc_config = {
+    network               = var.vpc.self_link
+    subnetwork            = var.subnet.self_link
+    secondary_range_names = { pods = "pods", services = "services" }
+  }
+  backup_configs = {
+    enable_backup_agent = true
+    backup_plans = {
+      "backup-1" = {
+        region   = "europe-west-2"
+        schedule = "0 9 * * 1"
+      }
+    }
+  }
+}
+# tftest modules=1 resources=2 inventory=backup.yaml
+```
+<!-- BEGIN TFDOC -->
+
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---:|:---:|:---:|
+| [location](variables.tf#L106) | Autopilot cluster are always regional. | <code>string</code> | ✓ |  |
+| [name](variables.tf#L141) | Cluster name. | <code>string</code> | ✓ |  |
+| [project_id](variables.tf#L167) | Cluster project id. | <code>string</code> | ✓ |  |
+| [vpc_config](variables.tf#L190) | VPC-level configuration. | <code title="object&#40;&#123;&#10;  network                &#61; string&#10;  subnetwork             &#61; string&#10;  master_ipv4_cidr_block &#61; optional&#40;string&#41;&#10;  secondary_range_blocks &#61; optional&#40;object&#40;&#123;&#10;    pods     &#61; string&#10;    services &#61; string&#10;  &#125;&#41;&#41;&#10;  secondary_range_names &#61; optional&#40;object&#40;&#123;&#10;    pods     &#61; string&#10;    services &#61; string&#10;  &#125;&#41;, &#123; pods &#61; &#34;pods&#34;, services &#61; &#34;services&#34; &#125;&#41;&#10;  master_authorized_ranges &#61; optional&#40;map&#40;string&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ |  |
+| [backup_configs](variables.tf#L17) | Configuration for Backup for GKE. | <code title="object&#40;&#123;&#10;  enable_backup_agent &#61; optional&#40;bool, false&#41;&#10;  backup_plans &#61; optional&#40;map&#40;object&#40;&#123;&#10;    region                            &#61; string&#10;    schedule                          &#61; string&#10;    retention_policy_days             &#61; optional&#40;string&#41;&#10;    retention_policy_lock             &#61; optional&#40;bool, false&#41;&#10;    retention_policy_delete_lock_days &#61; optional&#40;string&#41;&#10;  &#125;&#41;&#41;, &#123;&#125;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>&#123;&#125;</code> |
+| [description](variables.tf#L33) | Cluster description. | <code>string</code> |  | <code>null</code> |
+| [enable_addons](variables.tf#L39) | Addons enabled in the cluster (true means enabled). | <code title="object&#40;&#123;&#10;  cloudrun                   &#61; optional&#40;bool, false&#41;&#10;  config_connector           &#61; optional&#40;bool, false&#41;&#10;  dns_cache                  &#61; optional&#40;bool, false&#41;&#10;  horizontal_pod_autoscaling &#61; optional&#40;bool, false&#41;&#10;  http_load_balancing        &#61; optional&#40;bool, false&#41;&#10;  istio &#61; optional&#40;object&#40;&#123;&#10;    enable_tls &#61; bool&#10;  &#125;&#41;&#41;&#10;  kalm           &#61; optional&#40;bool, false&#41;&#10;  network_policy &#61; optional&#40;bool, false&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  horizontal_pod_autoscaling &#61; true&#10;  http_load_balancing        &#61; true&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [enable_features](variables.tf#L60) | Enable cluster-level features. Certain features allow configuration. | <code title="object&#40;&#123;&#10;  binary_authorization &#61; optional&#40;bool, false&#41;&#10;  dns &#61; optional&#40;object&#40;&#123;&#10;    provider &#61; optional&#40;string&#41;&#10;    scope    &#61; optional&#40;string&#41;&#10;    domain   &#61; optional&#40;string&#41;&#10;  &#125;&#41;&#41;&#10;  database_encryption &#61; optional&#40;object&#40;&#123;&#10;    state    &#61; string&#10;    key_name &#61; string&#10;  &#125;&#41;&#41;&#10;  gateway_api         &#61; optional&#40;bool, false&#41;&#10;  groups_for_rbac     &#61; optional&#40;string&#41;&#10;  l4_ilb_subsetting   &#61; optional&#40;bool, false&#41;&#10;  mesh_certificates   &#61; optional&#40;bool&#41;&#10;  pod_security_policy &#61; optional&#40;bool, false&#41;&#10;  resource_usage_export &#61; optional&#40;object&#40;&#123;&#10;    dataset                              &#61; string&#10;    enable_network_egress_metering       &#61; optional&#40;bool&#41;&#10;    enable_resource_consumption_metering &#61; optional&#40;bool&#41;&#10;  &#125;&#41;&#41;&#10;  tpu &#61; optional&#40;bool, false&#41;&#10;  upgrade_notifications &#61; optional&#40;object&#40;&#123;&#10;    topic_id &#61; optional&#40;string&#41;&#10;  &#125;&#41;&#41;&#10;  vertical_pod_autoscaling &#61; optional&#40;bool, false&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;&#10;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [issue_client_certificate](variables.tf#L94) | Enable issuing client certificate. | <code>bool</code> |  | <code>false</code> |
+| [labels](variables.tf#L100) | Cluster resource labels. | <code>map&#40;string&#41;</code> |  | <code>null</code> |
+| [maintenance_config](variables.tf#L112) | Maintenance window configuration. | <code title="object&#40;&#123;&#10;  daily_window_start_time &#61; optional&#40;string&#41;&#10;  recurring_window &#61; optional&#40;object&#40;&#123;&#10;    start_time &#61; string&#10;    end_time   &#61; string&#10;    recurrence &#61; string&#10;  &#125;&#41;&#41;&#10;  maintenance_exclusions &#61; optional&#40;list&#40;object&#40;&#123;&#10;    name       &#61; string&#10;    start_time &#61; string&#10;    end_time   &#61; string&#10;    scope      &#61; optional&#40;string&#41;&#10;  &#125;&#41;&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  daily_window_start_time &#61; &#34;03:00&#34;&#10;  recurring_window        &#61; null&#10;  maintenance_exclusion   &#61; &#91;&#93;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [min_master_version](variables.tf#L135) | Minimum version of the master, defaults to the version of the most recent official release. | <code>string</code> |  | <code>null</code> |
+| [node_locations](variables.tf#L146) | Zones in which the cluster's nodes are located. | <code>list&#40;string&#41;</code> |  | <code>&#91;&#93;</code> |
+| [private_cluster_config](variables.tf#L153) | Private cluster configuration. | <code title="object&#40;&#123;&#10;  enable_private_endpoint &#61; optional&#40;bool&#41;&#10;  master_global_access    &#61; optional&#40;bool&#41;&#10;  peering_config &#61; optional&#40;object&#40;&#123;&#10;    export_routes &#61; optional&#40;bool&#41;&#10;    import_routes &#61; optional&#40;bool&#41;&#10;    project_id    &#61; optional&#40;string&#41;&#10;  &#125;&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>null</code> |
+| [release_channel](variables.tf#L172) | Release channel for GKE upgrades. | <code>string</code> |  | <code>null</code> |
+| [service_account](variables.tf#L178) | The Google Cloud Platform Service Account to be used by the node VMs created by GKE Autopilot. | <code>string</code> |  | <code>null</code> |
+| [tags](variables.tf#L184) | Network tags applied to nodes. | <code>list&#40;string&#41;</code> |  | <code>null</code> |
+
+## Outputs
+
+| name | description | sensitive |
+|---|---|:---:|
+| [ca_certificate](outputs.tf#L17) | Public certificate of the cluster (base64-encoded). | ✓ |
+| [cluster](outputs.tf#L23) | Cluster resource. | ✓ |
+| [endpoint](outputs.tf#L29) | Cluster endpoint. |  |
+| [id](outputs.tf#L34) | Cluster ID. |  |
+| [location](outputs.tf#L39) | Cluster location. |  |
+| [master_version](outputs.tf#L44) | Master version. |  |
+| [name](outputs.tf#L49) | Cluster name. |  |
+| [notifications](outputs.tf#L54) | GKE PubSub notifications topic. |  |
+| [self_link](outputs.tf#L59) | Cluster self link. | ✓ |
+| [workload_identity_pool](outputs.tf#L65) | Workload identity pool. |  |
+
+<!-- END TFDOC -->