Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform 1.3.7 removes experimental optional() variable - breaking tf apply on shell container but not cloud build container running 1.0.10 - without preset optional values for "Error: Optional object type attributes are experimental" - cloudbuild dockerfile move to 1.5.2 and remove experimental code #224

Closed
obriensystems opened this issue Jan 30, 2023 · 11 comments

Comments

@obriensystems
Copy link
Collaborator

obriensystems commented Jan 30, 2023

20231014: see 1.0 specific issue that needs a 1.5 upgrade as a first step
hashicorp/terraform-provider-google#16217
and
#264

We will need to look at upgrading from 1.0 to 1.5.7 or the new 1.6.0 https://releases.hashicorp.com/terraform/1.6.0/ as a first step in the https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding codebase

20230724: Some private forks of this repo are working around the 1.3.7 issue by downgrading the client.
While this is an ok fix and allows the initial bootstrap.sh to proceed - it does not fix the underlying issue that the code is only 1.0.10 compliant.
The full fix is to upgrade the tf files to terraform 1.5.x
This is in progress.

As part of working out perimeter/peering work in #220
Reproduction:

run any of the bootstrap, common, nprod, prod builds and you will fail on a currently built TF container.
An older one from July 2022 will still work
Getting last working terraform version - currently default to 1.0.10 in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/modules/cloudbuild/cloudbuild_builder/Dockerfile#L20

see
Fix hashicorp/terraform#31692 and hashicorp/terraform#31355 first

In prep of the Terraform 1.4.0 release - on 5 Jan 2023 TF 1.3.7 introduced the breaking change on experimental use of optional(). This only breaks our bootstrap step in the PBMM LZ V1 LZ and the GR. The fix is to remove the experimental declaration for bootstrap so we can run in gcloud/shell again. For the gitops cloud build jobs - I'll keep these hardcoded to TF 1.0.10 for now until I refactor the defaults - so CB jobs are unaffected.

Fix is only required for bootstrap (which runs outside the CB container directly on the gcloud shell (which runs 1.3.7), for all other common/non-prod/prod we can continue to use terraform 1.0.10 and experimental optional attributes

Fix terraform version in the GitOps docker container

ARG TERRAFORM_VERSION=1.0.10

all terraform.tf

terraform {
  #required_version = ">= 1.3.0"
  # comment below only in bootstrap for now
  experiments = [module_variable_optional_attrs]
}

for example usage in
tfstate_buckets = object({
      common = object(
        {
          name          = string
          labels        = optional(map(string)),

on new install to terraform.landing.systems

Screen Shot 2023-01-29 at 19 17 18

Screen Shot 2023-01-29 at 19 18 28

Step #0 - "tf init": - vpc-svc-ctl.regular_service_perimeter_names.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": There are some problems with the configuration, described below.
Step #0 - "tf init": 
Step #0 - "tf init": The Terraform configuration must be valid before initialization so that
Step #0 - "tf init": Terraform can determine which modules and providers need to be installed.
Step #0 - "tf init": ╷
Step #0 - "tf init": │ Error: Optional object type attributes are experimental
Step #0 - "tf init": │ 
Step #0 - "tf init": │   on variables.tf line 25:
Step #0 - "tf init": │   25: variable "prod_host_net" {
Step #0 - "tf init": │ 
Step #0 - "tf init": │ This feature is currently an opt-in experiment, subject to change in future
Step #0 - "tf init": │ releases based on feedback.
Step #0 - "tf init": │ 
Step #0 - "tf init": │ Activate the feature for this module by adding
Step #0 - "tf init": │ module_variable_optional_attrs to the list of active experiments.
Step #0 - "tf init": ╵
Step #0 - "tf init": 
Finished Step #0 - "tf init"
ERROR
ERROR: build step 0 "northamerica-northeast1-docker.pkg.dev/tspe-tls-tls-dv/tls-tf-runners/terraform" failed: step exited with non-zero status: 1

on existing install on cloudnuage.dev (last update was Nov 2023

Screen Shot 2023-01-29 at 19 15 01

Screen Shot 2023-01-29 at 19 15 13

starting build "deee51de-2631-4822-b0d0-937ffd61e3ee"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://source.developers.google.com/p/cnpe-cnd-cndev-sbx/r/cndevlzd
 * branch            29697e3dce82da64ba70399833e3c55095db95be -> FETCH_HEAD
HEAD is now at 29697e3 #208 - gr 12 marketplace
BUILD
Starting Step #0 - "tf init"
Step #0 - "tf init": Pulling image: northamerica-northeast1-docker.pkg.dev/cnpe-cnd-cndev-sbx/cndev-tf-runners/terraform
Step #0 - "tf init": Using default tag: latest
Step #0 - "tf init": latest: Pulling from cnpe-cnd-cndev-sbx/cndev-tf-runners/terraform
Step #0 - "tf init": 5ce9eba43f65: Pulling fs layer
Step #0 - "tf init": 811b74c92401: Pulling fs layer
Step #0 - "tf init": 3c2cba919283: Pulling fs layer
Step #0 - "tf init": c434b03bdf75: Pulling fs layer
Step #0 - "tf init": 26aaf0ef6bab: Pulling fs layer
Step #0 - "tf init": 1c3df16aadbc: Pulling fs layer
Step #0 - "tf init": d647853d6554: Pulling fs layer
Step #0 - "tf init": 9376e9bc837e: Pulling fs layer
Step #0 - "tf init": 8010cce88aee: Pulling fs layer
Step #0 - "tf init": dde4f7a3e07c: Pulling fs layer
Step #0 - "tf init": 81aed9c488c5: Pulling fs layer
Step #0 - "tf init": c434b03bdf75: Waiting
Step #0 - "tf init": 8010cce88aee: Waiting
Step #0 - "tf init": dde4f7a3e07c: Waiting
Step #0 - "tf init": 81aed9c488c5: Waiting
Step #0 - "tf init": 26aaf0ef6bab: Waiting
Step #0 - "tf init": 1c3df16aadbc: Waiting
Step #0 - "tf init": d647853d6554: Waiting
Step #0 - "tf init": 9376e9bc837e: Waiting
Step #0 - "tf init": 3c2cba919283: Verifying Checksum
Step #0 - "tf init": 3c2cba919283: Download complete
Step #0 - "tf init": 811b74c92401: Verifying Checksum
Step #0 - "tf init": 811b74c92401: Download complete
Step #0 - "tf init": 5ce9eba43f65: Verifying Checksum
Step #0 - "tf init": 5ce9eba43f65: Download complete
Step #0 - "tf init": 26aaf0ef6bab: Verifying Checksum
Step #0 - "tf init": 26aaf0ef6bab: Download complete
Step #0 - "tf init": 1c3df16aadbc: Verifying Checksum
Step #0 - "tf init": 1c3df16aadbc: Download complete
Step #0 - "tf init": 9376e9bc837e: Verifying Checksum
Step #0 - "tf init": 9376e9bc837e: Download complete
Step #0 - "tf init": 8010cce88aee: Download complete
Step #0 - "tf init": dde4f7a3e07c: Verifying Checksum
Step #0 - "tf init": dde4f7a3e07c: Download complete
Step #0 - "tf init": 81aed9c488c5: Download complete
Step #0 - "tf init": c434b03bdf75: Verifying Checksum
Step #0 - "tf init": c434b03bdf75: Download complete
Step #0 - "tf init": d647853d6554: Verifying Checksum
Step #0 - "tf init": d647853d6554: Download complete
Step #0 - "tf init": 5ce9eba43f65: Pull complete
Step #0 - "tf init": 811b74c92401: Pull complete
Step #0 - "tf init": 3c2cba919283: Pull complete
Step #0 - "tf init": c434b03bdf75: Pull complete
Step #0 - "tf init": 26aaf0ef6bab: Pull complete
Step #0 - "tf init": 1c3df16aadbc: Pull complete
Step #0 - "tf init": d647853d6554: Pull complete
Step #0 - "tf init": 9376e9bc837e: Pull complete
Step #0 - "tf init": 8010cce88aee: Pull complete
Step #0 - "tf init": dde4f7a3e07c: Pull complete
Step #0 - "tf init": 81aed9c488c5: Pull complete
Step #0 - "tf init": Digest: sha256:67df843a28021b6960cfd48fae37092fa9a7a44b5c6dfce7457aef4413af87a3
Step #0 - "tf init": Status: Downloaded newer image for northamerica-northeast1-docker.pkg.dev/cnpe-cnd-cndev-sbx/cndev-tf-runners/terraform:latest
Step #0 - "tf init": northamerica-northeast1-docker.pkg.dev/cnpe-cnd-cndev-sbx/cndev-tf-runners/terraform:latest
Step #0 - "tf init": 
Step #0 - "tf init": *************** TERRAFORM INIT ******************
Step #0 - "tf init": ******* At environment: environments/prod *************
Step #0 - "tf init": *************************************************
Step #0 - "tf init": Initializing modules...
Step #0 - "tf init": - firewall in ../../modules/firewall
Step #0 - "tf init": - firewall.allow_admins_name in ../../modules/naming-standard/modules/gcp/firewall_rule
Step #0 - "tf init": - firewall.allow_admins_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - firewall.allow_admins_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - firewall.allow_internal_name in ../../modules/naming-standard/modules/gcp/firewall_rule
Step #0 - "tf init": - firewall.allow_internal_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - firewall.allow_internal_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - firewall.allow_zone_internal_ingress_name in ../../modules/naming-standard/modules/gcp/firewall_rule
Step #0 - "tf init": - firewall.allow_zone_internal_ingress_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - firewall.allow_zone_internal_ingress_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - firewall.bastion_rule_name in ../../modules/naming-standard/modules/gcp/firewall_rule
Step #0 - "tf init": - firewall.bastion_rule_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - firewall.bastion_rule_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - firewall.custom_rules_names in ../../modules/naming-standard/modules/gcp/firewall_rule
Step #0 - "tf init": - firewall.custom_rules_names.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - firewall.custom_rules_names.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - firewall.deny_all_egress_name in ../../modules/naming-standard/modules/gcp/firewall_rule
Step #0 - "tf init": - firewall.deny_all_egress_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - firewall.deny_all_egress_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - firewall.iap_rules_names in ../../modules/naming-standard/modules/gcp/firewall_rule
Step #0 - "tf init": - firewall.iap_rules_names.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - firewall.iap_rules_names.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - net-host-prj in ../../modules/network-host-project
Step #0 - "tf init": - net-host-prj.network in ../../modules/network-host-project/modules/network
Step #0 - "tf init": - net-host-prj.network.router in ../../modules/network-host-project/modules/router
Step #0 - "tf init": - net-host-prj.network.router.router_name in ../../modules/naming-standard/modules/gcp/router
Step #0 - "tf init": - net-host-prj.network.router.router_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - net-host-prj.network.router.router_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - net-host-prj.network.router.vpn in ../../modules/network-host-project/modules/vpn
Step #0 - "tf init": - net-host-prj.network.router.vpn.ext_vpn_name in ../../modules/naming-standard/modules/gcp/external_vpn_name
Step #0 - "tf init": - net-host-prj.network.router.vpn.ext_vpn_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - net-host-prj.network.router.vpn.ext_vpn_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - net-host-prj.network.router.vpn.ha_vpn_name in ../../modules/naming-standard/modules/gcp/ha_vpn_name
Step #0 - "tf init": - net-host-prj.network.router.vpn.ha_vpn_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - net-host-prj.network.router.vpn.ha_vpn_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - net-host-prj.network.router.vpn.vpn_tunnel_name in ../../modules/naming-standard/modules/gcp/vpn_tunnel_name
Step #0 - "tf init": - net-host-prj.network.router.vpn.vpn_tunnel_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - net-host-prj.network.router.vpn.vpn_tunnel_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - net-host-prj.network.routes in ../../modules/network-host-project/modules/routes
Step #0 - "tf init": - net-host-prj.network.routes.route_name in ../../modules/naming-standard/modules/gcp/route
Step #0 - "tf init": - net-host-prj.network.routes.route_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - net-host-prj.network.subnets in ../../modules/network-host-project/modules/subnets
Step #0 - "tf init": - net-host-prj.network.subnets.subnet_name in ../../modules/naming-standard/modules/gcp/subnet
Step #0 - "tf init": - net-host-prj.network.subnets.subnet_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - net-host-prj.network.subnets.subnet_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - net-host-prj.network.vpc in ../../modules/network-host-project/modules/vpc
Step #0 - "tf init": - net-host-prj.network.vpc.network_name in ../../modules/naming-standard/modules/gcp/virtual_private_cloud
Step #0 - "tf init": - net-host-prj.network.vpc.network_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - net-host-prj.network.vpc.network_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - net-host-prj.network.vpc.peer in ../../modules/network-host-project/modules/network-peering
Step #0 - "tf init": - net-host-prj.project in ../../modules/project
Step #0 - "tf init": - net-host-prj.project.project_name in ../../modules/naming-standard/modules/gcp/project
Step #0 - "tf init": - net-host-prj.project.project_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - net-host-prj.project.project_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - vpc-svc-ctl in ../../modules/vpc-service-controls
Step #0 - "tf init": - vpc-svc-ctl.access_level in ../../modules/vpc-service-controls/modules/access_level
Step #0 - "tf init": - vpc-svc-ctl.access_list_names in ../../modules/naming-standard/modules/gcp/vpc_svc_ctl
Step #0 - "tf init": - vpc-svc-ctl.access_list_names.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - vpc-svc-ctl.access_list_names.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - vpc-svc-ctl.bridge_service_perimeter in ../../modules/vpc-service-controls/modules/bridge_service_perimeter
Step #0 - "tf init": - vpc-svc-ctl.bridge_service_perimeter_names in ../../modules/naming-standard/modules/gcp/vpc_svc_ctl
Step #0 - "tf init": - vpc-svc-ctl.bridge_service_perimeter_names.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - vpc-svc-ctl.bridge_service_perimeter_names.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - vpc-svc-ctl.policy_name in ../../modules/naming-standard/modules/gcp/vpc_svc_ctl
Step #0 - "tf init": - vpc-svc-ctl.policy_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - vpc-svc-ctl.policy_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": - vpc-svc-ctl.regular_service_perimeter in ../../modules/vpc-service-controls/modules/regular_service_perimeter
Step #0 - "tf init": - vpc-svc-ctl.regular_service_perimeter_names in ../../modules/naming-standard/modules/gcp/vpc_svc_ctl
Step #0 - "tf init": - vpc-svc-ctl.regular_service_perimeter_names.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
Step #0 - "tf init": - vpc-svc-ctl.regular_service_perimeter_names.name_generation in ../../modules/naming-standard/modules/common/name_generator
Step #0 - "tf init": 
Step #0 - "tf init": Initializing the backend...
Step #0 - "tf init": 
Step #0 - "tf init": Successfully configured the backend "gcs"! Terraform will automatically
Step #0 - "tf init": use this backend unless the backend configuration changes.
Step #0 - "tf init": 
Step #0 - "tf init": Initializing provider plugins...
Step #0 - "tf init": - terraform.io/builtin/terraform is built in to Terraform
Step #0 - "tf init": - Finding latest version of hashicorp/null...
Step #0 - "tf init": - Finding latest version of hashicorp/random...
Step #0 - "tf init": - Finding hashicorp/google versions matching ">= 2.12.0, < 4.0.0"...
Step #0 - "tf init": - Finding latest version of hashicorp/google-beta...
Step #0 - "tf init": - Installing hashicorp/random v3.4.3...
Step #0 - "tf init": - Installed hashicorp/random v3.4.3 (signed by HashiCorp)
Step #0 - "tf init": - Installing hashicorp/google v3.90.1...
Step #0 - "tf init": - Installed hashicorp/google v3.90.1 (signed by HashiCorp)
Step #0 - "tf init": - Installing hashicorp/google-beta v4.50.0...
Step #0 - "tf init": - Installed hashicorp/google-beta v4.50.0 (signed by HashiCorp)
Step #0 - "tf init": - Installing hashicorp/null v3.2.1...
Step #0 - "tf init": - Installed hashicorp/null v3.2.1 (signed by HashiCorp)
Step #0 - "tf init": 
Step #0 - "tf init": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #0 - "tf init": selections it made above. Include this file in your version control repository
Step #0 - "tf init": so that Terraform can guarantee to make the same selections by default when
Step #0 - "tf init": you run "terraform init" in the future.
Step #0 - "tf init": 
Step #0 - "tf init": ╷
Step #0 - "tf init": │ Warning: Experimental feature "module_variable_optional_attrs" is active
Step #0 - "tf init": │ 
Step #0 - "tf init": │   on terraform.tf line 7, in terraform:
Step #0 - "tf init": │    7:   experiments = [module_variable_optional_attrs]
Step #0 - "tf init": │ 
Step #0 - "tf init": │ Experimental features are subject to breaking changes in future minor or
Step #0 - "tf init": │ patch releases, based on feedback.
Step #0 - "tf init": │ 
Step #0 - "tf init": │ If you have feedback on the design of this feature, please open a GitHub
Step #0 - "tf init": │ issue to discuss it.
Step #0 - "tf init": │ 
Step #0 - "tf init": │ (and 6 more similar warnings elsewhere)
Step #0 - "tf init": ╵
Step #0 - "tf init": 
Step #0 - "tf init": Terraform has been successfully initialized!
Step #0 - "tf init": 
Step #0 - "tf init": You may now begin working with Terraform. Try running "terraform plan" to see
Step #0 - "tf init": any changes that are required for your infrastructure. All Terraform commands
Step #0 - "tf init": should now work.
Step #0 - "tf init": 
Step #0 - "tf init": If you ever set or change modules or backend configuration for Terraform,
Step #0 - "tf init": rerun this command to reinitialize your working directory. If you forget, other
Step #0 - "tf init": commands will detect it and remind you to do so if necessary.
Finished Step #0 - "tf init"
Starting Step #1 - "tf validate"
Step #1 - "tf validate": Already have image (with digest): northamerica-northeast1-docker.pkg.dev/cnpe-cnd-cndev-sbx/cndev-tf-runners/terraform
Step #1 - "tf validate": 
Step #1 - "tf validate": *************** TERRAFORM VALIDATE ******************
Step #1 - "tf validate": ******* At environment: environments/prod *************
Step #1 - "tf validate": *************************************************
ep #1 - "tf validate": ╷
Step #1 - "tf validate": │ Warning: Experimental feature "module_variable_optional_attrs" is active
Step #1 - "tf validate": │ 
Step #1 - "tf validate": │   on terraform.tf line 7, in terraform:
Step #1 - "tf validate": │    7:   experiments = [module_variable_optional_attrs]
Step #1 - "tf validate": │ 
Step #1 - "tf validate": │ Experimental features are subject to breaking changes in future minor or
Step #1 - "tf validate": │ patch releases, based on feedback.
Step #1 - "tf validate": │ 
Step #1 - "tf validate": │ If you have feedback on the design of this feature, please open a GitHub
Step #1 - "tf validate": │ issue to discuss it.
Step #1 - "tf validate": │ 
Step #1 - "tf validate": │ (and 6 more similar warnings elsewhere)
Step #1 - "tf validate": ╵
Step #1 - "tf validate": Success! The configuration is valid, but there were some
Step #1 - "tf validate": validation warnings as shown above.
Step #1 - "tf validate": 
Finished Step #1 - "tf validate"
Starting Step #2 - "tf plan"
Step #2 - "tf plan": Already have image (with digest): northamerica-northeast1-docker.pkg.dev/cnpe-cnd-cndev-sbx/cndev-tf-runners/terraform
Step #2 - "tf plan": 
Step #2 - "tf plan": *************** TERRAFORM PLAN ******************
Step #2 - "tf plan": ******* At environment: environments/prod *************
Step #2 - "tf plan": *************************************************

Step #2 - "tf plan": module.net-host-prj.module.project.google_project.project: Refreshing state... [id=projects/cnpe-cnd-cnprod-host2]
Step #2 - "tf plan": module.net-host-prj.module.project.google_project_iam_member.tf_sa_project_perms[4]: Refreshing state... [id=cnpe-cnd-cnprod-host2/roles/resourcemanager.projectIamAdmin/serviceAccount:terraform0627@cnpe-cnd-cndev-sbx.iam.gserviceaccount.com]
Step #2 - "tf plan": module.net-host-prj.module.project.google_project_iam_member.tf_sa_project_perms[2]: Refreshing state... [id=cnpe-cnd-cnprod-host2/roles/resourcemanager.projectMover/serviceAccount:terraform0627@cnpe-cnd-cndev-sbx.iam.gserviceaccount.com]
Step #2 - "tf plan": module.net-host-prj.module.project.google_project_service.project_services["logging.googleapis.com"]: Refreshing state... [id=cnpe-cnd-cnprod-host2/logging.googleapis.com]
Step #2 - "tf plan": module.net-host-prj.module.project.google_project_iam_member.tf_sa_project_perms[0]: Refreshing state... [id=cnpe-cnd-cnprod-host2/roles/source.admin/serviceAccount:terraform0627@cnpe-cnd-cndev-sbx.iam.gserviceaccount.com]
Step #2 - "tf plan": module.net-host-prj.module.project.google_project_iam_member.tf_sa_project_perms[3]: Refreshing state... [id=cnpe-cnd-cnprod-host2/roles/editor/serviceAccount:terraform0627@cnpe-cnd-cndev-sbx.iam.gserviceaccount.com]
Step #2 - "tf plan": module.net-host-prj.module.project.google_project_iam_member.tf_sa_project_perms[1]: Refreshing state... [id=cnpe-cnd-cnprod-host2/roles/secretmanager.secretAccessor/serviceAccount:terraform0627@cnpe-cnd-cndev-sbx.iam.gserviceaccount.com]
Step #2 - "tf plan": module.net-host-prj.module.network["prod-shvpc"].module.vpc.google_compute_network.network: Refreshing state... [id=projects/cnpe-cnd-cnprod-host2/global/networks/cnpecnr-prod-shvpc-vpc]
Step #2 - "tf plan": module.net-host-prj.module.network["prod-shvpc"].module.vpc.google_compute_shared_vpc_host_project.shared_vpc_host[0]: Refreshing state... [id=cnpe-cnd-cnprod-host2]
Step #2 - "tf plan": module.net-host-prj.module.network["prod-shvpc"].module.subnets["subnet01"].google_compute_subnetwork.subnetwork: Refreshing state... [id=projects/cnpe-cnd-cnprod-host2/regions/northamerica-northeast1/subnetworks/cnpecnr-subnet01-host2-snet]
Step #2 - "tf plan": module.firewall.google_compute_firewall.deny-all-egress: Refreshing state... [id=projects/cnpe-cnd-cnprod-host2/global/firewalls/cnpefwl-cnpecnr-prod-shvpc-vpc-deny-all-egress-fwr]
Step #2 - "tf plan": module.firewall.google_compute_firewall.bastion_rule[0]: Refreshing state... [id=projects/cnpe-cnd-cnprod-host2/global/firewalls/cnpefwl-cnpecnr-prod-shvpc-vpc-iap-bastian-ports-fwr]
Step #2 - "tf plan": module.firewall.google_compute_firewall.custom["allow-ssh-ingress"]: Refreshing state... [id=projects/cnpe-cnd-cnprod-host2/global/firewalls/cnpefwl-allow-ssh-ingress-fwr]
Step #2 - "tf plan": module.firewall.google_compute_firewall.custom["allow-egress-internet"]: Refreshing state... [id=projects/cnpe-cnd-cnprod-host2/global/firewalls/cnpefwl-allow-egress-internet-fwr]
Step #2 - "tf plan": module.vpc-svc-ctl.module.regular_service_perimeter["regular_service_perimeter_1"].google_access_context_manager_service_perimeter.regular_service_perimeter: Refreshing state... [id=accessPolicies/128275356538/servicePerimeters/cnpevsc_regular_service_perimeter_prod_1_vsc]
Step #2 - "tf plan": 
Step #2 - "tf plan": Terraform used the selected providers to generate the following execution
Step #2 - "tf plan": plan. Resource actions are indicated with the following symbols:
Step #2 - "tf plan":   ~ update in-place
Step #2 - "tf plan": 
Step #2 - "tf plan": Terraform will perform the following actions:
Step #2 - "tf plan": 
Step #2 - "tf plan":   # module.net-host-prj.module.project.google_project.project will be updated in-place
Step #2 - "tf plan":   ~ resource "google_project" "project" {
Step #2 - "tf plan":         id                  = "projects/cnpe-cnd-cnprod-host2"
Step #2 - "tf plan":       ~ labels              = {
Step #2 - "tf plan":           - "date_modified" = "2022-11-23"
Step #2 - "tf plan":         } -> (known after apply)
Step #2 - "tf plan":         name                = "CnPe-cnd-cnprod-host2"
Step #2 - "tf plan":         # (5 unchanged attributes hidden)
Step #2 - "tf plan":     }
Step #2 - "tf plan": 
Step #2 - "tf plan":   # module.net-host-prj.module.network["prod-shvpc"].module.subnets["subnet01"].google_compute_subnetwork.subnetwork will be updated in-place
Step #2 - "tf plan":   ~ resource "google_compute_subnetwork" "subnetwork" {
Step #2 - "tf plan":         id                         = "projects/cnpe-cnd-cnprod-host2/regions/northamerica-northeast1/subnetworks/cnpecnr-subnet01-host2-snet"
Step #2 - "tf plan":         name                       = "cnpecnr-subnet01-host2-snet"
Step #2 - "tf plan":         # (13 unchanged attributes hidden)
Step #2 - "tf plan": 
Step #2 - "tf plan":       ~ log_config {
Step #2 - "tf plan":           - metadata             = "EXCLUDE_ALL_METADATA" -> null
Step #2 - "tf plan":             # (4 unchanged attributes hidden)
Step #2 - "tf plan":         }
Step #2 - "tf plan":     }
Step #2 - "tf plan": 
Step #2 - "tf plan": Plan: 0 to add, 2 to change, 0 to destroy.
Step #2 - "tf plan": ╷
Step #2 - "tf plan": │ Warning: Experimental feature "module_variable_optional_attrs" is active
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │   on terraform.tf line 7, in terraform:
Step #2 - "tf plan": │    7:   experiments = [module_variable_optional_attrs]
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │ Experimental features are subject to breaking changes in future minor or
Step #2 - "tf plan": │ patch releases, based on feedback.
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │ If you have feedback on the design of this feature, please open a GitHub
Step #2 - "tf plan": │ issue to discuss it.
Step #2 - "tf plan": │ 
Step #2 - "tf plan": │ (and 6 more similar warnings elsewhere)
Step #2 - "tf plan": ╵
Step #2 - "tf plan": 
Step #2 - "tf plan": ─────────────────────────────────────────────────────────────────────────────
Step #2 - "tf plan": 
Step #2 - "tf plan": Saved the plan to: deee51de-2631-4822-b0d0-937ffd61e3ee_tfplan.plan
Step #2 - "tf plan": 
Step #2 - "tf plan": To perform exactly these actions, run the following command to apply:
Step #2 - "tf plan":     terraform apply "deee51de-2631-4822-b0d0-937ffd61e3ee_tfplan.plan"
Finished Step #2 - "tf plan"
Starting Step #3 - "tf apply"
Step #3 - "tf apply": Already have image (with digest): northamerica-northeast1-docker.pkg.dev/cnpe-cnd-cndev-sbx/cndev-tf-runners/terraform
Step #3 - "tf apply": 
Step #3 - "tf apply": *************** TERRAFORM APPLY ******************
Step #3 - "tf apply": ******* At environment: environments/prod ***********
Step #3 - "tf apply": *************************************************
tep #3 - "tf apply": module.net-host-prj.module.project.google_project.project: Modifying... [id=projects/cnpe-cnd-cnprod-host2]
Step #3 - "tf apply": module.net-host-prj.module.project.google_project.project: Modifications complete after 1s [id=projects/cnpe-cnd-cnprod-host2]
Step #3 - "tf apply": module.net-host-prj.module.network["prod-shvpc"].module.subnets["subnet01"].google_compute_subnetwork.subnetwork: Modifying... [id=projects/cnpe-cnd-cnprod-host2/regions/northamerica-northeast1/subnetworks/cnpecnr-subnet01-host2-snet]
Step #3 - "tf apply": module.net-host-prj.module.network["prod-shvpc"].module.subnets["subnet01"].google_compute_subnetwork.subnetwork: Modifications complete after 1s [id=projects/cnpe-cnd-cnprod-host2/regions/northamerica-northeast1/subnetworks/cnpecnr-subnet01-host2-snet]
Step #3 - "tf apply": ╷
Step #3 - "tf apply": │ Warning: Experimental feature "module_variable_optional_attrs" is active
Step #3 - "tf apply": │ 
Step #3 - "tf apply": │   on terraform.tf line 7, in terraform:
Step #3 - "tf apply": │    7:   experiments = [module_variable_optional_attrs]
Step #3 - "tf apply": │ 
Step #3 - "tf apply": │ Experimental features are subject to breaking changes in future minor or
Step #3 - "tf apply": │ patch releases, based on feedback.
Step #3 - "tf apply": │ 
Step #3 - "tf apply": │ If you have feedback on the design of this feature, please open a GitHub
Step #3 - "tf apply": │ issue to discuss it.
Step #3 - "tf apply": │ 
Step #3 - "tf apply": │ (and 6 more similar warnings elsewhere)
Step #3 - "tf apply": ╵
Step #3 - "tf apply": 
Step #3 - "tf apply": Apply complete! Resources: 0 added, 2 changed, 0 destroyed.
Finished Step #3 - "tf apply"
PUSH

Artifacts will be uploaded to gs://cndev-cloudbuild_artifacts using gsutil cp
environments/prod/*.plan: Uploading path....
Copying file://environments/prod/deee51de-2631-4822-b0d0-937ffd61e3ee_tfplan.plan [Content-Type=application/octet-stream]...
/ [0/1 files][    0.0 B/183.8 KiB]   0% Done                                    
/ [1/1 files][183.8 KiB/183.8 KiB] 100% Done                                    
Operation completed over 1 objects/183.8 KiB.                                    
environments/prod/*.plan: 1 matching files uploaded
1 total artifacts uploaded to gs://cndev-cloudbuild_artifacts/terraform/cloudbuild/apply/landing-zone-prod/deee51de-2631-4822-b0d0-937ffd61e3ee/
Uploading manifest artifacts-deee51de-2631-4822-b0d0-937ffd61e3ee.json
Artifact manifest located at gs://cndev-cloudbuild_artifacts/terraform/cloudbuild/apply/landing-zone-prod/deee51de-2631-4822-b0d0-937ffd61e3ee/artifacts-deee51de-2631-4822-b0d0-937ffd61e3ee.json
DONE
Step #3 - "tf apply": 
@obriensystems
Copy link
Collaborator Author

Triage older system for tf version

  • add to the push cloud build yaml
root_@cloudshell:~/cloudshell_open/pbmm-on-gcp-onboarding (cnpe-cnd-cndev-sbx)$ git diff
diff --git a/modules/cloudbuild/templates/cloudbuild-push-request.yaml b/modules/cloudbuild/templates/cloudbuild-push-request.yaml
index 3495bbe..acc0fbf 100644
--- a/modules/cloudbuild/templates/cloudbuild-push-request.yaml
+++ b/modules/cloudbuild/templates/cloudbuild-push-request.yaml
@@ -14,6 +14,19 @@ steps:
               echo "*************************************************"
               terraform init || exit 1

+- id: 'tf version'
+  name: ${_DEFAULT_REGION}-docker.pkg.dev/${_SEED_PROJECT_ID}/${_GAR_REPOSITORY}/terraform
+  dir: "${_WORKSTREAM_PATH}"
+  entrypoint: 'sh'
+  args:
+  - '-c'
+  - |
+              echo ""
+              echo "*************** TERRAFORM VERSION ******************"
+              echo "******* At environment: ${_WORKSTREAM_PATH} *************"
+              echo "*************************************************"
+              terraform --version || exit 1

results
https://console.cloud.google.com/cloud-build/builds;region=global/1ec94347-7cae-41f6-8d22-0d57878f2a08;step=1?project=cnpe-cnd-cndev-sbx&supportedpurview=project

Terraform v1.0.10
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v3.90.1
+ provider registry.terraform.io/hashicorp/google-beta v4.50.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/random v3.4.3

Your version of Terraform is out of date! The latest version
is 1.3.7. You can update by downloading from https://www.terraform.io/downloads.html

@obriensystems
Copy link
Collaborator Author

Fix is only required for bootstrap (which runs outside the CB container directly on the gcloud shell (which runs 1.3.7), for all other common/non-prod/prod we can continue to use terraform 1.0.10 and experimental optional attributes

@obriensystems obriensystems changed the title Terraform 1.3.7 removes experimental optional() variable - breaking tf apply - without preset optional values for "Error: Optional object type attributes are experimental" Terraform 1.3.7 removes experimental optional() variable - breaking tf apply on shell container but not cloud build container running 1.0.10 - without preset optional values for "Error: Optional object type attributes are experimental" Jan 30, 2023
@fmichaelobrien fmichaelobrien self-assigned this Jan 30, 2023
obriensystems added a commit to cloud-quickstart/pbmm-on-gcp-onboarding that referenced this issue Jan 30, 2023
fmichaelobrien added a commit that referenced this issue Jan 30, 2023
#224 - workaround TF 1.3.7 from 1.0.10
@obriensystems
Copy link
Collaborator Author

working
Screen Shot 2023-01-30 at 23 32 01

@obriensystems
Copy link
Collaborator Author

Reran clean install - the TF check on 1.3.7 needs all experiments sections commented to initially run in the shell
after that we can revert

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git restore ..." to discard changes in working directory)
modified: ../common/terraform.tf
modified: ../nonprod/terraform.tf
modified: ../prod/terraform.tf
modified: ../../modules/audit-bunker/variables.tf
modified: ../../modules/iam/terraform.tf
modified: ../../modules/landing-zone-bootstrap/terraform.tf
modified: ../../modules/landing-zone-bootstrap/variables.tf
modified: ../../modules/network-host-project/modules/network/variables.tf
modified: ../../modules/network-host-project/modules/subnets/variables.tf
modified: ../../modules/network-host-project/modules/vpn/variables.tf
modified: ../../modules/network-host-project/variables.tf
modified: ../../modules/network/modules/network/variables.tf
modified: ../../modules/network/modules/router/variables.tf
modified: ../../modules/network/modules/subnets/variables.tf
modified: ../../modules/network/modules/vpn/variables.tf
modified: ../../modules/network/variables.tf
modified: ../../modules/virtual-machine/terraform.tf
modified: ../../modules/vpc-service-controls/terraform.tf

Untracked files:

@obriensystems
Copy link
Collaborator Author

Plan: 98 to add, 0 to change, 0 to destroy.

Changes to Outputs:

  • csr_name = "tlscsr2"
  • organization_config = {
    • billing_account = "01BFA8-80CDE5-0ADE3B"
    • contacts = {}
    • default_region = "northamerica-northeast1"
    • department_code = "Tz"
    • environment = "P"
    • labels = {}
    • location = "northamerica-northeast1"
    • org_id = "131880894992"
    • owner = "tlz"
    • root_node = "folders/594210056063"
      }
  • project_id = "tzpe-tlz-tlz-de"
  • service_account_email = (known after apply)
  • terraform_deployment_account = (known after apply)
  • tfstate_bucket_names = {
    • common = "tzpetlslzcom2"
    • nonprod = "tzpetlslznprd2"
    • prod = "tzpetlslzprd2"
      }

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: launchpad.2023-01-31.1501.plan

To perform exactly these actions, run the following command to apply:
terraform apply "launchpad.2023-01-31.1501.plan"
Please confirm that you have reviewed the plan and wish to apply it. Type 'yes' to proceed

@obriensystems
Copy link
Collaborator Author

delete existing org level services like

@obriensystems
Copy link
Collaborator Author

delete vpc sc
Screen Shot 2023-02-01 at 12 00 06

add IAM role
https://cloud.google.com/access-context-manager/docs/manage-access-policy#delete

get ID from asset inventory under identity.AccessPolicy

Display name
[tspevsc_tlsacm_vsc](https://console.cloud.google.com/#)
Resource type
identity.accesscontextmanager.googleapis.com/AccessPolicy
Name
[//accesscontextmanager.googleapis.com/accessPolicies/1021375921638](https://console.cloud.google.com/#)
Organization
[organizations/131880894992](https://console.cloud.google.com/#)
Parent asset type
[cloudresourcemanager.googleapis.com/Organization](https://console.cloud.google.com/#)
Parent full resource name
[//cloudresourcemanager.googleapis.com/organizations/131880894992](https://console.cloud.google.com/#)

enable role, delete vpc sc


root_@cloudshell:~ (lz-tls)$ gcloud access-context-manager policies delete accessPolicies/1021375921638
You are about to delete policy [1021375921638]

Do you want to continue (Y/n)?  y
API [accesscontextmanager.googleapis.com] not enabled on project [308673020059]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  y

Enabling service [accesscontextmanager.googleapis.com] on project [308673020059]...
Operation "operations/acat.p2-308673020059-bb01eee4-4f53-469f-987f-5179f0c8f6c6" finished successfully.
Deleted policy [1021375921638].

@obriensystems
Copy link
Collaborator Author

common ok
Screen Shot 2023-02-01 at 12 20 26

rerunning non-prod (2_folders is up now)
prod ok
Screen Shot 2023-02-01 at 12 56 04

@obriensystems
Copy link
Collaborator Author

Screen Shot 2023-02-01 at 13 25 33

prod - need to rename project - as deleted 30 day project still up after billing quota error earlier

Step #4 - "tf apply": │ Error: error creating project tzpe-tlz-tlzprod-host2 (TzPe-tlz-tlzprod-host2): googleapi: Error 409: Requested entity already exists, alreadyExists. If you received a 403 error, make sure you have the `roles/resourcemanager.projectCreator` permission
Step #4 - "tf apply": │ 
Step #4 - "tf apply": │   with module.net-host-prj.module.project.google_project.project,
Step #4 - "tf apply": │   on ../../modules/project/main.tf line 19, in resource "google_project" "project":
Step #4 - "tf apply": │   19: resource "google_project" "project" {


fix

prod_host_net = {
  user_defined_string            = "tlzprod" # Must be globally unique. Used to create project name
  additional_user_defined_string = "host3"#"host2"

root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ git status
On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   environments/prod/prod-network.auto.tfvars

no changes added to commit (use "git add" and/or "git commit -a")
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ git add environments/
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ git commit -m "new proj name for prod"
[main 6da67af] new proj name for prod
 1 file changed, 1 insertion(+), 1 deletion(-)
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ git push csr main

Screen Shot 2023-02-01 at 13 33 31

prod ok

@fmichaelobrien
Copy link
Contributor

fmichaelobrien commented Mar 23, 2023

Dont' use above until I update the dockerfile to Terraform 1.3.7 - above workaround breaks some IAM

peering hardcoded for now in deployment
terraform.landing.systems : csr2

Screenshot 2023-04-05 at 21 27 20

@obriensystems obriensystems changed the title Terraform 1.3.7 removes experimental optional() variable - breaking tf apply on shell container but not cloud build container running 1.0.10 - without preset optional values for "Error: Optional object type attributes are experimental" Terraform 1.3.7 removes experimental optional() variable - breaking tf apply on shell container but not cloud build container running 1.0.10 - without preset optional values for "Error: Optional object type attributes are experimental" - cloudbuild dockerfile move to 1.3.7 Apr 6, 2023
@obriensystems obriensystems changed the title Terraform 1.3.7 removes experimental optional() variable - breaking tf apply on shell container but not cloud build container running 1.0.10 - without preset optional values for "Error: Optional object type attributes are experimental" - cloudbuild dockerfile move to 1.3.7 Terraform 1.3.7 removes experimental optional() variable - breaking tf apply on shell container but not cloud build container running 1.0.10 - without preset optional values for "Error: Optional object type attributes are experimental" - cloudbuild dockerfile move to 1.5.2 and remove experimental code Jul 24, 2023
@fmichaelobrien
Copy link
Contributor

20240406: Closing issue during retrofit/rebase of this TEF V1 based/modified repo to TEF V4 standards
This issue may participate in the LZ refactor after rebase
Query on all issues related to the older V1 version via the tag
2024-pre-tef-v4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants