Example 258 fortigate perimeter package deploy procedure/verify for core lz unmanaged client

[fmichaelobrien]

updates

• review alternative serverless approach using https://cloud.google.com/blog/products/identity-security/introducing-google-cloud-firewall-plus-with-intrusion-prevention
• 20240125: move setup.sh to merge in phases in Demo: Script: Automated minimal landing zone with a (hub-env and core-landing-zone, client-setup, client-landing-zone, client-project-setup) GitOps/OCI based deployment on a clean GCP organization - walkthrough #766
• 20240220: investigate policy based routing for the hub-env package VPC - https://cloud.google.com/vpc/docs/policy-based-routes
• • refer to Ground to Cloud enablement through PSC (private service connect) or PGA (private google access) through an interconnect or VPN for private GCP API access - customer procedure using AWS as simulated groud #494
    dev = root landing systems
    test = nuage
    finops dev.. nua.cl.o
• training https://www.cloudskillsboost.google/focuses/77469?catalog_rank=%7B%22rank%22%3A1%2C%22num_filters%22%3A0%2C%22has_search%22%3Atrue%7D&parent=catalog&search_id=29853894

FinOps: PAYG + GKE + GCE costs will be $80/day above the normal $10/day for the GKE cluster alone.

The client requires deployment of the #258 perimeter on top of the core lz with additional DNS zones TBD

Document and reuse on top of #420 and and #421
gcloud deployment testing later 2022 - #158
See pre-kcc deployment run in #158

gcloud reference install: fortinet/fortigate-tutorial-gcp#1

see
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps

graph LR;
    style LZV2 fill:#44f,stroke:#f66,stroke-width:2px,color:#fff,stroke-dasharray: 5 5
    %% mapped and documented
    project/hub-env-->core-landing-zone;
    client-setup;
    client-setup-->dns-project;
    client-setup-->kcc-management-project;
    client-landing-zone-->client-setup;
    client-project-setup-->client-landing-zone;
    client-project-setup-->client-management-project;
    gatekeeper-policies;

    kcc-management-project;
    core-landing-zone-->kcc-management-project;
    dns-project-->core-landing-zone;
    logging-project-->core-landing-zone;
    client-management-project-->client-setup;
    host-project-->client-landing-zone;
    

mermaid - diagrams as code
See

• 20231027 clean org demo run of core-landing-zone and hub-env Demo: Automated minimal landing zone with a (hub-env and core-landing-zone) KPT based deployment on a clean GCP organization - walkthrough #611
• Compliance: ITSG-22 Network Security Zones and ITSG-38 Network Security Zoning - Design Considerations for Placement of Services #78
• Add KCC port of fortigate-tutorial-gcp example - backport collaboration - while we integrate into the KCC Landing zone #168
• Fortigate configuration: VM in mgmt vpc/subnet can have internet access without using a NAT? The reconfigured fortigate 4th mgmt NIC can not be used for firewall policies #166
• Investigate L7 packet inspection via packet mirroring all or filtered - to IDS, PaloAlto or Fortigate NGFW appliances  #177
• Add KCC yakima service account as Billing Account User for shared billing account in deployment.sh with 60 sec sleep - to avoid project/billing association error on 5 projects #207
• 258/446 fortigate hub-env package: missing role yakima gke service account role - kpt render failed to set org id  #573
  todo:
• add client-landing-zone on top of existing core-landing-zone before hub-env

Package Inventory

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone
• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/client-landing-zone
• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/project/hub-env

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Architecture

Notes:

• .sh linter failures Fix Lint bash failure when a .sh script is in the PR patch  #728

[fmichaelobrien]

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/project/hub-env/fortigate
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone

• no interconnects

[obriensystems]

org states

kcc.landing.systems

• 3 month old main core-landing-zone
• 3 month old hub-env partially up with a fixes
• #573 - hub-env setters.yaml missing org-id #574
• 258/446 fortigate hub-env package: missing role yakima gke service account role - kpt render failed to set org id  #573

obrien.industries

• dev box only for automation creation
• 1 week old - was master - now 0.3.2 core-landing-zone - more stable
• 8+ lz reinstalls - system unclean for hub-env

landing systems

• target test box only for automation of lcls
• full automation via sh script only
• derived setters.yaml
• 20231022 0.3.2 core-landing-zone
• will manually run hub-env

new org

• deploy clz + hub-env together as changes come in from ls above

TODO

• add an onboarding readme for the hub-env like
• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/onboarding-admin.md

edit
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/setters.yaml
via
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/README.md

Known Issues Workarounds

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigates.md#known-issues
• turn down security on the management VPC - turn back on public access for googleapis.com public access without putting in a private service connect endpoint like
• Interconnect: Add private DNS zones with domain forwarding/peering to enable Private Google Access or Private Service Connect workload calls from on-prem - using AWS VPN for Interconnect simulation #468

[obriensystems]

know oci and gitops are the core - and I agree having the code in github/ado/gitlab/csr is preferred - but some clients have requested the easier kpt option and it should be there as a base deployment option - since mid 2022

To be fair the base case deployment option is actually pure kubernetes krm yaml like in https://cloud.google.com/config-connector/docs/how-to/getting-started

see original gitops docs in https://cloud.google.com/anthos-config-management/docs/concepts/config-controller-overview and https://cloud.google.com/anthos-config-management/docs/how-to/unstructured-repo
see original kpt docs in https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt

20230814: revisit kls

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ export PROJECT_ID=kcc-kls-cluster3
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ ls
core-landing-zone  setters.yaml
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ rm -rf core-landing-zone/
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/core-landing-zone@main
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 + 52f93a3...ea2e57f main       -> origin/main  (forced update)
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ cp setters.yaml core-landing-zone/

re-add kpt documentation at the end of section 2
see #409

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#2-create-your-landing-zone
needs
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/solutions/core-landing-zone/0.3.0/docs/landing-zone-v2#kpt

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$   kpt live init core-landing-zone --namespace config-control
initializing "resourcegroup.yaml" data (namespace: config-control)...success

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt fn render core-landing-zone
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[FAIL] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.1s
  Results:
    [error]: failed to apply setters: values for setters [${platform-and-component-log-bucket}] must be provided
  Stderr:
    "values for setters [${platform-and-component-log-bucket}] must be providedvalues for setters [${platform-and-component-log-bucket}] must be provided"
  Exit code: 1
  
  fix: did not have the latest version of setters.yaml - updated
  
  mirroring changes to my local repo from the core-landing-zone kpt folder download

root_@cloudshell:~/kcc-kls/lz-20230803-gh/pubsec-declarative-toolkit (kcc-kls-cluster3)$ git diff
diff --git a/solutions/core-landing-zone/setters.yaml b/solutions/core-landing-zone/setters.yaml
index f3168d3..ca53ae4 100644
--- a/solutions/core-landing-zone/setters.yaml
+++ b/solutions/core-landing-zone/setters.yaml
@@ -14,10 +14,11 @@
 #########
 apiVersion: v1
 kind: ConfigMap
-metadata:
+metadata: # kpt-merge: /setters
   name: setters
   annotations:
     config.kubernetes.io/local-config: "true"
+    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
 data:
   ##########################
   # Instructions
@@ -38,9 +39,9 @@ data:
   # General Settings Values
   ##########################
   #
-  org-id: "0000000000"
-  lz-folder-id: '0000000000'
-  billing-id: "AAAAAA-BBBBBB-CCCCCC"
+  org-id: "15....993"
+  lz-folder-id: '444....332'
+  billing-id: "01....833"
   #
   ##########################
   # Management Project
@@ -48,8 +49,8 @@ data:
   #
   # This is the project where the config controller instance is running
   # Values can be viewed in the Project Dashboard
-  management-project-id: management-project-12345
-  management-project-number: "0000000000"
+  management-project-id: kcc-kls-cluster3
+  management-project-number: "53....547"
   management-namespace: config-control
   #
   ##########################
@@ -68,14 +69,14 @@ data:
   # org/org-policies/essentialcontacts-allowed-contact-domains.yaml
   # this setting MUST be changed
   allowed-contact-domains: |
-    - "@example.com"
+    - "@kcc.landing.systems"
   #
   # a list of directory customer IDs from which users can be added to IAM policies, see YAML file for more info:
   # org/org-policies/iam-allowed-policy-member-domains.yaml
   # this setting MUST be changed to include the GCP org's directory ID and any other directory containing users that will need IAM roles assigned
   # run 'gcloud organizations list' as described in https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains#retrieving_customer_id
   allowed-policy-domain-members: |
-    - "DIRECTORY_CUSTOMER_ID"
+    - "C0....m1"
   #
   # a list of allowed projects, folders, networks for VPC peering, see YAML file for more info:
   # org/org-policies/compute-restrict-vpc-peering.yaml
@@ -87,13 +88,13 @@ data:
   # Logging
   ##########################
   #
-  logging-project-id: logging-project-12345
+  logging-project-id: logging-project-kls
   #
   # Log Buckets
   # Security Logs Bucket
-  security-log-bucket: security-log-bucket-12345
+  security-log-bucket: security-log-bucket-kls
   # Platform and Component Log Bucket
-  platform-and-component-log-bucket: platform-and-component-log-bucket-12345
+  platform-and-component-log-bucket: platform-and-component-log-bucket-kls
   #
   # Retention settings
   # Set the number of days to retain logs in Cloud Logging buckets
@@ -110,8 +111,9 @@ data:
   # DNS
   ##########################
   #
-  dns-project-id: dns-project-12345
-  dns-name: "example.com."
+  dns-project-id: dns-project-kls
+  # the appended . is required by google cloud domain zones
+  dns-name: "kcc.landing.systems."

kpt rendering ok

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt fn render core-landing-zone
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 600ms
  Results:
    [info] spec.folderRef.external: set field value to "444332200332"
    [info] metadata.name: set field value to "security-log-bucket-kls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-kls"
    [info] spec.projectRef.name: set field value to "logging-project-kls"
    ...(213 line(s) truncated, use '--truncate-output=false' to disable)

Successfully executed 1 function(s) in 1 package(s).

kpt live apply (20230414:1552)

kpt live apply core-landing-zone --reconcile-timeout=2m --output=table

coming up

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/hierarchy                       Successful    Current                 <None>                                    2m      Resource is current                     
            Namespace/logging                         Successful    Current                 <None>                                    2m      Resource is current                     
            Namespace/networking                      Successful    Current                 <None>                                    2m      Resource is current                     
            Namespace/policies                        Successful    Current                 <None>                                    2m      Resource is current                     
            Namespace/projects                        Successful    Current                 <None>                                    2m      Resource is current                     
config-con  IAMCustomRole/gke-firewall-admin          Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier2-dnsrecord-admin       Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier2-vpcpeering-admin      Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier3-dnsrecord-admin       Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier3-firewallrule-admin    Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier3-vpcsc-admin           Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/gatekeeper-admin-sa-wor  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/hierarchy-sa-workload-i  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/logging-sa-workload-ide  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/networking-sa-workload-  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/policies-sa-workload-id  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/projects-sa-workload-id  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-orgrol  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/gatekeeper-admin-sa-metr  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/hierarchy-sa-folderadmin  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/logging-sa-bigqueryadmin  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/logging-sa-logadmin-perm  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-dns-permis  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-networkadm  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-security-p  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-service-co  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-servicedir  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-xpnadmin-p  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/policies-sa-orgpolicyadm  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/projects-sa-billinguser-  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectcreat  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectdelet  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
config-con  IAMPolicyMember/projects-sa-projectiamad  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
config-con  IAMPolicyMember/projects-sa-projectmover  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
config-con  IAMPolicyMember/projects-sa-serviceusage  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
config-con  IAMServiceAccount/gatekeeper-admin-sa     Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/hierarchy-sa            Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/logging-sa              Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/networking-sa           Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/policies-sa             Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/projects-sa             Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  Service/kcc-kls-cluster3-accesscontextma  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  Service/kcc-kls-cluster3-cloudbilling     Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  Service/kcc-kls-cluster3-cloudresourcema  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  Service/kcc-kls-cluster3-serviceusage     Successful    Current                 Ready                                     2m      Resource is Current                     
gatekeeper  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
hierarchy   ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
hierarchy   RoleBinding/allow-folders-resource-refer  Successful    Current                 <None>                                    37s     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    37s     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    37s     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    36s     Resource is current                     
hierarchy   Folder/audits                             Successful    Current                 Ready                                     34s     Resource is Current                     
hierarchy   Folder/clients                            Successful    Current                 Ready                                     33s     Resource is Current                     
hierarchy   Folder/services                           Successful    Current                 Ready                                     33s     Resource is Current                     
hierarchy   Folder/services-infrastructure            Successful    Current                 Ready                                     33s     Resource is Current                     
logging     ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
logging     LoggingLogBucket/platform-and-component-  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogBucket/security-log-bucket-kls  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/logging-project-kls-secur  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-disa  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-plat  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     RoleBinding/allow-logging-resource-refer  Successful    Current                 <None>                                    36s     Resource is current                     
networking  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
networking  DNSManagedZone/dns-project-kls-standard-  Skipped       Unknown                 -                                         -                                               
policies    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
policies    ResourceManagerPolicy/compute-disable-gu  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-ne  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-vp  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-os  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     31s     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     31s     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     31s     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-s  Successful    Current                 Ready                                     31s     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Failed                  Ready                                     30s     Update call failed: error applying desir
policies    ResourceManagerPolicy/compute-skip-defau  Successful    Current                 Ready                                     30s     Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     30s     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     30s     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     30s     Resource is Current                     
policies    ResourceManagerPolicy/essentialcontacts-  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/gcp-restrict-resou  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/iam-allowed-policy  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/iam-disable-servic  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/sql-restrict-publi  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/storage-public-acc  Successful    Current                 Ready                                     28s     Resource is Current                     
policies    ResourceManagerPolicy/storage-uniform-bu  Successful    Current                 Ready                                     28s     Resource is Current                     
projects    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
projects    IAMAuditConfig/logging-project-data-acce  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/mgmt-project-cluster-pl  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/platform-and-component-  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/platform-and-component-  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/security-log-bucket-wri  Skipped       Unknown                 -                                         -                                               
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    36s     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    36s     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    36s     Resource is current                     
projects    Project/dns-project-kls                   Successful    Failed                  Ready                                     2s      Update call failed: error fetching live 
projects    Project/logging-project-kls               Successful    Failed                  Ready                                     33s     Update call failed: error fetching live 
projects    Service/dns-project-kls-dns               Skipped       Unknown                 -                                         -                                               

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get pods --all-namespaces
NAMESPACE                         NAME                                                       READY   STATUS    RESTARTS       AGE
cnrm-system                       cnrm-controller-manager-3fo6phebqgg23knqq5qq-0             2/2     Running   0              4m2s
cnrm-system                       cnrm-controller-manager-7c4rehlik7xgxc2utq6a-0             2/2     Running   0              4m2s
cnrm-system                       cnrm-controller-manager-fiqj4dqbgpwy6mlvh25q-0             2/2     Running   0              4m
cnrm-system                       cnrm-controller-manager-ghhiigeeussitzq7mfza-0             2/2     Running   0              4m
cnrm-system                       cnrm-controller-manager-gnunqke5gjhr55wngr7q-0             2/2     Running   0              4m1s
cnrm-system                       cnrm-controller-manager-sgfj3cxgisp6jdsfy7qq-0             2/2     Running   0              5d3h
cnrm-system                       cnrm-controller-manager-swyfekd4gcdftjnvc2qa-0             2/2     Running   0              5m29s
cnrm-system                       cnrm-deletiondefender-0                                    1/1     Running   0              5d3h
cnrm-system                       cnrm-resource-stats-recorder-88bfdfd56-kqdq2               2/2     Running   0              5d3h
cnrm-system                       cnrm-unmanaged-detector-0                                  1/1     Running   0              5d3h
cnrm-system                       cnrm-webhook-manager-54c8477885-cr54f                      1/1     Running   0              5d3h
cnrm-system                       cnrm-webhook-manager-54c8477885-plgkd                      1/1     Running   0              4m36s
cnrm-system                       cnrm-webhook-manager-54c8477885-ssldj                      1/1     Running   0              5d3h
config-management-monitoring      otel-collector-865b4f4968-l89bt                            1/1     Running   0              5d3h
config-management-system          config-management-operator-5db59f7f8f-5fb4p                1/1     Running   0              5d3h
config-management-system          reconciler-manager-5cddc57f5-bxc86                         2/2     Running   0              5d3h
configconnector-operator-system   configconnector-operator-0                                 1/1     Running   0              5d3h
gatekeeper-system                 gatekeeper-audit-6d686f5467-zlwzr                          1/1     Running   0              5d3h
gatekeeper-system                 gatekeeper-controller-manager-6b47854cf5-nsmzs             1/1     Running   0              5d3h
gke-gmp-system                    alertmanager-0                                             2/2     Running   2 (11d ago)    11d
gke-gmp-system                    collector-bb4st                                            2/2     Running   2 (11d ago)    11d
gke-gmp-system                    collector-h4j24                                            2/2     Running   1 (11d ago)    11d
gke-gmp-system                    collector-szhxn                                            2/2     Running   2 (11d ago)    11d
gke-gmp-system                    gmp-operator-7645bc584f-5d8gf                              1/1     Running   0              30h
gke-gmp-system                    rule-evaluator-767c5ccc99-7mbnt                            2/2     Running   2 (11d ago)    11d
krmapihosting-monitoring          krmapihosting-metrics-agent-55glj                          1/1     Running   0              11d
krmapihosting-monitoring          krmapihosting-metrics-agent-9nlw9                          1/1     Running   0              11d
krmapihosting-monitoring          krmapihosting-metrics-agent-d8xm9                          1/1     Running   0              11d
krmapihosting-system              bootstrap-5d5578f758-sh76w                                 1/1     Running   0              5d3h
kube-system                       anetd-cg6g9                                                1/1     Running   0              11d
kube-system                       anetd-f2gpt                                                1/1     Running   0              11d
kube-system                       anetd-r7gr2                                                1/1     Running   0              11d
kube-system                       antrea-controller-horizontal-autoscaler-7b69d9bfd7-rqq8r   1/1     Running   0              11d
kube-system                       egress-nat-controller-98648bc69-fm8nk                      1/1     Running   0              11d
kube-system                       event-exporter-gke-7bf6c99dcb-c5dd9                        2/2     Running   0              11d
kube-system                       filestore-node-4p9cx                                       3/3     Running   0              11d
kube-system                       filestore-node-5jlfv                                       3/3     Running   0              11d
kube-system                       filestore-node-74pm4                                       3/3     Running   1 (7d6h ago)   11d
kube-system                       fluentbit-gke-big-6hsk5                                    2/2     Running   0              11d
kube-system                       fluentbit-gke-big-sxkh2                                    2/2     Running   0              11d
kube-system                       fluentbit-gke-big-vm26j                                    2/2     Running   0              11d
kube-system                       gcsfusecsi-node-7k76l                                      2/2     Running   0              11d
kube-system                       gcsfusecsi-node-j8r4b                                      2/2     Running   0              11d
kube-system                       gcsfusecsi-node-sq62q                                      2/2     Running   0              11d
kube-system                       gke-metadata-server-btb9x                                  1/1     Running   0              30h
kube-system                       gke-metadata-server-l447p                                  1/1     Running   0              30h
kube-system                       gke-metadata-server-w7brs                                  1/1     Running   0              30h
kube-system                       gke-metrics-agent-9hvwg                                    2/2     Running   0              11d
kube-system                       gke-metrics-agent-j4xvr                                    2/2     Running   0              11d
kube-system                       gke-metrics-agent-spdl8                                    2/2     Running   0              11d
kube-system                       ip-masq-agent-cphwd                                        1/1     Running   0              11d
kube-system                       ip-masq-agent-n7nbw                                        1/1     Running   0              11d
kube-system                       ip-masq-agent-r8pvq                                        1/1     Running   0              11d
kube-system                       konnectivity-agent-5b687c8dcb-d64h7                        1/1     Running   0              5d3h
kube-system                       konnectivity-agent-5b687c8dcb-dkrth                        1/1     Running   0              11d
kube-system                       konnectivity-agent-5b687c8dcb-vgmkm                        1/1     Running   0              11d
kube-system                       konnectivity-agent-autoscaler-5d9dbcc6d8-2s5dp             1/1     Running   0              11d
kube-system                       kube-dns-865c4fb86d-k5b2c                                  4/4     Running   0              11d
kube-system                       kube-dns-865c4fb86d-skmk6                                  4/4     Running   0              11d
kube-system                       kube-dns-autoscaler-84b8db4dc7-h47j6                       1/1     Running   0              11d
kube-system                       l7-default-backend-58c4fb8884-7n45b                        1/1     Running   0              2d6h
kube-system                       metrics-server-v0.5.2-6bf74b5d5f-fknxl                     2/2     Running   0              11d
kube-system                       netd-dtqvj                                                 1/1     Running   0              11d
kube-system                       netd-l5wgc                                                 1/1     Running   0              11d
kube-system                       netd-nhgl9                                                 1/1     Running   0              11d
kube-system                       node-local-dns-5wzzk                                       1/1     Running   0              11d
kube-system                       node-local-dns-bxqzh                                       1/1     Running   0              11d
kube-system                       node-local-dns-fkfln                                       1/1     Running   0              11d
kube-system                       pdcsi-node-h8jzw                                           2/2     Running   0              9d
kube-system                       pdcsi-node-hl6m6                                           2/2     Running   0              9d
kube-system                       pdcsi-node-sxfns                                           2/2     Running   0              9d
resource-group-system             resource-group-controller-manager-5594cd7b8-l87bc          2/2     Running   0              5d3h

just 1 org policy has an issue
missed a setters.yaml var
under:organizations/ORGANIZATION_ID]

  allowed-vpc-peering: |
    - "under:organizations/15..."
    
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt live status core-landing-zone
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/audits is Current: Resource is Current
inventory-85852139/project.resourcemanager.cnrm.cloud.google.com/projects/logging-project-kls is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services is Current: Resource is Current
inventory-85852139/project.resourcemanager.cnrm.cloud.google.com/projects/dns-project-kls is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services-infrastructure is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-mgt-project is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-cloudbilling is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-cloudresourcemanager is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-serviceusage is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-accesscontextmanager is Current: Resource is Current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-metric-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/hierarchy-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//hierarchy is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-projects is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-config-control is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-folders-resource-reference-to-logging is Current: Resource is current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/logging-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-logadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-bigqueryadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/logging-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//logging is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/logging/allow-logging-resource-reference-from-projects is Current: Resource is current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-orgroleadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-editor-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-serviceaccountadmin-permissions is Current: Resource is Current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/networking-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-security-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-dns-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-service-control-org-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/networking-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//networking is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/policies-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/policies-sa-orgpolicyadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/policies-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//policies is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/policies/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/projects-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectmover-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-billinguser-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/projects-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//projects is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-logging is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-networking is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-policies is Current: Resource is current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/gke-firewall-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-dnsrecord-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-vpcpeering-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-dnsrecord-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-firewallrule-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-vpcsc-admin is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-guest-attribute-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-nested-virtualization is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-vpc-external-ipv6 is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-os-login is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-shared-vpc-lien-removal is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering is Failed: Update call failed: error applying desired state: summary: googleapi: Error 400: One or more values is invalid.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.BadRequest",
    "fieldViolations": [
      {
        "description": "Invalid value: [under:organizations/ORGANIZATION_ID]",
        "field": "policy.list_policy.allowed_values[0]"
      }
    ]
  }
]
, badRequest
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-skip-default-network-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/essentialcontacts-allowed-contact-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/gcp-restrict-resource-locations is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-allowed-policy-member-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/sql-restrict-public-ip is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-public-access-prevention is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-uniform-bucket-level-access is Current: Resource is Current

fixing

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/hierarchy                       Successful    Current                 <None>                                    61m     Resource is current                     
            Namespace/logging                         Successful    Current                 <None>                                    61m     Resource is current                     
            Namespace/networking                      Successful    Current                 <None>                                    61m     Resource is current                     
            Namespace/policies                        Successful    Current                 <None>                                    61m     Resource is current                     
            Namespace/projects                        Successful    Current                 <None>                                    61m     Resource is current                     
config-con  IAMCustomRole/gke-firewall-admin          Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier2-dnsrecord-admin       Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier2-vpcpeering-admin      Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier3-dnsrecord-admin       Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier3-firewallrule-admin    Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier3-vpcsc-admin           Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/gatekeeper-admin-sa-wor  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/hierarchy-sa-workload-i  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/logging-sa-workload-ide  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/networking-sa-workload-  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/policies-sa-workload-id  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/projects-sa-workload-id  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-orgrol  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/gatekeeper-admin-sa-metr  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/hierarchy-sa-folderadmin  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/logging-sa-bigqueryadmin  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/logging-sa-logadmin-perm  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-dns-permis  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-networkadm  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-security-p  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-service-co  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-servicedir  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-xpnadmin-p  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/policies-sa-orgpolicyadm  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-billinguser-  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectcreat  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectdelet  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectiamad  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectmover  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-serviceusage  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/gatekeeper-admin-sa     Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/hierarchy-sa            Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/logging-sa              Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/networking-sa           Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/policies-sa             Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/projects-sa             Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  Service/kcc-kls-cluster3-accesscontextma  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  Service/kcc-kls-cluster3-cloudbilling     Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  Service/kcc-kls-cluster3-cloudresourcema  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  Service/kcc-kls-cluster3-serviceusage     Successful    Current                 Ready                                     61m     Resource is Current                     
gatekeeper  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    61m     status.healthy is true                  
hierarchy   ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
hierarchy   RoleBinding/allow-folders-resource-refer  Successful    Current                 <None>                                    60m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    60m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    60m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    60m     Resource is current                     
hierarchy   Folder/audits                             Successful    Current                 Ready                                     60m     Resource is Current                     
hierarchy   Folder/clients                            Successful    Current                 Ready                                     60m     Resource is Current                     
hierarchy   Folder/services                           Successful    Current                 Ready                                     60m     Resource is Current                     
hierarchy   Folder/services-infrastructure            Successful    Current                 Ready                                     60m     Resource is Current                     
logging     ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
logging     LoggingLogBucket/platform-and-component-  Successful    Current                 Ready                                     3m      Resource is Current                     
logging     LoggingLogBucket/security-log-bucket-kls  Successful    Current                 Ready                                     3m      Resource is Current                     
logging     LoggingLogSink/logging-project-kls-secur  Successful    Current                 Ready                                     32s     Resource is Current                     
logging     LoggingLogSink/mgmt-project-cluster-disa  Successful    Current                 Ready                                     3m      Resource is Current                     
logging     LoggingLogSink/mgmt-project-cluster-plat  Successful    Current                 Ready                                     31s     Resource is Current                     
logging     LoggingLogSink/platform-and-component-se  Successful    Current                 Ready                                     31s     Resource is Current                     
logging     LoggingLogSink/platform-and-component-se  Successful    Current                 Ready                                     31s     Resource is Current                     
logging     RoleBinding/allow-logging-resource-refer  Successful    Current                 <None>                                    60m     Resource is current                     
networking  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
networking  DNSManagedZone/dns-project-kls-standard-  Successful    InProgress              Ready                                     32s     Update in progress                      
policies    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
policies    ResourceManagerPolicy/compute-disable-gu  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-ne  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-vp  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-os  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-s  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-skip-defau  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/essentialcontacts-  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/gcp-restrict-resou  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/iam-allowed-policy  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/iam-disable-servic  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/sql-restrict-publi  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/storage-public-acc  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/storage-uniform-bu  Successful    Current                 Ready                                     60m     Resource is Current                     
projects    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
projects    IAMAuditConfig/logging-project-data-acce  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    IAMPartialPolicy/mgmt-project-cluster-pl  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    IAMPartialPolicy/platform-and-component-  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    IAMPartialPolicy/platform-and-component-  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    IAMPartialPolicy/security-log-bucket-wri  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    60m     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    60m     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    60m     Resource is current                     
projects    Project/dns-project-kls                   Successful    Current                 Ready                                     59m     Resource is Current                     
projects    Project/logging-project-kls               Successful    Current                 Ready                                     60m     Resource is Current                     
projects    Service/dns-project-kls-dns               Successful    Current                 Ready                                     30s     Resource is Current                     

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt live status core-landing-zone
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/audits is Current: Resource is Current
inventory-85852139/logginglogbucket.logging.cnrm.cloud.google.com/logging/security-log-bucket-kls is Current: Resource is Current
inventory-85852139/logginglogbucket.logging.cnrm.cloud.google.com/logging/platform-and-component-log-bucket-kls is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is Current: Resource is Current
inventory-85852139/iamauditconfig.iam.cnrm.cloud.google.com/projects/logging-project-data-access-log-config is Current: Resource is Current
inventory-85852139/project.resourcemanager.cnrm.cloud.google.com/projects/logging-project-kls is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services is Current: Resource is Current
inventory-85852139/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-kls-standard-core-public-dns is Current: Resource is Current
inventory-85852139/project.resourcemanager.cnrm.cloud.google.com/projects/dns-project-kls is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-kls-dns is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services-infrastructure is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-mgt-project is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-disable-default-bucket is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-cloudbilling is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-cloudresourcemanager is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-serviceusage is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-accesscontextmanager is Current: Resource is Current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-metric-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/hierarchy-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//hierarchy is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-projects is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-config-control is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-folders-resource-reference-to-logging is Current: Resource is current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/logging-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-logadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-bigqueryadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/logging-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//logging is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/logging/allow-logging-resource-reference-from-projects is Current: Resource is current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-orgroleadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-editor-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-serviceaccountadmin-permissions is Current: Resource is Current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/networking-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-security-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-dns-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-service-control-org-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/networking-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//networking is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/policies-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/policies-sa-orgpolicyadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/policies-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//policies is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/policies/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/projects-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectmover-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-billinguser-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/projects-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//projects is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-logging is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-networking is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-policies is Current: Resource is current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/gke-firewall-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-dnsrecord-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-vpcpeering-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-dnsrecord-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-firewallrule-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-vpcsc-admin is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-guest-attribute-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-nested-virtualization is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-vpc-external-ipv6 is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-os-login is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-shared-vpc-lien-removal is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-skip-default-network-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/essentialcontacts-allowed-contact-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/gcp-restrict-resource-locations is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-allowed-policy-member-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/sql-restrict-public-ip is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-public-access-prevention is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-uniform-bucket-level-access is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-kls-security-sink is Current: Resource is Current
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ 

deploying hub package

kubectl get gcp --all-namespaces
kubectl get gcp -n projects


root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/project/hub-env@main
Package "hub-env":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
Adding package "solutions/project/hub-env".

Fetched 1 package(s).


modifying setters
 apiVersion: v1
 kind: ConfigMap
-metadata:
+metadata: # kpt-merge: /setters
   name: setters
   annotations:
     config.kubernetes.io/local-config: "true"
+    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
 data:
   # Billing Account ID to be associated with this project
-  project-billing-id: "AAAAAA-BBBBBB-CCCCCC"
+  project-billing-id: "01A4...699F"
   # GCP folder to use as parent to this project, lowercase K8S resource name
-  project-parent-folder: project-parent-folder
+  project-parent-folder: services-infrastructure
   # Naming Convention for project-id : <tenant-code><environment-code>m<data-classification>-<project-owner>-<user defined string>
-  # Max 30 characters
-  hub-project-id: xxdmu-admin1-projectname
+  # Max 30 characters - must be unique for name to match id
+  hub-project-id: dmu-admin1-hub-kls
   # Identity that should be allowed to access the management VM using IAP TCP forwarding
   # https://cloud.google.com/iap/docs/using-tcp-forwarding
-  hub-admin: group:group@domain.com
+  hub-admin: group:org-admins@kcc.landing.systems
   #################
   # Org Policies
   #######
@@ -23,19 +39,19 @@ data:
   # org-policies/exceptions/compute-restrict-vpc-peering-except-hub-project.yaml
   # this setting MUST be changed to include the ORG ID
   project-allowed-restrict-vpc-peering: |
-    - under:organizations/ORGANIZATION_ID
+    - under:organizations/156...93
   # This list constraint defines the set of Compute Engine VM instances that are allowed to use external IP addresses, see YAML file for more info:
   # org-policies/exceptions/compute-vm-external-ip-access-except-hub-project.yaml
   # this setting MUST be changed to include the hub project ID
   project-allowed-vm-external-ip-access: |
-    - "projects/HUB_PROJECT_ID/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
-    - "projects/HUB_PROJECT_ID/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
+    - "projects/dmu-admin1-hub-kls/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
+    - "projects/dmu-admin1-hub-kls/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
   # This list constraint defines the set of VM instances that can enable IP forwarding., see YAML file for more info:
   # org-policies/exceptions/compute-vm-can-ip-forward-except-hub-project.yaml
   # this setting MUST be changed to include the hub project ID
   project-allowed-vm-can-ip-forward: |
-    - "projects/HUB_PROJECT_ID/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
-    - "projects/HUB_PROJECT_ID/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
+    - "projects/dmu-admin1-hub-kls/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
+    - "projects/dmu-admin1-hub-kls/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
   #################
   # Fortigate
   #################
@@ -46,13 +62,15 @@ data:
   # Primary
   # Having disctinct images allows one to use a Licensed Fortigate for the primary and a Pay-as-you-Go license for the secondary
   # and run the secondary just a couple of minutes each day for synching purposes thus obtaining an affordable cold standby.
-  fgt-primary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
+  #fgt-primary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
+  fgt-primary-image: projects/dmu-admin1-hub-kls/global/images/fortinet-fgtondemand-724-20230201-001-w-license
   # replace the word LICENSE below with the actual license value. Not required if using the Pay-as-you-Go image.
   fgt-primary-license: |
     LICENSE
   #######
   # Secondary
-  fgt-secondary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
+  #fgt-secondary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
+  fgt-secondary-image: projects/dmu-admin1-hub-kls/global/images/fortinet-fgtondemand-724-20230201-001-w-license
   # replace the word LICENSE below with the actual license value. Not required if using the Pay-as-you-Go image.
   fgt-secondary-license: |
     LICENSE

forgot to init - do this first

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live init hub-env --namespace config-control
initializing "resourcegroup.yaml" data (namespace: config-control)...success

render

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt fn render hub-env
Package "hub-env": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.3s
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    ...(102 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 2s
  Results:

apply

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live apply hub-env --reconcile-timeout=2m --output=table
installing inventory ResourceGroup CRD.
error: invalid object: "projects_dmu-admin1-hub-kls_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dmu-admin1-hub-kls -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure

getting a depends error on an existing resource - the folder reference - it exists - triaging

checking it it requires the folder id

  #project-parent-folder: services-infrastructure
  project-parent-folder: "176411558066"

but fix folder.yaml manually
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure
to
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/176411558066


same thing after a render and apply

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live apply hub-env --reconcile-timeout=2m --output=table
installing inventory ResourceGroup CRD.
error: invalid object: "projects_dmu-admin1-hub-kls_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dmu-admin1-hub-kls -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/176411558066

comment out dependency - rerun

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live apply hub-env --reconcile-timeout=2m --output=table
installing inventory ResourceGroup CRD.
NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-computeins  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-serviceacc  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-serviceacc  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-ext-addre  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-int-addre  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-transit-a  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-ext-add  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-int-add  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-transit  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-ilb-address            Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-ilb-proxy-address      Pending       Unknown                 -                                         -                                               
networking  ComputeBackendService/hub-ilb-bes         Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-primary-log-disk      Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-secondary-log-disk    Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-mgmt-data-disk            Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-external-fwr    Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-fortigates-ha-  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-spokes-to-fort  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-elb-allow-health-che  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-iap-allow-rdp-to-man  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-ilb-allow-health-che  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-managementvm-allow-s  Pending       Unknown                 -                                         -                                               
networking  ComputeForwardingRule/hub-ilb-fwdrule     Pending       Unknown                 -                                         -                                               
networking  ComputeForwardingRule/hub-ilb-proxy-fwdr  Pending       Unknown                 -                                         -                                               
networking  ComputeHTTPHealthCheck/hub-http-8008-htt  Pending       Unknown                 -                                         -                                               
networking  ComputeHealthCheck/hub-http-8008-hc       Pending       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-fgt-primary-instance  Pending       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-fgt-secondary-instan  Pending       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-management-instance   Pending       Unknown                 -                                         -                                               
networking  ComputeInstanceGroup/hub-fgt-primary-umi  Pending       Unknown                 -                                         -                                               
networking  ComputeInstanceGroup/hub-fgt-secondary-u  Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-external-vpc    Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-internal-vpc    Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-mgmt-vpc        Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-transit-vpc     Pending       Unknown                 -                                         -                                               
networking  ComputeRoute/hub-external-vpc-internet-e  Pending       Unknown                 -                                         -                                               
networking  ComputeRoute/hub-internal-vpc-internet-e  Pending       Unknown                 -                                         -                                               
networking  ComputeRouter/hub-nane1-external-router   Pending       Unknown                 -                                         -                                               
networking  ComputeRouterNAT/hub-nane1-external-nat   Pending       Unknown                 -                                         -                                               
NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE 

nothing deployed


root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live status hub-env
no resources found in the inventory                            

returning simpler folder

  project-parent-folder: kcc

render to adjust previous

  folderRef:
    name: 176411558066 # kpt-set: ${project-parent-folder}
    namespace: hierarchy

running

                                             
policies    ResourceManagerPolicy/compute-vm-externa  Pending       Unknown                 -                                         -                                               
projects    Project/dmu-admin1-hub-kls                Successful    InProgress              Ready                                     18s     reference Folder hierarchy/kcc is not fo
projects    Service/dmu-admin1-hub-kls-compute        Pending       Unknown                 -                                         -                                               
projects    Service/dmu-admin1-hub-kls-dns            Pending       Unknown                 -                                         -                                               

the folder is still the issue because of the missing namespace

namespace: hierarchy

but it is there

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE     READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    6h27m   True    UpToDate   6h27m
folder.resourcemanager.cnrm.cloud.google.com/clients                   6h27m   True    UpToDate   6h27m
folder.resourcemanager.cnrm.cloud.google.com/services                  6h27m   True    UpToDate   6h27m
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   6h27m   True    UpToDate   6h27m

of course - kcc is not in scope of the package - returning to services-infrastructure

rerunning after render / apply

projects    Project/dmu-admin1-hub-kls                Successful    Failed                  Ready                                     5m      Update call failed: error applying desir

is IAM permissions

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live status hub-env
inventory-38012504/project.resourcemanager.cnrm.cloud.google.com/projects/dmu-admin1-hub-kls is Failed: Update call failed: error applying desired state: summary: failed pre-requisites: missing permission on "billingAccounts/01A...99F": billing.resourceAssociations.create

switching back to local BID

015***

rerunning

                                             
projects    Project/dmu-admin1-hub-kls                Successful    InProgress              Ready                                     7m      Update in progress                      

2226: project created

dmu-admin1-hub-kls | dmu-admin1-hub-kls

                                            
NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Successful    Failed                  Ready                                     26s     Update call failed: error fetching live 
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Successful    Current                 Ready                                     26s     Resource is Current                     
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Successful    Current                 Ready                                     26s     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-computeins  Successful    Failed                  Ready                                     26s     Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     25s     Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     25s     Update call failed: error setting policy
networking  ComputeAddress/hub-fgt-primary-ext-addre  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-int-addre  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-transit-a  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-ext-add  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-int-add  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-transit  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-ilb-address            Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-ilb-proxy-address      Pending       Unknown                 -                                         -                                               
networking  ComputeBackendService/hub-ilb-bes         Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-primary-log-disk      Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-secondary-log-disk    Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-mgmt-data-disk            Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-external-fwr    Pending       Unknown                 -                                         -                                               


                                         

2233

working through failures/iam issues

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live status hub-env
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-ext-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-ext-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-int-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-int-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-mgmt-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-mgmt-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-transit-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-transit-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-ilb-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-ilb-proxy-address is Current: Resource is Current
inventory-38012504/iamcustomrole.iam.cnrm.cloud.google.com/config-control/hub-fortigatesdnreader-role is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing organizations/123456789012/roles/FortigateSdnViewer: googleapi: Error 403: You don't have permission to get the role at organizations/123456789012/roles/FortigateSdnViewer.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "iam.googleapis.com",
    "metadata": {
      "permission": "iam.roles.get",
      "resource": "organizations/123456789012/roles/FortigateSdnViewer"
    },
    "reason": "IAM_PERMISSION_DENIED"
  }
]
, forbidden
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-allow-external-fwr is Current: Resource is Current
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-elb-allow-health-checks-to-fortigate-fwr is InProgress: reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-allow-spokes-to-fortigates-fwr is InProgress: reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-ilb-allow-health-checks-to-fortigate-fwr is InProgress: reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-allow-fortigates-ha-fwr is InProgress: reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
inventory-38012504/computeinstance.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-instance is InProgress: reference ComputeDisk networking/hub-fgt-primary-log-disk is not found
inventory-38012504/computeinstance.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-instance is InProgress: reference ComputeDisk networking/hub-fgt-secondary-log-disk is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-iap-allow-rdp-to-managementvm-fwr is InProgress: reference IAMServiceAccount networking/hub-managementvm-sa is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-managementvm-allow-ssh-https-to-fortigates-fwr is InProgress: reference IAMServiceAccount networking/hub-managementvm-sa is not found
inventory-38012504/computeinstance.compute.cnrm.cloud.google.com/networking/hub-management-instance is InProgress: reference ComputeDisk networking/hub-mgmt-data-disk is not found
inventory-38012504/dnspolicy.dns.cnrm.cloud.google.com/networking/hub-external-logging-dnspolicy is Current: Resource is Current
inventory-38012504/dnspolicy.dns.cnrm.cloud.google.com/networking/hub-internal-logging-dnspolicy is Current: Resource is Current
inventory-38012504/dnspolicy.dns.cnrm.cloud.google.com/networking/hub-mgmt-logging-dnspolicy is Current: Resource is Current
inventory-38012504/dnspolicy.dns.cnrm.cloud.google.com/networking/hub-transit-logging-dnspolicy is Current: Resource is Current
inventory-38012504/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Current: Resource is Current
inventory-38012504/computerouter.compute.cnrm.cloud.google.com/networking/hub-nane1-external-router is Current: Resource is Current
inventory-38012504/computeroute.compute.cnrm.cloud.google.com/networking/hub-external-vpc-internet-egress-route is Current: Resource is Current
inventory-38012504/computesubnetwork.compute.cnrm.cloud.google.com/networking/hub-nane1-external-paz-snet is Current: Resource is Current
inventory-38012504/computesubnetwork.compute.cnrm.cloud.google.com/networking/hub-nane1-internal-paz-snet is Current: Resource is Current
inventory-38012504/computesubnetwork.compute.cnrm.cloud.google.com/networking/hub-nane1-mgmt-rz-snet is Current: Resource is Current
inventory-38012504/computesubnetwork.compute.cnrm.cloud.google.com/networking/hub-nane1-transit-paz-snet is Current: Resource is Current
inventory-38012504/computenetwork.compute.cnrm.cloud.google.com/networking/hub-global-external-vpc is Current: Resource is Current
inventory-38012504/computenetwork.compute.cnrm.cloud.google.com/networking/hub-global-internal-vpc is Current: Resource is Current
inventory-38012504/computenetwork.compute.cnrm.cloud.google.com/networking/hub-global-mgmt-vpc is Current: Resource is Current
inventory-38012504/computenetwork.compute.cnrm.cloud.google.com/networking/hub-global-transit-vpc is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access-except-hub-project is Current: Resource is Current
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-serviceaccountadmin-permissions is Failed: Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/iam.serviceAccountAdmin serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project "dmu-admin1-hub-kls"` returned error: Batch request and retried single request "Create IAM Members roles/iam.serviceAccountAdmin serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project \"dmu-admin1-hub-kls\"" both failed. Final error: Error applying IAM policy for project "dmu-admin1-hub-kls": Error setting IAM policy for project "dmu-admin1-hub-kls": googleapi: Error 400: Service account networking-sa@management-project-id.iam.gserviceaccount.com does not exist., badRequest
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-serviceaccountuser-permissions is Failed: Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/iam.serviceAccountUser serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project "dmu-admin1-hub-kls"` returned error: Batch request and retried single request "Create IAM Members roles/iam.serviceAccountUser serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project \"dmu-admin1-hub-kls\"" both failed. Final error: Error applying IAM policy for project "dmu-admin1-hub-kls": Error setting IAM policy for project "dmu-admin1-hub-kls": googleapi: Error 400: Service account networking-sa@management-project-id.iam.gserviceaccount.com does not exist., badRequest
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-computeinstanceadmin-permissions is Failed: Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project "dmu-admin1-hub-kls"` returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project \"dmu-admin1-hub-kls\"" both failed. Final error: Error applying IAM policy for project "dmu-admin1-hub-kls": Error setting IAM policy for project "dmu-admin1-hub-kls": googleapi: Error 400: Service account networking-sa@management-project-id.iam.gserviceaccount.com does not exist., badRequest
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/hub-admin-iaptunnelresourceaccessor-permissions is Current: Resource is Current
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/hub-admin-computeinstanceadmin-permissions is Current: Resource is Current
inventory-38012504/project.resourcemanager.cnrm.cloud.google.com/projects/dmu-admin1-hub-kls is Current: Resource is Current
inventory-38012504/service.serviceusage.cnrm.cloud.google.com/projects/dmu-admin1-hub-kls-compute is Current: Resource is Current
inventory-38012504/service.serviceusage.cnrm.cloud.google.com/projects/dmu-admin1-hub-kls-dns is Current: Resource is Current
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ 

working out iam permissions issues

[fmichaelobrien]

working additions
main not 0.2.0 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/releases/tag/solutions%2Fproject%2Fhub-env%2F0.2.0

custom FortigateSdnViewer role is in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigate/custom-role.yaml#L27

add
organizations/123456789012/roles/FortigateSdnViewer

and
IAMServiceAccount networking/hub-managementvm-sa

todo:

• add client-landing-zone on top of existing core-landing-zone before hub-env

Package Inventory

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone
• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/client-landing-zone
• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/project/hub-env

[obriensystems]

add packages

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/client-landing-zone@main
Package "client-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 + ea2e57f...10ca23d main       -> origin/main  (forced update)
Adding package "solutions/client-landing-zone".

Fetched 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/client-setup@main
Package "client-setup":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
Adding package "solutions/client-setup".

Fetched 1 package(s).

[obriensystems]

Current status (deploying hub) - move from #445
kcc.landing.zone
root_@cloudshell:/kcc-kls/lz-20230803$ ls
client-landing-zone client-setup core-landing-zone hub-env setters.yaml
root_@cloudshell:/kcc-kls/lz-20230803$

[obriensystems]

Known Issues Workarounds

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigates.md#known-issues
• turn down security on the management VPC - turn back on public access for googleapis.com public access without putting in a private service connect endpoint like
• Interconnect: Add private DNS zones with domain forwarding/peering to enable Private Google Access or Private Service Connect workload calls from on-prem - using AWS VPN for Interconnect simulation #468

[obriensystems]

restarting hub-env adjustment

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt fn render hub-env
Package "hub-env": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.1s
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    ...(102 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 1.7s
  Results:
    [info]: no matches

Successfully executed 2 function(s) in 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$  kpt live apply hub-env --reconcile-timeout=2m --output=table

config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Successful    Failed                  Ready                                     1167h   Update call failed: error fetching live 
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Skipped       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Successful    Current                 Ready                                     1167h   Resource is Current                     
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Successful    Current                 Ready                                     1167h   Resource is Current                     
config-con  IAMPolicyMember/networking-sa-computeins  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
networking  ComputeAddress/hub-fgt-primary-ext-addre  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-int-addre  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-transit-a  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-ext-add  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-int-add  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-transit  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-ilb-address            Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-ilb-proxy-address      Successful    Current                 Ready                                     1167h   Resource is Current                     
NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Successful    Failed                  Ready                                     1167h   Update call failed: error fetching live 
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Skipped       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Successful    Current                 Ready                                     1167h   Resource is Current                     
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Successful    Current                 Ready                                     1167h   Resource is Current                     
config-con  IAMPolicyMember/networking-sa-computeins  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
networking  ComputeAddress/hub-fgt-primary-ext-addre  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-int-addre  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-transit-a  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-ext-add  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-int-add  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-transit  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-ilb-address            Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-ilb-proxy-address      Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeBackendService/hub-ilb-bes         Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-primary-log-disk      Skipped       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-secondary-log-disk    Skipped       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-mgmt-data-disk            Skipped       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-external-fwr    Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeFirewall/hub-allow-fortigates-ha-  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-allow-spokes-to-fort  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-elb-allow-health-che  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-iap-allow-rdp-to-man  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-ilb-allow-health-che  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-managementvm-allow-s  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeForwardingRule/hub-ilb-fwdrule     Pending       Unknown                 -                                         -                                               
networking  ComputeForwardingRule/hub-ilb-proxy-fwdr  Pending       Unknown                 -                                         -                                               
networking  ComputeHTTPHealthCheck/hub-http-8008-htt  Skipped       Unknown                 -                                         -                                               
networking  ComputeHealthCheck/hub-http-8008-hc       Skipped       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-fgt-primary-instance  Successful    InProgress              Ready                                     1167h   reference ComputeDisk networking/hub-fgt
networking  ComputeInstance/hub-fgt-secondary-instan  Successful    InProgress              Ready                                     1167h   reference ComputeDisk networking/hub-fgt
networking  ComputeInstance/hub-management-instance   Successful    InProgress              Ready                                     1167h   reference ComputeDisk networking/hub-mgm
networking  ComputeInstanceGroup/hub-fgt-primary-umi  Pending       Unknown                 -                                         -                                               
networking  ComputeInstanceGroup/hub-fgt-secondary-u  Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-external-vpc    Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeNetwork/hub-global-internal-vpc    Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeNetwork/hub-global-mgmt-vpc        Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeNetwork/hub-global-transit-vpc     Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeRoute/hub-external-vpc-internet-e  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeRoute/hub-internal-vpc-internet-e  Pending       Unknown                 -                                         -                                               
networking  ComputeRouter/hub-nane1-external-router   Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeRouterNAT/hub-nane1-external-nat   Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-external-paz  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-internal-paz  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-mgmt-rz-snet  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-transit-paz-  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeTargetPool/hub-elb-pool            Pending       Unknown                 -                                         -                                               
networking  DNSPolicy/hub-external-logging-dnspolicy  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  DNSPolicy/hub-internal-logging-dnspolicy  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  DNSPolicy/hub-mgmt-logging-dnspolicy      Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  DNSPolicy/hub-transit-logging-dnspolicy   Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  IAMPolicyMember/hub-admin-serviceaccount  Skipped       Unknown                 -                                         -                                               
networking  IAMServiceAccount/hub-fortigatesdn-sa     Skipped       Unknown                 -                                         -                                               
networking  IAMServiceAccount/hub-managementvm-sa     Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     1167h   Resource is Current                     
projects    Project/dmu-admin1-hub-kls                Successful    Current                 Ready                                     1167h   Resource is Current                     
projects    Service/dmu-admin1-hub-kls-compute        Successful    Current                 Ready                                     1167h   Resource is Current                     
projects    Service/dmu-admin1-hub-kls-dns            Successful    Current                 Ready                                     1167h   Resource is Current          

[obriensystems]

Restarting clean org fortigate install for monday
obrien.industries

Deployment change - we will switch to an in-place kpt render (right in the github repo) - so we can track changes

see fine tuning of the wiki documentation in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#quickstart

Install the KCC cluster and minimal set of Landing Zone packages

• 0 overview https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/README.md#quickstart
• 1 see kcc cluster install script in https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh
• 2 see landing zone install docs in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/README.md#2-create-your-landing-zone

Scenarios

• workflow 1 - shut down cluster only (create kcc cluster - create lz (render kpt lz packages) - delete kcc cluster - recreate cluster (automatically acquire gcp resources)
• workflow 2 - shut down lz and cluster (create kcc cluster - create lz (render kpt lz packages) - delete lz packages - delete kcc cluster

prereq = billing quota above 5, liens commented in the code, org polices (gatekeeper) omitted

create kcc cluster =
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L107

gcloud anthos config controller get-credentials $CLUSTER  --location $REGION

but use
https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh

gcloud anthos config controller create "$CLUSTER" --location "$REGION" --network "$NETWORK" --subnet "$SUBNET" --master-ipv4-cidr-block="172.16.0.128/28" --full-management "${args[@]}"
else

create lz =
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L165

# packages core-landing-zone, client-landing-zone, client-setup, project/hub-env
kpt live init core-landing-zone --namespace config-control --force
kpt fn render core landing-zone
kpt live apply core-landing-zone --reconcile-timeout=2m --output=table

delete lz = (including liens)
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L198C8-L198C8

gcloud alpha resource-manager liens delete $NONPROD_LIEN # all 3
kpt live destroy core-landing-zone

delete kcc cluster =
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L206

gcloud anthos config controller delete --location $REGION $CLUSTER --quiet

Plan

• clean org
• install kcc cluster - via script
• install 4 lz packages via script (to be re-written from kcc lz 1 version at https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L164 under Add dev deployment.sh for KCC cluster CD create/delete and LZ initial deployment/delete with canary #192

    1  export EMAIL=michael@obrien.industries
    2  gcloud organizations get-iam-policy roles/resourcemanager.organizationAdmin --filter="bindings.members:$EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
    3  gcloud organizations list
    4  gcloud organizations list | grep ID
    5  gcloud organizations list --format="get(name)"
    6  export DOMAIN=obrien.industries
    7  ORG_ID=$(gcloud organizations list --format="get(name)" --filter=displayName=$DOMAIN)
    8  echo $ORG_ID
    9  gcloud organizations list --format="get(name)" --filter=displayName=$DOMAIN
   10  export ORG_ID=459065442144
   11  gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
   12  gcloud organizations add-iam-policy-binding $ORG_ID  --member=serviceAccount:$EMAIL --role=iam.serviceAccountTokenCreator
   13  gcloud organizations add-iam-policy-binding $ORG_ID  --member=serviceAccount:$EMAIL --role=roles/iam.serviceAccountTokenCreator
   14  gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/iam.serviceAccountTokenCreator
   15  gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/billing.projectManager
   16  mkdir kcc-oi
   17  cd kcc-oi
   18  mkdir github
   19  mkdir kpt
   20  cd github
   21  git clone https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git
   22  git clone https://github.com/ssc-spc-ccoe-cei/gcp-tools.git
   23  cd gcp-tools/scripts/bootstrap/
   24  cp .env.sample kcc.env
   25  export PROJECT_ID=kcc-oi
   26  gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default
   27  gcloud config set project "${PROJECT_ID}"
   28  echo $ORG_ID
   29  export ROOT_FOLDER=kcc
   30  gcloud resource-manager folders create --display-name=$ROOT_FOLDER --organization=$ORG_ID
   31  export BILLING_ID=$(gcloud alpha billing projects describe $PROJECT_ID '--format=value(billingAccountName)' | sed 's/.*\///')
   32  echo $BILLING_ID
   33  gcloud alpha billing projects describe $PROJECT_ID '--format=value(billingAccountName)' 
   34  echo $PROJECT_ID
   35  export BILLING_ID=014479-806359-2F5F85
   36  gcloud beta billing projects link "$PROJECT_ID" --billing-account "$BILLING_ID"
   37  ls
   38  chmod 777 setup-kcc.sh 
   39  ./setup-kcc.sh -af kcc.env
   40  history
   41  gcloud config set project kcc-oi
   42  cd kcc-oi/
   43  ls
   44  cd kpt/
   45  PACKAGE="solutions/gatekeeper-policies"
   46  VERSION=$(curl -s $URL | jq -r ".\"$PACKAGE\"")
   47  URL=https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
   48  VERSION=$(curl -s $URL | jq -r ".\"$PACKAGE\"")
   49  curl -s $URL
   50  https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
   51  curl -s $URL | jq -r ".\"$PACKAGE\""
   52  VERSION=main
   53  kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${PACKAGE}@${VERSION}
   54  PACKAGE="solutions/core-landing-zone"
   55  kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${PACKAGE}@${VERSION}
   56  gcloud organizations list
   57  kpt live init core-landing-zone --namespace config-control
   58  kpt fn render core-landing-zone
   59  gcloud config set project kcc-oi
   60  cd kcc-oi/
   61  cd kpt/
   62  ls
   63  kpt fn render core-landing-zone
   64  gcloud config set project kcc-oi
   65  gcloud config set project kcc-oi-cluster
   66  kubectl edit validatingwebhookconfiguration/gatekeeper-validating-webhook-configuration
   67  kubectl get nodes
   68  gcloud config set project kcc-oi
   69  kubectl get nodes
   70  kubectl get pods --all-namespaces
   71  gcloud anthos config controller get-credentials krmapihost-kcc-oi  --location northamerica-northeast1
   72  gcloud config set project kcc-oi-cluster
   73  gcloud anthos config controller get-credentials krmapihost-kcc-oi  --location northamerica-northeast1
   74  gcloud anthos config controller get-credentials kcc-oi  --location northamerica-northeast1
   75  kubens config-control
   76  kubectl get pods --all-namespaces
   77  cd kcc-oi/kpt/
   78  ls
   79  kpt live apply core-landing-zone --reconcile-timeout=2m --output=table
   80  gcloud config set project kcc-oi-cluster
   81  kubectl get pods
   82  ls
   83  ls -la
   84  ls -la .kube/
   85  cat .kube/config
   86  kubectl get nodes
   87  history
   88  gcloud config set project kcc-oi
   89  cd kcc-oi/github/
   90  cd gcp-tools/scripts/bootstrap/
   91  ./setup-kcc.sh -afp kcc.env
   92  kubectl get nodes
   93  history

gcp-tools

michael@cloudshell:~/kcc-oi/github/gcp-tools (kcc-oi)$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   scripts/bootstrap/setup-kcc.sh

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        scripts/bootstrap/kcc.env

kcc.env
export CLUSTER=kcc-oi2
export REGION=northamerica-northeast1
export PROJECT_ID=kcc-oi2-cluster
export LZ_FOLDER_NAME=kcc-lz-20230928b
export NETWORK=kcc-oi2-vpc
export SUBNET=kcc-oi2-sn
export ORG_ID=459065442144
export ROOT_FOLDER_ID=96269513997
export BILLING_ID=014479-806359-2F5F85
#export GIT_USERNAME=obriensystems
#export CONFIG_SYNC_REPO=<Repo for Config Sync> # tierX repo URL
#export CONFIG_SYNC_VERSION='HEAD'
#export CONFIG_SYNC_DIR=<Directory for config sync repo which syncs> # Should default to csync/deploy/<env>

refresh repo with main 20231019

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git pull
remote: Enumerating objects: 202, done.
remote: Counting objects: 100% (202/202), done.
remote: Compressing objects: 100% (99/99), done.
remote: Total 202 (delta 126), reused 163 (delta 102), pack-reused 0
Receiving objects: 100% (202/202), 80.52 KiB | 8.95 MiB/s, done.
Resolving deltas: 100% (126/126), completed with 38 local objects.
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
   8370e06..6260c22  main                                              -> origin/main
 * [new branch]      dependabot/go_modules/cli/golang.org/x/net-0.17.0 -> origin/dependabot/go_modules/cli/golang.org/x/net-0.17.0
 * [new branch]      gh540-fmichaelobrien-temp-fix-kpt-readme          -> origin/gh540-fmichaelobrien-temp-fix-kpt-readme
 * [new branch]      gh563-fix-cleanup-tier1                           -> origin/gh563-fix-cleanup-tier1
 * [new branch]      https-elb-example                                 -> origin/https-elb-example
 * [new tag]         solutions/client-landing-zone/0.4.6               -> solutions/client-landing-zone/0.4.6
 * [new tag]         solutions/client-landing-zone/0.4.7               -> solutions/client-landing-zone/0.4.7
 * [new tag]         solutions/client-setup/0.6.1                      -> solutions/client-setup/0.6.1
Updating 8370e06..6260c22
Fast-forward
 .release-please-manifest.json                                                                        |   4 +-
 docs/landing-zone-v2/README.md                                                                       |   7 +-
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/README.md                |  27 +++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/address.yaml             |  27 +++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/backend-service.yaml     |  60 ++++++++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/elb.yaml                 |  27 +++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/firewall.yaml            |  38 ++++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/forwarding-rule.yaml     |  34 ++++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/health-check.yaml        |  32 +++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/target-https-proxy.yaml  |  33 +++++
 examples/landing-zone-v2/setters.yaml                                                                |   2 +
 solutions/client-landing-zone/CHANGELOG.md                                                           |  14 +++
 solutions/client-landing-zone/README.md                                                              |  10 +-
 solutions/client-landing-zone/client-folder/firewall-policy/policy.yaml                              |   9 +-
 solutions/client-landing-zone/client-folder/firewall-policy/rules/defaults.yaml                      |  17 ++-
 solutions/client-landing-zone/client-folder/firewall-policy/rules/iap.yaml                           |   6 +-
 solutions/client-landing-zone/client-folder/firewall-policy/rules/lb-health-checks.yaml              |   6 +-
 solutions/client-landing-zone/client-folder/firewall-policy/rules/os-updates.yaml                    |  10 +-
 solutions/client-landing-zone/client-folder/folder-iam.yaml                                          |   3 +-
 solutions/client-landing-zone/client-folder/folder-sink.yaml                                         |   8 +-
 .../client-folder/standard/applications-infrastructure/firewall-policy/policy.yaml                   |   7 +-
 .../client-folder/standard/applications-infrastructure/firewall-policy/rules/defaults.yaml           |  17 ++-
 .../client-folder/standard/applications-infrastructure/firewall-policy/rules/iap.yaml                |   6 +-
 .../client-folder/standard/applications-infrastructure/firewall-policy/rules/lb-health-checks.yaml   |   6 +-
 .../client-folder/standard/applications-infrastructure/host-project/network/firewall.yaml            |  46 +++++--
 .../standard/applications-infrastructure/host-project/network/psc/google-apis/firewall.yaml          |   3 +-
 .../client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml              |   5 +-
 .../client-folder/standard/applications-infrastructure/host-project/network/vpc.yaml                 |   2 +-
 solutions/client-landing-zone/client-folder/standard/firewall-policy/policy.yaml                     |   5 +-
 solutions/client-landing-zone/client-folder/standard/firewall-policy/rules/network-isolation.yaml    |  11 +-
 solutions/client-landing-zone/logging-project/cloud-logging-bucket.yaml                              |   8 +-
 solutions/client-landing-zone/logging-project/project-iam.yaml                                       |   3 +-
 solutions/client-landing-zone/securitycontrols.md                                                    | 326 +++++++++++++++++++++++++++++++++++++++++++++++---
 solutions/client-landing-zone/setters.yaml                                                           |  24 ++--
 solutions/client-setup/CHANGELOG.md                                                                  |   7 ++
 35 files changed, 764 insertions(+), 86 deletions(-)
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/README.md
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/address.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/backend-service.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/elb.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/firewall.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/forwarding-rule.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/health-check.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/target-https-proxy.yaml

create dev branch

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git pull
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * [new branch]      gh446-hub  -> origin/gh446-hub
Already up to date.

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git checkout gh446-hub
Branch 'gh446-hub' set up to track remote branch 'gh446-hub' from 'origin'.
Switched to a new branch 'gh446-hub'

add setters.yaml changes

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git status
On branch gh446-hub
Your branch is up to date with 'origin/gh446-hub'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   solutions/core-landing-zone/setters.yaml
        
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git add solutions/core-landing-zone/setters.yaml 
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git commit -m "#446 - add clz setters.yaml"

undeploy any lz packages up

deploy changes - clz package of 4

push to remote repo

write part 2 lz sh script

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -c false -l false -r false -d false -p kcc-oi2-cluster

see #567

Delete cluster only

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -c false -l false -r false -d true -p kcc-oi2-cluster
Date: Fri 20 Oct 2023 01:37:16 AM UTC
Timestamp: 1697765836
running with: -b kcc-oi -u oi -c false -l false -r false -d true -p kcc-oi2-cluster
Deleting cluster kcc-oi2 in region northamerica-northeast1
Delete Cluster kcc-oi2 in region northamerica-northeast1
Delete request issued for: [kcc-oi2]
Waiting for operation [projects/kcc-oi2-cluster/locations/northamerica-northeast1/operations/operation-1697765858524-6081beae8be0d-93150922-7d4fd45a] to com
plete...done.                                                                                                                                               
Deleted instance [kcc-oi2].
Cluster delete time: 405 sec
Total Duration: 425 sec
Date: Fri 20 Oct 2023 01:44:22 AM UTC
Timestamp: 1697766262
Updated property [core/project].
Switched back to boot project kcc-oi
**** Done ****

Recreate KCC cluster

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi2-cluster)$ ./setup.sh -b kcc-oi -u oi -c true -l false -r false -d false -p kcc-oi2-cluster
Date: Fri 20 Oct 2023 01:55:26 AM UTC
Timestamp: 1697766926
running with: -b kcc-oi -u oi -c true -l false -r false -d false -p kcc-oi2-cluster

Reusing project: kcc-oi2-cluster
Updated property [core/project].
Creating Anthos KCC autopilot cluster kcc-oi2 in region northamerica-northeast1 in subnet kcc-oi2-sn off VPC kcc-oi2-vpc
Create request issued for: [kcc-oi2]
Waiting for operation [projects/kcc-oi2-cluster/locations/northamerica-northeast1/operations/operation-1697766931672-6081c2adface2-48f8ae7f-81f8305c] to com
plete...working.. 

plete...done.                                                                                                                                               
Created instance [kcc-oi2].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi2.
Cluster create time: 1105 sec
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi2.
Context "gke_kcc-oi2-cluster_northamerica-northeast1_krmapihost-kcc-oi2" modified.
Active namespace is "config-control".
List Clusters:
NAME: kcc-oi2
LOCATION: northamerica-northeast1
STATE: RUNNING
Total Duration: 1111 sec
Date: Fri 20 Oct 2023 02:13:58 AM UTC
Timestamp: 1697768038
Updated property [core/project].
Switched back to boot project kcc-oi

deploy changes - clz package of 4

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi2-cluster)$ ./setup.sh -b kcc-oi -u oi -c false -l true -r false -d false -p kcc-oi2-cluster

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/hierarchy                       Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/logging                         Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/networking                      Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/policies                        Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/projects                        Successful    Current                 <None>                                    4m      Resource is current                     
config-con  IAMCustomRole/gke-firewall-admin          Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier2-dnsrecord-admin       Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier2-vpcpeering-admin      Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier3-dnsrecord-admin       Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier3-firewallrule-admin    Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier3-subnetwork-admin      Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier3-vpcsc-admin           Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier4-secretmanager-admin   Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPartialPolicy/gatekeeper-admin-sa-wor  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/hierarchy-sa-workload-i  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/logging-sa-workload-ide  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/networking-sa-workload-  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/policies-sa-workload-id  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/projects-sa-workload-id  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/config-control-sa-orgrol  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/gatekeeper-admin-sa-metr  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/hierarchy-sa-folderadmin  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/logging-sa-bigqueryadmin  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/logging-sa-logadmin-perm  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-dns-permis  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-networkadm  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-security-p  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-service-co  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-servicedir  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-xpnadmin-p  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/policies-sa-orgpolicyadm  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-billinguser-  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-projectcreat  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-projectdelet  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-projectiamad  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-projectmover  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-serviceusage  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMServiceAccount/gatekeeper-admin-sa     Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/hierarchy-sa            Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/logging-sa              Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/networking-sa           Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/policies-sa             Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/projects-sa             Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  Service/kcc-oi-cluster-accesscontextmana  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  Service/kcc-oi-cluster-cloudbilling       Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  Service/kcc-oi-cluster-cloudresourcemana  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  Service/kcc-oi-cluster-serviceusage       Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
gatekeeper  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    4m      status.healthy is true                  
hierarchy   ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
hierarchy   RoleBinding/allow-folders-resource-refer  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   Folder/audits                             Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
hierarchy   Folder/clients                            Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
hierarchy   Folder/services                           Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
hierarchy   Folder/services-infrastructure            Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
logging     ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
logging     LoggingLogBucket/platform-and-component-  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogBucket/security-log-bucket-oi   Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/logging-project-oi-securi  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-disa  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-plat  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     RoleBinding/allow-logging-resource-refer  Successful    Current                 <None>                                    2m      Resource is current                     
networking  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
networking  DNSManagedZone/dns-project-oi-standard-c  Skipped       Unknown                 -                                         -                                               
policies    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
policies    ResourceManagerPolicy/compute-disable-gu  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-disable-ne  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-disable-se  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-disable-vp  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-require-os  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-require-sh  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-require-sh  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-restrict-s  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-skip-defau  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/essentialcontacts-  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/gcp-restrict-resou  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/iam-allowed-policy  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/iam-disable-servic  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/sql-restrict-publi  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/storage-public-acc  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/storage-uniform-bu  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
projects    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
projects    IAMAuditConfig/logging-project-data-acce  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/mgmt-project-cluster-pl  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/platform-and-component-  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/platform-and-component-  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/security-log-bucket-wri  Skipped       Unknown                 -                                         -                                               
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    2m      Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    2m      Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    2m      Resource is current                     
projects    Project/dns-project-oi                    Skipped       Unknown                 -                                         -                                               
projects    Project/logging-project-oi                Successful    InProgress              Ready                                     2m      reference Folder hierarchy/audits is not
projects    Service/dns-project-oi-dns                Skipped       Unknown                 -                                         -                                               

wait for cnrm workloads to come up - 5 min first

triage - folder errors

ierarchy   Folder/audits                             Skipped       Failed                  Ready                                     13m     Update call failed: error applying desir
hierarchy   Folder/clients                            Skipped       Failed                  Ready                                     13m     Update call failed: error applying desir
hierarchy   Folder/services                           Skipped       Failed                  Ready                                     13m     Update call failed: error applying desir
hierarchy   Folder/services-infrastructure            Skipped       Failed                  Ready                                     13m     Update call failed: error applying desir

switched setters.yaml
management-project-id: kcc-oi-cluster
management-project-id: kcc-oi2-cluster

delete partial lz

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi2-cluster)$ ./setup.sh -b kcc-oi -u oi -c false -l false -r true -d false -p kcc-oi2-cluster

resource-group-system             resource-group-controller-manager-7dbf5b5766-s9sr7         2/2     Running   0             32m
deleting lz on kcc-oi2 in region northamerica-northeast1
delete phase started
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-serviceusage delete successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudresourcemanager delete successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudbilling delete successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-accesscontextmanager delete successful
delete phase finished
reconcile phase started
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-serviceusage reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudresourcemanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudbilling reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-accesscontextmanager reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 4 attempted, 4 successful, 0 skipped, 0 failed
reconcile result: 4 attempted, 4 successful, 0 skipped, 0 failed, 0 timed out
Total Duration: 21 sec
Date: Fri 20 Oct 2023 02:40:22 AM UTC
Timestamp: 1697769622
Updated property [core/project].
Switched back to boot project kcc-oi

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl delete gcp --all
iamcustomrole.iam.cnrm.cloud.google.com "gke-firewall-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier2-dnsrecord-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier2-vpcpeering-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-dnsrecord-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-firewallrule-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-subnetwork-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-vpcsc-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier4-secretmanager-admin" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "gatekeeper-admin-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "hierarchy-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "logging-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "networking-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "policies-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "projects-sa-workload-identity-binding" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-management-project-editor-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-management-project-serviceaccountadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-orgroleadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "gatekeeper-admin-sa-metric-writer-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "hierarchy-sa-folderadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "logging-sa-bigqueryadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "logging-sa-logadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-dns-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-networkadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-security-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-service-control-org-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-servicedirectoryeditor-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-xpnadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "policies-sa-orgpolicyadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-billinguser-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectcreator-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectdeleter-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectiamadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectmover-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-serviceusageadmin-permissions" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "gatekeeper-admin-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "hierarchy-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "logging-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "networking-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "policies-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "projects-sa" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-cluster-accesscontextmanager" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-cluster-cloudbilling" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-cluster-cloudresourcemanager" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-cluster-serviceusage" deleted

adjust setters.yaml

delete/recreate cluster

 dns-name: "obrien.industries."
  management-project-number: "180205379034"

Triage pod failures

kubectl get gcp
kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa

  Warning  UpdateFailed  36s (x12 over 12m)  iamserviceaccount-controller  Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Caller does not have required permission to use project kcc-oi2-cluster. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=kcc-oi2-cluster and then retry. Propagation of the new permission may take a few minutes.

Adding security Admin to super admin (is in kcc.landing.systems)
and adding service usage consumer role

2400: got it - should not have commented out the gKE service account - the yakima one

 # Assign Permissions to the KCC Service Account - will need a currently running kcc cluster
#  export SA_EMAIL="$(kubectl get ConfigConnectorContext -n config-control -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"

#  echo "SA_EMAIL: ${SA_EMAIL}"
#  ROLES=("roles/bigquery.dataEditor" "roles/serviceusage.serviceUsageAdmin" "roles/logging.configWriter" "roles/resourcemanager.projectIamAdmin" "roles/resourcemanager.organizationAdmin" "roles/iam.organizationRoleAdmin" "roles/compute.networkAdmin" "roles/resourcemanager.folderAdmin" "roles/resourcemanager.projectCreator" "roles/resourcemanager.projectDeleter" "roles/resourcemanager.projectMover" "roles/iam.securityAdmin" "roles/orgpolicy.policyAdmin" "roles/serviceusage.serviceUsageConsumer" "roles/billing.user" "roles/accesscontextmanager.policyAdmin" "roles/compute.xpnAdmin" "roles/iam.serviceAccountAdmin" "roles/serviceusage.serviceUsageConsumer" "roles/logging.admin") 
#  for i in "${ROLES[@]}" ; do
    # requires iam.securityAdmin
    #ROLE=`gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$SA_EMAIL" --flatten="bindings[].members" --format="table(bindings.role)" | grep $i`
    #echo $ROLE
    #if [ -z "$ROLE" ]; then
#        echo "Applying role $i to $SA_EMAIL"
#        gcloud organizations add-iam-policy-binding $ORG_ID  --member=serviceAccount:$SA_EMAIL --role=$i --quiet > /dev/null 1>&1
    #else
    #    echo "Role $i already set on $USER"
    #fi
#  done


running
kube-system                       netd-w5m97                                                 1/1     Running   0               69m
kube-system                       node-local-dns-5gfds                                       1/1     Running   0               73m
kube-system                       node-local-dns-flq8w                                       1/1     Running   0               69m
kube-system                       node-local-dns-krw4v                                       1/1     Running   0               2m19s
kube-system                       node-local-dns-mm2k8                                       1/1     Running   0               69m
kube-system                       node-local-dns-pqrqb                                       1/1     Running   0               6m47s
kube-system                       pdcsi-node-cjjs9                                           2/2     Running   0               69m
kube-system                       pdcsi-node-hdz4x                                           2/2     Running   0               73m
kube-system                       pdcsi-node-ntc5r                                           2/2     Running   0               2m23s
kube-system                       pdcsi-node-svd24                                           2/2     Running   0               6m51s
kube-system                       pdcsi-node-tpmmv                                           2/2     Running   0               69m
resource-group-system             resource-group-controller-manager-7dbf5b5766-z9ncd         2/2     Running   0               5m19s
SA_EMAIL: service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Applying role roles/bigquery.dataEditor to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/serviceusage.serviceUsageAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/logging.configWriter to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.projectIamAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.organizationAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/iam.organizationRoleAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/compute.networkAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.folderAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.projectCreator to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.projectDeleter to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.projectMover to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/iam.securityAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/orgpolicy.policyAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/serviceusage.serviceUsageConsumer to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/billing.user to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/accesscontextmanager.policyAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/compute.xpnAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/iam.serviceAccountAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/serviceusage.serviceUsageConsumer to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/logging.admin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
kpt live init

raised
#568

services coming up now

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp
NAME                                                                AGE     READY   STATUS         STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          7m27s   False   UpdateFailed   7m26s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       7m27s   True    UpToDate       77s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      7m26s   False   UpdateFailed   7m26s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       7m26s   False   UpdateFailed   7m26s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    7m26s   False   UpdateFailed   7m26s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      7m26s   True    UpToDate       77s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           7m26s   False   UpdateFailed   7m25s
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   7m25s   False   UpdateFailed   7m25s

NAME                                                                                       AGE     READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   7m26s   False   DependencyNotReady   7m26s
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          7m26s   False   DependencyNotFound   7m26s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            7m26s   False   DependencyNotFound   7m26s
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         7m26s   False   DependencyNotReady   7m25s
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           7m25s   False   DependencyNotReady   7m25s
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           7m25s   False   DependencyNotFound   7m25s

NAME                                                                                                             AGE     READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                7m25s   False   UpdateFailed   7m24s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   7m25s   False   UpdateFailed   7m24s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             7m24s   True    UpToDate       76s
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          7m24s   True    UpToDate       63s
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   7m24s   False   UpdateFailed   7m24s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   7m24s   False   UpdateFailed   7m24s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        7m24s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     7m23s   False   UpdateFailed   7m22s
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 7m22s   False   UpdateFailed   7m22s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    7m22s   False   UpdateFailed   7m22s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 7m22s   False   UpdateFailed   7m22s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 7m22s   False   UpdateFailed   7m21s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                7m21s   False   UpdateFailed   7m21s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   7m21s   False   UpdateFailed   7m20s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              7m20s   False   UpdateFailed   7m20s

NAME                                                              AGE     READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   7m20s   False   UpdateFailed   7m19s
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          7m20s   False   UpdateFailed   7m19s
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            7m19s   False   UpdateFailed   7m19s
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         7m19s   False   UpdateFailed   7m19s
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           7m19s   False   UpdateFailed   7m17s
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           7m18s   False   UpdateFailed   7m17s

NAME                                                                              AGE     READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-accesscontextmanager   7m19s   True    UpToDate   30s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudbilling           7m19s   True    UpToDate   2m44s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudresourcemanager   7m18s   True    UpToDate   41s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-serviceusage           7m18s   True    UpToDate   2m44s


michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp | grep UpToDate | wc -l
40
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp | grep UpToDate | wc -l
44


michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi2-cluster)$ kubectl get gcp
NAME                                                                AGE     READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          4m25s   True    UpToDate   4m25s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       4m25s   True    UpToDate   4m25s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      4m25s   True    UpToDate   4m24s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       4m25s   True    UpToDate   4m24s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    4m24s   True    UpToDate   4m24s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      4m24s   True    UpToDate   4m24s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           4m24s   True    UpToDate   4m23s
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   4m24s   True    UpToDate   4m23s

NAME                                                                                       AGE     READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   4m23s   True    UpToDate   4m16s
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          4m23s   True    UpToDate   4m15s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            4m23s   True    UpToDate   4m16s
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         4m23s   True    UpToDate   4m14s
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           4m23s   True    UpToDate   4m14s
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           4m22s   True    UpToDate   3m31s

NAME                                                                                                             AGE     READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                4m22s   True    UpToDate   4m4s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   4m22s   True    UpToDate   4m4s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             4m22s   True    UpToDate   4m16s
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          4m22s   True    UpToDate   4m4s
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   4m21s   True    UpToDate   3m52s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   4m21s   True    UpToDate   3m43s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        4m21s   True    UpToDate   4m9s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          4m20s   True    UpToDate   3m42s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 4m20s   True    UpToDate   3m42s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     4m20s   True    UpToDate   4m5s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          4m20s   True    UpToDate   3m56s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       4m20s   True    UpToDate   4m4s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     4m19s   True    UpToDate   3m56s
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 4m19s   True    UpToDate   3m55s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    4m19s   True    UpToDate   3m19s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 4m19s   True    UpToDate   3m23s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 4m19s   True    UpToDate   3m5s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                4m19s   True    UpToDate   3m4s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   4m18s   True    UpToDate   3m4s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              4m18s   True    UpToDate   3m4s

NAME                                                              AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   4m18s   True    UpToDate   4m17s
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          4m18s   True    UpToDate   4m16s
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            4m17s   True    UpToDate   4m16s
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         4m17s   True    UpToDate   4m16s
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           4m17s   True    UpToDate   4m15s
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           4m17s   True    UpToDate   3m32s

NAME                                                                              AGE     READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-accesscontextmanager   4m20s   True    UpToDate   4m15s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudbilling           4m19s   True    UpToDate   4m16s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudresourcemanager   4m19s   True    UpToDate   4m16s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-serviceusage           4m19s   True    UpToDate   4m16s

[obriensystems]

ssc-spc-ccoe-cei/gcp-tools#53
and
#568

editupdate: found them in the new 2nd script

https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/configure-kcc-access.sh#L35

Issue is that the access script assumes rootsync usage - it leaves out the kpt optionI recommend we put the yakima service account role additions back to the generic setup script.

[obriensystems]

automation test target env root at landing.systems
dev target obrien.enginnering
partial fortigate kcc.landing.systems
pull/run
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/gh446-hub

skip #296 (comment)
move to #445 (comment)

need to be done manually

verify org level sa roles in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/solutions/core-landing-zone/0.3.0/docs/landing-zone-v2/README.md#1-complete-the-bootstrap-procedure
create landing-zone folder
create kcc-boot-ls project in the folder

switch to automation

repo already cloned
mkdir kpt folder at the root
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit$ cd solutions/
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions$ gcloud config set project kcc-boot-ls
Updated property [core/project].
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls)$ 
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls)$ mkdir ../../kpt

push super admin changes in #570

[obriensystems]

merge main

michaelobrien@mbp7 pubsec-declarative-toolkit % git merge main
Merge made by the 'ort' strategy.
 .github/workflows/scorecards.yml                                                                           |   2 +-
 .release-please-manifest.json                                                                              |   2 +-
 docs/landing-zone-v2/README.md                                                                             |  71 ++++++++++++++++++++++++++--
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/Kptfile                                  |  18 +++++++
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/README.md                                |  77 ++++++++++++++++++++++++++++++
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/iam-tier4-sa.yaml                        |  33 +++++++++++++
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/iam-users.yaml                           |  63 +++++++++++++++++++++++++
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/setters.yaml                             |  46 ++++++++++++++++++
 examples/landing-zone-v2/configconnector/tier3/cloud-armor/security-policy.yaml                            |  57 ++++++++++++++++++----
 solutions/core-landing-zone/CHANGELOG.md                                                                   |   7 +++
 solutions/core-landing-zone/org/org-sink.yaml                                                              |   3 ++
 solutions/experimentation/core-landing-zone/README.md                                                      | 180 +++++++++++++++++++++++++++++++++++++---------------------------------
 solutions/experimentation/core-landing-zone/lz-folder/audits/logging-project/monitoring/metrics-scope.yaml |  23 +++++++++
 solutions/experimentation/core-landing-zone/lz-folder/audits/logging-project/services.yaml                 |  41 ++++++++++++++++
 solutions/experimentation/core-landing-zone/namespaces/config-management-monitoring.yaml                   |  81 ++++++++++++++++++++++++++++++++
 solutions/experimentation/core-landing-zone/namespaces/logging.yaml                                        |  35 ++++++++++++++
 16 files changed, 639 insertions(+), 100 deletions(-)
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/client-project-iam/Kptfile
 create mode 100755 examples/landing-zone-v2/configconnector/tier3/client-project-iam/README.md
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/client-project-iam/iam-tier4-sa.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/client-project-iam/iam-users.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/client-project-iam/setters.yaml
 create mode 100644 solutions/experimentation/core-landing-zone/lz-folder/audits/logging-project/monitoring/metrics-scope.yaml
 create mode 100644 solutions/experimentation/core-landing-zone/lz-folder/audits/logging-project/services.yaml
 create mode 100644 solutions/experimentation/core-landing-zone/namespaces/config-management-monitoring.yaml

[fmichaelobrien]

From #654

KCC_PROJECT_ID reset on run KCC GKE cluster only without LZ deploy

  SA_EMAIL="$(kubectl get ConfigConnectorContext -n config-control -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"
  echo "post GKE cluster create - applying 2 roles to org: ${ORG_ID} and project: ${KCC_PROJECT_ID} on the yakima gke service account to prep for kpt deployment: $SA_EMAIL"
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/resourcemanager.organizationAdmin --condition=None --quiet  > /dev/null 1>&1
  gcloud projects add-iam-policy-binding "${KCC_PROJECT_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/serviceusage.serviceUsageConsumer" --project "${KCC_PROJECT_ID}" --quiet  > /dev/null 1>&1
  # need service account admin for kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.organizationRoleAdmin --condition=None --quiet > /dev/null 1>&1
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.serviceAccountAdmin --condition=None --quiet > /dev/null 1>&1
fi

line 88

  # set KCC project id for case where we initially create the KCC cluster without rerunning with passed in -p project_id
  KCC_PROJECT_ID=$CC_PROJECT_ID

[fmichaelobrien]

State:
Hi, there is a tracking issue on bringing up the fortigates that details every workaround/fix (3 so far) involved in deploying hub-env on top of core-landing-zone over the weekend of Oct 20th in prep of posting the deployment steps for wed the 25th.  Most of the changes in the gh446-hub branch were merged into main at that time.  The hub-env package is still being adjusted to bring it up to a full prod state in that branch.

#446 (comment)
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/gh446-hub

such as
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/612/files#diff-c0fda181f1f31975bd6d6ccb1d9a1bb827a8ff54dac4ac7757bc6de13c0303d2L37

  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
-  namespace: config-control # kpt-set: ${management-namespace}
+  namespace: networking

in the larger set of issues
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues?q=is%3Aissue+is%3Aopen+label%3Afortinet

There is WIP automation going into automating the hub-env setters.yaml in 
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L357
The script above is a combination of KCC cluster bootstrap (reuse or recreation of the GKE cluster is optional) and deployment of the clz and hub-env packages - but it is still in dev.

The yakima role associations are in both scripts in addition to the readme at
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L209
and
https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/configure-kcc-access.sh#L28

thank you
/michael

[obriensystems]

Note: config-control namespace override

• Currently retesting in Add single Fortigate VM partner-ready minimal extract for dev - separate from production level hub-env/fortigate package and without need for core-landing-zone #654 and Demo: Automated minimal landing zone with a (hub-env and core-landing-zone) KPT based deployment on a clean GCP organization - walkthrough #611
  https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigate/service-account.yaml#L37
  needs
  https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/612/files#diff-0453fd24870bbd1f648b0d14dcbd3877e4eceafba5cf535c1b962ac63969a94fR22
  on top of existing
  Update to Fortinet Hub ENV Package #609
  for
  hub-env: reference networking-sa service account in config controller project from hub-env setters.yaml #607

project/hub-env/fortigate/service-account.yaml:37

kind: IAMPolicyMember
metadata:
  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
  namespace: config-control # kpt-set: ${management-namespace}
 

via
project/hub-env/setters.yaml:22
  # keep config-control as the default
  management-namespace: config-control 

[obriensystems]

generated kcc project_id propagation to the end in yakima/sa role additions retested in #654

0648
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u ar -n true -c true -l false -h false -d false -j false
existing project: 
Date: Wed 06 Dec 2023 11:48:39 AM UTC
Timestamp: 1701863319
running with: -b kcc-oi -u ar -c true -l false -h false -r false -d false -p 
Updated property [core/project].
Switched back to boot project kcc-oi
Start: 1701863320
unique string: ar
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc
Creating project: kcc-oi-6475
CC_PROJECT_ID: kcc-oi-6475
BOOT_PROJECT_ID: kcc-oi
BILLING_ID: 014479-806359-2F5F85
ORG_ID: 459..44
applying roles to the super admin SUPER_ADMIN_EMAIL: michael@obrien.industries
Updated IAM policy for organization [4..44].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..].
Updated IAM policy for organization [4.4]..
Updated IAM policy for organization [4..144].
Updated IAM policy for organization [459..44].
Creating KCC project: kcc-oi-6475 on folder: 38862..43
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-oi-6475].
Waiting for [operations/cp.5638443903817105010] to finish...done.                                                                                           
Enabling service [cloudapis.googleapis.com] on project [kcc-oi-6475]...
Operation "operations/acat.p2-993154031891-29201c86-a034-44cc-a146-92e3e696b676" finished successfully.
Updated property [core/project] to [kcc-oi-6475].
Updated property [core/project].
Enabling billing on account: 014..85
billingAccountName: billingAccounts/014..5
billingEnabled: true
name: projects/kcc-oi-6475/billingInfo
projectId: kcc-oi-6475
sleep 45 sec before enabling services
Enabling APIs
Operation "operations/acf.p2-993154031891-7d0764e3-2cd3-49e7-8fb3-102ebcc9c323" finished successfully.
Operation "operations/acat.p2-993154031891-d64f4422-74fd-48c8-a84b-c664d443bb03" finished successfully.
Operation "operations/acat.p2-993154031891-512f8af5-90e8-42e4-8ec0-5b6ad758cf31" finished successfully.
Operation "operations/acat.p2-993154031891-cf30917a-8316-439f-b3c4-67035ae22681" finished successfully.
Operation "operations/acat.p2-993154031891-de537f80-1838-463a-991e-5dfb9fbcd191" finished successfully.
Operation "operations/acat.p2-993154031891-fc32e1ef-6444-4b10-af5a-73a29e981b21" finished successfully.
name: organizations/459065442144/settings
storageLocation: northamerica-northeast1
Create VPC: kcc-ls-vpc
Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-6475/global/networks/kcc-ls-vpc].
NAME: kcc-ls-vpc
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp:22,tcp:3389,icmp

Create subnet kcc-ls-sn off VPC: kcc-ls-vpc using 192.168.0.0/16 on region: northamerica-northeast1
Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-6475/regions/northamerica-northeast1/subnetworks/kcc-ls-sn].
NAME: kcc-ls-sn
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 
create default firewalls
Creating Anthos KCC autopilot cluster kcc in region northamerica-northeast1 in subnet kcc-ls-sn off VPC kcc-ls-vpc on project kcc-oi-6475
Create request issued for: [kcc]
Waiting for operation [projects/kcc-oi-6475/locations/northamerica-northeast1/operations/operation-1701863484478-60bd5f872d47b-3203efc2-fe59f81d] to complet
e...working        

Waiting for operation [projects/kcc-oi-6475/locations/northamerica-northeast1/operations/operation-1701863484478-60bd5f872d47b-3203efc2-fe59f81d] to complet
e...working.                                                                                                                                                
e...working.                                                                                                                                                
e...working..                                                                                                                                               
e...done.                                                                                                                                                   
Created instance [kcc].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
Cluster create time: 1107 sec
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
List Clusters:
NAME: kcc
LOCATION: northamerica-northeast1
STATE: RUNNING
post GKE cluster create - applying 2 roles to org: 459065442144 and project: kcc-oi-6475 on the yakima gke service account to prep for kpt deployment: service-993154031891@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Updated IAM policy for project [kcc-oi-6475].
Updated IAM policy for organization [459065442144].
Updated IAM policy for organization [459065442144].
Total Duration: 1282 sec
Date: Wed 06 Dec 2023 12:10:02 PM UTC
Timestamp: 1701864602
Updated property [core/project].
Switched back to boot project kcc-oi
**** Done ****
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$                 

711

[obriensystems]

delete/recreate KCC GKE cluster - then re-acquire resources by id
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L524

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit (kcc-boot-ls-8704)$ gcloud anthos config controller delete --location northamerica-northeast1 kcc-oi4
You are about to delete instance [kcc-oi4]

Do you want to continue (Y/n)?  Y

Delete request issued for: [kcc-oi4]
Waiting for operation [projects/kcc-boot-ls-8704/locations/northamerica-northeast1/operations/operation-1703862526504-60da768a36c60-f3b99c97-6bd1b089] to complete...working..   

Deleted instance [kcc-oi4].        

[fmichaelobrien]

move the partially completed kpt version script in 446 that completed the core-landing-zone and was mid way through hub-env

to #766

• Add client-landing-zone and client-setup
• merge periodic package automations into main in phases

[fmichaelobrien]

Restarting hub-env deployment
existing deployment moved from oi to ls

[obriensystems]

all 4 core-landing-zone, client-setup, client-landing-zone and client-project-setup done/fixed

client-project-setup setters generation

data:
  org-id: "${ORG_ID}"  
  management-project-id: "${KCC_PROJECT_ID}"
  management-namespace: "${MANAGEMENT_NAMESPACE}"
  client-name: client-${PREFIX_CLIENT_SETUP}
  client-management-project-id: client-management-project-${PREFIX_CLIENT_SETUP}
  host-project-id: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}
  # see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml#L26
  #allowed-nane1-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  #allowed-nane2-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane1-main-subnet: nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane2-main-subnet: nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  project-id: client-project-${PREFIX_CLIENT_PROJECT_SETUP}
  project-billing-id: "${BILLING_ID}"
#  project-parent-folder: clients.client-${PREFIX_CLIENT_SETUP}.standard.applications-infrastructure.${CLIENT_PROJECT_PARENT_FOLDER}
  project-parent-folder: standard.applications.${CLIENT_PROJECT_PARENT_FOLDER}
  repo-url: git-repo-to-observe
  repo-branch: main
  tier3-repo-dir: csync/tier3/configcontroller/deploy/env
  tier4-repo-dir: csync/tier4/configcontroller/deploy/env
EOF

see #766
see 84afc61

[fmichaelobrien]

add to setup.sh - Anoop's RBAC addition in
#834
#766 deprecated this #446

[obriensystems]

Nat issue fixed by adding a restrictCloudNATUsage project level override for hub-env in #837