hub-env: Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env #837

obriensystems · 2024-02-20T17:20:33Z

Likely due to the recent - couple of weeks ago - forced organization policy additions across GCP orgs - investigating/documenting (will need to check all deployed landing zones as this will only occur on redeployment or new GKE config connector object browser applies)

Part of landing zone automation in #766 and the older #446
see PR #846

Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env

First time redeploying the projects/hub-env package on top of the full 4 package core-lz, client*3

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Failed: Update call failed: error applying desired state: summary: Error creating RouterNat: googleapi: Error 412: Constraint constraints/compute.restrictCloudNATUsage violated for projects/xxdmu-admin1-hub-cso2. projects/xxdmu-admin1-hub-cso2/regions/northamerica-northeast1/subnetworks/nane1-external-paz-snet is not allowed to use Cloud NAT., conditionNotMet

org level

project level - example at the client-landing-zone host-project

need the same for hub-env

The text was updated successfully, but these errors were encountered:

obriensystems · 2024-02-21T20:49:43Z

we need the same project level override on the org policy that cloud-landing-zone has for it's host project
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/org-policies/exceptions/compute-restrict-cloud-nat-usage-except-host-project.yaml#L39

place it on
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/project/hub-env/org-policies/exceptions

with an additional

  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/hub-project-id # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}

obriensystems · 2024-02-21T20:54:10Z

Testing

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#########
#
# GCP Organization Policies
# Org policies that correspond with a Guardrail will contain a label indicating what Guardrails it helps in enforcing
# https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints
#
# Constraint: constraints/compute.restrictCloudNATUsage
#
# This list constraint defines the set of subnetworks that are allowed to use Cloud NAT.
#
# This exception is for the host project as it requires use of Cloud NAT.
#
#########
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata:
  name: compute-restrict-cloud-nat-usage-except-host-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${host-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/hub-project-id # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/PROJECT_ID # kpt-set: under:projects/${hub-project-id}}
  projectRef:
    external: "0000000000" # kpt-set: ${hub-project-id}}

obriensystems · 2024-02-21T21:13:44Z

rendering/applying change

michael@cloudshell:~/kcc-cso (kcc-cso-4380)$ ls
github  kpt
michael@cloudshell:~/kcc-cso (kcc-cso-4380)$ cd kpt/
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render hub-env --truncate-output=false

Successfully executed 2 function(s) in 1 package(s).
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live apply hub-env --reconcile-timeout=15m --output=table


missed one
  name: compute-restrict-cloud-nat-usage-except-host-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${host-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-cso2 # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
    cnrm.cloud.google.com/blueprint: 'kpt-fn'
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/xxdmu-admin1-hub-cso2} # kpt-set: under:projects/${hub-project-id}}
  projectRef:
    external: "xxdmu-admin1-hub-cso2}" # kpt-set: ${hub-project-id}}

fixed

 name: compute-restrict-cloud-nat-usage-except-host-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${hub-project-id}

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render hub-env --truncate-output=false
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live apply hub-env --reconcile-timeout=15m --output=table

checking

restarting vms

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Failed: Update call failed: error applying desired state: summary: Error creating RouterNat: googleapi: Error 412: Constraint constraints/compute.restrictCloudNATUsage violated for projects/xxdmu-admin1-hub-cso2. projects/xxdmu-admin1-hub-cso2/regions/northamerica-northeast1/subnetworks/nane1-external-paz-snet is not allowed to use Cloud NAT., conditionNotMet
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Organization policy for projects/xxdmu-admin1-hub-cso2}: googleapi: Error 400: Request contains an invalid argument., badRequest


found it - double end bracket
        - under:projects/xxdmu-admin1-hub-cso2} # kpt-set: under:projects/${hub-project-id}}
      
rerun
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render hub-env --truncate-output=false
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live apply hub-env --reconcile-timeout=15m --output=table

never mind - found second typo

        - under:projects/xxdmu-admin1-hub-cso2 # kpt-set: under:projects/${hub-project-id}
  projectRef:
    external: "xxdmu-admin1-hub-cso2}" # kpt-set: ${hub-project-id}}

fixed projectRef: - rerunning

failed

this is what we have
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata:
  name: compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 # kpt-set: compute-restrict-cloud-nat-usage-except-${hub-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-cso2 # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
    cnrm.cloud.google.com/blueprint: 'kpt-fn'
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/xxdmu-admin1-hub-cso2 # kpt-set: under:projects/${hub-project-id}
  projectRef:
    external: "xxdmu-admin1-hub-cso2" # kpt-set: ${hub-project-id}

this is a working reference
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata: # kpt-merge: policies/compute-restrict-cloud-nat-usage-except-host-project-id
  name: compute-restrict-cloud-nat-usage-except-net-host-project-cso3 # kpt-set: compute-restrict-cloud-nat-usage-except-${host-project-id}
  namespace: policies
  labels:
    guardrail: "false"
  annotations:
    internal.kpt.dev/upstream-identifier: 'resourcemanager.cnrm.cloud.google.com|ResourceManagerPolicy|policies|compute-restrict-cloud-nat-usage-except-host-project-id'
    cnrm.cloud.google.com/blueprint: 'kpt-pkg-fn-live'
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/net-host-project-cso3 # kpt-set: under:projects/${host-project-id}
  projectRef:
    external: "net-host-project-cso3" # kpt-set: ${host-project-id}


metadata different
add kpt-merge: policies/compute-restrict-cloud-nat-usage-except-host-project-id
remove depends-on

obriensystems · 2024-02-21T21:50:40Z

removing kpt folder yaml - recopy/render to fix collision in policies namespace
to fix

-  name: compute-restrict-cloud-nat-usage-except-host-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${hub-project-id}
+  name: compute-restrict-cloud-nat-usage-except-hub-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${hub-project-id}


resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage                                                       21d   True    UpToDate       21d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380                                   21d   True    UpToDate       21d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-net-host-project-cso3                          15d   True    UpToDate       15d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2                          39m   False   UpdateFailed   39m

render
apply

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Failed: Update call failed: error applying desired state: summary: Error creating RouterNat: googleapi: Error 412: Constraint constraints/compute.restrictCloudNATUsage violated for projects/xxdmu-admin1-hub-cso2. projects/xxdmu-admin1-hub-cso2/regions/northamerica-northeast1/subnetworks/nane1-external-paz-snet is not allowed to use Cloud NAT., conditionNotMet
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Organization policy for projects/xxdmu-admin1-hub-cso2}: googleapi: Error 400: Request contains an invalid argument., badRequest

checking gke object browser - still on old version - deleting manually

obriensystems · 2024-02-21T21:58:56Z

repasting clean yaml to kpt folder, render, apply (renamed file host to hub)

apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata:
  name: compute-restrict-cloud-nat-usage-except2-hub-project-id # kpt-set: compute-restrict-cloud-nat-usage-except2-${hub-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/hub-project-id # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/PROJECT_ID # kpt-set: under:projects/${hub-project-id}
  projectRef:
    external: "0000000000" # kpt-set: ${hub-project-id}

rendered

apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata:
  name: compute-restrict-cloud-nat-usage-except2-xxdmu-admin1-hub-cso2 # kpt-set: compute-restrict-cloud-nat-usage-except2-${hub-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-cso2 # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
    cnrm.cloud.google.com/blueprint: 'kpt-fn'
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/xxdmu-admin1-hub-cso2 # kpt-set: under:projects/${hub-project-id}
  projectRef:
    external: "xxdmu-admin1-hub-cso2" # kpt-set: ${hub-project-id}

fixed - but remove older gke object

ichael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep nat
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Current: Resource is Current
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Organization policy for projects/xxdmu-admin1-hub-cso2}: googleapi: Error 400: Request contains an invalid argument., badRequest
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except2-xxdmu-admin1-hub-cso2 is Current: Resource is Current

obriensystems · 2024-02-21T21:59:48Z

only the trailing original resource left - the nat is fixed

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Organization policy for projects/xxdmu-admin1-hub-cso2}: googleapi: Error 400: Request contains an invalid argument., badRequest

see extended live testing on a deployed hub for #766 under #837

obriensystems · 2024-02-21T22:07:01Z

branch
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/gh837-natpolicyforhub
PR
#846

obriensystems · 2024-02-21T22:10:06Z

gcp

gke object

see extended live testing on a deployed hub for #766 under #837

obriensystems added automation fortinet labels Feb 20, 2024

obriensystems self-assigned this Feb 20, 2024

obriensystems mentioned this issue Feb 20, 2024

Demo: Script: Automated minimal landing zone with a (hub-env and core-landing-zone, client-setup, client-landing-zone, client-project-setup) GitOps/OCI based deployment on a clean GCP organization - walkthrough #766

Open

obriensystems assigned fmichaelobrien Feb 21, 2024

obriensystems added a commit that referenced this issue Feb 21, 2024

Create compute-restrict-cloud-nat-usage-except-hub-project.yaml

d758ec1

see extended live testing on a deployed hub for #766 under #837

This was referenced Feb 21, 2024

Add constraints/compute.restrictCloudNATUsage policy override for hub-env fortigate project - to allow NAT to deploy #837 #846

Merged

Example 258 fortigate perimeter package deploy procedure/verify for core lz unmanaged client #446

Open

fmichaelobrien pushed a commit that referenced this issue Feb 22, 2024

Create compute-restrict-cloud-nat-usage-except-hub-project.yaml (#846)

7e71ec4

see extended live testing on a deployed hub for #766 under #837

fmichaelobrien added the client label Feb 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hub-env: Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env #837

hub-env: Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env #837

obriensystems commented Feb 20, 2024 •

edited by fmichaelobrien

Loading

obriensystems commented Feb 21, 2024 •

edited

Loading

obriensystems commented Feb 21, 2024 •

edited

Loading

obriensystems commented Feb 21, 2024 •

edited

Loading

obriensystems commented Feb 21, 2024

obriensystems commented Feb 21, 2024

obriensystems commented Feb 21, 2024

obriensystems commented Feb 21, 2024

obriensystems commented Feb 21, 2024

hub-env: Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env #837

hub-env: Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env #837

Comments

obriensystems commented Feb 20, 2024 • edited by fmichaelobrien Loading

obriensystems commented Feb 21, 2024 • edited Loading

obriensystems commented Feb 21, 2024 • edited Loading

obriensystems commented Feb 21, 2024 • edited Loading

obriensystems commented Feb 21, 2024

obriensystems commented Feb 21, 2024

obriensystems commented Feb 21, 2024

obriensystems commented Feb 21, 2024

obriensystems commented Feb 21, 2024

obriensystems commented Feb 20, 2024 •

edited by fmichaelobrien

Loading

obriensystems commented Feb 21, 2024 •

edited

Loading

obriensystems commented Feb 21, 2024 •

edited

Loading

obriensystems commented Feb 21, 2024 •

edited

Loading