[BUG]: selective failure applying to the cc via kpt live apply - for those that fail add --inventory-policy adopt

[obriensystems]

Describe the bug
kpt live apply - having periodic issues applying to the cc (after a successful replacement render (see #103 and #111)
same issue as
kptdev/kpt#1724

To Reproduce

michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kubens
cnrm-system
config-control
config-management-monitoring
config-management-system
configconnector-operator-system
default
gatekeeper-system
krmapihosting-monitoring
krmapihosting-system
kube-node-lease
kube-public
kube-system
resource-group-system
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ cd landing-zone/
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103/landing-zone (lz-20220910-oldev)$ kpt fn render
Package "landing-zone/environments/common/guardrails-policies":
Package "landing-zone/environments/common":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 500ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone/environments/nonprod":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 400ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone/environments/prod":
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 1.6s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-logging: recreated service
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 400ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 600ms
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "583675367868"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "583675367868"
    [info] spec.projectID: set field value to "net-perimeter-prj-common-old1"
    [info] spec.parentRef.external: set field value to "583675367868"
    ...(87 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/generate-folders:v0.1.1"
[PASS] "gcr.io/kpt-fn/generate-folders:v0.1.1" in 900ms
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 1.8s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-compute: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-dns: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-logging: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    ...(3 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/gatekeeper:v0.2.1"
[PASS] "gcr.io/kpt-fn/gatekeeper:v0.2.1" in 1.3s
[RUNNING] "gcr.io/kpt-fn/kubeval:v0.3.0"
[PASS] "gcr.io/kpt-fn/kubeval:v0.3.0" in 14s

Successfully executed 9 function(s) in 5 package(s).
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103/landing-zone (lz-20220910-oldev)$ cd ..
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live init landing-zone --namespace config-control
initializing Kptfile inventory info (namespace: config-control)...failed
Error: Inventory information has already been added to the package Kptfile. Changing it after a package has been applied to the cluster can lead to undesired results. Use the --force flag to suppress this error. 
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live apply landing-zone
installing inventory ResourceGroup CRD.
namespace/config-control apply failed: can't adopt an object without the annotation config.k8s.io/owning-inventory
namespace/config-control reconcile skipped
configmap/setters unchanged

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Solution

running with
--inventory-policy adopt

looks to work
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live apply landing-zone --inventory-policy adopt
installing inventory ResourceGroup CRD.
namespace/config-control configured
namespace/config-control reconcile pending
namespace/config-control reconciled
configmap/setters created

folders coming up

using --inventory-policy adopt via https://github.com/GoogleContainerTools/kpt/issues/1724
works well

before

michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live apply landing-zone
installing inventory ResourceGroup CRD.
namespace/config-control apply failed: can't adopt an object without the annotation config.k8s.io/owning-inventory
namespace/config-control reconcile skipped
configmap/setters unchanged
..
iampolicymember.iam.cnrm.cloud.google.com/log-sink-writer reconcile skipped
0 resource(s) reconciled, 90 skipped, 0 failed to reconcile, 0 timed out
1 resources failed


after
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live apply landing-zone --inventory-policy adopt
installing inventory ResourceGroup CRD.
namespace/config-control configured
namespace/config-control reconcile pending
namespace/config-control reconciled
configmap/setters created
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels created
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/nonprodperimaccesslevel created
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/prodaccesslevels created
accesscontextmanageraccesspolicy.accesscontextmanager.cnrm.cloud.google.com/orgaccesspolicy created
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet created
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pr created
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pu created
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingress-pr created
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingressp created
computefirewall.compute.cnrm.cloud.google.com/computefirewall-sample-deny created
computefirewall.compute.cnrm.cloud.google.com/deny-ssh-ingress created
computefirewall.compute.cnrm.cloud.google.com/prod-firewall-default-deny created
computenetwork.compute.cnrm.cloud.google.com/common-ha-perimeter created
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter created
computenetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc created
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter created
computenetwork.compute.cnrm.cloud.google.com/prod-sharedvpc created
computenetwork.compute.cnrm.cloud.google.com/public-perimeter created
computeprojectmetadata.compute.cnrm.cloud.google.com/nonprod-oslogin-meta created
computeroute.compute.cnrm.cloud.google.com/egress-internet-nonprod created
computeroute.compute.cnrm.cloud.google.com/egress-internet-prod created
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample created
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host created
computesubnetwork.compute.cnrm.cloud.google.com/common-ha-perimeter-subnet created
computesubnetwork.compute.cnrm.cloud.google.com/management created
computesubnetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc-subnet created
computesubnetwork.compute.cnrm.cloud.google.com/priv-perimeter-subnet created
computesubnetwork.compute.cnrm.cloud.google.com/prod-sharedvpc-subnet created
computesubnetwork.compute.cnrm.cloud.google.com/public-perimeter-subnet created
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer created
iampolicymember.iam.cnrm.cloud.google.com/billing-iam-member created
iampolicymember.iam.cnrm.cloud.google.com/log-reader created
iampolicymember.iam.cnrm.cloud.google.com/log-writer created
iampolicymember.iam.cnrm.cloud.google.com/organization-viewer created
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account created
logginglogsink.logging.cnrm.cloud.google.com/audit-bucket-sink created
logginglogsink.logging.cnrm.cloud.google.com/logs-bucket-sink created
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security created
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit created
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security created
folder.resourcemanager.cnrm.cloud.google.com/automation created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure created
folder.resourcemanager.cnrm.cloud.google.com/sandbox created
folder.resourcemanager.cnrm.cloud.google.com/shared-services created
folder.resourcemanager.cnrm.cloud.google.com/workloads created
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev created
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod created
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat created
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1 created
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 created
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-nonprod-old1 created
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-prod-old1 created
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-old1 created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-contact-domains created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-policy-member-domain created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-guest-attribute-access created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-nested-virtualization created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serial-port-access created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serviceaccount-key-creation created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-vpc-external-ipv6 created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-shielded-vm created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-trusted-images created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-os-login created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-sql-public-ip created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vm-external-access created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward created
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute created
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging created
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute created
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns created
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging created
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute created
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging created
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-old1 created
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-old1 created
configmap/setters reconciled
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels reconcile pending
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/nonprodperimaccesslevel reconcile pending
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/prodaccesslevels reconcile pending
accesscontextmanageraccesspolicy.accesscontextmanager.cnrm.cloud.google.com/orgaccesspolicy reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pu reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingress-pr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingressp reconcile pending
computefirewall.compute.cnrm.cloud.google.com/computefirewall-sample-deny reconcile pending
computefirewall.compute.cnrm.cloud.google.com/deny-ssh-ingress reconcile pending
computefirewall.compute.cnrm.cloud.google.com/prod-firewall-default-deny reconcile pending
computenetwork.compute.cnrm.cloud.google.com/common-ha-perimeter reconcile pending
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter reconcile pending
computenetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter reconcile pending
computenetwork.compute.cnrm.cloud.google.com/prod-sharedvpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/public-perimeter reconcile pending
computeprojectmetadata.compute.cnrm.cloud.google.com/nonprod-oslogin-meta reconcile pending
computeroute.compute.cnrm.cloud.google.com/egress-internet-nonprod reconcile pending
computeroute.compute.cnrm.cloud.google.com/egress-internet-prod reconcile pending
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample reconcile pending
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/common-ha-perimeter-subnet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/management reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc-subnet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/priv-perimeter-subnet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/prod-sharedvpc-subnet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/public-perimeter-subnet reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/billing-iam-member reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/log-reader reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/log-writer reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/organization-viewer reconciled
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account reconcile pending
logginglogsink.logging.cnrm.cloud.google.com/audit-bucket-sink reconcile pending
logginglogsink.logging.cnrm.cloud.google.com/logs-bucket-sink reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/automation reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/sandbox reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/shared-services reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/workloads reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat reconcile pending
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-nonprod-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-prod-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-old1 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-contact-domains reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-policy-member-domain reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-guest-attribute-access reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-nested-virtualization reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serial-port-access reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serviceaccount-key-creation reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-vpc-external-ipv6 reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-shielded-vm reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-trusted-images reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-os-login reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-sql-public-ip reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vm-external-access reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward reconcile pending
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute reconcile pending
E0910 15:28:03.551790    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_common-nethost-service-logging_serviceusage.cnrm.cloud.google.com_Service
E0910 15:28:03.553439    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_nonprod-nethost-service-compute_serviceusage.cnrm.cloud.google.com_Service
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging reconcile pending
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute reconcile pending
E0910 15:28:03.553505    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_nonprod-nethost-service-dns_serviceusage.cnrm.cloud.google.com_Service
E0910 15:28:03.553533    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_nonprod-nethost-service-logging_serviceusage.cnrm.cloud.google.com_Service
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns reconcile pending
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging reconcile pending
E0910 15:28:03.553578    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_prod-nethost-service-compute_serviceusage.cnrm.cloud.google.com_Service
E0910 15:28:03.553607    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_prod-nethost-service-logging_serviceusage.cnrm.cloud.google.com_Service
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute reconcile pending
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging reconcile pending
E0910 15:28:03.553645    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_audit-audit-prj-id-old1_storage.cnrm.cloud.google.com_StorageBucket
E0910 15:28:03.553666    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_log-bucket-audit-prj-id-old1_storage.cnrm.cloud.google.com_StorageBucket
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-old1 reconcile pending
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-old1 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile failed
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile failed
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation reconciled
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-old1 reconcile failed
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-old1 reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security reconciled
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter reconcile failed
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconciled
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute reconcile failed
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging reconcile failed
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/automation reconciled
folder.resourcemanager.cnrm.cloud.google.com/infrastructure reconciled
folder.resourcemanager.cnrm.cloud.google.com/sandbox reconciled
folder.resourcemanager.cnrm.cloud.google.com/shared-services reconciled
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns reconcile failed
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging reconcile failed
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute reconcile failed
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/workloads reconciled
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit reconciled
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security reconciled
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter reconcile failed
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host reconcile failed
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1 reconcile failed
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 reconcile failed
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-old1 reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/log-reader reconciled
iampolicymember.iam.cnrm.cloud.google.com/log-writer reconciled
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod reconciled
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking reconciled
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat reconciled
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample reconcile failed
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking reconciled
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking reconciled
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure reconciled
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev reconciled

will take over an hour to bring up the system
state so far is

michael@cloudshell:~ (magellan-01)$ kubectl get gcp
NAME                                                                                          AGE   READY   STATUS         STATUS AGE
accesscontextmanageraccesspolicy.accesscontextmanager.cnrm.cloud.google.com/orgaccesspolicy   21m   False   UpdateFailed   21m

NAME                                                                                                 AGE   READY   STATUS               STATUS AGE
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels        21m   False   DependencyNotReady   21m
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/nonprodperimaccesslevel   21m   False   DependencyNotReady   21m
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/prodaccesslevels          21m   False   DependencyNotReady   21m

NAME                                                                        AGE   READY   STATUS         STATUS AGE
computeprojectmetadata.compute.cnrm.cloud.google.com/nonprod-oslogin-meta   21m   False   UpdateFailed   21m

NAME                                                                         AGE   READY   STATUS               STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/common-ha-perimeter-subnet   21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/management                   21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc-subnet     21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/priv-perimeter-subnet        21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/prod-sharedvpc-subnet        21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/public-perimeter-subnet      21m   False   DependencyNotReady   21m

NAME                                                                                           AGE   READY   STATUS         STATUS AGE
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample   21m   False   UpdateFailed   21m
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host              21m   False   UpdateFailed   21m

NAME                                                                 AGE   READY   STATUS         STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/common-ha-perimeter     21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter   21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc       21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter          21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/prod-sharedvpc          21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/public-perimeter        21m   False   UpdateFailed   21m

NAME                                                                 AGE   READY   STATUS               STATUS AGE
computeroute.compute.cnrm.cloud.google.com/egress-internet-nonprod   21m   False   DependencyNotReady   21m
computeroute.compute.cnrm.cloud.google.com/egress-internet-prod      21m   False   DependencyNotReady   21m

NAME                                                                        AGE   READY   STATUS               STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet         21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pr      21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pu      21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingress-pr          21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingressp            21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/computefirewall-sample-deny   21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/deny-ssh-ingress              21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/prod-firewall-default-deny    21m   False   DependencyNotReady   21m

NAME                                                            AGE   READY   STATUS               STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer          21m   True    UpToDate             21m
iampolicymember.iam.cnrm.cloud.google.com/billing-iam-member    21m   False   DependencyNotReady   21m
iampolicymember.iam.cnrm.cloud.google.com/log-reader            21m   True    UpToDate             20m
iampolicymember.iam.cnrm.cloud.google.com/log-writer            21m   True    UpToDate             20m
iampolicymember.iam.cnrm.cloud.google.com/organization-viewer   21m   True    UpToDate             21m

NAME                                                                  AGE   READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account   21m   False   UpdateFailed   21m

NAME                                                             AGE   READY   STATUS               STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/audit-bucket-sink   21m   False   DependencyNotReady   21m
logginglogsink.logging.cnrm.cloud.google.com/logs-bucket-sink    21m   False   DependencyNotReady   21m

NAME                                                                                               AGE   READY   STATUS         STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-contact-domains                21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-policy-member-domain           21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-guest-attribute-access         21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-nested-virtualization          21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serial-port-access             21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serviceaccount-key-creation    21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-vpc-external-ipv6              21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-shielded-vm                    21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-trusted-images                 21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types   21m   False   UpdateFailed   21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-os-login                      21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations            21m   False   UpdateFailed   21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-sql-public-ip                 21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vm-external-access            21m   False   UpdateFailed   21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal              21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering                   21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation          21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention       21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access            21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward                      21m   True    UpToDate       21m

NAME                                                                                       AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security                            21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit                      21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security                   21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/automation                                    21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure                                21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking                     21m   True    UpToDate   20m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking   21m   True    UpToDate   19m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking      21m   True    UpToDate   19m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure           21m   True    UpToDate   19m
folder.resourcemanager.cnrm.cloud.google.com/sandbox                                       21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/shared-services                               21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/workloads                                     21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev                                 21m   True    UpToDate   19m
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod                                21m   True    UpToDate   20m
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat                                 21m   True    UpToDate   20m

NAME                                                                          AGE   READY   STATUS         STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1               21m   False   UpdateFailed   21m
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1         21m   False   UpdateFailed   21m
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-nonprod-old1       21m   False   UpdateFailed   21m
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-prod-old1          21m   False   UpdateFailed   21m
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-old1   21m   False   UpdateFailed   21m

NAME                                                                         AGE   READY   STATUS         STATUS AGE
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute    21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging    21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute   21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns       21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging   21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute      21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging      21m   False   UpdateFailed   21m

NAME                                                                       AGE   READY   STATUS         STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-old1        21m   False   UpdateFailed   21m
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-old1   21m   False   UpdateFailed   21m

[cartyc]

What error are the projects giving you?

kubectl describe project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1

[cartyc]

Re-reading this issue. Are you running kpt live init before every deploy?

[fmichaelobrien]

Yes, understand this is a known issue - documenting for the automated script in
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L113

kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/landing-zone landing-zone
# cp the setters.yaml
cp pubsec-declarative-toolkit/solutions/landing-zone/setters.yaml landing-zone/ 
kpt live init landing-zone --namespace config-control
kpt live apply landing-zone --reconcile-timeout=2m --output=table

michael@cloudshell:~/dev/pdt-oldev/obriensystems (controller-oldev-3495)$ kpt live apply landing-zone --reconcile-timeout=2m --output=table
I1207 18:56:53.675976   12418 request.go:601] Waited for 1.165556178s due to client-side throttling, not priority and fairness, request: GET:https://35.203.38.53/apis/spanner.cnrm.cloud.google.com/v1beta1?timeout=32s
Error: 4 resource types could not be found in the cluster or as CRDs among the applied resources.

Resource types:
[constraints.gatekeeper.sh/v1beta1](http://constraints.gatekeeper.sh/v1beta1), Kind=NamingPolicy
[constraints.gatekeeper.sh/v1beta1](http://constraints.gatekeeper.sh/v1beta1), Kind=DataLocation
[constraints.gatekeeper.sh/v1beta1](http://constraints.gatekeeper.sh/v1beta1), Kind=LimitEgressTraffic
[constraints.gatekeeper.sh/v1beta1](http://constraints.gatekeeper.sh/v1beta1), Kind=CloudMarketPlaceConfig

[fmichaelobrien]

The suggested addition to the root .krmignore works
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/landing-zone/.krmignore#L1
cicd-examples/
+constraint.yaml

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE
            ConstraintTemplate/cloudmarketplaceconfi  Successful    Current                 <None>                                    81s     Resource is current
            ConstraintTemplate/datalocation           Successful    Current                 <None>                                    81s     Resource is current
            ConstraintTemplate/limitegresstraffic     Successful    Current                 <None>                                    81s     Resource is current
            ConstraintTemplate/namingpolicy           Successful    Current                 <None>                                    80s     Resource is current
common      ComputeFirewall/allow-egress-internet-pr                Unknown                 -                                         -
common      ComputeFirewall/allow-egress-internet-pu                Unknown                 -                                         -
common      ComputeFirewall/allow-ssh-ingress-pr                    Unknown                 -                                         -
common      ComputeFirewall/allow-ssh-ingressp                      Unknown                 -                                         -
common      ComputeNetwork/common-ha-perimeter                      Unknown                 -                                         -
common      ComputeNetwork/common-mgmt-perimeter                    Unknown                 -                                         -
common      ComputeNetwork/priv-perimeter                           Unknown                 -                                         -

[obriensystems]

Freed up billing quota - working

Status:
  Conditions:
    Last Transition Time:  2022-12-08T01:46:49Z
    Message:               The resource is up to date
    Reason:                UpToDate
    Status:                True
    Type:                  Ready
  Number:                  1013829665443
  Observed Generation:     2
Events:
  Type     Reason              Age                   From                Message
  ----     ------              ----                  ----                -------
  Warning  DependencyNotReady  49m                   project-controller  reference Folder config-control/audit-and-security.audit is not ready
  Warning  UpdateFailed        34m (x13 over 48m)    project-controller  Update call failed: error applying desired state: summary: failed pre-requisites: missing permission on "billingAccounts/011D7E-BD499C-CF71C5": billing.resourceAssociations.create
  Normal   Updating            2m11s (x29 over 48m)  project-controller  Update in progress
michael@cloudshell:~/dev/pdt-oldev/obriensystems (kcc-lz-8597)$ kubectl describe project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-oldv1

michael@cloudshell:/dev/pdt-oldev/obriensystems (kcc-lz-8597)$ kubectl get gcp | grep UpdateFailed
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations 50m False UpdateFailed 50m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention 50m False UpdateFailed 50m
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-oldv1 50m False UpdateFailed 50m
michael@cloudshell:/dev/pdt-oldev/obriensystems (kcc-lz-8597)$ kubectl get gcp | grep UpdateFailed | wc -l
3

[fmichaelobrien]

dup to #114