Regression: The networking-sa service account used by the hub-env requires the role roles/iam.serviceAccountAdmin

[obriensystems]

part of #446
The networking-sa service account used by the hub-env requires the role roles/iam.serviceAccountAdmin
Fix was to add the same permissions as for core-landing-zone in the config-control namespace to the hub-env in the networking namespace to the networking-sa service account

before
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                                       AGE    READY   STATUS               STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     128m   False   DependencyNotFound   128m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   128m   False   DependencyNotFound   128m

NAME                                                                                            AGE    READY   STATUS               STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   128m   False   UpdateFailed         128m
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions              128m   False   DependencyNotReady   128m

NAME                                                              AGE    READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   128m   False   UpdateFailed   128m
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   128m   False   UpdateFailed   128m


michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ gcloud projects list --filter="kcc-oi-3552" '--format=value(PROJECT_NUMBER)'
850340197245

iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   119m   False   UpdateFailed   119m
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa  -n networking

    Message:               Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).


service-850340197245@gcp-sa-yakima.iam.gserviceaccount.com | Yakima Service Account for Project 850340197245 | Organization AdministratorOrganization Role AdministratorService Account Admin
-- | -- | --

delete it and wait for recreation after an apply
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl delete iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa  -n networking
iamserviceaccount.iam.cnrm.cloud.google.com "hub-fortigatesdn-sa" deleted

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
...
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed

iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   51s    False   UpdateFailed   51s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   131m   False   UpdateFailed   131m

another approach to get the permission set per project instead of org

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ export SA_EMAIL="$(kubectl get ConfigConnectorContext -n networking -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ echo $SA_EMAIL
networking-sa@kcc-oi-3552.iam.gserviceaccount.com

this one


networking-sa@kcc-oi-3552.iam.gserviceaccount.com | networking-sa | Access Context Manager AdminCompute Shared VPC Admin
-- | -- | --

add role
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.serviceAccountAdmin --condition=None --quiet > /dev/null 1>&1
Updated IAM policy for organization [459065442144].

from

to

networking-sa@kcc-oi-3552.iam.gserviceaccount.com | networking-sa | Access Context Manager AdminCompute Shared VPC AdminService Account Admin
-- | -- | --

fixed without any delete/render
  Warning  UpdateFailed  2m26s (x8 over 6m34s)  iamserviceaccount-controller  Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "iam.googleapis.com",
    "metadata": {
      "permission": "iam.serviceAccounts.create"
    },
    "reason": "IAM_PERMISSION_DENIED"
  }
]
, forbidden
  Normal  Updating  26s (x9 over 6m34s)  iamserviceaccount-controller  Update in progress
  Normal  UpToDate  24s                  iamserviceaccount-controller  The resource is up to date
  
  fixed
  NAME                                                              AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   7m35s   True    UpToDate   84s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   147m    True    UpToDate   51s

after
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                           AGE   READY   STATUS         STATUS AGE
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk   17m   False   UpdateFailed   17m

NAME                                                                       AGE    READY   STATUS               STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     148m   False   DependencyNotFound   148m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   148m   False   DependencyNotFound   148m

NAME                                                                                            AGE    READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   148m   False   UpdateFailed   148m



these 3 are fixed in the queue
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions              148m   True    UpToDate       104s



NAME                                                              AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   8m33s   True    UpToDate   2m22s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   148m    True    UpToDate   109s