GCP: Updated install flow and managed infra

[samos123]

• Changes the Service Account that the controller-manager and gcpmanager use to `substratus
• Removed all terraform except parts related to cluster and nodepool creation
• Updated gcp-up.sh to create bucket, gar, GSA and required permissions
• Add skaffold support gcp for both the controller-manager and csi

The GCP branch is working very good so far. Here is what I tested:

• New installation flow brings up cluster and script with gcloud commands adds required permissions
  make dev-run-gcp and verifying workload identity things got installed
• in-cluster deployment by using make dev-skaffold-gcp and doing the following in a new namespace:
  • Load a model
  • Finetune a model
  • Load a dataset
  • Serve a model
  • Run a notebook using upload

Issue has been resolved by retrying and sleeping for 5 seconds during initial startup of sci-gcp. There was a long delay after annotating the K8s SA for it to be able to utilize the GSA. As a result the sci-gcp container will restart many times until GKE workload identity finally starts working. You would originally see the following error message:

ERROR   setup   failed to validate server       {"error": "Post \"https://iam.googleapis.com/v1/projects/sam-XX/serviceAccounts/substratus@sam-XX.iam.gserviceaccount.com:getIamPolicy?alt=json&prettyPrint=false\": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\thttps://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`"}

Issue: Kaniko builder gets stuck when creating a new namespace

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "us-central1-docker.pkg.de...

Solved by running an initContainer that validates GKE metadata is ready for the pod: https://cloud.google.com/kubernetes-engine/docs/troubleshooting/troubleshooting-security#workload-identity

[nstogner]

Can you remove config/sci-kind and config/install-kind? These will cause merge conflicts later. Other than that, I dont see blockers to merge.

[brandonjbjelland]

I expected to see a companion gcp-down.sh. What, do you work for Google or something? 😛

[samos123]

oh right I hadn't gotten to it. Good catch!

[brandonjbjelland]

If this is fully working, awesome work and ship it when you're ready.