kfctl apply failed

[lluunn]

kfctl init kfctl1 --platform gcp --skip-init-gcp-project
kfctl generate --email=XXX --ipName=kfctl1 --hostname=kfctl1
kfctl apply --oauth_id=XX --oauth_secret=XX

Got error:

couldn't apply KfApp: gcp apply could not update deployment manager Error could not update storage-kubeflow.yaml: Insert deployment error: Post https://www.googleapis.com/deploymentmanager/v2/projects/kai-test2/global/deployments?alt=json&prettyPrint=false: oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error": "invalid_grant",
  "error_description": "Bad Request"
}

cc @gabrielwen @kunmingg

[kkasravi]

I got this too, my workaround was to download my @.iam.gserviceaccount.com as a json file and set the envvar GOOGLE_APPLICATION_CREDENTIALS to point to it

eg: export GOOGLE_APPLICATION_CREDENTIALS=/Users/kdkasrav/auth.json

[gabrielwen]

I think this is duplicate issue of #2535

[lluunn]

Thanks, the workaround works.

But then I got another error:

...
WARN[0023] Deployment service is not ready: RUNNING
WARN[0024] Deployment service is not ready: RUNNING
ERRO[0026] couldn't apply KfApp: gcp apply could not update deployment manager Error could not update cluster-kubeflow.yaml: Deployment error(400): BAD REQUEST

From the deployment manager page:

Manifest expansion encountered the following errors: Resource name 'kfctl1' is not unique in cluster.jinja. Resource: {'type': 'compute.v1.globalAddress', 'name': 'kfctl1', 'properties': {'description': 'Static IP for Kubeflow ingress.'}} Resource: config

I think the problem is the ip name is the same as something else. We should not have param ipName, but just use {deployment} + "-ip" to avoid this.

[lluunn]

And then I got

ERRO[0232] couldn't apply KfApp: gcp apply could not update deployment manager Error SetIamPolicy error: googleapi: Error 403: The caller does not have permission, forbidden

It was because my service account was only project editor, and it needs to be project owner

[lluunn]

Next error:

ERRO[0288] couldn't apply KfApp: gcp apply could not create secrets Error At least one of --oauth_id or ENV `CLIENT_ID` needs to be set.

But my command is: kfctl apply --oauth_id=xxx --oauth_secret=xxx

[lluunn]

I printed it out, options[string(kftypes.OAUTH_ID)] is nil.
https://github.com/kubeflow/kubeflow/blob/master/bootstrap/pkg/client/gcp/gcp.go#L816

[lluunn]

Next error, seems transient

WARN[0141] Deployment service is not ready: RUNNING
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2bb6d1e]

goroutine 1 [running]:
github.com/kubeflow/kubeflow/bootstrap/pkg/client/gcp.blockingWait.func1(0x0, 0x0)
	/Users/lunkai/go/src/github.com/kubeflow/kubeflow/bootstrap/pkg/client/gcp/gcp.go:266 +0x18e
github.com/cenkalti/backoff.RetryNotify(0xc0000dc100, 0x396ef60, 0xc0001f8420, 0x0, 0x0, 0x0)
	/Users/lunkai/go/pkg/mod/github.com/cenkalti/backoff@v2.1.1+incompatible/retry.go:37 +0xb9
github.com/cenkalti/backoff.Retry(0xc0000dc100, 0x396ef60, 0xc0001f8420, 0x0, 0x0)
	/Users/lunkai/go/pkg/mod/github.com/cenkalti/backoff@v2.1.1+incompatible/retry.go:24 +0x50
github.com/kubeflow/kubeflow/bootstrap/pkg/client/gcp.blockingWait(0xc001d7a0f3, 0x9, 0xc0000420c0, 0x37, 0xc0022280a0, 0x398c6a0, 0xc0000da008, 0x0, 0x0)
	/Users/lunkai/go/src/github.com/kubeflow/kubeflow/bootstrap/pkg/client/gcp/gcp.go:264 +0x1dc
github.com/kubeflow/kubeflow/bootstrap/pkg/client/gcp.(*Gcp).updateDeployment(0xc001d66c40, 0xc001d38fa0, 0x6, 0x30e445f, 0x15, 0x0, 0x0)
	/Users/lunkai/go/src/github.com/kubeflow/kubeflow/bootstrap/pkg/client/gcp/gcp.go:319 +0xc26

https://github.com/kubeflow/kubeflow/blob/master/bootstrap/pkg/client/gcp/gcp.go#L319

[lluunn]

next error:

INFO[0296] ks.Apply: project = kai-test2, zone = us-east1-d name = kfctl1
INFO[0296] ServerVersion: https://35.229.123.230
INFO[0296] namespace: kubeflow
INFO[0297] Writing deploying config to /Users/lunkai/kai/ksapps/kfctl1/ks_app/default.yaml
ERRO[0304] couldn't apply KfApp: gcp apply failed for ksonnet: Writing config file error: find objects: RUNTIME ERROR: Unexpected type null, expected object
	<std>:1184:5-33	builtin function <objectHasEx>
	/Users/lunkai/kai/ksapps/kfctl1/ks_app/vendor/kubeflow/application/application.libsonnet:195:13-48	object <anonymous>
	/Users/lunkai/kai/ksapps/kfctl1/ks_app/vendor/kubeflow/application/application.libsonnet:(191:35)-(200:13)	function <anonymous>

[lluunn]

I was able to apply if I skip the "iap-ingress" component in application component.
Will use #2642 to track.