kfctl apply fails to create k8s resources; tries to deploy to 127.0.0.1

[jlewi]

When I first run kfctl apply -v all it fails with the following error

ERRO[0054] (Will retry) Component metacontroller apply failed; Error: Attempting to deploy to environment 'default' at '127.0.0.1', but cannot locate a server at that address  filename="ksonnet/ksonnet.go:191"

If I rerun kfctl apply -v all it works.

So it looks like there is a problem getting the cluster to talk when we first run kfctl apply and call ksonnet apply.

I'm using version:
v0.4.0-rc.1-264-g20a80512

/cc @gabrielwen @kkasravi

[kkasravi]

I believe this was fixed by #2763 (and #2719)

[jlewi]

@kkasravi Doesn't look like it. I'm using v0.4.0-rc.1-264-g20a80512
20a80512

Which should include those fixes.

@gabrielwen @kkasravi How does the ksonnet KfApp get the credentials and cluster info to talk to?

My conjecture is that when we first call kfctl apply we initialize the ksonnet KfApp with a dummy address like 127.0.0.1 because the cluster doesn't exist yet; but on rerunning it the cluster exists and we now have the cluster configured correctly.

Are we relying on the KUBECONFIG file and gcloud container clusters get-credentials?

[gabrielwen]

• credentials is from either .config/gcloud/application_default.json or GOOGLE_APPLICATION_CREDENTIALS.
• we don't really use gcloud container clusters get-credentials to get cluster info but Ksonnet still uses KUBECONFIG anyway. This might be the root cause of the problem.
• I've created a function that could get rid of KUBECONFIG file: link

[kkasravi]

@jlewi what were your kfctl init and kfctl generate args?

we don't really use gcloud container clusters get-credentials to get cluster info but Ksonnet still uses KUBECONFIG anyway. This might be the root cause of the problem.

Ksonnet shouldn't use KUBECONFIG (and use the config from the platform if the platform has set it - either by expanding the KfDefSpec to include credentials or a function pointer). This is the pattern in general - so that

• adding new platforms will be transparent to ksonnet (or kustomize).
• not specifying a platform won't break default behavior in ksonnet

I can pick this up

[kkasravi]

/assign @kkasravi

[jlewi]

I looked at a recent successful postsubmit run.
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/kubeflow_kubeflow/kubeflow-postsubmit/1294

Logs are attached below. You can see in the logs that the error occurs.

kubeflow-postsubmit-kfctl-go-cefb791-1294-5f69-1259696250.log.txt

[kkasravi]

The error is due to the following:

• we set the ksonnet serverURL to 127.0.0.1 during generate
• in gcp.apply, we update .kube/config with a new entry of the deployment cluster using gcloud
• in ksonnet.apply we call ksonnet envSet with the new serverUrl but without an override flag and w/o reloading the env.