Broken test runs: gcloud crashing

[krousey]

gcloud is crashing due to ERROR: gcloud crashed (error): [Errno 111] Connection refused

see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10711/ for an example.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/19753/ seems to have some backtraces. This may indicate that it has something to do with the metadata server.

At least one of the failures in:
#28336
#28083
#28013
#27839
#27767

[krousey]

Seems we're having metadata issues in general. Also filed #28608 this morning.

[krousey]

cc @fejta @spxtr possibly related to #23545

[spxtr]

We don't run the metadata cache anymore now that we only run one test per VM. That we're still seeing Error 111 is worrying.

[fabioy]

This may be a known issue and it's being worked on internally on GCE.

[fejta]

In the mean time I'll work on kubernetes/test-infra#266 to mitigate this.

[fejta]

Please reactivate next time you see this happen.

[ncdc]

failed to find client for version v1: error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. happened again yesterday in #27715 (comment)

[fejta]

#28794 is also needed -- right now we're starting clusters and then blowing up.

[zeg-io]

According to Google, their instructions state to run the following two commands:

gcloud container clusters get-credentials projectname-cluster-1 \
    --zone us-central1-c --project projectname

kubectl proxy

however, the kubectl command returns:

error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

This issue is where all google searches lead. Anything I can do to fix it?

[aeneasr]

Same issue here as @zeg-io - Did you solve that yet?

[zeg-io]

Nope. I have an open issue with Google as well, been emailing back and forth oh-so-slowly. Nothing resolved.

Latest from Google: "Well, hmm, that should work..."

Me: "Gee, thanks"

[aeneasr]

That's unfortunate, thanks for the heads up

[zeg-io]

If I get it resolved will try to remember to post resolution here.

[jedipunkz]

I have same issue as @zeg-io .

[zeg-io]

Gentlemen, Here is more information from Google rep, read ALL of it, because the first interaction didn't resolve it as I had no idea where my "keyfile.json" file was.

Your ~/.kube/config is missing some value for authentication. To resolve this, run the command below on your gcloud shell (Google Console):

$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"

Next, you'll need to run the get-credentials command below again to fetch credentials for a running cluster.

$ gcloud container clusters get-credentials project-cluster-1 --zone us-central1-c

Then, you may now run 'kubectl proxy' command again and let me know if you're still having trouble.

I asked where the keyfile.json file was...

The keyfile.json is a service account key that can be manually generated under IAM & Admin --> Service accounts. Currently, you have not created such key.

However, there's still an alternative and easier way to that. Run the command below on your Google Console.

$ gcloud config set container/use_client_certificate True

Then, rerun:

$ gcloud container clusters get-credentials project-cluster-1 --zone us-central1-c

Let me know if you'll get expected results as you run kubectl proxy again.

So this time when I ran that I no longer got an error... BUT...

Now when I run kubectl proxy it returns:

Starting to serve on 127.0.0.1:8001

But when I use a browser to access that it returns:

<h3>Unauthorized</h3>

Incidentally, I also tried the first method, generating the key and pointing at it to no effect.

[crockpotveggies]

Following instructions from @zeg-io I then followed some discussion from this issue kubernetes/dashboard#692 to find a solution.

kubectl proxy --port=9090

Then navigate to:

localhost:9090/ui

And then the UI successfully opens. Why is the base domain failing to redirect?

[zeg-io]

@crockpotveggies adding the /ui actually worked for me without changing the port... but... it doesn't seem to be connected to my project, at least, not in a recognizable way for someone who is unfamiliar with the tool.