New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken test runs: gcloud crashing #28612

Closed
krousey opened this Issue Jul 7, 2016 · 17 comments

Comments

Projects
None yet
9 participants
@krousey
Member

krousey commented Jul 7, 2016

gcloud is crashing due to ERROR: gcloud crashed (error): [Errno 111] Connection refused

see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10711/ for an example.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/19753/ seems to have some backtraces. This may indicate that it has something to do with the metadata server.

At least one of the failures in:
#28336
#28083
#28013
#27839
#27767

@krousey

This comment has been minimized.

Member

krousey commented Jul 7, 2016

Seems we're having metadata issues in general. Also filed #28608 this morning.

@krousey

This comment has been minimized.

Member

krousey commented Jul 7, 2016

cc @fejta @spxtr possibly related to #23545

@spxtr

This comment has been minimized.

Member

spxtr commented Jul 7, 2016

We don't run the metadata cache anymore now that we only run one test per VM. That we're still seeing Error 111 is worrying.

@fabioy

This comment has been minimized.

Member

fabioy commented Jul 7, 2016

This may be a known issue and it's being worked on internally on GCE.

@fejta

This comment has been minimized.

Contributor

fejta commented Jul 7, 2016

In the mean time I'll work on kubernetes/test-infra#266 to mitigate this.

@fejta

This comment has been minimized.

Contributor

fejta commented Jul 9, 2016

Please reactivate next time you see this happen.

@fejta fejta closed this Jul 9, 2016

k8s-merge-robot added a commit that referenced this issue Jul 9, 2016

Merge pull request #28634 from fejta/auth
Automatic merge from submit-queue

Activate a service account if this is defined

For kubernetes/test-infra#266 and #28612

This will activate a service account if the service account file exists.
@ncdc

This comment has been minimized.

Member

ncdc commented Jul 11, 2016

failed to find client for version v1: error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. happened again yesterday in #27715 (comment)

k8s-merge-robot added a commit that referenced this issue Jul 11, 2016

Merge pull request #28780 from fejta/auth
Automatic merge from submit-queue

Inject service-account.json into test container

Add a volume with the service account credentials. This should cause e2e-runner.sh to use them.

Fixes #28612

@fejta fejta reopened this Jul 11, 2016

@fejta

This comment has been minimized.

Contributor

fejta commented Jul 11, 2016

#28794 is also needed -- right now we're starting clusters and then blowing up.

@zeg-io

This comment has been minimized.

zeg-io commented Sep 29, 2016

According to Google, their instructions state to run the following two commands:

gcloud container clusters get-credentials projectname-cluster-1 \
    --zone us-central1-c --project projectname

kubectl proxy

however, the kubectl command returns:

error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

This issue is where all google searches lead. Anything I can do to fix it?

@aeneasr

This comment has been minimized.

aeneasr commented Oct 3, 2016

Same issue here as @zeg-io - Did you solve that yet?

@zeg-io

This comment has been minimized.

zeg-io commented Oct 3, 2016

Nope. I have an open issue with Google as well, been emailing back and forth oh-so-slowly. Nothing resolved.

Latest from Google: "Well, hmm, that should work..."

Me: "Gee, thanks"

@aeneasr

This comment has been minimized.

aeneasr commented Oct 3, 2016

That's unfortunate, thanks for the heads up

@zeg-io

This comment has been minimized.

zeg-io commented Oct 3, 2016

If I get it resolved will try to remember to post resolution here.

@jedipunkz

This comment has been minimized.

jedipunkz commented Oct 5, 2016

I have same issue as @zeg-io .

@zeg-io

This comment has been minimized.

zeg-io commented Oct 5, 2016

Gentlemen, Here is more information from Google rep, read ALL of it, because the first interaction didn't resolve it as I had no idea where my "keyfile.json" file was.

Your ~/.kube/config is missing some value for authentication. To resolve this, run the command below on your gcloud shell (Google Console):

$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"

Next, you'll need to run the get-credentials command below again to fetch credentials for a running cluster.

$ gcloud container clusters get-credentials project-cluster-1 --zone us-central1-c

Then, you may now run 'kubectl proxy' command again and let me know if you're still having trouble.

I asked where the keyfile.json file was...

The keyfile.json is a service account key that can be manually generated under IAM & Admin --> Service accounts. Currently, you have not created such key.

However, there's still an alternative and easier way to that. Run the command below on your Google Console.

$ gcloud config set container/use_client_certificate True

Then, rerun:

$ gcloud container clusters get-credentials project-cluster-1 --zone us-central1-c

Let me know if you'll get expected results as you run kubectl proxy again.

So this time when I ran that I no longer got an error... BUT...

Now when I run kubectl proxy it returns:

Starting to serve on 127.0.0.1:8001

But when I use a browser to access that it returns:

<h3>Unauthorized</h3>

Incidentally, I also tried the first method, generating the key and pointing at it to no effect.

@crockpotveggies

This comment has been minimized.

crockpotveggies commented Oct 7, 2016

Following instructions from @zeg-io I then followed some discussion from this issue kubernetes/dashboard#692 to find a solution.

kubectl proxy --port=9090

Then navigate to:

localhost:9090/ui

And then the UI successfully opens. Why is the base domain failing to redirect?

@zeg-io

This comment has been minimized.

zeg-io commented Oct 7, 2016

@crockpotveggies adding the /ui actually worked for me without changing the port... but... it doesn't seem to be connected to my project, at least, not in a recognizable way for someone who is unfamiliar with the tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment