Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken test runs: gcloud crashing #28612

Closed
krousey opened this issue Jul 7, 2016 · 17 comments
Closed

Broken test runs: gcloud crashing #28612

krousey opened this issue Jul 7, 2016 · 17 comments

Comments

@krousey
Copy link
Member

@krousey krousey commented Jul 7, 2016

gcloud is crashing due to ERROR: gcloud crashed (error): [Errno 111] Connection refused

see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10711/ for an example.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/19753/ seems to have some backtraces. This may indicate that it has something to do with the metadata server.

At least one of the failures in:
#28336
#28083
#28013
#27839
#27767

@krousey
Copy link
Member Author

@krousey krousey commented Jul 7, 2016

Seems we're having metadata issues in general. Also filed #28608 this morning.

@krousey
Copy link
Member Author

@krousey krousey commented Jul 7, 2016

cc @fejta @spxtr possibly related to #23545

@spxtr
Copy link
Contributor

@spxtr spxtr commented Jul 7, 2016

We don't run the metadata cache anymore now that we only run one test per VM. That we're still seeing Error 111 is worrying.

@fabioy
Copy link
Contributor

@fabioy fabioy commented Jul 7, 2016

This may be a known issue and it's being worked on internally on GCE.

@fejta
Copy link
Contributor

@fejta fejta commented Jul 7, 2016

In the mean time I'll work on kubernetes/test-infra#266 to mitigate this.

@fejta
Copy link
Contributor

@fejta fejta commented Jul 9, 2016

Please reactivate next time you see this happen.

@fejta fejta closed this Jul 9, 2016
k8s-github-robot added a commit that referenced this issue Jul 9, 2016
Automatic merge from submit-queue

Activate a service account if this is defined

For kubernetes/test-infra#266 and #28612

This will activate a service account if the service account file exists.
@ncdc
Copy link
Member

@ncdc ncdc commented Jul 11, 2016

failed to find client for version v1: error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. happened again yesterday in #27715 (comment)

k8s-github-robot added a commit that referenced this issue Jul 11, 2016
Automatic merge from submit-queue

Inject service-account.json into test container

Add a volume with the service account credentials. This should cause e2e-runner.sh to use them.

Fixes #28612
@fejta fejta reopened this Jul 11, 2016
@fejta
Copy link
Contributor

@fejta fejta commented Jul 11, 2016

#28794 is also needed -- right now we're starting clusters and then blowing up.

@zeg-io
Copy link

@zeg-io zeg-io commented Sep 29, 2016

According to Google, their instructions state to run the following two commands:

gcloud container clusters get-credentials projectname-cluster-1 \
    --zone us-central1-c --project projectname

kubectl proxy

however, the kubectl command returns:

error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

This issue is where all google searches lead. Anything I can do to fix it?

@aeneasr
Copy link

@aeneasr aeneasr commented Oct 3, 2016

Same issue here as @zeg-io - Did you solve that yet?

@zeg-io
Copy link

@zeg-io zeg-io commented Oct 3, 2016

Nope. I have an open issue with Google as well, been emailing back and forth oh-so-slowly. Nothing resolved.

Latest from Google: "Well, hmm, that should work..."

Me: "Gee, thanks"

@aeneasr
Copy link

@aeneasr aeneasr commented Oct 3, 2016

That's unfortunate, thanks for the heads up

@zeg-io
Copy link

@zeg-io zeg-io commented Oct 3, 2016

If I get it resolved will try to remember to post resolution here.

@jedipunkz
Copy link

@jedipunkz jedipunkz commented Oct 5, 2016

I have same issue as @zeg-io .

@zeg-io
Copy link

@zeg-io zeg-io commented Oct 5, 2016

Gentlemen, Here is more information from Google rep, read ALL of it, because the first interaction didn't resolve it as I had no idea where my "keyfile.json" file was.

Your ~/.kube/config is missing some value for authentication. To resolve this, run the command below on your gcloud shell (Google Console):

$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"

Next, you'll need to run the get-credentials command below again to fetch credentials for a running cluster.

$ gcloud container clusters get-credentials project-cluster-1 --zone us-central1-c

Then, you may now run 'kubectl proxy' command again and let me know if you're still having trouble.

I asked where the keyfile.json file was...

The keyfile.json is a service account key that can be manually generated under IAM & Admin --> Service accounts. Currently, you have not created such key.

However, there's still an alternative and easier way to that. Run the command below on your Google Console.

$ gcloud config set container/use_client_certificate True

Then, rerun:

$ gcloud container clusters get-credentials project-cluster-1 --zone us-central1-c

Let me know if you'll get expected results as you run kubectl proxy again.

So this time when I ran that I no longer got an error... BUT...

Now when I run kubectl proxy it returns:

Starting to serve on 127.0.0.1:8001

But when I use a browser to access that it returns:

<h3>Unauthorized</h3>

Incidentally, I also tried the first method, generating the key and pointing at it to no effect.

@crockpotveggies
Copy link

@crockpotveggies crockpotveggies commented Oct 7, 2016

Following instructions from @zeg-io I then followed some discussion from this issue kubernetes/dashboard#692 to find a solution.

kubectl proxy --port=9090

Then navigate to:

localhost:9090/ui

And then the UI successfully opens. Why is the base domain failing to redirect?

@zeg-io
Copy link

@zeg-io zeg-io commented Oct 7, 2016

@crockpotveggies adding the /ui actually worked for me without changing the port... but... it doesn't seem to be connected to my project, at least, not in a recognizable way for someone who is unfamiliar with the tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants