New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubernetes-dashboard is crashing on new deployments #227

Closed
Manifaust opened this Issue Nov 16, 2017 · 4 comments

Comments

Projects
None yet
5 participants
@Manifaust
Member

Manifaust commented Nov 16, 2017

On kubo/0.9.0-dev.35, the dashboard is stuck in a crash loop. Here're the logs:

$ kubectl log kubernetes-dashboard-6bb5695ccf-cswg9 -n kube-system
W1116 12:40:49.375945   45756 cmd.go:392] log is DEPRECATED and will be removed in a future version. Use logs instead.
2017/11/16 17:38:32 Starting overwatch
2017/11/16 17:38:32 Using kubeconfig file: /var/vcap/jobs/kubeconfig/config/kubeconfig-kubelet
2017/11/16 17:38:32 Skipping in-cluster config
2017/11/16 17:38:32 Using random key for csrf signing
2017/11/16 17:38:32 No request provided. Skipping authorization
2017/11/16 17:39:02 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://master.kubo:8443/version: dial tcp: i/o timeout

I'm running on GCP.

@cf-gitbot

This comment has been minimized.

cf-gitbot commented Nov 16, 2017

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/152932856

The labels on this github issue will be updated when the story is started.

@jasonkeene

This comment has been minimized.

Member

jasonkeene commented Nov 16, 2017

I was getting the same issue yesterday. From my poking around I was able to resolve master.kubo from all the worker nodes:

worker/f730ddbc-2935-42bb-b13f-66c666c57a0a:~$ dig master.kubo +short
10.0.1.7

The port is open and reachable from the worker as well:

PORT     STATE SERVICE        VERSION
8443/tcp open  ssl/https-alt?
| ssl-cert: Subject: commonName=35.203.164.41/organizationName=system:masters
| Not valid before: 2017-11-15T18:00:45+00:00
|_Not valid after:  2018-11-15T18:00:45+00:00

The actual image is built FROM scratch so it has no shell to exec. I didn't feel like copying over a rootfs to see if it is DNS issue inside the container, seems likely though.

@tomsherrod

This comment has been minimized.

tomsherrod commented Nov 16, 2017

kubo/0.9.0-dev.33, on vpshere...same issue, same log output, node digs fine for master.kubo.

@tvs

This comment has been minimized.

Member

tvs commented Nov 16, 2017

This was a leftover issue from the swap to RBAC/service accounts. We forgot to add the ABAC policies for the kube-scheduler, kube-proxy, and route-sync users.

This should be fixed as of cloudfoundry-incubator/kubo-release@aa0dd3e and cloudfoundry-incubator/kubo-release@739f341.

@tvs tvs closed this Nov 16, 2017

@cf-gitbot cf-gitbot added accepted and removed delivered labels Nov 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment