Fails to start after a while #718

sgandon · 2020-05-18T13:56:03Z

Describe the bug
running the version 0.19.5 I am having some issues appearing first after a while and then blocking completly the start of the tool (see at the botton of the issue for logs).
So from a fresh cluster (docker-for-mac or K3d) eveything is running fine until I have some error messages appearing at the bottom like [list watch] access denied on resource "default":"v1/pods"
Then if I quit K9s and start to relaunch it it fails with the logs below.
I am doing some experiments with a webhook admission controller so I wonder if this could be related.
If I delete my cluster and start a fresh one the issue disapear and come later somehow.

To Reproduce
it is hard to describe some steps, I am playing with Kyvero ClusterPolicies but and this issu happen's after a while

Expected behavior
not to crash

Screenshots

Versions (please complete the following information):
On MacOs 10.15.3
It fails on both Docker-for-Desktop : 2.3.0.2
or on
k3d version v1.7.0
k9s version:
Version: 0.19.5
Commit: 9f1b099
Date: 2020-05-15T22:35:38Z
Additional context
start logs

3:40PM INF 🐶 K9s starting up...
3:40PM DBG Active Context "k3s-default"
3:40PM DBG Connecting to API Server https://localhost:6443
3:40PM DBG RESETING CON!!
3:40PM INF ✅ Kubernetes connectivity
3:40PM DBG [Config] Saving configuration...
3:40PM INF No context specific skin file found -- /Users/myname/.k9s/k3s-default_skin.yml
3:40PM DBG CURRENT-NS "" -- No active namespace specified
3:40PM INF No namespace specified using cluster default namespace
3:40PM DBG Factory START with ns `""
3:40PM DBG Connecting to API Server https://localhost:6443
3:40PM WRN   Dial Failed! error="Post \"https://localhost:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": context deadline exceeded"
3:40PM WRN Fail CRDs load error="Post \"https://localhost:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": context deadline exceeded"
3:40PM DBG SkinWatcher watching `/Users/myname/.k9s/skin.yml
3:40PM DBG CustomView watching `/Users/myname/.k9s/views.yml
3:40PM WRN   Dial Failed! error="Post \"https://localhost:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": context deadline exceeded"
3:40PM ERR Saved command load failed. Loading default view error="Post \"https://localhost:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": context deadline exceeded"
3:40PM ERR Boom! [list watch] access denied on resource "default":"v1/pods"
3:40PM ERR goroutine 1 [running]:
runtime/debug.Stack(0x4195040, 0x2c1c703, 0x0)
	runtime/debug/stack.go:24 +0x9d
github.com/derailed/k9s/cmd.run.func1()
	github.com/derailed/k9s/cmd/root.go:73 +0x11d
panic(0x2936e00, 0xc00043b450)
	runtime/panic.go:969 +0x166
github.com/derailed/k9s/cmd.run(0x41753c0, 0xc00000da80, 0x0, 0x2)
	github.com/derailed/k9s/cmd/root.go:89 +0x1ef
github.com/spf13/cobra.(*Command).execute(0x41753c0, 0xc00004c0d0, 0x2, 0x2, 0x41753c0, 0xc00004c0d0)
	github.com/spf13/cobra@v1.0.0/command.go:846 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0x41753c0, 0x0, 0x0, 0x0)
	github.com/spf13/cobra@v1.0.0/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.0.0/command.go:887
github.com/derailed/k9s/cmd.Execute()
	github.com/derailed/k9s/cmd/root.go:64 +0x2d
main.main()
	github.com/derailed/k9s/main.go:27 +0x1a6

The text was updated successfully, but these errors were encountered:

derailed · 2020-05-18T21:11:24Z

@sgandon I think there is something hoarked with this cluster. At first glance the connection to te api server somehow times out. K9s default timeout is set to 5secs. Can u connect to this cluster using kubectl? If I recall correctly there is something funky in the way you connect to k3s in terms of where the kubeconfig is loaded ie the command is wrapped k3s kubectl get po?? Could it be the issue here? ie you need to set KUBECONFIG explicitly. If not, I will suspect an issue with your adm-controller but you can validate this with kubectl get po and see if you can get the resource list given the same user and kubeconfig.

sgandon · 2020-05-19T07:10:22Z

@derailed I forgot in my description that I have no issue at all using the kubectl command and I eventually did use the kubectl command for inspecting my resources.
I did re-create the cluster many times and as I mentionned also had the issue on the docker-for-desktop cluster many times.
As for k3d, the command for the config is the following export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')" which will create a config file locally and assign the path to the KUBECONFIG environment variable, nothing really special on that side.
I wonder why there are some thumbs up on my description, does this mean I am not the only one experiencing this issue ?

leandroandrade-hotmart · 2020-05-19T12:48:02Z

@sgandon Yes, you're not the only one with this problem :/

matheussilva-hotmart · 2020-05-19T14:00:35Z

We are running our k8s cluster on AWS EKS and facing the same issue.

matheussilva-hotmart · 2020-05-19T14:30:43Z

It looks like this change introduced the issue.
If you run k9s -n all it works as intended.

derailed · 2020-05-19T14:52:34Z

@sgandon Thanks for the details! Just wanted to make sure it was not a k3s config issue. I am baffled! I have created a docker-for cluster and left it running for ~ an hour and of course No issues! So I am thinking it's not a cluster origination issue. Same revs as what you are running. I keep going back to a kube config issue either api server url or certs are messed up but I can't seem to repro here at the ranch ;(

Of the folks that up voted here would you mind sharing some repros and logs so you can help me track this down? Any details here would help... Thank you!!

derailed · 2020-05-19T14:55:19Z

@matheussilva-hotmart This might be different issue as it would indicate RBAC restriction on a given namespace. Would you mind including your logs when the error occurs. Thank you Matheus!

matheussilva-hotmart · 2020-05-19T15:09:56Z

matheussilva-hotmart · 2020-05-19T15:20:02Z

When i start using -n all i can switch namespaces with no issues, and that includes the default namespace

sgandon · 2020-05-19T15:50:00Z

I have updated k9s to 0.19.6 and also tried a -n all without success, it still crashes upon start.

matheussilva-hotmart · 2020-05-19T17:17:50Z

Our issues seems also to be related to a VPN/Firewall. Turning off the VPN greatly improves the connection stability.

nat2k5us · 2020-05-19T18:06:32Z

I have been experiencing the same issue today - when I connect to AWS EKS - which was working fine yesterday - only thing I remember doing was trying the : ctx from k9s.

Kubectl get po - works fine
I can still switch to docker desktop but not to any of the aws:eks clusters

[90m12:58PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:58PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/arn:aws:eks:us-east-1:idxxxxxxx:cluster/name_skin.yml
�[90m12:58PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m12:58PM�[0m �[1m�[31mERR�[0m�[0m [list watch] access denied on resource "":"v1/pods"
�[90m12:58PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:58PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:58PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/arn:aws:eks:us-east-1:idxxxxxxx:cluster/name_skin.yml
�[90m12:58PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m12:58PM�[0m �[1m�[31mERR�[0m�[0m [list watch] access denied on resource "":"v1/pods"
�[90m12:58PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m12:59PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:00PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:01PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:02PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:02PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:02PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:02PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:02PM�[0m �[32mINF�[0m 🐶 K9s starting up...
�[90m1:02PM�[0m �[32mINF�[0m ✅ Kubernetes connectivity
�[90m1:02PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/docker-desktop_skin.yml
�[90m1:02PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m1:02PM�[0m �[31mWRN�[0m   Dial Failed! �[31merror=�[0m�[31mUnauthorized�[0m
�[90m1:02PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31mUnauthorized�[0m
�[90m1:02PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/arn:aws:eks:us-east-1:idxxxxxxx:cluster/name_skin.yml
�[90m1:02PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m1:02PM�[0m �[31mWRN�[0m   Dial Failed! �[31merror=�[0m�[31mUnauthorized�[0m
�[90m1:02PM�[0m �[1m�[31mERR�[0m�[0m Unauthorized
�[90m1:02PM�[0m �[31mWRN�[0m   Dial Failed! �[31merror=�[0m�[31mUnauthorized�[0m
�[90m1:02PM�[0m �[1m�[31mERR�[0m�[0m List metrics failed �[31merror=�[0m�[31mUnauthorized�[0m
�[90m1:02PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:02PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:02PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:03PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:03PM�[0m �[32mINF�[0m 🐶 K9s starting up...
�[90m1:03PM�[0m �[32mINF�[0m ✅ Kubernetes connectivity
�[90m1:03PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/docker-desktop_skin.yml
�[90m1:03PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m1:03PM�[0m �[1m�[31mERR�[0m�[0m List metrics failed �[31merror=�[0m�[31m"nodes.metrics.k8s.io is forbidden: User \"user:user-tradestation.com\" cannot list resource \"nodes\" in API group \"metrics.k8s.io\" at the cluster scope"�[0m
�[90m1:03PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"`list access denied for user on \"\":apiextensions.k8s.io/v1beta1/customresourcedefinitions"�[0m
�[90m1:03PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/arn:aws:eks:us-east-1:idxxxxxxx:cluster/name_skin.yml
�[90m1:03PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m1:03PM�[0m �[1m�[31mERR�[0m�[0m `list access denied for user on "":v1/pods
�[90m1:03PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:03PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:03PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:03PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:03PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/arn:aws:eks:us-east-1:idxxxxxxx:cluster/name_skin.yml
�[90m1:03PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m1:03PM�[0m �[1m�[31mERR�[0m�[0m [list watch] access denied on resource "":"v1/pods"
�[90m1:03PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:03PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/arn:aws:eks:us-east-1:idxxxxxxx:cluster/name_skin.yml
�[90m1:03PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m1:03PM�[0m �[1m�[31mERR�[0m�[0m [list watch] access denied on resource "":"v1/pods"
�[90m1:03PM�[0m �[31mWRN�[0m Fail CRDs load �[31merror=�[0m�[31m"[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""�[0m
�[90m1:03PM�[0m �[32mINF�[0m No context specific skin file found -- /Users/user/.k9s/arn:aws:eks:us-east-1:idxxxxxxx:cluster/name_skin.yml
�[90m1:03PM�[0m �[32mINF�[0m No skin file found -- /Users/user/.k9s/skin.yml. Loading stock skins.
�[90m1:03PM�[0m �[1m�[31mERR�[0m�[0m [list watch] access denied on resource "":"v1/pods"

derailed · 2020-05-19T19:10:03Z

@nat2k5us Thank you for sending this extra info! Let's try a few things here and see if we can shed some light... Seems to me this is a different issue. Guessing RBAC rules changed??

Also don't think this will change anything... but try moving your $HOME/.k9s/config.yml and see if k9s will start from scratch with a new config.

In the same shell and same user as you're launching k9s can you try these and report back on the results.

kubectl auth can-i list pods -A --as user-tradestation.com
kubectl auth can-i list crds -A --as user-tradestation.com

derailed · 2020-05-19T19:17:13Z

@sgandon Don't think this will change much but try moving your $HOME/.k9s/config.yml. Also is it possible your api server on the cluster take more than 5s to respond ie in the same shell and user you launch k9s in, what does time kubectl get pods -A returns? Looking at the errors in the initial issue it sure seems to me your api-server is timing out for some reason??

derailed · 2020-05-19T19:20:16Z

@sgandon Also do you currently have any RBAC rules defined on this cluster? Is your admission controller active or is this just a fresh and plain ol' docker-deskop cluster??

elordahl · 2020-05-19T19:29:29Z

fwiw, when i run k9s locally from my mac, i run into similar issues. When i run from docker, using the command in the README, i'm able to connect.

derailed · 2020-05-19T19:32:39Z

@elordahl Tx for the info Eric !! Can you attach the logs when running locally? Hard to pin point this issue at the moment as I can't seem to repro on any of my clusters local or remote ;(
Thank you!!

elordahl · 2020-05-19T19:51:08Z

yep -- see below:

local/mac

3:39PM INF 🐶 K9s starting up...
3:39PM DBG Active Context "CLUSTERNAME"
3:39PM DBG Connecting to API Server https://api.CLUSTERNAME
3:39PM DBG Unable to access servergroups &url.Error{Op:"Get", URL:"https://api.CLUSTERNAME/api?timeout=32s", Err:(*net.OpError)(0xc0009c00f0)}
3:40PM ERR K9s can't connect to cluster error="Get \"https://api.CLUSTERNAME/version?timeout=10s\": dial tcp: lookup api.CLUSTERNAME on 192.168.1.1:53: server misbehaving"
3:40PM PNC K9s can't connect to cluster
3:40PM ERR Boom! K9s can't connect to cluster
3:40PM ERR goroutine 1 [running]:
runtime/debug.Stack(0x4195040, 0x2c1c603, 0x0)
	runtime/debug/stack.go:24 +0x9d
github.com/derailed/k9s/cmd.run.func1()
	github.com/derailed/k9s/cmd/root.go:73 +0x11d
panic(0x28b7c00, 0xc000113000)
	runtime/panic.go:969 +0x166
github.com/rs/zerolog.(*Logger).Panic.func1(0xc0001ad9a0, 0x1c)
	github.com/rs/zerolog@v1.18.0/log.go:338 +0x4f
github.com/rs/zerolog.(*Event).msg(0xc00035b5c0, 0xc0001ad9a0, 0x1c)
	github.com/rs/zerolog@v1.18.0/event.go:146 +0x200
github.com/rs/zerolog.(*Event).Msgf(0xc00035b5c0, 0x2c42366, 0x1c, 0x0, 0x0, 0x0)
	github.com/rs/zerolog@v1.18.0/event.go:126 +0x83
github.com/derailed/k9s/cmd.loadConfiguration(0x1de46e4)
	github.com/derailed/k9s/cmd/root.go:138 +0x536
github.com/derailed/k9s/cmd.run(0x41753c0, 0xc0008083c0, 0x0, 0x2)
	github.com/derailed/k9s/cmd/root.go:81 +0x85
github.com/spf13/cobra.(*Command).execute(0x41753c0, 0xc00004c0d0, 0x2, 0x2, 0x41753c0, 0xc00004c0d0)
	github.com/spf13/cobra@v1.0.0/command.go:846 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0x41753c0, 0x0, 0x0, 0x0)
	github.com/spf13/cobra@v1.0.0/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.0.0/command.go:887
github.com/derailed/k9s/cmd.Execute()
	github.com/derailed/k9s/cmd/root.go:64 +0x2d
main.main()
	github.com/derailed/k9s/main.go:27 +0x1a6

docker

7:48PM INF 🐶 K9s starting up...
7:48PM WRN Unable to locate K9s config. Generating new configuration...
7:48PM DBG Active Context "CLUSTERNAME"
7:48PM DBG Connecting to API Server https://api.CLUSTERNAME
7:48PM DBG RESETING CON!!
7:48PM INF ✅ Kubernetes connectivity
7:48PM DBG [Config] Saving configuration...
7:48PM INF No context specific skin file found -- /root/.k9s/CLUSTERNAME_skin.yml
7:48PM INF No skin file found -- /root/.k9s/skin.yml. Loading stock skins.
7:48PM DBG CURRENT-NS "" -- No active namespace specified
7:48PM INF No namespace specified using cluster default namespace
7:48PM DBG Factory START with ns `""
7:48PM DBG Connecting to API Server https://api.CLUSTERNAME
7:48PM DBG CustomView watching `/root/.k9s/views.yml
7:48PM ERR Custom view load failed /root/.k9s/views.yml error="open /root/.k9s/views.yml: no such file or directory"
7:48PM ERR CustomView watcher failed error="no such file or directory"
7:48PM DBG Setting active ns "default"
7:48PM DBG [Config] Saving configuration...
7:48PM DBG [Config] Saving configuration...

elordahl · 2020-05-19T19:56:10Z

to be clear, i ran the following to get logs, which is a slight variation of the README command (which works):

# first
docker run --entrypoint='' --rm -it -v ~/.kube/config:/root/.kube/config derailed/k9s sh

# then, from inside docker container
k9s -l debug
cat /tmp/k9s-root.log

nat2k5us · 2020-05-19T20:23:35Z

@derailed moving the config.yml out of .k9s works - I am able to list pods in eks cluster.

sgandon · 2020-05-20T07:30:01Z

Hello,
moving the config.yaml out ot the folder does not change anything.
here is the timing for getting pods

kubectl get pods -A  0.09s user 0.05s system 60% cpu 0.229 total

And yes my admission controller is running when I am experiencing this issue.
Please have a look this to know what I installed on the cluster. I then install specific policies related to the CRDs of this tool.

derailed · 2020-05-21T01:36:58Z

@sgandon Boy this is a mix bag ;( Having a hard time figuring out what is what here. So on a fresh cluster without kiverno running are you still unable to connect? The initial logs you've send on this issue indicates a connection failure to the api server not an actual RBAC issue. So are the k9s logs you're seeing now the same as the original logs or are they different? ie k9s may fail to launch under different scenarios we just need to figure out what's failing here. Also did you try k9s -A and see if anything is different or still getting the same connectivity issue with the api server?? I'd love to track this down. To be clear, I am not saying there is no K9s issue here, just trying to get to root cause of the failure. Tx!! Please attach any debug logs you can so I can narrow this down. Tx!

sgandon · 2020-05-21T08:33:42Z

Hi @derailed
I am sure you are doing your best at pinning down the issue.
I may not have been clear before but when I start from a fresh cluster k9s starts fine. This is after a while it gets stalled. Then if I close it I cannot restart it.
I have stopped using it for my current tests of kyvrrno.
I’ll try to help you as much as I can and hope to find some time for this during this long weekend in France.

sgandon · 2020-05-22T22:06:03Z

I have tried k9s -Awithout luck.
I have pasted more log including a few previous sessions at this url : https://pastebin.com/wxqWnNYe

derailed · 2020-05-23T17:43:48Z

@sgandon Thank you for the logs info. Not sure I've got this but please give v0.19.7 a shot and see if we're happier. If not please reopen. Thanks!

sgandon · 2020-05-25T08:15:12Z

@derailed, I have update to 0.19.7
and I have the ui starting after a while and showing a nice cow :)

 MEOW! Post "https://localhost:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded

So I suppose you could re-open this ticket, I am sorry about this.
I still keep my k3d cluster to make sure I can test k9s. with this issue.

derailed · 2020-05-25T16:42:15Z

@sgandon I think we should rename this issue k9s does not work with k3d clusters. I've try to get a k3d cluster up this am and just can't for the life of me get anywhere with it. The rancher docs seems out of date and most of the commands are duds?? Thus I can't validate this issue. I do know that k9s works with docker-4-mac and KiND clusters.

Here are my steps:

brew install k3d
k3d create -n k9s --workers 2
export KUBECONFIG="$(k3d get-kubeconfig --name='k9s')"
kubectl get no // => connection refused??

What am I missing??

derailed · 2020-05-25T18:42:38Z

@sgandon So after a reboot ;( I was able to get a k3d cluster up and running. Left k9s running for ~1hours and no issues... Bare cluster running...

K9s v0.19.7
K8s v1.18.2
K3d v1.7.0

What am I missing?

sgandon · 2020-05-25T19:10:47Z

Well I knew that was going to ba a tough one :)
As I mentionned in my first post, I did have the issue with docker-for-mac too so I don't think this k3d related.
Is there away I can enable some debug logs ?
If I find some time I'll try to clone the repo and debug it myself.
I am also looking for a way to export my setup with k3d. I have commited a docker image but I am failling to restart it properly, the idea was to be able to provide you with a reproducable environment.

derailed · 2020-05-26T16:21:11Z

@sgandon Thank you!! That would be super helpful if you can as I can't see any issue here at the ranch with either k3d or docker k8s. Again I am not saying there is not a bug under all this... but can't seem to repro ;(

sgandon · 2020-06-04T22:32:22Z

@derailed, it seems this is a timeout issue because when I increase the timeout to 10 seconds here, k9s eventually starts.
I have enabled the klog logs to trace the k8s rest calls and dumped it here.
From what I see all the POST done apis/authorization.k8s.io/v1/selfsubjectaccessreviews take 2 to 3 seconds and I don't know if all those calls are considered together with the timeout but it seems pretty slow for simple posts.
These logs are done with the base timeout of 5s and K9s fails, but as you can see none of the api calls to the api server fail.

derailed · 2020-06-04T23:28:45Z

@sgandon Thank you for sending the details! Hum.. obviously a 3sec post is highly dubious. It appears you're still running kyverno on your cluster. Seems suspicious given the nature of the fmk and the api server slow down on these SubjectAccessReview calls. My best guess here is they are getting routed thru an admission controller that puts the brakes on. Could you try removing it from your cluster and retry k9s and see if that make a difference? Also checkout the api server/kyverno deployment logs and see if these posts are somehow getting routed/decorated or trapped by their adm controller. My best guess is they are given I was able to run k9s with no issues on vanillas k3 or dockerFor clusters...

derailed · 2020-06-04T23:31:55Z

@sgandon Also take a quick look at whether pods are restarting on your cluster. It could be some kyverno components are in trouble on that cluster??

sgandon · 2020-06-05T07:10:39Z

@derailed, indeed when I removed kyverno the POST taking 3s went down to 4ms. So we found our guilty.
I'll close this issue as K9s is not responsible for this.

elordahl · 2020-06-05T20:42:54Z

fwiw, i am no longer having issues after upgrading to v0.20.x (tested with 0.20.0 and 0.20.5). There are still some connectivity issues, which may be unrelated, but for the most part it's functional.

jmound · 2020-06-22T17:38:44Z

A note to others that may view this with a similar issue: the MEOW! Post "https://<MASTER-IP>/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded issue, at least on GCP/GKE was resolved by by refreshing credentials (gcloud container clusters get-credentials ...).

OmegaVVeapon · 2021-01-13T19:03:03Z

I'm having this issue intermittently as well.
Is there a way to configure the k9s timeout to increase it from the default 5 seconds without recompiling the tool?

2811299 · 2021-09-01T16:42:50Z

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │
│ < Ruroh? Post "https://:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded > │
│ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │
│ \ ^^ │
│ \ (oo)_____ │
│ (__)\ )/\ │
│ ||----w | │
│ || || │
│

mac-chaffee · 2021-09-03T19:23:17Z

For anyone else finding this through keyword searches, my issues was similar: a misbehaving ValidatingWebhook.

I was using Gatekeeper, and had installed an over-restrictive NetworkPolicy which was blocking the webhook.

Check for any Validating Webhooks with kubectl get Validatingwebhookconfigurations, then check for network policies in those namespaces with kubectl get netpol -A.

…ailed#743 derailed#728 derailed#718

derailed added the question Further information is requested label May 18, 2020

derailed closed this as completed May 23, 2020

derailed reopened this May 25, 2020

derailed closed this as completed Jun 4, 2020

derailed reopened this Jun 4, 2020

derailed added a commit that referenced this issue Jun 5, 2020

add pvc usage check + bugz #642 #754 #753 #743 #728 #718

e2f1626

sgandon closed this as completed Jun 5, 2020

mbarrien mentioned this issue Aug 14, 2020

[BUG] kyverno slowing made K9s timedout kyverno/kyverno#918

Closed

placintaalexandru pushed a commit to placintaalexandru/k9s that referenced this issue Apr 3, 2024

add pvc usage check + bugz derailed#642 derailed#754 derailed#753 der…

79e6254

…ailed#743 derailed#728 derailed#718

Fails to start after a while #718

Fails to start after a while #718

Comments

sgandon commented May 18, 2020

derailed commented May 18, 2020

sgandon commented May 19, 2020

leandroandrade-hotmart commented May 19, 2020 • edited Loading

matheussilva-hotmart commented May 19, 2020

matheussilva-hotmart commented May 19, 2020

derailed commented May 19, 2020

derailed commented May 19, 2020

matheussilva-hotmart commented May 19, 2020

matheussilva-hotmart commented May 19, 2020

sgandon commented May 19, 2020

matheussilva-hotmart commented May 19, 2020

nat2k5us commented May 19, 2020 • edited Loading

derailed commented May 19, 2020

derailed commented May 19, 2020

derailed commented May 19, 2020

elordahl commented May 19, 2020 • edited Loading

derailed commented May 19, 2020

elordahl commented May 19, 2020

local/mac

docker

elordahl commented May 19, 2020

nat2k5us commented May 19, 2020

sgandon commented May 20, 2020

derailed commented May 21, 2020

sgandon commented May 21, 2020 • edited Loading

sgandon commented May 22, 2020

derailed commented May 23, 2020

sgandon commented May 25, 2020

derailed commented May 25, 2020

derailed commented May 25, 2020 • edited Loading

sgandon commented May 25, 2020

derailed commented May 26, 2020

sgandon commented Jun 4, 2020 • edited Loading

derailed commented Jun 4, 2020

derailed commented Jun 4, 2020

sgandon commented Jun 5, 2020

elordahl commented Jun 5, 2020

jmound commented Jun 22, 2020

OmegaVVeapon commented Jan 13, 2021

2811299 commented Sep 1, 2021

mac-chaffee commented Sep 3, 2021

leandroandrade-hotmart commented May 19, 2020 •

edited

Loading

nat2k5us commented May 19, 2020 •

edited

Loading

elordahl commented May 19, 2020 •

edited

Loading

sgandon commented May 21, 2020 •

edited

Loading

derailed commented May 25, 2020 •

edited

Loading

sgandon commented Jun 4, 2020 •

edited

Loading