Namespaces list is empty

[morigs]

Describe the bug

After some version of k9s list of namespaces stopped working. When switching to the view using :ns main part of the screen is totally empty, even the header is missing. When I try searching using / on the screen, K9S crashes with the following logs:

zerolog: could not write event: write /tmp/k9s-morigs.log: file already closed
zerolog: could not write event: write /tmp/k9s-morigs.log: file already closed
 ____  __.________
|    |/ _/   __   \______
|      < \____    /  ___/
|    |  \   /    /\___ \
|____|__ \ /____//____  >
        \/            \/

Boom!! runtime error: index out of range [-1].

To Reproduce
Steps to reproduce the behavior:

1. Type :ns

Expected behavior
List of namespaces is shown

Screenshots

Versions (please complete the following information):

• OS: WSL2 on Windows 10 (10.0.19043.1348).
• K9s: v0.25.8
• K8s: 1.21

[derailed]

@morigs Something else must be going on... What happens if you list other resources? Also could you include k9s debug logs? Tx!

[morigs]

Hello, @derailed . Other resources are working well, except for nodes.
The logs are following:

Logs

9:38AM DBG Factory START with ns `"otelemetry-develop"
9:38AM WRN Cluster metrics failed error="`list access denied for user on \"\":v1/nodes"
9:38AM DBG Fetching latest k9s rev...
9:38AM DBG K9s latest rev: "v0.25.8"
9:38AM DBG CustomView watching `/home/morigs/.config/k9s/views.yml
9:38AM WRN Custom view load failed /home/morigs/.config/k9s/views.yml error="open /home/morigs/.config/k9s/views.yml: no such file or directory"
9:38AM ERR CustomView watcher failed error="no such file or directory"
9:38AM WRN Cluster metrics failed error="user is not authorized to list nodes"
9:38AM DBG No DAO registry entry for "v1/namespaces". Using generics!
9:38AM ERR Watcher failed for v1/namespaces error="`list access denied for user on \"\":v1/namespaces"

This looks like a permissions issue. However, kubectl get ns, kubectl get nodes are working well (with --watch too)

[derailed]

@morigs Can u give v0.25.10 a rinse and see if we're happier? Tx!

[morigs]

It looks like nothing has changed 😞
Perhaps I can debug this, but I don't know where to start

[softstuff]

This version (v0.25.10) breaks it for me. I do not have access to list namespaces or nodes so I learned the blind way :ns xxx :po but this version seams to fail to do the namespace change so it keeps being black after :po

Downgraded to v0.25.8 and it worked again.

Love the tool, keep up the good work.

k9s-stefan.log

[derailed]

@softstuff Thank you!! Let's see if v0.25.12 helps. @morigs I think something else about your cluster setup is throwing k9s off but I can't seem to find a good repro. More details about configs, env and cluster setup could help??

[softstuff]

@derailed v0.25.12 works fine, still in the dark but the pods pops up.

[morigs]

@derailed Still the same... Which details can I provide (pls note it's a private cluster)? Maybe I can debug it myself?

[dicolasi]

same issue here this morning

[derailed]

@morigs No sure what's happening on v0.25.12. @softstuff seemed better on this rev. Does it make any difference with v0.25.15? Also do you have rbac policies set on your user? How is kubeconfig configured for your cluster, where does it reside and how are you launching k9s vs kubectl? Tx!

[morigs]

0.25.15 still the same 😢
My group has admin role with the following rules

rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

All my configs are in the ~/.kube/config file.

Some differences in kubeconfig between cluster that works fine in k9s and the clusters which dodn't work:

1. working clusters use IP address, while non-working domain name
2. context name of working clusters is in the kebab-case, while non-working - in mixed (dashes, dots, Uppercased letters)
3. users for non-working clusters include auth-provider settings (OIDC using Dex)

[derailed]

@morigs Thank you Igor for the follow up! So I don't think the casing is going to matter however I am dubious about the auth-provider given the inherent privilege issue in the initial logs. Another thing I think would be worth is trying to isolate the use case you've highlighted. ie potential dns issue? does a given non-working cluster using an ip vs domain works? Also to eliminate a casing issues, you could try to rename a non working context with a plain name and see if that makes any difference (tho I would be surprised) ?? or is it any issue with caching ie does your token expires and we fail to renew. Isolating use case between k9s and kubectl on a given context could help shade some light too...

[morigs]

Ok, I'll try to debug. What's really strange is that kubectl works fine in all cases

[morigs]

I finished my debugging here.
It looks like our clusters return error on create SelfSubjectAccessReview request when request for all namespaces (namespace: '').

Example

$ kubectl create -f - -o yaml << EOF
apiVersion: authorization.k8s.io/v1
kind: SelfSubjectAccessReview
spec:
  resourceAttributes:
    group: ''
    resource: namespaces
    verb: watch
    namespace: ''
EOF

apiVersion: authorization.k8s.io/v1
kind: SelfSubjectAccessReview
metadata:
  creationTimestamp: null
spec:
  resourceAttributes:
    resource: namespaces
    verb: watch
status:
  allowed: false
  denied: true
  reason: 'webhook: bad request, group and groupVersion are empty'

I'll investigate our k8s api-server hereinafter.
However, I see the following possible improvements in K9S:

1. If it's possible, it would be awesome to log all api-server requests and responses on debug level
2. K9S should report authorization errors to the user to avoid confusion
3. Perhaps it's a good idea to return empty list of namespaces in case of errors (to avoid crash on / command)

[derailed]

@morigs You are awesome Igor! Thank you for this great analysis! I think you've highlighted some good issues and bugs in k9s. I think the access review is indeed an issue, along with the crash you'd mentioned above. I'll push a patch up and let's see if we can zero in a bit more on this deal. I'll also add a trace level for debugging access issue on your clusters it won't log everything (for now!) but will help us zero in if that's still a deal after this next drop ie k9s -l trace...

[knisbet]

1. If it's possible, it would be awesome to log all api-server requests and responses on debug level

AFAICT this isn't currently possible via flags. If you build k9s locally, the hack I do to turn on client-go logging and get per request/response logging is:

diff --git a/main.go b/main.go
index dd4d99ad..2d10dab0 100644
--- a/main.go
+++ b/main.go
@@ -20,7 +20,7 @@ func init() {
        if err := flag.Set("stderrthreshold", "fatal"); err != nil {
                panic(err)
        }
-       if err := flag.Set("v", "0"); err != nil {
+       if err := flag.Set("v", "99"); err != nil {
                panic(err)
        }
 }

This should work like passing -v=99 to kubectl. Of course, you can use lower verbosity levels depending on what you're troubleshooting. The output ends up in a debug log in a temp directory (previously it was part of k9s own log, but when I build off master it's now in a separate file in the same directory).

[derailed]

No response. Closing...