[SOLVED] chectl server: commands return "Error Unauthorized" after update

[Mbd06b]

I've been using chectl successfully for a while.
chectl updated itself from 7.28.0 to 7.29.0 when I was going to perform a chectl server:update.

chectl version
chectl/7.29.0 linux-x64 node-v12.22.1

almost every chectl che commands that I use that interact with che returns:
"Error: Unauthorized"

without any other details.

any ideas?

I use microk8s on ubuntu 18 LTS.
kubectl version
Client Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.0-3+121713cef81e03", GitCommit:"121713cef81e0316fb97af2111e1d3aeac3fd14a", GitTreeState:"clean", BuildDate:"2021-04-11T13:30:54Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.0-3+121713cef81e03", GitCommit:"121713cef81e0316fb97af2111e1d3aeac3fd14a", GitTreeState:"clean", BuildDate:"2021-04-11T13:26:38Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}

[l0rd]

@Mbd06b can you please provide the full command, output and log of the command that is failing? The previous and working version of chectl was 7.28.0?

[tolusha]

@Mbd06b
Do you have any updates on this issue?

[Mbd06b]

@tolusha @l0rd

I guess the problem is that these outputs aren't helpful. Yes I was originally coming from 7.28.x chectl. But this is exactly what started happening when I attempted to update.

For some reason, I have to reset the microk8s alias to kubectl to check kubectl commands. but the chectl command still doesn't work.

[tolusha]

Could you check:

1. ~/.cache/chectl/error.log if there is any stacktrace there?
2. Pass --chenamespace flag to server:update command: chectl server:update --chenamespace <YOUR_NAMESPACE>

[tolusha]

It might be related to #15331

[Mbd06b]

@tolusha

Here's a sample from ~/.cache/chetcl/error.log .the messages are all following similar pattern...

chenamespace doesn't seem to do anything...

[tolusha]

I dig in the code and I didn't see anything suspicious.
It simply tries to list checlusters objects in namespace.

Could you try the previous version if it still works (ignore chectl update)?

cd ~/.local/share/chectl/client/7.28.0/bin
./run server:status

[Mbd06b]

@tolusha

Could you try the previous version if it still works (ignore chectl update)?

the only client in that directory is the most up-to-date chectl version.

Are there instructions on how to downgrade chectl, or how to remove entirely to perform a clean install of chectl?

[Mbd06b]

My best guess at this point is that there is some kind of disagreement between kubectl and chectl.

I assume chectl uses kubectl somehow to do it's work, but every time it attempts to do something kubernetes is rejecting chectl's requests.

'KubeHelper.wrapK8sClientError()'

[tolusha]

To remove chectl completely:

1. whereis chectl
2. sudo rm -rf /usr/local/bin/chectl /usr/local/lib/chectl (I guess the same on your side)

To install from the scratch:

1. bash <(curl -sL https://www.eclipse.org/che/chectl/)

[Mbd06b]

This is the key: Why is this failing?
2021-05-16T19:33:32.281Z Error: Unauthorized 2021-05-16T19:33:32.281Z at KubeHelper.wrapK8sClientError (/usr/local/lib/chectl/lib/api/kube.js:2784:20) 2021-05-16T19:33:32.281Z at KubeHelper.<anonymous> (/usr/local/lib/chectl/lib/api/kube.js:1742:28) 2021-05-16T19:33:32.281Z at Generator.throw (<anonymous>) 2021-05-16T19:33:32.281Z at rejected (/usr/local/lib/chectl/node_modules/tslib/tslib.js:115:69)

[tolusha]

@Mbd06b
It is hard to say.
It would be nice if you debug in your environment.

git clone https://github.com/che-incubator/chectl.git
cd chectl
yarn
cd bin
./debug

Use VS Code and Attach to remote debug configuration or any other IDE.

[Mbd06b]

@tolusha I've been working on setting up a remote debug. I've not done a remote debug before so it will take me a little time to set it up and try it out.

In a remote debug situation, do you have a copy of the source on your localmachine while connecting to the remote debug over the internet? Or do you have to run the debug locally on the same machine where you are trying to use chectl, connecting to the port on localhost?

[Mbd06b]

ok, I'm able to run the ./debug on the remote server, just needed to specify to node inspect port argument in the chectl/bin/debug command.

https://stackoverflow.com/questions/18910452/how-to-change-node-js-debug-port

[Update] even better, update argument --inspect-brk=9030 (see: https://nodejs.org/en/docs/guides/debugging-getting-started/ )

This article helped enormously on how to attach a node debugger to VS code. Using the SSH tunnel seems like more of a requirement if you want to do any "really remote" debugging. https://maikthulhu.github.io/2019-05-17-remote-debugging-node-vscode/

[Mbd06b]

So far this is the error that I can see from stepping through debug.
Error: Unauthorized\n at KubeHelper.wrapK8sClientError (/home/user/chectl/src/api/kube.ts:2617:74)\n at KubeHelper.<anonymous> (/home/user/chectl/src/api/kube.ts:1646:18)\n at Generator.throw (<anonymous>)\n at rejected (/home/user/chectl/src/api/kube.ts:15:65)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)

[Mbd06b]

So I'm a bit stuck at the moment as to how to set a breakpoint (at that location) on the remote node application itself.
during the debug the source of the files in VS code I'm walking through look like this... (<node internals>\internal\per_context\primoridals.js)
But this looks like the location..

Does my kube context look ok? I'm not finding much on the wrapK8sClientError. (note: I added the "caFile": undefined to show something I noticed debugging vs the actual JSON.stringify() output. ) It looks suspicious.

[tolusha]

Everything looks find. I see a breakpoint mark on 1626 line.
So, debugger must stop at that line when you run it.
To run a debugger:

1. cd bin && ./debug
2. Run VS Code Attach to remote debug configuration

[Mbd06b]

Ok, I figured it out, and it's not chectl's fault (however chectl errors had no clues to the feedback)

What was happening was that my kube config credentials were not being read properly, this was preventing not only chectl from working, but helm was throwing errors too.

This config is found in ~/.kube/config
Thes config file should to be set with the right file access permissions, and with a cluster context that will allow 3rd party tools like chectl or helm to authenticate against the cluster to perform operations. Something about this file wasn't working with helm or chectl as expected.

I haven't been able to figure out what exactly triggered my problem (maybe a microk8s update),
and I'm not exactly sure how I fixed it. I just remember I found someone with a similar issue with helm, and they recommended taking the Kube Config from my running kubernetes cluster and dumping it out with a "raw >" command that looked like that, and then deleting a duplicated configuration in the .kube/config diretory which got everything working again for chectl and helm.

hopefully that's enough clues for anyone to Google for them if you're experiencing a similar issue, google my problem and surf similar issues in the helm forums as chectl and helm interact similarly with the kubernetes cluster

[tolusha]

@Mbd06b
Thank you for your feedback!