Improve microk8s support

[benoitf]

From chectl created by l0rd: che-incubator/chectl#196

A first PR has been merged to support microk8s but there are some issues to address. In particular

• Use microk8s.kubectl instead of kubectl. Use kubectl if microk8s.kubectl is not found. Fail if none of them is found
• chectl should not have the responsibility to enable microk8s. we should simply fail if microk8s is not enabled and provide the command line to enable it
• when enabling the storage addon chectl should wait until it is enabled or timeout if it's not enabled after 60s

[TheRealChssAddct]

Starting with the refactor in commit b9f6ac7, chectl server:start -p microk8s began failing for me during k8sTasks.testApiTasks with the following:

[13:33:41] Verify Kubernetes API [started]
[13:33:41] Verify Kubernetes API [failed]
[13:33:41] → Failed to connect to Kubernetes API. Error: connect ECONNREFUSED 127.0.0.1:8080

I found a couple of workarounds after poking around and seeing the loadFromDefault code in chectl/node_modules/@kubernetes/client-node/dist/config.js.

Workaround #1
I found I didn't have a $HOME/.kube/config file. This is because microk8s doesn't create one. (I don't have kubectl installed, because microk8s.kubectl.) The microk8s readme mentions how to create this file:

To use MicroK8s with your existing kubectl:
sudo microk8s.kubectl config view --raw > $HOME/.kube/config"

however this gives me this error: "error: write /dev/stdout: permission denied"

This alternative:

sudo microk8s.config > $HOME/.kube/config

gives me this error: "/snap/microk8s/1079/bin/sed: couldn't flush stdout: Permission denied"

What I found actually works is this:

microk8s.config | cat - > $HOME/.kube/config

Update: I found that the errors I mentioned above were somehow due to running the commands in the integrated terminal in vscode. When I run in a regular old external non managed terminal, I do not get the error.

Workaround #2
Guided by this nicja comment I looked in /var/snap/microk8s/current/args/kubectl to find the pointer to the microk8s kubeconfig file (mine is here: /var/snap/microk8s/current/credentials/client.config).

So, theoretically another alternative would be to set the KUBECONFIG environment variable.
Note: I haven't actually tried it, I'm asserting this based on the aforementioned loadFromDefaults code.

Suggested Fix
These workarounds shouldn't be necessary. Perhaps the KubeHelper constructor could be modified to something like this:

if (!context) {
   if (flags.platform === 'microk8s') {
      const { stdout } = yield execa('microk8s.config', { timeout: 10000 });
      this.kc.loadFromString(stdout)

however, that can introduce an undesireable wait-time in a constructor, so maybe callers should pass in an already retrieved context to new KubeHelper.
Perhaps the change is as simple as not running the k8sTasks.testApiTasks task when platform = microk8s.

[mentos1386]

I can't makechectl work with microk8s. It's always failing at storage check.

[TheRealChssAddct]

@mentos1386

This is actually this bug (#16408) I previously submitted and then closed, and which I will now reopen.

Re: chectl failing the storage check, it is because chectl uses the original microk8s.status format (some might say 'contract'). I previously submitted a bug here (canonical/microk8s#1041) and the microk8s team (@ktsakalozos) very obligingly fixed it, quite quickly (PR here: canonical/microk8s#1042), by having the default be the original format, which che uses.

However, on Aug 20, this commit (canonical/microk8s@7bb1b25) and PR (canonical/microk8s#1208) changed the default to 'pretty', probably unintentionally.

@balchua did notice it and commented here (canonical/microk8s#1127 (comment)). I missed it, but @ktsakalozos actually did give me a heads up here (canonical/microk8s#1041 (comment))

I'm not happy with the breaking change, but the comment above makes it clear they intend the default to be 'pretty'.

I already changed my own scripts to accommodate this by explicitly setting the --format flag of the microk8s.status call.

This must be fixed on the che side as well.

[TheRealChssAddct]

@mentos1386

fyi, if you are using snaps, this is how you would go back to a version of microk8s that should work with chectl:

snap install microk8s --classic --channel=1.18/stable

[peterlobster]

I've been trying to install che on microk8s but it constantly seems to fail because it can't find kubectl. I've tried 1.17, 1.18, and latest, and no cigar. I've also tried to export out kubectl as both a alias and a function, and still no cigar. Can anyone one help me out?

# chectl server:deploy -m --installer=helm --platform=microk8s
› Current Kubernetes context: 'microk8s'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not
  ❯ ✈️  MicroK8s preflight checklist
    ✖ Verify if kubectl is installed
      → E_REQUISITE_NOT_FOUND
      Verify if microk8s is installed
      Verify if microk8s is running
      Start microk8s
      Check Kubernetes version
      Verify if microk8s ingress and storage addons is enabled
      Enable microk8s ingress addon
      Enable microk8s storage addon
      Retrieving microk8s IP and domain for ingress URLs
      Check if cluster accessible
 ›   Error: Error: E_REQUISITE_NOT_FOUND
 ›   Command server:deploy failed. Error log: /root/.cache/chectl/error.log Eclipse Che logs: /tmp/chectl-logs/1619115234533

# cat /root/.cache/chectl/error.log
2021-04-22T18:13:54.762Z Error: Error: E_REQUISITE_NOT_FOUND
2021-04-22T18:13:54.762Z Command server:deploy failed. Error log: /root/.cache/chectl/error.log Eclipse Che logs: /tmp/chectl-logs/1619115234533
2021-04-22T18:13:54.762Z     at Object.error (/usr/local/lib/chectl/node_modules/@oclif/errors/lib/index.js:26:15)
2021-04-22T18:13:54.762Z     at Deploy.error (/usr/local/lib/chectl/node_modules/@oclif/command/lib/command.js:60:23)
2021-04-22T18:13:54.762Z     at Deploy.<anonymous> (/usr/local/lib/chectl/lib/commands/server/deploy.js:253:22)
2021-04-22T18:13:54.762Z     at Generator.throw (<anonymous>)
2021-04-22T18:13:54.762Z     at rejected (/usr/local/lib/chectl/node_modules/tslib/tslib.js:115:69)

# cat /tmp/chectl-logs/1619115234533
cat: /tmp/chectl-logs/1619115234533: No such file or directory

[ItaloQualisoni]

@peterlobster

By default you should use microk8s.kubectl and looking the code it's expects kubectl only.

https://github.com/che-incubator/chectl/blob/fcbd2b3990289f017528a1fb42b6a928a8a25687/src/tasks/platforms/microk8s.ts#L31

This got fixed after creating an alias for me:

sudo snap alias microk8s.kubectl kubectl

Hope this helps you!