Issue in enabling kubeflow in ubuntu 18.04 azure vm

[thirubeyond]

On enabling kubeflow in azure vm, I get error

ERROR cannot load ssh client keys: open /var/snap/microk8s/2052/juju/share/juju/ssh: permission denied

OS: 18.04.5 LTS (Bionic Beaver)
microk8s.kubectl config view --raw > $HOME/.kube/config
export KUBECONFIG=$HOME/.kube/config
microk8s enable dns dashboard storage worked fine
snap list
microk8s v1.20.2 2052 latest/edge canonical✓ classic
inspection-report-20210217_122634.tar.gz

Tried:

• I retried in fresh azure vm and still getting the same issue
• I tried as per Link but still facing the issue.
• Commands are run without sudo
• output of: microk8s kubectl cluster-info
  Kubernetes control plane is running at https://127.0.0.1:16443
  CoreDNS is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
  Metrics-server is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

-output of microk8s.kubectl get all --all-namespaces give all pods are running fine

• output of microk8s inspect gave a warning
  WARNING: Port 16443 seems to be in use by another application.

I have been trying several times still hitting the same issue.

[knkski]

@thirubeyond: that's odd that you'd encounter this permissions issue if you're not running any command with sudo . Can you try running these commands to see if they fix the permissions issue?

sudo chgrp -R microk8s /var/snap/microk8s/current/juju
sudo chmod -R 775 /var/snap/microk8s/current/juju

If that doesn't work, can you try including the --debug flag and posting the output? e.g.

microk8s enable kubeflow --debug

[thirubeyond]

@knkski : I had run the above chgrp/chmod commands last around, did it again now in fresh azure vm.

Experiment 1: enable kubeflow with debug (had set KUBECONFIG but not kept the kubeconfig file), got output Log1 (attached)
Experiment 2: generated kubeconfig file and enabled kubeflow with debug, got output Log 2 (attached)

Log2.pdf
Log1.pdf

After seeing log2 I ran below command:
sudo chgrp -R microk8s /tmp
sudo chmod -R 775 /tmp

Again I enabled kubeflow,
Now it proceeded and reached a stage showing " waiting for operator pod " for very long time (30mins) after which I have to stop

Attaching inspect log at 30 mins:
inspection-report-20210219_120910.tar.gz

Please advise.

[knkski]

@thirubeyond: You shouldn't have to set KUBECONFIG or modify the kubeconfig file. What happens if you leave both of those unset?

I see in the inspection log messages such as The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?, which looks to me like a deeper MicroK8s issue, instead of a Kubeflow-specific error.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.