Enable Kubeflow ask for authorization website (Ubuntu One) - access denied

[ereil]

Hi

since a few days when enabling kubeflow, the script is opening firefox (crazy..) and opens the webpage https://api.jujucharms.com/identity/login, and asks for a login.
I've never had this before.
I have tried to register an account and login during kubeflow enabling. But also this is not working, i get the error:

Opening an authorization web page in your browser.
If it does not open, please open this URL:
https://api.jujucharms.com/identity/login-legacy?did=be0659fb7b0fc2f12588fc28bc2XxXxXxxXXxxXxXxXxx3496442e464a5efef30e722
[2020-12-22T13:14:18Z ERROR audio_thread_priority::rt_linux] setrlimit64: 1
ERROR cannot deploy bundle: cannot resolve URL "cs:~kubeflow-charmers/modeldb-backend-86": cannot resolve charm URL "cs:~kubeflow-charmers/modeldb-backend-86": cannot get "/~kubeflow-charmers/modeldb-backend-86/meta/any?include=id&include=supported-series&include=published": access denied for user "XXxxXXxx"

Command '('microk8s-juju.wrapper', 'deploy', 'cs:kubeflow-195', '--channel', 'stable', '--overlay', '/tmp/tmplq2baq_c')' returned non-zero exit status 1
Failed to enable kubeflow

I'm using microk8s (1.18/stable) v1.18.13

I've also tried newer versions of microk8s 1.19 and also 1.20. But both versions had also problems to enable kubeflow.
inspection-report-20201222_142215.tar.gz

[ereil]

And here is the output when enabling kubeflow with newest microk8s 1.20 stable:

caduser@cad660895:~$ microk8s.enable kubeflow
Password [4DTEFG85ZOWXGOTL4B6RJV58TIOSTI]: 
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling metallb:10.64.140.43-10.64.140.49...
Waiting for DNS and storage plugins to finish setting up
DNS and storage setup complete. Checking connectivity...
Bootstrapping...
Kubeflow could not be enabled:
ERROR cannot load ssh client keys: open /var/snap/microk8s/1864/juju/share/juju/ssh: permission denied

Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 2

install steps are following:

sudo snap install microk8s --classic
sudo microk8s.status --wait-ready
sudo usermod -a -G microk8s $USER
su - $USER
sudo microk8s.enable dns storage dashboard
sudo snap alias microk8s.kubectl kubectl
sudo chown -R $USER ~/.kube
microk8s.kubectl config view --raw > $HOME/.kube/config
microk8s.enable gpu
sudo microk8s.inspect
microk8s.enable kubeflow

Ubuntu is 18.04.4 LTS

[ereil]

same problem in Ubuntu 18.04.4 LTS + 18.04.5 LTS and 20.04.1 LTS
also with newest microk8s latest/edge: v1.20.1 2021-01-04 (1891)

There are several permission denied errors for /var/snap/microk8s/xxxx/juju... and tmp/juju_xxx pathes
These could be fixed with chown the pathes to local user

But then the final issue with authorization over webpage Ubuntu One occours.

[knkski]

@ereil: can you try with the latest version of microk8s? sudo snap switch microk8s --channel latest/edge && sudo snap refresh

[knkski]

@ereil: Also, you shouldn't have to use sudo with microk8s, that's probably what's responsible for the error you got about ERROR cannot load ssh client keys: open /var/snap/microk8s/1864/juju/share/juju/ssh: permission denied

To fix that you can run these commands:

sudo chgrp -R microk8s /var/snap/microk8s/current/juju/
sudo chmod -R 775 /var/snap/microk8s/current/juju/

[ereil]

@knkski: with latest/edge (1894) it's working.
And also the commands to fix the permissions is working.
I've done a complete new installation of microk8s without sudo, but it has also the permission issue. Probably there were old pathes from old microk8s installation. I would check it with a fresh ubuntu installation.

Is it possible to fix also kubeflow integration for microk8s 1.18?
Because we are using the kubeflow/arena toolkit, but its not compatible with 1.20 now.
Thanks

[knkski]

@ktsakalozos: what is the policy on backporting addon changes into older microk8s versions?

[knkski]

@ereil: the latest changes to microk8s enable kubeflow are being backported to 1.18 here:

#1974

Going to close this issue as it works with latest/stable, and the above PR will backport that fix.