Error encountered, when the user is set other than jovyan, while executing voila

[ricky-lim]

Hi Dan,

In my setup, the user of the jupyter is set with her/his username coupled to the permissions from an active directory.

With each user has its own homedir, such as '/home/ril' instead of home/jovyan

Unfortunately while trying to execute voila, I encountered the following error, which I think might be caused by trying to set username to jovyan.

I was wondering if you have encountered this issue before and I'll be very grateful if you could help, please.

Thank you in advance for your time and consideration.

The log with its error messages

Set username to: jovyan
usermod: no changes
Executing the command: python3 -m jhsingle_native_proxy.main --destport=0 python3 {-}m voila {presentation_path} {--}port={port} {--}no-browser {--}Voila.base_url={base_url}/ {--}Voila.server_url=/ --presentation-path=./gaussian_process_regression.ipynb --ip=0.0.0.0 --port=8888 --request-timeout=600 --last-activity-interval=600 --allow-root
sudo: setrlimit(RLIMIT_CORE): Operation not permitted

[danlester]

Thank you for getting in touch.

Please could you share more about your config, especially the docker image you are using, and I'll take a look.

I may have some of this already, but it could be good to get the latest anyway, and also useful for anyone else reading.

[ricky-lim]

Yes sure. Here is with more detail specs

• hub: jupyterhub/k8s-hub:0.9.0
• singleuser: jupyterhub/singleuser:1.2
• cdsdashboards==0.3.5

Below I received the error message, that the notebook file (*.ipynb) is not found.
I checked into the dashboard-pod, and the notebook is located in /home/ril/gaussian_process_regression.ipynb
instead of in /home/jovyan/gaussian_process_regression.ipynb

I was wondering if there is away to set the home directory, other than /home/jovyan, during the dashboard build step ?

I have tried to run the start.sh at c.VariableKubeSpawner.default_presentation_cmd = ['start.sh', 'python3', '-m', 'jhsingle_native_proxy.main', '--allow-root'], unfortunately I still received the same error

Here is my config:

hub:
  allowNamedServers: true
  extraConfig:
    00-auth: |
      # Custom authenticator to set NB_USER, NB_UID and NB_GID
      c.JupyterHub.authenticator_class = RZAzureADAuthenticator
      c.Authenticator.enable_auth_state = True
    01-spawners: |
      c.Spawner.cmd = ['start.sh', 'jupyterhub-singleuser', '--allow-root']
      c.KubeSpawner.http_timeout = 600
      c.KubeSpawner.start_timeout = 600
    02-cds-handlers: |
      from cdsdashboards.hubextension import cds_extra_handlers


      c.JupyterHub.extra_handlers = cds_extra_handlers
      c.VariableMixin.proxy_force_alive = True
      c.VariableMixin.proxy_last_activity_interval = 600
      c.VariableMixin.proxy_request_timeout = 600
    04-cds-kube: |
      c.JupyterHub.spawner_class = 'cdsdashboards.hubextension.spawners.variablekube.VariableKubeSpawner'
      c.CDSDashboardsConfig.builder_class = 'cdsdashboards.builder.kubebuilder.KubeBuilder'
      c.JupyterHub.redirect_to_server = False
      c.JupyterHub.default_url = '/hub/dashboards'
      c.Spawner.default_url = '/lab'
      c.VariableKubeSpawner.default_presentation_cmd = ['start.sh', 'python3', '-m', 'jhsingle_native_proxy.main', '--allow-root']


singleuser:
  uid: 0
  storage:
    type: dynamic
    capacity: 10Gi
    dynamic:
      pvcNameTemplate: claim-{username}
      volumeNameTemplate: volume-{username}
      storageAccessModes: [ ReadWriteMany ]
    extraVolumes:
      - name: jupyterhub-workspace
        persistentVolumeClaim:
          claimName: jupyterhub-workspace
    extraVolumeMounts:
      - name: jupyterhub-workspace
        mountPath: /home/jovyan/shared
    homeMountPath: /home/{username}
    extraEnv:
      CHOWN_HOME: 'yes'

Below is the Error message

Error report from ContainDS Dashboards
Command Running:

python3 -m voila ./gaussian_process_regression.ipynb --port=54658 --no-browser --Voila.base_url=/user/ril/dash-gaussian-process-regression/ --Voila.server_url=/
    

Error output:

None
    

Standard output:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/voila/__main__.py", line 12, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 663, in launch_instance
    app.initialize(argv)
  File "/opt/conda/lib/python3.7/site-packages/voila/app.py", line 340, in initialize
    raise ValueError('argument is neither a file nor a directory: %r' % arg)
ValueError: argument is neither a file nor a directory: './gaussian_process_regression.ipynb'

Thanks you again in advance for your help :)

[danlester]

You've probably just omitted these lines from your config for brevity, but it doesn't quite make sense to be running:

• hub: jupyterhub/k8s-hub:0.9.0
• singleuser: jupyterhub/singleuser:1.2

because these don't have the ContainDS Dashboards components in them by default. (Or maybe you have another way to install them?)

Do you mean:

• hub: ideonate/cdsdashboards-jupyter-k8s-hub:0.9.0-0.3.5
• singleuser: ideonate/jh-voila-oauth-singleuser:0.3.5 or simiar?

In particular, I don't believe your singleuser image contains the latest jhsingle_native_proxy 0.5.6 (because that passes through absolute paths, whereas the one in your 'error message' section above is relative). So the singleuser image definitely needs updating.

No images are specified in your config (although clearly cdsdashboards is installed at some level because you're seeing the cdsdashboards error page). So it would be great if you can just clarify that? Or are you building your own Docker images - if so, it would be really useful to see the Dockerfiles.

Mainly as background, what is the main reason for wanting to change the name from jovyan inside the container? (Assuming you mount /home/jovyan to the correct {username} on the volume, they should still see the correct files. But maybe you want to see correct usernames propagated into the central storage, or just make it more user-friendly within the container.)

Why have you mounted the extra shared folder at /home/jovyan/shared if you don't want jovyan to be the username? (Maybe extraVolumeMounts doesn't allow substitution of {username} - I haven't checked.)

Finally, are you seeing both the 'error message' web output ('Error report from ContainDS Dashboards') from your last post AND the log output from the very first post?

Sorry for all the questions! But if we can get you onto the latest versions, and fully understand your config, and then you're still seeing the errors I should be able to try it out for myself to see what's going on...

Thanks,

Dan

[ricky-lim]

Hi Dan,

Correct, I built as my own docker images. I upgraded the jhsingle-native-proxy==0.5.6

Here is the dockerfile for singleuser

ARG TAG=1.2
ARG BASE_REPO=jupyterhub/singleuser

FROM $BASE_REPO:$TAG

# Add conda environment
USER root
COPY conda-env conda-env
RUN fix-permissions conda-env
USER $NB_UID

RUN conda env update --name base --file conda-env/base.yaml


# Set the voila default template
COPY jupyter_notebook_config_singleuser.py /etc/jupyter/
RUN cat /etc/jupyter/jupyter_notebook_config_singleuser.py >> /etc/jupyter/jupyter_notebook_config.py
RUN rm /etc/jupyter/jupyter_notebook_config_singleuser.py

# Set conda and pip environment
RUN conda init bash
COPY conda-env/.condarc /etc/conda/.condarc
COPY conda-env/conda-init.sh /etc/profile.d/conda-init.sh

# Fix permissions on /etc/jupyter as root
USER root
RUN fix-permissions /etc/jupyter/

Here is base.yaml

name: base
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.7
  - pip=20.1.1
  - matplotlib=3.3.0
  - voila=0.1.21
  - bqplot=0.12.15
  - vim>=8.2
  - git>=2.28.0
  - pip:
      - ipyfetch==0.1.3
      - voila-material==0.3.0
      - pyprojroot==0.2.0
      - dvc==1.4.0
      - jhsingle-native-proxy==0.5.6

I needed to set the uid and guid as I mounted an existing NFS volume that requires a set of correct permissions.
However, the username is indeed is more of a convenience.

I also tried to keep using jovyan as its user and get the /home/jovyan instead of /home/ril.
Unfortunately I still get the permission error while trying to serve a voila notebook (as shown below)

I think it's because of the unmatched uid between jupyter-server and dashboard-server on /home/jovyan/.jupyter

INFO:tornado.application:b"    with open(os.path.join(env['jupyter_config'], 'migrated'), 'w') as f:\n"
INFO:tornado.application:b"PermissionError: [Errno 13] Permission denied: '/home/jovyan/.jupyter/migrated'\n"
INFO:tornado.application:b'[Voila] Using contents: services/contents\n'
INFO:tornado.application:b'[Voila] Using contents: services/contents\n'
INFO:tornado.application:b'ERROR:tornado.access:500 GET / (127.0.0.1) 8.11ms\n'

[ricky-lim]

With the updated jhsingle-native-proxy==0.5.6, however I got the unexpected absolute path as

/home/jovyan/./gaussian_process_regression.ipynb

I was wondering, is there a way I could set the absolute path in the dashboard UI, such as /home/ril/gaussian_process_regression.ipynb.

I think that will solve my issue.

Thanks again in advance for your time and support.

Standard output:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/voila/__main__.py", line 12, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 663, in launch_instance
    app.initialize(argv)
  File "/opt/conda/lib/python3.7/site-packages/voila/app.py", line 340, in initialize
    raise ValueError('argument is neither a file nor a directory: %r' % arg)
ValueError: argument is neither a file nor a directory: '/home/jovyan/./gaussian_process_regression.ipynb'

    

[ricky-lim]

A little update.

If I execute the command with the expected home directory on /home/ril/gaussian_process_regression.ipynb instead of /home/jovyan/gaussian_process_regression.ipynb , on the pod directly. It works.

# Get into the pod
kubectl exec -it jupyter-ril-dash-2dgaussian-2dprocess-2dregression /bin/bash

# Voila command
python3 -m voila /home/ril/gaussian_process_regression.ipynb --port=36206 --no-browser --Voila.base_url=/user/ril/dash-gaussian-process-regression/ --Voila.server_url=/ --debug

I was wondering if there is a way in the configuration to set a dynamic home directory for the dashboard ?
Another alternative, I think is to allow the dashboard developer to set an absolute path.

What is your advice, please?

Thanks again for your time. Look forward to hearing from you.

[danlester]

Thank you for working on this further!

Please could you let us know more about RZAzureADAuthenticator? That seems to be where NB_USER etc are set.

I don't believe 'jovyan' is hard-coded into the cdsdashboards or jhsingle-native-proxy code, but of course /home/jovyan is the default working folder for the Docker image you are using.

If you start a normal Jupyter notebook server, does that start up in /home/ril or /home/jovyan?

If this all works for standard Jupyter servers, we need to understand why CHOWN_HOME and NB_USER etc aren't being recognized by start.sh script when run through cdsdashboards.

If it doesn't work for standard Jupyter servers, then we should fix that first!

[ricky-lim]

For the jupyter notebook server, the start.sh seems to work and it does start up in /home/ril.

Aha, for the authenticator, the NB_USER is set during pre_spawn_start hook

Below is the snippet of RZAzureADAuthenticator

 class RZAzureADAuthenticator(AzureAdOAuthenticator):


          @gen.coroutine
          def pre_spawn_start(self, user, spawner):
              auth_state = yield user.get_auth_state()
              self.log.info('pre_spawn_start auth_state: %s' % auth_state)
              self.log.info('pre_spawn_start user: %s' % user)
              if not auth_state:
                  return

              spawner.environment['NB_UID'] = auth_state['uid']
              # This is set to 'ril'
              spawner.environment['NB_USER'] = user.name

# On the config.yaml
auth:
  type: custom
  custom:
    className: 'RZAzureADAuthenticator'
    config:
      client_id: xxxxx
      client_secret: xxxxx
      tenant_id: xxxxx
      oauth_callback_url: https://jupyterhub.tst/hub/oauth_callback
  state:
    enabled: true
    cryptoKey: xxxxx
  admin:
    users:
      - ril

As I looked into the log from the hub during the creation of the dashboard, the pre_spawn_start seems to work.

[I 2020-10-07 06:52:19.651 JupyterHub <string>:28] pre_spawn_start user: <User(ril 2/2 running)>
[I 2020-10-07 06:52:19.652 JupyterHub <string>:29] pre_spawn_start username: ril

Although there was an error prior to that.

ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<DashboardBaseMixin.pipe_spawner_progress() done, defined at /usr/local/lib/python3.6/dist-packages/cdsdashboards/hubextension/base.py:129> exception=AttributeError("'NoneType' object has no attribute 'done'",)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/cdsdashboards/hubextension/base.py", line 136, in pipe_spawner_progress
    if builder._build_future.done():
AttributeError: 'NoneType' object has no attribute 'done'

Here is the log from the dashboard-server:

Set username to: jovyan
usermod: no changes
Executing the command: python3 -m jhsingle_native_proxy.main --destport=0 python3 {-}m voila {presentation_path} {--}port={port} {--}no-browser {--}Voila.base_url={base_url}/ {--}Voila.server_url=/ --presentation-path=./hom
e/ril/gaussian_process_regression.ipynb --ip=0.0.0.0 --port=8888 {--}debug --debug --request-timeout=600 --last-activity-interval=600
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
INFO:tornado.application:SuperviseAndProxyHandler http_get 51385 
DEBUG:tornado.application:No user identified

Along with your pointer, it seems the start.sh does not seem to run with cdsdashboard because of NB_USER was not set properly.

Looking into the container's pod, it is run as root which we would like to avoid and NB_USER is jovyan

(base) root@jupyter-ril-dash-2dgaussian-2dprocess-2dregression:~# id
uid=0(root) gid=1000 groups=1000
(base) root@jupyter-ril-dash-2dgaussian-2dprocess-2dregression:~# echo $HOME
/home/jovyan
(base) root@jupyter-ril-dash-2dgaussian-2dprocess-2dregression:~# echo $NB_USER
jovyan

I was wondering, is there a specific hook for the dashboard spawner to ensure the NB_USER is set from authentication ?

I think the problem is that the environment variables, such as NB_USER, set during pre_spawn_start does not seem to propagate to singleuser pod's container with cdsdashboards.hubextension.spawners.variablekube.VariableKubeSpawner

Thank you in advance for your kind guidance. I look forward to hearing from you.

[danlester]

@ricky-lim sorry for the delay here, but I think I've reproduced most of your setup.

I think it might make sense to add GRANT_SUDO: 'yes' in addition to CHOWN_HOME: 'yes'.
I'm not too sure if you need the --allow-root bit...

In any case, cdsdashboards was causing you a problem. The pre_spawn_hook was being called, but then the environment dict was subsequently overwritten. Instead, I've checked in a change that attempts to merge it. This preserves the NB_USER in my tests, although I haven't thought too hard about any other side effects yet.

Please could you try it out by using this commit. If you're using my standard images, just try using the hub image from Docker Hub: ideonate/cdsdashboards-jupyter-k8s-hub:sha-d60c383

[ricky-lim]

Hi Dan,

Thank you for your effort to reproduce the setup.

Yes, I have tried it and it works as expected, now

Are you going to tag the release for cdsdashaboard ?

Cheers

[danlester]

Great to hear that worked - thank you for letting me know.

I will make sure this functionality is brought into the next tagged release, but I'm not sure exactly when.

Hopefully you are OK to use the commit version instead for now - let me know if not for some reason.

Thanks,

Dan

[danlester]

This is now in version 0.4.0