New conda environment issue with another username

[masip85]

What docker image(s) are you using?

scipy-notebook

OS system and architecture running docker image

aarch64

What Docker command are you running?

docker run -it --name myproject --rm
-p 8888:8888
--user root
-e NB_USER=vicente
-e NB_UID="$(id -u)"
-e NB_GID="$(id -g)"
-e CHOWN_HOME=yes
-e CHOWN_HOME_OPTS="-R"
-w "/home/${NB_USER}"
-v "${PWD}"/test:/home/vicente/work
jupyter/myproject-notebook

How to Reproduce the problem?

1. Create Dockerfile with conda environment following doc:
   https://jupyter-docker-stacks.readthedocs.io/en/latest/using/recipes.html#add-a-custom-conda-environment-and-jupyter-kernel
   adding at the end the commented line:
   RUN echo "conda activate ${conda_env}" >> "${HOME}/.bashrc"

2. Execute docker run with new NB_USER.

Command output

No response

Expected behavior

I want my user has the conda environments in the list of kernels from the beginning (as is documented, but for any other user)

Actual behavior

# create Python kernel and link it to jupyter
RUN "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --user --name="${conda_env}" && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"

and

RUN echo "conda activate ${conda_env}" >> "${HOME}/.bashrc" ,

in its context is executed only for jovyan home. So I guess is only being executed when jovyan starts a session. When my notebook starts with my user, if I open the terminal I only get default conda environment at the jupyter kernelspec list.

the And I dont want to use jovyan.

Anything else?

I can't use install without --user for install kernel it system wide:
RUN "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --name="${conda_env}" && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}"
[Errno 13] Permission denied: '/usr/local/share/jupyter'

And in that case, I would have to hardcode this in dockerfile,
RUN echo "conda activate ${conda_env}" >> "/home/vicente/.bashrc"

which is not very pleasant. If you could solve the first issue, I'd like to get a nice way for doing the second one.

If the solution is not easy,maybe the best way is to giveup to conda environments, and always overwrite default conda environment with custom conda yaml or whatever. But in any of these cases,(if I am right) everything should be documented.

[Bidek56]

Can you use pip? This command works fine, and I can install packages with no issues.
docker run -it --name myproject --rm -p 8888:8888 --user root -e NB_USER=vicente -e NB_UID="$(id -u)" -e NB_GID="$(id -g)" -e CHOWN_HOME=yes -e CHOWN_HOME_OPTS="-R" -w "/home/${NB_USER}" -v "${PWD}":/home/vicente/work jupyter/pyspark-notebook

[mathbunnyru]

@Bidek56 I guess the problem here is that .bashrc is not run when changing username.
In your run command bashrc is more-or-less default, so you don't think any issues.
In the issue, your .bashrc is used to activate non-default environment.

[masip85]

@mathbunnyru , Yes, I'd do it, but first ipykernel install must be successful for all users, and it's only done for jovyan. How should I solve this?

[mathbunnyru]

@masip85 I understand this, my message was addressed to @Bidek56.

I don't know a good solution for your issue, but I might have a hack that might work (I haven't tested it).
Could you maybe mount /home/vicente folder, instead of the work folder and then put .bashrc in this folder with the activate line.

It's a bit ugly, because you will have all the config files in your mounted folder.
Also, I highly recommend making a backup, before testing, just in case.

[masip85]

And ipykernel installation? El El lun, 3 oct 2022 a las 19:05, Ayaz Salikhov ***@***.***> escribió:

…

@masip85 <https://github.com/masip85> I understand this, my message was addressed to @Bidek56 <https://github.com/Bidek56>. I don't know a good solution for your issue, but I might have a hack that might work (I haven't tested it). Could you maybe mount /home/vicente folder, instead of the work folder and then put .bashrc in this folder with the activate line. It's a bit ugly, because you will have all the config files in your mounted folder. Also, I highly recommend making a backup, before testing, just in case. — Reply to this email directly, view it on GitHub <#1792 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADCES3WLFWFZHKTZ52MYGSTWBMG53ANCNFSM6AAAAAAQY2AN64> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mathbunnyru]

I don't know the answer yet, sorry :(

[masip85]

This is a solution, not a perfect one, but the best one by now:
Jupyter Forum

USER 0:0
RUN sudo "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --name="${conda_env}" && \
	fix-permissions "${CONDA_DIR}" && \
	fix-permissions "/home/${NB_USER}"
USER 1000:100

Now we could find a cleaner way for this:
Could you maybe mount /home/vicente folder, instead of the work folder and then put .bashrc in this folder with the activate line.

I don't understand why .bashrc is not functional as the doc says.This should copy it to my user:

    # Move or symlink the jovyan home directory to the desired users home
    # directory if it doesn't already exist, and update the current working
    # directory to the new location if needed.
    if [[ "${NB_USER}" != "jovyan" ]]; then
        if [[ ! -e "/home/${NB_USER}" ]]; then
            _log "Attempting to copy /home/jovyan to /home/${NB_USER}..."
            mkdir "/home/${NB_USER}"
            if cp -a /home/jovyan/. "/home/${NB_USER}/"; then
                _log "Success!"
            else
                _log "Failed to copy data from /home/jovyan to /home/${NB_USER}!"
                _log "Attempting to symlink /home/jovyan to /home/${NB_USER}..."
                if ln -s /home/jovyan "/home/${NB_USER}"; then
                    _log "Success creating symlink!"
                else
                    _log "ERROR: Failed copy data from /home/jovyan to /home/${NB_USER} or to create symlink!"
                    exit 1
                fi
            fi
        fi

[njacobson-nci]

I am also having this issue in a gpu-jupyter install. The cuda environment doesn't transfer over cleanly to the new user.

iot-salzburg/gpu-jupyter#108 (comment)

[benz0li]

@njacobson-nci You may be interested in the CUDA-enabled JupyterLab Python docker stack, then:

[...]

Prerequisites

The same as the JupyterLab Python docker stack plus

• NVIDIA GPU
• NVIDIA Linux driver
• NVIDIA Container Toolkit

ℹ️ The host running the GPU accelerated images only requires the NVIDIA driver, the CUDA toolkit does not have to be installed.

[...]

Similar projects

• iot-salzburg/gpu-jupyter
• prp/jupyter-stack

What makes this project different:

1. Multi-arch: linux/amd64, linux/arm64/v8
2. Derived from nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04
   • including development libraries and headers
3. TensortRT and TensorRT plugin libraries
   • including development libraries and headers
4. IDE: code-server next to JupyterLab
5. Just Python – no Conda / Mamba

See Notes for tweaks, settings, etc.

[njacobson-nci]

@benz0li
Thanks, I will try out your library

[benz0li]

@njacobson-nci Follow-up to iot-salzburg/gpu-jupyter#108 (comment). Using one of my images and after pip install bitsandbytes torch:

$ python
Python 3.10.10 (main, Feb  8 2023, 13:06:32) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bitsandbytes

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/home/benz0li/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/extras/CUPTI/lib64'), PosixPath('/usr/local/cuda/lib'), PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /home/benz0li/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118.so...
>>> 

[mathbunnyru]

@masip85 I merged the fix.
New images will be ready in approximately 1 hour.
The recipe has also been updated.

Sorry for making you wait a year.