In bundled python package installation is performed in user location

[Czaki]

Problem

When the download and install .deb package and try run pip install using pip from bundled python there is information that the package will be installed in user mode because site-package is not writable.

Packages installed in user mode will be available in all python3.8 environments and may introduce unexpected side effects.

pip bundled in app should use different location to install a new packages.

Proposed Solution

set alternative install location in user directory when run in non administrator mode and add this location to PYTHONPATH

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[psychemedia]

This issue also affects %pip install run from notebook code cells in notebooks running in the app, and which would ordinarily be expected to guarantee the availability of the installed package to the kernel from which the magic was run.

[Czaki]

This issue also affects %pip install run from notebook code cells in notebooks running in the app, and which would ordinarily be expected to guarantee the availability of the installed package to the kernel from which the magic was run.

It will be available, but it will be also available in all other user python3.8 (in my case) environments.

[psychemedia]

@Czaki Ah... seems it is there but I had to restart the kernel before I could see the newly installed packages, whereas typically (for me at least...) packages seem to be available without the need to restart (although a warning that a kernel restart may be required is typically given.)

I also found I had one package that errored on a line:

_base = os.environ['JUPYTER_SERVER_ROOT'] if 'JUPYTER_SERVER_ROOT' in os.environ else os.environ['HOME']

with os.environ['HOME'] not found; this is probably an issue with how the package works as much as anything, but I note it here anyway...

[Czaki]

@Czaki Ah... seems it is there but I had to restart the kernel before I could see the newly installed packages, whereas typically (for me at least...) packages seem to be available without the need to restart (although a warning that a kernel restart may be required is typically given.)

If I good remember it was caused by installation outside site-packages. I meet this when performing installation in developer mode (pip install -e).

[jpivarski]

I'm watching this issue, too, because it's the one capability that's necessary to make jupyterlab_app useful. (What's Python without packages?)

As I understand the current status,

• running pip install X in the JupyterLab terminal or in a notebook cell starting with ! runs pip in some global mode, putting the installed package in ~/.local, rather than in the conda environment
• running conda install X in the JupyterLab terminal or in a notebook cell starting with ! doesn't work at all because the install area is not writable.

Oddly, considering this error, the pip and conda commands are coming from the right place (this output came from the JupyterLab terminal; the output from a notebook cell is the same):

$ which pip
/opt/JupyterLab/resources/jlab_server/bin/pip
$ which conda
/opt/JupyterLab/resources/jlab_server/bin/conda

Aha! The problem is that the site-packages is not writable, and so that's why pip is deferring to ~/.local:

$ pip install uproot
Defaulting to user installation because normal site-packages is not writeable
Collecting uproot
  Using cached uproot-4.1.2-py2.py3-none-any.whl (290 kB)
Requirement already satisfied: numpy in /opt/JupyterLab/resources/jlab_server/lib/python3.8/site-packages (from uproot) (1.21.2)
Requirement already satisfied: setuptools in /opt/JupyterLab/resources/jlab_server/lib/python3.8/site-packages (from uproot) (58.0.4)
Installing collected packages: uproot
Successfully installed uproot-4.1.2

We see where it is when uninstalling the package:

$ pip uninstall uproot
Found existing installation: uproot 4.1.2
Uninstalling uproot-4.1.2:
  Would remove:
    /home/jpivarski/.local/lib/python3.8/site-packages/uproot-4.1.2.dist-info/*
    /home/jpivarski/.local/lib/python3.8/site-packages/uproot/*
Proceed (Y/n)? y
  Successfully uninstalled uproot-4.1.2

The error message from conda seems to be from the same underlying cause:

$ conda install uproot
Collecting package metadata (current_repodata.json): failed

NotWritableError: The current user does not have write permissions to a required path.
  path: /opt/JupyterLab/resources/jlab_server/pkgs/urls.txt
  uid: 1000
  gid: 1000

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

  $ sudo chown 1000:1000 /opt/JupyterLab/resources/jlab_server/pkgs/urls.txt

In general, it's not advisable to use 'sudo conda'.

Interestingly, the conda in JupyterLab_app sees my normal conda environments:

$ conda env info
usage: conda-env [-h] {create,export,list,remove,update,config} ...
conda-env: error: invalid choice: 'info' (choose from 'create', 'export', 'list', 'remove', 'update', 'config')
$ conda env list
# conda environments:
#
                         /home/jpivarski/miniconda3
                         /home/jpivarski/miniconda3/envs/np13
                         /home/jpivarski/miniconda3/envs/np21
                         /home/jpivarski/miniconda3/envs/numba-py38
                         /home/jpivarski/miniconda3/envs/pandas-0.22
                         /home/jpivarski/miniconda3/envs/py27
                         /home/jpivarski/miniconda3/envs/py39
                         /home/jpivarski/miniconda3/envs/rapids
                         /home/jpivarski/miniconda3/envs/test-fastjet
base                  *  /opt/JupyterLab/resources/jlab_server

But the conda I use outside of JuptyerLab_app, in a normal terminal, for instance, does not see the JupyterLab_app one:

% conda env list
# conda environments:
#
base                  *  /home/jpivarski/miniconda3
np13                     /home/jpivarski/miniconda3/envs/np13
np21                     /home/jpivarski/miniconda3/envs/np21
numba-py38               /home/jpivarski/miniconda3/envs/numba-py38
pandas-0.22              /home/jpivarski/miniconda3/envs/pandas-0.22
py27                     /home/jpivarski/miniconda3/envs/py27
py39                     /home/jpivarski/miniconda3/envs/py39
rapids                   /home/jpivarski/miniconda3/envs/rapids
test-fastjet             /home/jpivarski/miniconda3/envs/test-fastjet

If there's some way now to pip and conda into /opt/JupyterLab/resources/jlab_server, I'd love to know it! If not, this is a bug-fix/feature-request. Thanks!

(Or do I just have to install it not from the .deb? That seemed like the appropriate way to install it, as I'm an end-user, not a developer.)

[fperez]

BTW - as a rule of thumb, instructions that call for using pip/conda from within a notebook should in general use the %pip and %conda magics instead of the !pip/!conda shell escapes. The latter are far more likely to cause path/environment problems like the ones being discussed here...

In addition to other improvements, I think a) the suggestion (until better environment management and integration with one's base Conda install is developed) to install in the user's home directory should be always emphasized; b) using %pip or %conda instead of the ! options should be the default in all docs, especially in the customization guide.

Here's a quick illustration that the experience with the magics works quite well:

[mbektas]

BTW - as a rule of thumb, instructions that call for using pip/conda from within a notebook should in general use the %pip and %conda magics instead of the !pip/!conda shell escapes. The latter are far more likely to cause path/environment problems like the ones being discussed here...

Thank you @fperez!, I will update the documentation to use %pip / %conda.

[psychemedia]

I'm all for the UX that @fperez suggests. When things are going well there is no need to restart the kernel.

You can mitigate against a restart in a linear interaction by always putting all the import statements at the top of a notebook, %pip installing them if you have to install any missing packages, restarting the kernel, then trying to import all the packages again in the next cell. (This sort of interaction may not make sense to folk using notebooks for development, but if you are using notebooks for teaching and want to make a reproducible linear document that explicitly shows a sequence of steps, it does!)

[Czaki]

@fperez But this what you wrote still does not address the core of the issue that such installed packages will be available in all environments of such python version (currently 3.8) which breaks environments isolation.