Which env should be first when there's two?

[minrk]

Proposed change

When running with an older Python (just 2.7 prior to #1239, <= 3.6 and more in the future), there are two environments:

1. $NB_PYTHON_PREFIX - the env where the notebook server runs
2. $KERNEL_PYTHON_PREFIX - the env where the kernel runs

These two variables are both always defined, so absolute paths with env variables can always be used to disambiguate. In most cases, these two are the same, but in the too-old-python cases, they will differ. The question is: which should be first, and when?

There's ambiguity on the command-line, because both envs are on $PATH. Right now, we are in a bit of a peculiar situation, because $NB is first on path so that e.g. jupyter-lab will launch in the server env, but we set CONDA_DEFAULT_ENV so that conda install installs in the kernel env -it is assumed that runtime installation will want to go in the kernel environment. However, this means tha conda install and pip install will install to different environments, and python script.py, ipython, etc. will launch in the server env, not the kernel one. (aside: %pip install from a notebook will work, because that reliably resolves to the kernel's env, regardless of $PATH).

I think in most cases, when the two differ the least surprise will occur if $KERNEL_PYTHON_PREFIX comes first. This is easy enough to switch, but we need to be careful to ensure that the launch command finds the NB prefix first, even if the rest of the time the KERNEL prefix should come first. Anything that relied on the server env being first may also break. This should only come up, though, for commands that are present in both prefixes - python, pip, jupyter, etc. It's not a pure win, though - common postBuild commands operate on the server env, like jupyter labextension install. Some of these could break if we switch.

Whether we change it or not, I think we should surface these environment variables, and encourage folks to use them when running commands to be explicit about which env they are running in, especially in postBuild.

Alternative options

• Keep the status quo - server env comes first, kernel comes second, special-case conda install
• Only remove conda special-case, so conda and pip are both pointed to the same environment (the server env)

Who would use this feature?

Users who execute jupyter or python or pip commands in the terminal, either at runtime in the container (e.g. in a terminal), or in steps like postBuild.

One of the more common postBuild steps is something like jupyter labextension install. This relies on the current $NB prefix being first. It will actually work most of the time in both cases, as long as jupyterlab is not also in the kernel env (jupyter [subcommand] works across envs if both are on your path - it just picks the first it finds).

How much effort will adding it take?

It may be tricky to get the details right in keeping our default launch command working, if we switch the kernel to be in the front, since there are a lot of cases:

• no command specified
• user specified, e.g. jupyter-lab (which should be in the front here?)
• BinderHub, which currently specifies jupyter-notebook

Who can do this work?

Folks with path/shell environment and possibly conda env activation experience

[betatim]

I think we have issues from a while ago about people being surprised that if they run python in the terminal (of jupyterlab) that they don't find the dependencies that they thought they installed. Which adds some data points to the "the NB env should be first".