Prepend PATH with environment's bin when loading kernel

[tritemio]

Linking to a jupyter notebook issue which may be more appropriate here:

See jupyter/notebook#4527

Please advice on where you prefer to continue the discussion.

[tritemio]

Summarizing jupyter/notebook#4527 here as it is more relevant to ipykernel than notebook

Issue summary

When activating an env in the terminal, the PATH is prepended with the environment's bin dir. Instead, when running the notebook from a base environment and loading the kernel of another env, the PATH is not changed.

This causes issues with tools that need to call executables, expecting them to be in the PATH. This is the case for pydot (requires the dot executable) and the %tensorboard notebook magic (requires the tensorboard executable).

Stop-gap solution

@kevin-bates suggested to setup a shell wrapper that modifies the PATH before launching the kernel (details below). This works, but it is cumbersome and error-prone to setup, especially for non-experts and/or for multiple environments.

Ideal solution

Ideally, we would like to create the kernel_spec in the usual way with

python -m ipykernel install ...

with a new flag (like --prepend_env_bin_path) for prepending the PATH with the bin dir of current env. Would be great if ipykernel could detect the current active env and find the bin dir automatically. Otherwise, specifying the dir manually would still be a big improvement.

---

Details of solution proposed by @kevin-bates in jupyter/notebook#4527:

Modify the kernel_spec to:

{
 "env": {
  "MY_ENV_BIN_DIR": "/opt/anaconda/envs/my_special_env/bin"
 },
 "argv": [
  "/usr/local/share/jupyter/kernels/python3_with_env/prepend_and_launch.sh",
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 with env",
 "language": "python"
}

Note that we specify the launcher wrapper prepend_and_launch.sh with the full path (it is inside the kernel_spec dir). python, instead, is specified with no path, because the right python will be in the env bin's dir.

The wrapper prepend_and_launch.sh is simply:

#!/bin/bash
export PATH=${MY_ENV_BIN_DIR}:${PATH}
"$@"

Finally, prepend_and_launch.sh should have execution rights.

Note that this solution may not work on windows.

EDIT: Replaced $* with "$@" as suggested by @wchargin in https://github.com/tensorflow/tensorboard/pull/2088

[soxofaan]

Alternative simpler solution: avoid the prepend script and just hard-code a PATH "env" field in the kernel spec file e.g.

{
 "env": {
   "PATH": "/opt/anaconda/envs/my_special_env/bin:/usr/local/bin:/usr/bin/:etcetera/etcetera"
 }
 ....
}

This has the disadvantage that the PATH in the kernel does not adapt automatically to changes in the user's PATH. But it can also be an advantage for some use cases: it's more explicit and could improve reproducability.

[schmoelder]

Has anyone got this running in a JupyterHub environment with DockerSpawner?