Environment setup before starting the kernel

[rbiswas4]

We are trying to use the JupyterHub setup on nersc, but we need to setup our own code (LSST stack) by running a couple of setup scripts (which work on the NERSC machines) before starting an ipython/jupyter notebook, so that our package is setup. How can we achieve this?

[yuvipanda]

Hello!

What spawner are you using? You can probably use c.Spawner.cmd to do what you want. This is the command that is run by JupyterHub spawners. You can override that to be another command that you provision which does whatever setup you want and then calls the jupyterhub-singleuser command (which is the default). This will all be run as the user who is being spawned.

[rbiswas4]

Hi,

Sorry, I was not clear. I was not using a spawner explicitly, I was simply trying to point the browser at nersc.jupyter.gov which already spawns a server internally (I think). I would be happy to spawn something myself, if that helps.

[minrk]

Do you mean before starting a single notebook, or before starting your whole notebook server?

Depending on what the commands are and how early they need to run, perhaps the simplest thing to do is to put a Python script in ~/.ipython/profile_default/startup/init_lsst.py, which will be executed by IPython early in its setup, but after initializing IPython.

If it needs to be shell commands executed before starting Python, perhaps the thing to do is to create a kernelspec for your setup. You can start by creating an lsst kernelspec starting with IPython's default:

ipython kernel install --user --name lsst

which will create a directory like:

~/.local/share/jupyter/kernels/lsst

containing a kernel.json

You can then write a script (lsst-kernel.sh) that does your setup followed by the IPython kernel launch:

#!/usr/bin/env bash
module lsst
# setup lsst, environment variables, etc.

# this is the critical part, and should be at the end of your script:
exec python -m ipykernel $@

Finally, you can modify the ~/.local/share/jupyter/kernels/lsst/kernel.json file to call your script, changing:

 "argv": [
  "/usr/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],

to

 "argv": [
  "/path/to/lsst-kernel.sh",
  "-f",
  "{connection_file}"
 ],

(replacing python -m ipykernel with /path/to/lsst-kernel.sh).

[jbkalmbach]

Hi @minrk I am picking up on this for @rbiswas4 and whenever I try to get this to run the kernel says it is loading and then says it is ready, but as soon as that happens I get a dead kernel. Any idea why?

[minrk]

Thanks for picking it up! Can you see the logs for the single-user server? If the first execution dies right away, it's likely that there's a low-level error that wouldn't propagate all the way to the client.

[jbkalmbach]

Hi @minrk it was actually some of what was going on behind the scenes at NERSC that wouldn't let us execute the commands in the shell script we wanted. We got in touch with them and sorted it out and now your steps from November 12 work perfectly. Thanks for your help!

[minrk]

Great to hear! Closing then.