Environment setup before starting the kernel

[rbiswas4]

We are trying to use the JupyterHub setup on nersc, but we need to setup our own code (LSST stack) by running a couple of setup scripts (which work on the NERSC machines) before starting an ipython/jupyter notebook, so that our package is setup. How can we achieve this?

[yuvipanda]

Hello!

What spawner are you using? You can probably use c.Spawner.cmd to do what you want. This is the command that is run by JupyterHub spawners. You can override that to be another command that you provision which does whatever setup you want and then calls the jupyterhub-singleuser command (which is the default). This will all be run as the user who is being spawned.

[rbiswas4]

Hi,

Sorry, I was not clear. I was not using a spawner explicitly, I was simply trying to point the browser at nersc.jupyter.gov which already spawns a server internally (I think). I would be happy to spawn something myself, if that helps.

[minrk]

Do you mean before starting a single notebook, or before starting your whole notebook server?

Depending on what the commands are and how early they need to run, perhaps the simplest thing to do is to put a Python script in ~/.ipython/profile_default/startup/init_lsst.py, which will be executed by IPython early in its setup, but after initializing IPython.

If it needs to be shell commands executed before starting Python, perhaps the thing to do is to create a kernelspec for your setup. You can start by creating an lsst kernelspec starting with IPython's default:

ipython kernel install --user --name lsst

which will create a directory like:

~/.local/share/jupyter/kernels/lsst

containing a kernel.json

You can then write a script (lsst-kernel.sh) that does your setup followed by the IPython kernel launch:

#!/usr/bin/env bash
module lsst
# setup lsst, environment variables, etc.

# this is the critical part, and should be at the end of your script:
exec python -m ipykernel $@

Finally, you can modify the ~/.local/share/jupyter/kernels/lsst/kernel.json file to call your script, changing:

 "argv": [
  "/usr/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],

to

 "argv": [
  "/path/to/lsst-kernel.sh",
  "-f",
  "{connection_file}"
 ],

(replacing python -m ipykernel with /path/to/lsst-kernel.sh).

[jbkalmbach]

Hi @minrk I am picking up on this for @rbiswas4 and whenever I try to get this to run the kernel says it is loading and then says it is ready, but as soon as that happens I get a dead kernel. Any idea why?

[minrk]

Thanks for picking it up! Can you see the logs for the single-user server? If the first execution dies right away, it's likely that there's a low-level error that wouldn't propagate all the way to the client.

[jbkalmbach]

Hi @minrk it was actually some of what was going on behind the scenes at NERSC that wouldn't let us execute the commands in the shell script we wanted. We got in touch with them and sorted it out and now your steps from November 12 work perfectly. Thanks for your help!

[minrk]

Great to hear! Closing then.

[GrantFinneman]

I understand this is closed but I am running into the same exact issue but using The Littlest Jupyterhub server instead of a single user install. My kernel.json is different but I need the same thing. I have a couple compiled programs that have libraries sourced by my .bashrc in the jupyter account but since the notebooks are spawned by the default spawner they do not have the same environment variables. How would I accomplish this?

[VladimirFokow]

It wasn't working for me. This fixed it:

Replace #!/usr/bin/env bash with #!/bin/bash - it worked.

(then if changed back to #!/usr/bin/env bash, weirdly it worked too.
But I'd rather not change it back - because I don't know why it wasn't working in the beginning
so I don't know if it can break again.)

(Note: instead of ipykernel, I was using ipykernel_launcher because it's the default today.)