can't start ipykernel on HPC node

[swnesbitt]

Strange issue on our linux HPC cluster. Running

python -m ipykernel --log-level DEBUG

on head node of our cluster works fine.
On a compute node, the process hangs

snesbitt@keeling-d01 ~$ python -m ipykernel --log-level DEBUG
[IPKernelApp] IPYTHONDIR set to: /data/keeling/a/snesbitt/.ipython
[IPKernelApp] Using existing profile dir: u'/data/keeling/a/snesbitt/.ipython/profile_default'
[IPKernelApp] Searching path [u'/data/keeling/a/snesbitt', u'/data/keeling/a/snesbitt/.ipython/profile_default', '/data/keeling/a/snesbitt/anaconda2/etc/ipython', '/usr/local/etc/ipython', '/etc/ipython'] for config files
[IPKernelApp] Attempting to load config file: ipython_config.py
[IPKernelApp] Looking for ipython_config in /etc/ipython
[IPKernelApp] Looking for ipython_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_config in /data/keeling/a/snesbitt/anaconda2/etc/ipython
[IPKernelApp] Looking for ipython_config in /data/keeling/a/snesbitt/.ipython/profile_default
[IPKernelApp] Looking for ipython_config in /data/keeling/a/snesbitt
[IPKernelApp] Attempting to load config file: ipython_kernel_config.py
[IPKernelApp] Looking for ipython_kernel_config in /etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /data/keeling/a/snesbitt/anaconda2/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /data/keeling/a/snesbitt/.ipython/profile_default
[IPKernelApp] Looking for ipython_kernel_config in /data/keeling/a/snesbitt
[IPKernelApp] Connection file not found: kernel-27529.json
[IPKernelApp] Starting the kernel at pid: 27529

Brand new miniconda python 3.6 install. Only addition was pip install jupyter

I tried ulimit -s -S unlimited to see if there were issues starting threads, but doesn't help.

Any help is greatly appreciated!

[swnesbitt]

This turned out to be an issue with /dev/random on our systems. We solved it with a strace.

There was insufficient entropy on our system in /dev/random for ipykernel. When we made /dev/random inaccessible (through a chmod 640), ipykernel accessed /dev/urandom, which allowed the kernel to initialize.

Closing.

[ltalirz]

For others running into this issue, the problem is likely not in the ipykernel code, which (at least today) seems to rely on os.urandom() which relies on /dev/urandom, which should never block.
However, ipykernel calls Socket.bind_to_random_port from pyzmq, and there is a related open issue on that repo zeromq/pyzmq#1224

P.S. See jupyter/help#480 (comment) for instructions on how to check whether you are indeed suffering from this issue.
I found that the code to start the ipython kernel does not quite work anymore; if you're using jupyter_client, this worked for me:

from jupyter_client.manager import start_new_kernel
km, kc = start_new_kernel()
km.shutdown_kernel()