Long running notebook died with a coredump

[wasade]

WARNING:root:Read error on 993: [Errno 104] Connection reset by peer
WARNING:root:error on read
Traceback (most recent call last):
File "/Users/mcdonadt/lib/python2.6/site-packages/tornado-2.4-py2.6.egg/tornado/iostream.py", line 355, in _handle_read
if self._read_to_buffer() == 0:
File "/Users/mcdonadt/lib/python2.6/site-packages/tornado-2.4-py2.6.egg/tornado/iostream.py", line 422, in _read_to_buffer
chunk = self._read_from_socket()
File "/Users/mcdonadt/lib/python2.6/site-packages/tornado-2.4-py2.6.egg/tornado/iostream.py", line 403, in _read_from_socket
chunk = self.socket.recv(self.read_chunk_size)
error: [Errno 104] Connection reset by peer
[NotebookApp] Using existing kernel: 4a9ae6fa-7518-43cd-9007-483967e7d152
[NotebookApp] Connecting to: tcp://127.0.0.1:59706
[NotebookApp] Connecting to: tcp://127.0.0.1:42576
Too many open files (bundled/zeromq/src/signaler.cpp:330)
Aborted (core dumped)
$ ipython --version
0.14.dev

[wasade]

Note, it is possible this is associated with some stale NFS handles. We did have this server running for a few months though w/o issue

[minrk]

running out of FDs can be a problem, but there isn't too much IPython can do. I presume this is on OS X? Check ulimit -n. You may want to up this limit, if you have permission, it is often quite small. Do you perhaps have many running kernels at the same time?

[wasade]

Thanks, Min. This was on RHEL6. Here is the output from ulimit -n

07:28:47 (mcdonadt@stein):~$ ulimit -n
1024

We were only running a single instance of ipython from this host. We aren't
planning on rerunning the existing setup on this particular host, but we
can up the open file descriptors on the final host if you think that will
cover it.

On Fri, Jul 5, 2013 at 10:36 PM, Min RK notifications@github.com wrote:

running out of FDs can be a problem, but there isn't too much IPython can
do. I presume this is on OS X? Check ulimit -n. You may want to up this
limit, if you have permission, it is often quite small. Do you perhaps have
many running kernels at the same time?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3213#issuecomment-20548847
.

[minrk]

There could be a bug in IPython (or tornado or pyzmq or libzmq) preventing the closing of file descriptors, but that is the limit being hit, so raising it will help. If it's a bug, raising the limit will only increase the longevity of the process before it dies again. If it's not a bug, and it really is concurrent actions going on (running kernels plus web frontends), then it will increase the amount of concurrent action you can have before it balks.