zmq "too many open files" (sockets?)

[hjoliver]

After leaving the end-to-end system running for a while (with a small test suite at the back end) I get this error repeatedly:

ERROR:tornado.application:Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7fd3aa9857b8>>, <Task finished coro=<WorkflowsManager.gather_workflows() done, defined at /home/oliverh/cylc/cylc-uiserver/workflows_mgr.py:83> exception=ZMQError('Too many open files')>)
Traceback (most recent call last):
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/home/oliverh/cylc/cylc-uiserver/workflows_mgr.py", line 94, in gather_workflows
    items = await asyncio.gather(*gathers)
  File "/home/oliverh/cylc/cylc-uiserver/workflows_mgr.py", line 69, in est_workflow
    client = SuiteRuntimeClient(reg, host=host, port=port, timeout=timeout)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/client.py", line 254, in __init__
    timeout_handler=partial(self._timeout_handler, suite, host, port)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/client.py", line 104, in __init__
    self.socket = CONTEXT.socket(zmq.REQ)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/sugar/context.py", line 146, in socket
    s = self._socket_class(self, socket_type, **kwargs)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/_future.py", line 134, in __init__
    super(_AsyncSocket, self).__init__(context, socket_type, **kwargs)
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/sugar/socket.py", line 59, in __init__
    super(Socket, self).__init__(*a, **kw)
  File "zmq/backend/cython/socket.pyx", line 328, in zmq.backend.cython.socket.Socket.__init__
zmq.error.ZMQError: Too many open files

and separately:

ERROR:asyncio:Exception in callback BaseAsyncIOLoop._handle_events(9, 1)
handle: <Handle BaseAsyncIOLoop._handle_events(9, 1)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 138, in _handle_events
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/netutil.py", line 260, in accept_handler
  File "/usr/local/lib/python3.7/socket.py", line 212, in accept
OSError: [Errno 24] Too many open files

[hjoliver]

May be relevant? vatlab/sos#1208

[kinow]

Quite sure I had same issue. Thought I had created an issue for that already

[kinow]

#44