Voila leaks memory

[jpswinski]

The python container running on the node manager slowly eats away at all of the available memory on the system until no memory is left at voila stops working.

The first symptom was that plot on the interactive map would not show any of the elevations.

Looking at the logs, the following error was observed to be scrolling:

ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-12502345' coro=<WebSocketProtocol13.write_message.<locals>.wrapper() done, defined at /opt/conda/envs/sliderule/lib/python3.8/site-packages/tornado/websocket.py:1100> exception=WebSocketClosedError()>
Traceback (most recent call last):
  File "/opt/conda/envs/sliderule/lib/python3.8/site-packages/tornado/websocket.py", line 1102, in wrapper
    await fut
  File "/opt/conda/envs/sliderule/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/sliderule/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/opt/conda/envs/sliderule/lib/python3.8/site-packages/tornado/websocket.py", line 1104, in wrapper
    raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError

Looking at the system metrics, the memory usage plot showed a slow decline in available memory:

Using docker stats it was confirmed that the python container was using all of the memory.

[jpswinski]

It looks like adding the following options to start voila fixes this problem:

--MappingKernelManager.cull_interval=60 --MappingKernelManager.cull_idle_timeout=120

This will clean up kernels every 60 seconds if the kernel has been idle for 120 seconds. In testing, it looks like a kernel is idle only after the browser window with voila open is closed.

[jpswinski]