Jupyter Notebook silently hanging after importing Client in Docker images #2414
Description
Problem
There seems to be an issue with running Dask + Jupyter on a Ubuntu 16.04 + Python 3.5 docker image. After a cell that contains from distributed import Client or from dask.distributed import Client, all subsequent Jupyter notebook cells will hang and be unresponsive. The kernel seems to never die nor report any issues. The only recovery is to restart the kernel entirely. This results in never being able to use dask distributed via a notebook on the machine.
Steps to Cause
Dockerfile
FROM ubuntu:16.04
RUN apt-get update
RUN apt-get install locales
# Set the locale
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN apt-get install -y python python-pip python3 python3-pip python3-wheel python3-venv git curl lsb-release
RUN mkdir /app
RUN mkdir /app/work
WORKDIR /app
RUN python3 -m venv /app/venv
RUN /app/venv/bin/pip3 install --upgrade pip setuptools wheel
RUN /app/venv/bin/pip3 install jupyter "dask[complete]" distributed
CMD /app/venv/bin/jupyter notebook --allow-root --ip=0.0.0.0
Commands to run:
docker build -t dask-in-docker:latest .
docker run -p 8888:8888 dask-in-docker:latest
Connect to the Jupyter notebook:
in[1]:
from dask.distributed import Client
print(5)
out[1]: 5
in[2]:
print(6)
--> Entire notebook hangs now and seems to be unrecoverable.
Of specific note: running the exact same commands via
docker exec -it $CONTAINER_NAME bash
$ source /app/venv/bin/activate
$ ipython
And running the same import's work as intended, showing that the issue seems to be coupled with how Jupyter is actually working and not related to ipython or the environment.
Solution (18.04 and Python3.7)
After trying many different versions and combinations of tornado, jupyter, ipython, ipykernel, dask and distributed, I eventually tried updating the OS + Python version and was able to avoid the problem.
Dockerfile
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install locales
# Set the locale
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN apt-get install -y python3.7 python3-pip python3.7-venv python3.7-dev git curl lsb-release
RUN mkdir /app
RUN mkdir /app/work
WORKDIR /app
RUN python3.7 -m venv /app/venv
RUN /app/venv/bin/pip3 install --upgrade pip setuptools wheel
RUN /app/venv/bin/pip3 install jupyter "dask[complete]" distributed
CMD /app/venv/bin/jupyter notebook --allow-root --ip=0.0.0.0
This dockerfile (18.04 + Python3.7) fixes the problem!
I'm not sure why, so I'm opening this up as a possible edge case that could be handled for backwards compatibility. I am personally unblocked and will be moving forward with newer versions of things, but I wanted to post this here for helping others who may come across a similar issue.