Starting a Local cluster in an existing jupyter notebook crashes

[jbusecke]

Describe the issue:
I suspect that some recent updates to dask/distributed introduced a bug when using a LocalCluster in an existing notebook.

Minimal Complete Verifiable Example:

This conda environment is sufficient to reproduce the problem:

mamba create -n test-dask jupyterlab dask distributed python

EDIT: the above actually works but we are not looking to use python 3.11 just yet.

this does not work:

mamba create -n test-dask "jupyterlab=3.5.3" dask distributed "python<3.11"

which resolves to the following versions

anyio 3.6.2 pyhd8ed1ab_0 conda-forge appnope 0.1.3 pyhd8ed1ab_0 conda-forge argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py310h8e9501a_3 conda-forge asttokens 2.2.1 pyhd8ed1ab_0 conda-forge attrs 22.2.0 pyh71513ae_0 conda-forge babel 2.11.0 pyhd8ed1ab_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.11.2 pyha770c72_0 conda-forge bleach 6.0.0 pyhd8ed1ab_0 conda-forge bokeh 2.4.3 pyhd8ed1ab_3 conda-forge brotlipy 0.7.0 py310h8e9501a_1005 conda-forge bzip2 1.0.8 h3422bc3_4 conda-forge ca-certificates 2022.12.7 h4653dfc_0 conda-forge certifi 2022.12.7 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py310h2399d43_3 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge click 8.1.3 unix_pyhd8ed1ab_2 conda-forge cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge comm 0.1.2 pyhd8ed1ab_0 conda-forge cryptography 39.0.1 py310hfc83b78_0 conda-forge cytoolz 0.12.0 py310h8e9501a_1 conda-forge dask 2023.1.1 pyhd8ed1ab_0 conda-forge dask-core 2023.1.1 pyhd8ed1ab_0 conda-forge debugpy 1.6.6 py310h0f1eb42_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge distributed 2023.1.1 pyhd8ed1ab_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge flit-core 3.8.0 pyhd8ed1ab_0 conda-forge freetype 2.12.1 hd633e50_1 conda-forge fsspec 2023.1.0 pyhd8ed1ab_0 conda-forge heapdict 1.0.1 py_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.0.0 pyha770c72_0 conda-forge importlib_metadata 6.0.0 hd8ed1ab_0 conda-forge importlib_resources 5.10.2 pyhd8ed1ab_0 conda-forge ipykernel 6.21.1 pyh736e0ef_0 conda-forge ipython 8.10.0 pyhd1c38e8_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.18.2 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge jpeg 9e he4db4b2_2 conda-forge json5 0.9.5 pyh9f0ad1d_0 conda-forge jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge jupyter_client 8.0.2 pyhd8ed1ab_0 conda-forge jupyter_core 5.2.0 py310hbe9552e_0 conda-forge jupyter_events 0.6.3 pyhd8ed1ab_0 conda-forge jupyter_server 2.2.1 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge jupyterlab 3.5.3 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jupyterlab_server 2.19.0 pyhd8ed1ab_0 conda-forge lcms2 2.14 h481adae_1 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libblas 3.9.0 16_osxarm64_openblas conda-forge libcblas 3.9.0 16_osxarm64_openblas conda-forge libcxx 14.0.6 h2692d47_0 conda-forge libdeflate 1.17 h1a8c8d9_0 conda-forge libffi 3.4.2 h3422bc3_5 conda-forge libgfortran 5.0.0 11_3_0_hd922786_27 conda-forge libgfortran5 11.3.0 hdaf2cc0_27 conda-forge liblapack 3.9.0 16_osxarm64_openblas conda-forge libopenblas 0.3.21 openmp_hc731615_3 conda-forge libpng 1.6.39 h76d750c_0 conda-forge libsodium 1.0.18 h27ca646_1 conda-forge libsqlite 3.40.0 h76d750c_0 conda-forge libtiff 4.5.0 h5dffbdd_2 conda-forge libwebp-base 1.2.4 h57fd34a_0 conda-forge libxcb 1.13 h9b22ae9_1004 conda-forge libzlib 1.2.13 h03a7124_4 conda-forge llvm-openmp 15.0.7 h7cfbb63_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4 4.3.2 py310ha6df754_0 conda-forge lz4-c 1.9.4 hb7217d7_0 conda-forge markupsafe 2.1.2 py310h8e9501a_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mistune 2.0.5 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.4 py310h2887b22_1 conda-forge nbclassic 0.5.1 pyhd8ed1ab_0 conda-forge nbclient 0.7.2 pyhd8ed1ab_0 conda-forge nbconvert 7.2.9 pyhd8ed1ab_0 conda-forge nbconvert-core 7.2.9 pyhd8ed1ab_0 conda-forge nbconvert-pandoc 7.2.9 pyhd8ed1ab_0 conda-forge nbformat 5.7.3 pyhd8ed1ab_0 conda-forge ncurses 6.3 h07bb92c_1 conda-forge nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge notebook 6.5.2 pyha770c72_1 conda-forge notebook-shim 0.2.2 pyhd8ed1ab_0 conda-forge numpy 1.24.2 py310h3d2048e_0 conda-forge openjpeg 2.5.0 hbc2ba62_2 conda-forge openssl 3.0.8 h03a7124_0 conda-forge packaging 23.0 pyhd8ed1ab_0 conda-forge pandas 1.5.3 py310h2b830bf_0 conda-forge pandoc 2.19.2 hce30654_1 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.3.0 pyhd8ed1ab_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 9.4.0 py310h5a7539a_1 conda-forge pip 23.0 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge platformdirs 3.0.0 pyhd8ed1ab_0 conda-forge prometheus_client 0.16.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.36 pyha770c72_0 conda-forge psutil 5.9.4 py310h8e9501a_0 conda-forge pthread-stubs 0.4 h27ca646_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.14.0 pyhd8ed1ab_0 conda-forge pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge pyrsistent 0.19.3 py310h8e9501a_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.10.9 h3ba56d0_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.16.2 pyhd8ed1ab_0 conda-forge python-json-logger 2.0.4 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytz 2022.7.1 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py310h8e9501a_5 conda-forge pyzmq 25.0.0 py310hc407298_0 conda-forge readline 8.1.2 h46ed386_0 conda-forge requests 2.28.2 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 67.1.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge terminado 0.17.1 pyhd1c38e8_0 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 he1e0b03_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.2 py310h8e9501a_1 conda-forge traitlets 5.9.0 pyhd8ed1ab_0 conda-forge typing-extensions 4.4.0 hd8ed1ab_0 conda-forge typing_extensions 4.4.0 pyha770c72_0 conda-forge tzdata 2022g h191b570_0 conda-forge urllib3 1.26.14 pyhd8ed1ab_0 conda-forge wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.5.1 pyhd8ed1ab_0 conda-forge wheel 0.38.4 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.9 h27ca646_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xz 5.2.6 h57fd34a_0 conda-forge yaml 0.2.5 h3422bc3_2 conda-forge zeromq 4.3.4 hbdafb3b_1 conda-forge zict 2.2.0 pyhd8ed1ab_0 conda-forge zipp 3.13.0 pyhd8ed1ab_0 conda-forge zstd 1.5.2 hf913c23_6 conda-forge

Now if you open a new notebook with jupyterlab and execute the following code in a cell twice

from distributed import LocalCluster
cluster = LocalCluster()

the notebook cell throws a horrible long warning (which makes the notebook unresponsive for a while and maybe? crashes the jupyterlab)

I was able to extract this bit from the warning/error trace:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/juliusbusecke/miniconda/envs/test-dask/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/juliusbusecke/miniconda/envs/test-dask/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/juliusbusecke/miniconda/envs/test-dask/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/juliusbusecke/miniconda/envs/test-dask/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Users/juliusbusecke/miniconda/envs/test-dask/lib/python3.10/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/juliusbusecke/miniconda/envs/test-dask/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/juliusbusecke/miniconda/envs/test-dask/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/juliusbusecke/Desktop/Untitled1.ipynb", line 27, in <module>
    "execution_count": null,
NameError: name 'null' is not defined

Which indicates to me that there is something incompatible happening between the notebook metadata and dask?

This bug currently prevents everyone at our Center to use a LocalCluster which is pretty bad for many of us. Does anyone know a workaround/fix for this? Thanks.

[emmanuel-ch]

I think I have the same issue - described on this post.

[keewis]

you can avoid this issue for now by pinning ipykernel<6.21 (see jupyter-server/jupyter_server#1198). As a summary, ipykernel is setting __file__ to the notebook path in ipykernel=6.21 if a specific environment variable is set – which is what jupyter_server is doing. The result is that multiprocessing is trying to execute the notebook as a python module.

If that does indeed help, this would make this issue a duplicate of #7515

Edit: another option besides downgrading would be to set the multiprocessing method to "fork" (as detailed in the other distributed issue)

[jrbourbeau]

Thanks @jbusecke for the issue and @keewis for the context / workaround. To consolidate discussion, I'm going to close this as a duplicate of #7515.

Also wanted to note that there's a new ipykernel release https://pypi.org/project/ipykernel/6.21.2 that should fix the underlying issue jupyter-server/jupyter_server#1198