upgrade images to jupyterlab 1.0

[scottyhq]

The Jupyterlab 1.0 release is coming up in the next couple weeks! (jupyterlab/jupyterlab#6504).

Seems like there have been a lot of changes since 0.35 that we've been using https://jupyterlab.readthedocs.io/en/latest/getting_started/changelog.html#v0-35-0

We should make sure our setup is ready for 1.0 with the various extensions we are using

[TomAugspurger]

FYI, I'm kinda debugging this in tandem at https://github.com/TomAugspurger/distributed-memory-pressure. Thus far

• jupyterlab -> 1.0.0rc0
• dask-labextension -> 0.4.0a1 (python) 1.0.0-rc.0 (js)
• hub-extension -> jupyterlab/hub-extension@1.0.0-alpha.6
• removed jupyter-widgets (waiting for Update to JLab 1.0RC jupyter-widgets/ipywidgets#2456 to be published)
• removed leaflet (blocked by jupyter widgets I think)
• ...

[TomAugspurger]

#57 makes the changes for jupyterlab & dask-labextension.

So just to make sure I have everything, after #57 is merged, what else needs to be updated to use it?

• pangeo-data/helm-chart
• pangeo-data/pangeo-cloud-federation

Any others?

[scottyhq]

Let's check with @jhamman who has been updating helm charts recently I think...

But once we rebuild these images here, we can just point to them without other modifications https://github.com/pangeo-data/pangeo-cloud-federation/blob/staging/deployments/nasa/image/binder/Dockerfile#L2

[jhamman]

The main place to update things is the pangeo-cloud-federation. The helm chart also references the base image here but most overwrite this in their config so it is less critical. I'm happy to let you two just roll forward and play things a bit fast and loose. We date tag all the built images so if a few days end up broken, that's okay.

[scottyhq]

new image from 2019.07.17 (#57), now has jupyterlab 1.0.1. The dask-labextension isn't working showing a 'cannot get address of non-running Server error.

**full list of packages and jupyerlab extensions**

# packages in environment at /srv/conda/envs/notebook: # # Name Version Build Channel _libgcc_mutex 0.1 main defaults adal 1.2.2 py_0 conda-forge alembic 1.0.9 pypi_0 pypi asn1crypto 0.24.0 py37_1003 conda-forge async_generator 1.10 py_0 conda-forge attrs 19.1.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge bleach 3.1.0 py_0 conda-forge blinker 1.4 py_1 conda-forge bokeh 1.2.0 py37_0 conda-forge bzip2 1.0.6 h14c3975_1002 conda-forge ca-certificates 2019.6.16 hecc5488_0 conda-forge cachetools 2.1.0 py_0 conda-forge cairo 1.16.0 h18b612c_1001 conda-forge certifi 2019.6.16 py37_1 conda-forge certipy 0.1.3 py_0 conda-forge cffi 1.12.3 py37h8022711_0 conda-forge chardet 3.0.4 pypi_0 pypi click 7.0 py_0 conda-forge cloudpickle 1.2.1 py_0 conda-forge configurable-http-proxy 4.1.0 node11_1 conda-forge cryptography 2.7 py37h72c5cf5_0 conda-forge cytoolz 0.10.0 py37h516909a_0 conda-forge dask 2.1.0 py_0 conda-forge dask-core 2.1.0 py_0 conda-forge dask-kubernetes 0.9.0 py_0 conda-forge dask-labextension 0.3.3 py_1 conda-forge dbus 1.13.6 he372182_0 conda-forge decorator 4.4.0 py_0 conda-forge defusedxml 0.5.0 py_1 conda-forge distributed 2.1.0 py_0 conda-forge entrypoints 0.3 py37_1000 conda-forge expat 2.2.5 he1b5a44_1003 conda-forge fontconfig 2.13.1 he4413a7_1000 conda-forge freetype 2.10.0 he983fc9_0 conda-forge fribidi 1.0.5 h516909a_1002 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge glib 2.58.3 h6f030ca_1002 conda-forge google-auth 1.6.3 py_0 conda-forge graphite2 1.3.13 hf484d3e_1000 conda-forge graphviz 2.40.1 h5933667_1 conda-forge gst-plugins-base 1.14.5 h0935bb2_0 conda-forge gstreamer 1.14.5 h36ae1b5_0 conda-forge harfbuzz 2.4.0 h37c48d4_1 conda-forge heapdict 1.0.0 py37_1000 conda-forge icu 58.2 hf484d3e_1000 conda-forge idna 2.8 pypi_0 pypi ipykernel 5.1.0 py37h24bf2e0_1002 conda-forge ipython 7.5.0 py37h24bf2e0_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.0 py_0 conda-forge jedi 0.13.3 py37_0 conda-forge jinja2 2.10.1 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge json5 0.8.5 py_0 conda-forge jsonschema 3.0.1 py37_0 conda-forge jupyter 1.0.0 py_2 conda-forge jupyter_client 5.2.4 py_3 conda-forge jupyter_console 6.0.0 py_0 conda-forge jupyter_core 4.4.0 py_0 conda-forge jupyterhub 0.9.4 pypi_0 pypi jupyterlab 1.0.1 py37_0 conda-forge jupyterlab_server 1.0.0 py_1 conda-forge krb5 1.16.3 h05b26f9_1001 conda-forge libblas 3.8.0 10_openblas conda-forge libcblas 3.8.0 10_openblas conda-forge libcurl 7.64.1 hda55be3_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 8.2.0 hdf63c60_1 defaults libgfortran-ng 7.3.0 hdf63c60_0 defaults libiconv 1.15 h516909a_1005 conda-forge liblapack 3.8.0 10_openblas conda-forge libopenblas 0.3.6 h6e990d7_4 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libsodium 1.0.16 h14c3975_1001 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 8.2.0 hdf63c60_1 defaults libtiff 4.0.10 h57b8799_1003 conda-forge libtool 2.4.6 h14c3975_1002 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.9 h13577e0_1 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge mako 1.0.9 pypi_0 pypi markupsafe 1.1.1 py37h14c3975_0 conda-forge mistune 0.8.4 py37h14c3975_1000 conda-forge msgpack-python 0.6.1 py37h6bb024c_0 conda-forge nbconvert 5.4.1 py_2 conda-forge nbformat 4.4.0 py_1 conda-forge nbserverproxy 0.8.8 py_1000 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge nodejs 11.14.0 he1b5a44_1 conda-forge notebook 5.7.8 py37_0 conda-forge nteract-on-jupyter 2.0.12 pypi_0 pypi numpy 1.16.4 py37h95a1406_0 conda-forge oauthlib 3.0.1 py_0 conda-forge olefile 0.46 py_0 conda-forge openblas 0.3.6 h6e990d7_4 conda-forge openssl 1.1.1c h516909a_0 conda-forge packaging 19.0 py_0 conda-forge pamela 1.0.0 py_0 conda-forge pandas 0.24.2 py37hb3f55d8_0 conda-forge pandoc 2.7.2 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge pango 1.42.4 he7ab937_0 conda-forge parso 0.4.0 py_0 conda-forge partd 1.0.0 py_0 conda-forge pcre 8.41 hf484d3e_1003 conda-forge pexpect 4.7.0 py37_0 conda-forge pickleshare 0.7.5 py37_1000 conda-forge pillow 6.1.0 py37he7afcd5_0 conda-forge pip 19.1 py37_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge prometheus_client 0.6.0 py_0 conda-forge prompt_toolkit 2.0.9 py_0 conda-forge psutil 5.6.3 py37h516909a_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge pyasn1 0.4.5 py_0 conda-forge pyasn1-modules 0.2.5 py_0 conda-forge pycparser 2.19 py37_1 conda-forge pycurl 7.43.0.2 py37h16ce93b_1 conda-forge pygments 2.3.1 py_0 conda-forge pyjwt 1.7.1 py_0 conda-forge pyopenssl 19.0.0 py37_0 conda-forge pyparsing 2.4.0 py_0 conda-forge pyqt 5.9.2 py37hcca6a23_0 conda-forge pyrsistent 0.15.1 py37h516909a_0 conda-forge pysocks 1.7.0 py37_0 conda-forge python 3.7.3 h33d41f4_1 conda-forge python-dateutil 2.8.0 py_0 conda-forge python-editor 1.0.4 pypi_0 pypi python-kubernetes 10.0.0 py37_0 conda-forge python-oauth2 1.1.0 pypi_0 pypi pytz 2019.1 py_0 conda-forge pyyaml 5.1.1 py37h516909a_0 conda-forge pyzmq 18.0.1 py37hc4ba49a_1 conda-forge qt 5.9.7 h52cfd70_2 conda-forge qtconsole 4.5.1 py_0 conda-forge readline 8.0 hf8c457e_0 conda-forge requests 2.21.0 pypi_0 pypi requests-oauthlib 1.2.0 py_0 conda-forge rsa 3.4.2 py_1 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 41.0.1 py37_0 conda-forge sip 4.19.8 py37hf484d3e_1000 conda-forge six 1.12.0 py37_1000 conda-forge sortedcontainers 2.1.0 py_0 conda-forge sqlalchemy 1.3.3 pypi_0 pypi sqlite 3.29.0 hcee41ef_0 conda-forge tblib 1.4.0 py_0 conda-forge terminado 0.8.2 py37_0 conda-forge testpath 0.4.2 py_1001 conda-forge tk 8.6.9 h84994c4_1001 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.3 py37h516909a_0 conda-forge traitlets 4.3.2 py37_1000 conda-forge urllib3 1.24.2 pypi_0 pypi wcwidth 0.1.7 py_1 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 0.56.0 py37_0 conda-forge wheel 0.33.1 py37_0 conda-forge widgetsnbextension 3.5.0 py37_0 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.8 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxpm 3.5.12 h14c3975_1002 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-libxt 1.1.5 h516909a_1003 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.1.7 h14c3975_1001 conda-forge zeromq 4.3.1 hf484d3e_1000 conda-forge zict 1.0.0 py_0 conda-forge zlib 1.2.11 h14c3975_1004 conda-forge zstd 1.4.0 h3b9ef0a_0 conda-forge

JupyterLab v1.0.1
Known labextensions:
app dir: /srv/conda/envs/notebook/share/jupyter/lab
@jupyter-widgets/jupyterlab-manager v1.0.1 enabled OK
dask-labextension v1.0.0-rc.0 enabled OK

Note that launching KubeClusters from a code cell is working:

import dask
from dask_kubernetes import KubeCluster
from dask.distributed import Client

cluster = KubeCluster(n_workers=2)
client = Client(cluster)
client

simple code to test dask

import dask.array as da
bigshape = (200000, 4000)
chunk_shape = (1000, 1000)
big_ones = da.ones(bigshape, chunks=chunk_shape)
big_ones.compute()

[scottyhq]

should we go back to dask-labextension 0.3.1 and jupyter labextension install dask-labextension@0.3.0? but maybe jupyter labextension install dask-labextension@1.0.0 is required for jupyterlab>1?

might also still be needing some things to be changed in dask packages (#48 (comment))

cc @ian-r-rose and @TomAugspurger

[TomAugspurger]

Hmm, I'm noticing

bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31 WebSocket connection to 'wss://hub.aws-uswest2-binder.pangeo.io/user/scottyhq-pangeo-binder-test-atjb2xa9/proxy/8787/profile/ws?bokeh-protocol-version=1.0&bokeh-session-id=zsMISZknq9JzxksDjZBlGZEteVzAVXcn2si8hLf2ATGS' failed: Error during WebSocket handshake: Unexpected response code: 502
t.connect @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
i.pull_session @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
i.add_document_from_session @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
f @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31 [bokeh] Failed to connect to Bokeh server Error: Could not open websocket
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
Promise.then (async)
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
i.pull_session @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
i.add_document_from_session @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
f @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
requestAnimationFrame (async)
i.defer @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
i.embed_items @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
embed_document @ profile:96
(anonymous) @ profile:100
(anonymous) @ profile:115
i.safely @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:33
fn @ profile:90
bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31 [bokeh] Lost websocket 0 connection, 1006 ()
bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31 Uncaught (in promise) Error: Could not open websocket
    at t._on_error (bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31)
    at WebSocket.t.socket.onerror (bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31)
t._on_error @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
t.socket.onerror @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
Promise.then (async)
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
i.pull_session @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
i.add_document_from_session @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
f @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
(anonymous) @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
requestAnimationFrame (async)
i.defer @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
i.embed_items @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31
embed_document @ profile:96
(anonymous) @ profile:100
(anonymous) @ profile:115
i.safely @ bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:33
fn @ profile:90
bokeh.min.js?v=1bfbafacfa847bc6589a4af73a904fef:31 [bokeh] Websocket connection 0 disconnected, will not attempt to reconnect

in the logs. Let's try pinning to tornado=5 for now. Will make a PR.

[ian-r-rose]

You may want to also try upgrading nbserverproxy to use jupyter-server-proxy, which is now name of the new package, and is more or less a complete rewrite. @rabernat ran into something similar last week and found that to be the fix.

[ian-r-rose]

(I'm really regretting making dask_labextension expect async clusters -- it's made supporting dask-kubernetes quite difficult while it gets asyncified)

[TomAugspurger]

Ah, thanks Ian. Let's try that first.

[scottyhq]

thanks @ian-r-rose! do you recommend we pip install from the jupyter-server-proxy master branch until the next release (jupyterhub/jupyter-server-proxy#139)?

also if i'm not mistaken we can remove these lines from our config now as long as we are installing from conda (swapping nbserverproxy or jupyter_server_proxy)?:

pangeo-stacks/base-notebook/binder/postBuild

Lines 4 to 7 in 72588b5

	# serverextensions
	jupyter serverextension enable --sys-prefix --py nbserverproxy
	# nbextensions
	jupyter nbextension enable --sys-prefix --py widgetsnbextension

[ian-r-rose]

@scottyhq I believe that issue is referring to the release of some client-side code that is not necessary for your case (I'd release it myself, but don't have permissions to do so). The PyPI package should be up-to-date.

[scottyhq]

getting the same errors even with jupyter-server-proxy or pinning notebook=5.7.4 and tornado=5.1.1 as suggested here:
dask/dask-labextension#51

current binder: https://github.com/scottyhq/pangeo-binder-test/tree/jlab1

[ian-r-rose]

Thanks @scottyhq, I'll try to play around with the binder this evening (though the server logs will be unavailable to me, right?)

[scottyhq]

@ian-r-rose please do experiment with the binder! how do you normally look at the server logs? it's possible we can set something up.

I have access to the kubernetes logs which aren't necessarily more informative. There is one suspect line from the kubernetes log which is:

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/deploy/spec.py:261: RuntimeWarning: coroutine 'SpecCluster._close' was never awaited
  self.close()

cc @jacobtomlinson as I think this is related to dask/dask-kubernetes#162

It's worth pointing out that launching a Kubernetes Cluster in a code cell and then passing the dashboard url to the labextension (Dashboard: /user/scottyhq-pangeo-binder-test-j40j0nnj/proxy/8787/status), the buttons and task graphs all work as expected (It's just the +New that doesn't work and refreshing doesn't find the active cluster). Once the dashboard url is manually entered logs I have access to then show:

[I 01:04:55.912 LabApp] Trying to establish websocket connection to ws://localhost:8787/status/ws?bokeh-protocol-version=1.0&bokeh-session-id=zQTxe6X81qLFZiB3onvMP7A0INon6lDadkl97LEtFb8I
[I 01:04:55.914 LabApp] Websocket connection established to ws://localhost:8787/status/ws?bokeh-protocol-version=1.0&bokeh-session-id=zQTxe6X81qLFZiB3onvMP7A0INon6lDadkl97LEtFb8I
[I 01:05:34.353 LabApp] Saving file at /notebooks/0-demo-aws.ipynb
[I 01:06:19.927 LabApp] Trying to establish websocket connection to ws://localhost:8787/individual-workers/ws?bokeh-protocol-version=1.0&bokeh-session-id=2lX3oPSYhzfMsWFfxGr8Ez8GXd8Rh7d3OglblQnnT5cy
[I 01:06:19.930 LabApp] Websocket connection established to ws://localhost:8787/individual-workers/ws?bokeh-protocol-version=1.0&bokeh-session-id=2lX3oPSYhzfMsWFfxGr8Ez8GXd8Rh7d3OglblQnnT5cy
[I 01:06:23.092 LabApp] Trying to establish websocket connection to ws://localhost:8787/individual-workers/ws?bokeh-protocol-version=1.0&bokeh-session-id=DD6ibqubTBQQJoBXlAgopo3lRfNj0QV6LECX5r6dbF5v
[I 01:06:23.094 LabApp] Websocket connection established to ws://localhost:8787/individual-workers/ws?bokeh-protocol-version=1.0&bokeh-session-id=DD6ibqubTBQQJoBXlAgopo3lRfNj0QV6LECX5r6dbF5v
[I 01:06:41.534 LabApp] Trying to establish websocket connection to ws://localhost:8787/individual-task-stream/ws?bokeh-protocol-version=1.0&bokeh-session-id=Ytk7m9KLdFiwL1PJiV6ozjKsKp0gEpGdPCnrMFSpuydv
[I 01:06:41.537 LabApp] Websocket connection established to ws://localhost:8787/individual-task-stream/ws?bokeh-protocol-version=1.0&bokeh-session-id=Ytk7m9KLdFiwL1PJiV6ozjKsKp0gEpGdPCnrMFSpuydv
[I 01:07:15.976 LabApp] Uploading file to /notebooks/dask.ipynb
[W 01:07:20.497 LabApp] Notebook notebooks/dask.ipynb is not trusted
[I 01:07:21.802 LabApp] Kernel started: c2fa981a-5d5c-4d00-9e52-380ef6274de9
[I 01:07:22.788 LabApp] Adapting from protocol version 5.1 (kernel c2fa981a-5d5c-4d00-9e52-380ef6274de9) to 5.3 (client).

[ian-r-rose]

Sorry for the slow response @scottyhq, didn't get the chance to look into this until now. I believe the problem is, unfortunately, that KubeCluster still doesn't support asynchronous starting and stopping of the cluster, and dask_labextension expects its clusters to be able to do that.

The relevant issue for this is here. I believe @jacobtomlinson has started looking into it.

[ian-r-rose]

@scottyhq I've successfully launched a KubeCluster using your jlab1 branch by installing an unpublished branch of dask_labextension that removes the requirement on asynchronous launching: https://github.com/ian-r-rose/pangeo-binder-test/tree/jlab1

The relevant branch is here, it's really a remarkably small change! I think until we get async KubeCluster up and running, it would be a reasonable thing to install from this.

[ian-r-rose]

I'm hesitant to move forwards too much with a non-async version, as things are only moving more and more towards an async universe, c.f. dask/dask-labextension#72

[scottyhq]

Thanks @ian-r-rose! Much appreciated. This is a great patch for the time being and we'll keep watching the dask-kubernetes developments.