dask labextension not working in ocean

[rabernat]

I am seeing this in my notebook pod logs:

404 GET /user/0000-0001-5999-4917/dask/clusters?1551920957641 (0000-0001-5999-4917@10.128.0.18) 2.41ms

after I click on the button to create a new cluster.

[ian-r-rose]

@rabernat Is it possible that the server extension is not installed or enabled? The newer version of the extension requires it. We also haven't hammered out all the kinks in usage with dask-kubernetes yet, so I'm quite curious to see how this works.

In a terminal:

jupyter serverextension list

[jhamman]

Just ran that on the ocean deployment:

$ jupyter serverextension list
config dir: /srv/conda/etc/jupyter
    jupyterlab  enabled
    - Validating...
      jupyterlab 0.35.4 OK
    nteract_on_jupyter  enabled
    - Validating...
Error loading server extension nteract_on_jupyter
      X is nteract_on_jupyter importable?
    nbserverproxy  enabled
    - Validating...
      nbserverproxy  OK

[ian-r-rose]

Is the nasa deployment working? I see that it includes the python package, while the ocean deployment does not: https://github.com/pangeo-data/pangeo-cloud-federation/blob/staging/deployments/nasa/image/binder/environment.yml#L39

[rabernat]

So in the latest configuration, the dask labextension works, but it launches LocalCluster, rather than kubeCluster, by default.

I think all that is needed is an extra piece of dask config. So the questions are:

• What is this config?
• Where is the best way to inject it into the environment. (The notebook image is just a straight copy of pangeo-notebook:2019-04-19 from pangeo-stacks.)

[jhamman]

@rabernat - you may want to look at the NASA deployment.

• we have the dask lab extension set as:

  pangeo-cloud-federation/deployments/nasa/image/binder/dask_config.yaml

  Lines 39 to 44 in 1bbab64

  	labextension:
  	factory:
  	module: dask_kubernetes
  	class: KubeCluster
  	args: []
  	kwargs: {}

• we overwrite the default config here:

  pangeo-cloud-federation/deployments/nasa/image/binder/postBuild

  Line 14 in 1bbab64

  cp binder/dask_config.yaml ${KERNEL_PYTHON_PREFIX}/etc/dask/dask.yaml

[ian-r-rose]

@jhamman Oh awesome, I wasn't aware that we had any deployments with KubeCluster up and running yet. Would you be willing to submit a PR to the dask-labextension README with an example config that is known to work? I'd like to start collecting these from community deployments, since I don't really have the bandwidth to try it out in all the variety of deployments that exist.

[rabernat]

@raphaeldussin - how did you determine this was resolved? It still doesn't work for me on staging.

When I try to launch a cluster with the labextension, I get this error in my notebook log:

[W 2019-04-27 01:39:04.298 SingleUserLabApp handlers:563] object KubeCluster can't be used in 'await' expression

[jhamman]

I think this is still a work in progress - dask/dask-kubernetes#138 - but I am still able to get workers to connect in my cluster.

[rabernat]

finally fixed by #258