Dask-distributed cloud issues

[mrocklin]

Some notes on recent experience with the dask-distributed chart. CC @danielfrg, the original author of this chart, and @noelbundick and @martindurant who were involved in finding these issues.

Jupyter environment

It would be good to ensure that the Jupyter container and Dask worker containers share the same environment

Customizable environment

When deploying on different cloud providers we often want to include a few extra packages. For example on Amazon we tend to want s3fs, on Google gcsfs, on Azure, adlfs, or nothing when deploying on non-cloud systems. This also comes up with people want a container with slightly different versions of libraries.

To resolve this I recommend that we include small conda and pip install steps in setup scripts of the Docker container that refer to environment variables that optionally contain a list of extra packages to install. This might also allow us to remove some of the existing packages in the image.

Alternatively @noelbundick recommended having environment variables for each of the cloud providers, which would trigger the installation of packages specific to that cloud platform. I could go either way, with a slight preference to providing a list of explicit packages.

Rename to Dask

Is it likely that the community will ever want a helm chart of the single-machine scheduler? This seems somewhat unlikely to me. I'm inclined to rename this chart to just dask if that's feasible.

[mrocklin]

Remove Jupyter

I remember when creating this chart we were curious if we should include Jupyter or not. In many cases people using Dask will also want a Jupyter notebook server, but not always. We may want to consider removing the Jupyter component of this chart if it is easy to also install an additional chart running Jupyter with the same image. Thoughts?

[danielfrg]

Now its easier to have a jupyter notebook chart, even the jupyter hub is relatively easy to install now.

The one nice thing about deploying one single user notebook server from this chart is that you can control the environment so it matches the dask one, one of the issues you mention above.

Also nothing is blocking from having or using two notebook servers. The chart was based on the spark one that also deploys a zeppelin server even if you can deploy an standalone zeppelin chart.

[mrocklin]

Thoughts on customizable environments?

Do the standard Jupyter charts support custom images? Maybe we just make our own jupyter-dask docker image and direct people to use the standard Jupyter helm chart with our image when then deploy?

[danielfrg]

Yes, the customisable environment is definitely something we have to do. That sounds like a good solution. Their image is relatively easy to customise.

[mrocklin]

We might also want to write the scheduler connection information to local disk in the Jupyter image. Currently the scheduler address is something like ``happy-panda-scheduler:8786'` with the first couple terms there being dependent on the deployment. Dask clients can read files with connection information like the following.

{
  "type": "Scheduler",
  "id": "Scheduler-dfc2c954-ac6b-4b69-b319-9078e63ab1c1",  
  "address": "tcp://192.168.50.100:8786",
  "services": {
    "bokeh": 8787
  },
  "workers": {}
}

Where the only thing we really need there is the type and address fields

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mrocklin]

/remove-lifecycle stale

This is being slowly but surely resolved in #2914