Add deployment update post #58
Conversation
This includes a summary and links to the various deployment efforts that have occurred in the last few months.
| Dask-Cloudprovider and Dask-CUDA libraries place them | ||
| all under the same `dask.distributed.SpecCluster` superclass. So we can expect a high degree of | ||
| uniformity from them. Additionally, all of the classes now inherit from the | ||
| `dask.distributed.Cluster` class, which standardizes things like adaptivity, |
There was a problem hiding this comment.
Note: dask-gateway and dask-yarn don't inherit from dask.distributed.Cluster, but do match the API.
| For cloud deployments we generally recommend using a hosted Kubernetes or Yarn | ||
| service, and then using Dask-Kubernetes or Dask-Yarn on top of these. | ||
|
|
||
| However in some institutions these hosted services aren't yet accessible, and |
There was a problem hiding this comment.
I would maybe reword this as something like "in some institutions they have made decisions or commitments to use certain vendor specific technologies."
I think a lot of the use cases for dask-cloudprovider are going to be folks who have gone "all in" on a certain cloud provider or set of technologies. I've seen a few organisations to this, sometimes for a reduced price, sometimes to encourage a constrained pallete of tools, etc.
| found in HPC centers and Dask-Kubernetes. These now share a common codebase | ||
| along with Dask SSH, and so are much more consistent and hopefully bug free. | ||
|
|
||
| Hopefully users shouldn't notice much difference with existing workloads, |
There was a problem hiding this comment.
You've got two hopefullys four words apart here. May want to swap one for something else.
|
|
||
| In some cases users may not have access to the cluster manager. For example | ||
| the institution may not give all of their data science users access to the Yarn | ||
| or Kubernetes cluster. In this the [Dask-Gateway](https://gateway.dask.org) |
There was a problem hiding this comment.
| or Kubernetes cluster. In this the [Dask-Gateway](https://gateway.dask.org) | |
| or Kubernetes cluster. In this case the [Dask-Gateway](https://gateway.dask.org) |
| 1. One Dask-worker per GPU on a machine | ||
| 2. Specify the `CUDA_VISIBLE_DEVICES` environment variable to pin that worker | ||
| to that GPU | ||
| 3. If your machine has multiple network interfaces then choose the network interface closest to that GPU |
There was a problem hiding this comment.
Maybe we should define what "closest" means in this context, which is relative to system's topology. Ideally, we would provide samples on how can users find that information, but I think this may be too out of context for this blog.
There was a problem hiding this comment.
Thanks fo this @mrocklin, sorry I did not notice it among all other notifications 🙂.
Did a few suggestions, take or leave, this is already very nice!
| the institution may not give all of their data science users access to the Yarn | ||
| or Kubernetes cluster. In this case the [Dask-Gateway](https://gateway.dask.org) | ||
| project may be useful. | ||
| It can launch and manage Dask jobs, |
Co-Authored-By: Guillaume Eynard-Bontemps <g.eynard.bontemps@gmail.com>
|
Thanks all. This is in. |
This includes a summary and links to the various deployment efforts that
have occurred in the last few months.
So far this is pretty rough. Any help, including direct edits, would be welcome.
cc @lesteve @guillaumeeb @jhamman @andersy005 @jacobtomlinson @jcrist @pentschev