[skip ci] Add Cloud deployment documentation#1611
[skip ci] Add Cloud deployment documentation#1611mrocklin wants to merge 3 commits intodask:masterfrom
Conversation
| ~~~~~~~~~~~~~ | ||
|
|
||
| First you will need to set up an account with one of Google, Amazon, or | ||
| Microsoft Azure, and launch a cluster running Kubernetes. This is |
There was a problem hiding this comment.
Although it isn't "Cloud", I found minikube helpful for testing things out locally before trying on Google. Both helm install stable/dask-distributed and helm install stable/spark worked for me locally.
There was a problem hiding this comment.
Cool. I wasn't aware of minikube. Should we mention here in these docs or is this just an FYI?
There was a problem hiding this comment.
Not sure... I think a parenthetical "or minikube for local testing" may be worthwhile to ease the burden of entry for people looking to test things out. It seems to be the officially blessed version of "local kubernetes", so I don't see much downside.
| Conda packages are installed before pip packages. These values are appended | ||
| directly to the ``conda install`` and ``pip install`` commands and so you can | ||
| also include flags, such as specifying conda channels. | ||
|
|
There was a problem hiding this comment.
There's another "Kubernetes" section on line 305 below, not sure if it should be removed or merged with this.
There was a problem hiding this comment.
Yeah, this and the dask-kubernetes section probably require some conversation to see what we want to do going forward
There was a problem hiding this comment.
I think that the decision here probably depends on how smooth we can make the Helm solution. Right now I don't think it's sufficiently smooth, but perhaps we can get it there?
There was a problem hiding this comment.
The section I'm talking about is below the Google Cloud section, and looks like it lists some previous (and possibly outdated?) work.
|
Current work on dask helm chart: helm/charts#2914 |
| might want to conda-install XArray and Arrow:: | ||
|
|
||
| helm upgrade iced-mule stable/dask \ | ||
| --set worker.pipPackages="gcsfs dask-ml" \ |
There was a problem hiding this comment.
gcsfs is now on conda-forge too (here and above)
There was a problem hiding this comment.
My intent was to show off both. Personally I use pip for gcsfs just for speed. Conda can take a while.
|
cc @danielfrg |
4 participants
This depends on a number of PRs currently in-flight