Start cluster over ssh

[basnijholt]

I work on a server with a Jupyterhub and have access to a pbs cluster, both machines have the same Python environments.

Right now I do the following (manual work):

1. I start workers on the cluster with dask-drmaa 10
2. I create a tunnel from the port at which the scheduler runs to the server on which I run my Juypter notebook with: ssh -NL 39408:localhost:39408 cluster
3. In my Jupyter notebook I do the following:

import distributed
client = distributed.Client('localhost:39408')

It would be great if this could somehow be automated 😄

[mrocklin]

I'm not sure that helping users with ssh tunneling is in scope for this project.

[jakirkham]

Are you familiar with Fabric, @basnijholt? It's designed exactly for this sort of stuff. Would recommend Fabric3 for Python 2/3 compatibility.

[basnijholt]

I will definitely check it out! Thanks for the recommendation.

I should have probably known this before because now I wrote my own tools to do this.