Most Dask notebooks start off with the following three lines:
from dask.distributed import Client
client = Client()
client
Then users click on the dashboard link, and do some work.
Eventually the users runs all the cells in their notebook, rerunning the Client() call. This is a problem because it creates a new LocalCluster (Client with no args creates a LocalCluster). Now we have two clusters running which causes two problems:
- The old cluster still takes up modest resources. This isn't that bad, but is slightly inefficient
- The new cluster serves the diagnostic dashboard on a random port, so the user's dashboard appears to be non-responsive.
There are a few potential solutions to this problem:
- We could change the policy of
Client() with no arguments from "Create a LocalCluster" to "Use a pre-existing cluster if present, otherwise make one". However this becomes complex if the keyword arguments passed down to LocalCluster change, like changing Client(processes=False) to Client(processes=True). In this case we might consider closing the old LocalCluster before starting the new one so that we can claim the correct port
- We might consider erring if we end up choosing a random bokeh port if the user didn't explicitly request this.
- We might consider erring hard when calling
Client() twice, encouraging the user to call client.cluster.close()
Most Dask notebooks start off with the following three lines:
Then users click on the dashboard link, and do some work.
Eventually the users runs all the cells in their notebook, rerunning the
Client()call. This is a problem because it creates a newLocalCluster(Clientwith no args creates aLocalCluster). Now we have two clusters running which causes two problems:There are a few potential solutions to this problem:
Client()with no arguments from "Create a LocalCluster" to "Use a pre-existing cluster if present, otherwise make one". However this becomes complex if the keyword arguments passed down to LocalCluster change, like changingClient(processes=False)toClient(processes=True). In this case we might consider closing the oldLocalClusterbefore starting the new one so that we can claim the correct portClient()twice, encouraging the user to callclient.cluster.close()