Remove LocalCluster from KubeCluster

[traubms]

I propose to allow KubeCluster to take a scheduler argument (or client and get scheduler indirectly), instead of always creating its own via LocalCluster.

This would enable spawning a scheduler, possibly remotely (see #84 and similar proposal #84 (comment)), and adding multiple KubeClusters that manage different pod specs.

Some use cases:

• I'm in a notebook, I have some light workload so I create many small workers, but then I have a heavy workload and require fewer, larger workers. I want one scheduler so I can shuffle the work onto the larger workers before scaling down the cluster with many small workers.
• I have a remote scheduler taking batch work, but different tasks require new pod specs. The process that submits the tasks could also create a KubeCluster to manage the workers that will run those jobs, but I still get the benefits of a centralized scheduler to monitor my work.

This might be related to dask/distributed#2235, but within the scope of dask-kubernetes this seems like a relatively minor change. If no client/scheduler is passed when constructing a KubeCluster, it could still create its own from LocalCluster.

[mrocklin]

This looks like a possible repeat of #84 ?

[traubms]

If I understand correctly, #84 would be resolved by using a remote scheduler, but this isn't enough to make the two example use cases work.

This proposal provides a way to begin using heterogeneous pod specs, regardless of whether the scheduler is local or remote.

[traubms]

For clarity, this change enables the relationship between cluster/client/scheduler below. These don't assume you have a remote scheduler, but if you did, you also could have multiple clients.

Before:

Cluster <- Scheduler <- Client 

Now:

Scheduler <- Client <- Cluster (large pods)
                 ^
                  `- Cluster (small pods)

where an arrow A <- B means "B relies on A existing".

[mrocklin]

For broader visibility I recommend that you engage on dask/distributed#2235 . Any redesign like this will probably engage all of the projects.

[jacobtomlinson]

As this issue is related to the wider design of cluster managers and we've been pointed to an appropriate place to discuss that I'm going to close this out.