Detach from running cluster

[jacobtomlinson]

The current behaviour of dask-kubernetes is to create ephemeral clusters and use them to perform calculation. There is a bunch of logic here to clean up the cluster when the Python object is removed.

This may not always be the desired use case and we may want to support detaching from a cluster. This also fits with #185 where we may want to attach to an already running cluster.

[mrocklin]

I think that @jcrist has conventions for this in dask-gateway and dask-yarn.

…

On Mon, Oct 14, 2019 at 6:07 AM Jacob Tomlinson ***@***.***> wrote: The current behaviour of dask-kubernetes is to create ephemeral clusters and use them to perform calculation. There is a bunch of logic here to clean up the cluster when the Python object is removed. This may not always be the desired use case and we may want to support detaching from a cluster. This also fits with #185 <#185> where we may want to attach to an already running cluster. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#186?email_source=notifications&email_token=AACKZTHB3EJF3SG5HWVCMTLQORHHDA5CNFSM4JANVI6KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HRR4F5A>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTAR4Q5HAB7EJTFTVDTQORHHDANCNFSM4JANVI6A> .

[jcrist]

We use the keyword shutdown_on_close in dask-gateway (defaults to True, except when connecting to a running cluster then the default is False). Typically a cluster object has a few things you'd want to cleanup:

• Remote cluster resources (e.g. kubernetes pods)
• Local async things (event loop, background tasks, comms, etc...)

The .close() method is always called on GC of an object, but the meaning of .close() differs based on the value of shutdown_on_close for the cluster.

# On .close()/GC of cluster object, remote cluster is cleaned up
cluster = GatewayCluster()
cluster.close()

# On .close()/GC of cluster object, only local resources are cleaned up
cluster = GatewayCluster(shutdown_on_close=False)
cluster.close()

# You can override this behavior by passing `shutdown=True` to close
# or using the `shutdown()` method which is an alias to this
cluster.close(shutdown=True)
cluster.shutdown()

# When connecting to a previously started cluster, the default for
# shutdown_on_close is False
cluster = GatewayCluster.from_name(cluster_name)

This option only makes sense if the scheduler is remote (which is always true for dask-gateway).

[jacobtomlinson]

Thanks @jcrist this is the kind of thing I want to implement here. Now that we have the option of creating remote schedulers it would be good to implement this functionality.

[jacobtomlinson]

I'm going to close this out as being closely related to #185.