More convenient way to create the scheduler-file

[jhamman]

The concept of a "scheduler file" is a nice, convenient way to share scheduler information when working on a system where you won't know the scheduler address a priori. I would like to suggest a simple method, probably on Client, that creates such a "scheduler file" with a simple interface like this:

from distributed import Client
client = Client()
client.to_scheduler_file('path-to-a-file.json')

then from another session one could easily connect to that same set of resources via:

from distributed import Client
client = Client(scheduler_file='path-to-a-file.json')

Under the hood this would just be something like

json.dump(client.scheduler_info(), 'path-to-a-file.json', indent=2)

[mrocklin]

I'm curious, do you find yourself doing the json.dump solution today?

[jhamman]

Yes - to some extent. I basically went searching for either a spec for the scheduler file and discovered that there really isn't a standard way for creating these files.

distributed/distributed/scheduler.py

Lines 1031 to 1034 in 3e1094f

	with open(self.scheduler_file, 'w') as f:
	json.dump(self.identity(), f, indent=2)
	fn = self.scheduler_file # remove file when we close the process

Fuller context is that I'd like to easily and programmatically share an autoscaling PBSCluster/scheduler between a number of applications. Dumping the scheduler info to a file seems like a pretty clean way to share this information.

[mrocklin]

I agree that we need to make it more clear how to build these files, including possibly providing a convenience function. Short-term I have a slight preference to have someone try this out with the manual process and see how it goes. I suspect that that will inform the creation of such a function.

…

On Wed, Feb 7, 2018 at 10:58 AM, Joe Hamman ***@***.***> wrote: Yes - to some extent. I basically went searching for either a spec for the scheduler file and discovered that there really isn't a standard way for creating these files. https://github.com/dask/distributed/blob/3e1094fe7e6ea3960bccc29770a5d9 2844cdd073/distributed/scheduler.py#L1031-L1034 Fuller context is that I'd like to easily and programmatically share an autoscaling PBSCluster/scheduler between a number of applications. Dumping the scheduler info to a file seems like a pretty clean way to share this information. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1741 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszFtfSb3fflBwfGmPVikvh4aQAOWmks5tScg1gaJpZM4R8B-8> .

[jhamman]

Fair enough, I'll give it a test run and report back. I'd also be keen to make this contribution myself if it seems to work.

[jhamman]

Just a quick report back. I've been using the following snippit all week with a fair bit of success:

from distributed import Client
import json

client = Client(...)

with open('~/workdir/scheduler_file.json', 'w') as f:
    json.dump(client.scheduler_info(), f, indent=2)

This has allowed me to run a single adaptive cluster from one Jupyter Notebook and programmatically attach to it from various other notebooks.

[mrocklin]

@jhamman if you'd like to submit a PR for a function to write this to a file I'd be open to it.

[jhamman]

@mrocklin - see #1778